Vehicle interaction method, vehicle, and storage medium

Abstract

The application relates to a vehicle interaction method, a vehicle and a storage medium. The method comprises the following steps: determining the identity of an object; acquiring a facial image of the object if the identity of the object matches preset identity information; determining an emotional state of the object according to the facial image; and presenting a facial mimicry suitable for the emotional state through a peripheral device of the vehicle. The method can make appropriate feedback according to the emotion of a driver, thereby improving the experience of vehicle-human interaction.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction for vehicles, and more specifically, to vehicle interaction methods, vehicles, and storage media. Background Technology

[0002] Currently, drivers and passengers can interact with vehicles through manual controls and voice commands. While these traditional methods allow control of vehicle functions, improving the user experience remains a priority. As vehicles become more common in households, their role extends beyond mere transportation, evolving into something akin to a family member. If a couple purchases a vehicle when their child is born, that car is likely to accompany the child through much of their childhood.

[0003] In view of this, it is necessary to propose a vehicle interaction mechanism to improve the user experience. Summary of the Invention

[0004] Embodiments of this application provide a vehicle interaction method, a vehicle, and a storage medium for making appropriate feedback based on the driver's emotions, thereby improving the vehicle-human interaction experience.

[0005] According to one aspect of this application, a vehicle interaction method is provided, comprising the following steps: determining the identity of an object; acquiring a facial image of the object if the object's identity matches preset identity information; determining the emotional state of the object based on the facial image; and presenting facial expressions adapted to the emotional state through peripheral devices of the vehicle.

[0006] In some embodiments of this application, optionally, the identity of the object is determined based on the radio frequency signals transmitted by the mobile device carried by the object and / or the unlocking device carried by the object.

[0007] In some embodiments of this application, the peripheral device may optionally include at least one of the following: a rearview mirror, headlights, horn, windshield wipers, and tires.

[0008] In some embodiments of this application, the method may optionally further include: unlocking the vehicle if the identity of the object matches preset identity information.

[0009] Optionally, in some embodiments of this application, the method further includes playing a prompt voice when the emotional state exhibits a tendency towards negative driving.

[0010] In some embodiments of this application, optionally, the emotional state exhibiting a tendency toward negative driving includes at least one of the following: fatigue, intoxication, and anger.

[0011] Optionally, in some embodiments of this application, the method further includes: unlocking the vehicle in response to an unlocking command from the object if the identity of the object matches preset identity information.

[0012] Optionally, in some embodiments of this application, the method further includes: rejecting the object's instruction to unlock the vehicle.

[0013] In some embodiments of this application, the emotional state may optionally include at least one of the following: happiness, calmness, sadness, fatigue, intoxication, and anger.

[0014] Optionally, in some embodiments of this application, the method further includes: restoring the peripheral device to a state prior to displaying the facial mimicry.

[0015] In some embodiments of this application, the method may optionally further include: acquiring a second facial image of the object in response to a locking command of the object; determining a second emotional state of the object based on the second facial image; and presenting a second facial expression mimicking the second emotional state via the peripheral device.

[0016] According to another aspect of this application, a vehicle is provided, comprising: an identity authentication module configured to determine the identity of an object; an image acquisition module configured to acquire a facial image of the object when the identity authentication module determines that the object's identity matches preset identity information; an emotion analysis module configured to determine the emotional state of the object based on the facial image; and a peripheral device configured to present facial expressions adapted to the emotional state.

[0017] According to another aspect of this application, a vehicle is provided, comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform any of the methods described above.

[0018] According to another aspect of this application, a computer-readable storage medium is provided, wherein instructions are stored therein, characterized in that, when executed by a processor, the instructions cause the processor to perform any of the methods described above.

[0019] According to some embodiments of this application, the vehicle interaction method, vehicle, and storage medium can provide corresponding feedback based on the driver's emotions, thereby using the vehicle's peripheral devices to present facial expressions that are adapted to the driver's emotions, thus improving the user experience. Attached Figure Description

[0020] The above and other objects and advantages of this application will become more fully clear from the following detailed description taken in conjunction with the accompanying drawings, wherein the same or similar elements are denoted by the same reference numerals.

[0021] Figure 1 A vehicle interaction method according to an embodiment of this application is shown;

[0022] Figure 2 A vehicle according to one embodiment of this application is shown. Detailed Implementation

[0023] For purposes of brevity and illustrativeness, the principles of this application are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equivalently applicable to all types of vehicle interaction methods, vehicles, and storage media, and that these same or similar principles can be implemented therein, without departing from the true spirit and scope of this application.

[0024] One aspect of this application provides a vehicle interaction method. For example... Figure 1 As shown, the vehicle interaction method 10 (hereinafter referred to as Method 10) includes the following steps: determining the identity of the object in step S102; obtaining the object's facial image if the determined identity matches preset identity information in step S104; determining the object's emotional state based on the facial image in step S106; and presenting facial expressions adapted to the emotional state through the vehicle's peripheral devices in step S108. Method 10, through the above steps, enables the vehicle to make corresponding facial expressions based on the emotional state of the driver and passengers. This allows for actions such as celebrating together when the driver and passengers are happy, or encouraging them when they are frustrated. This improves the customer's driving experience and can also alleviate or even prevent the dangers caused by negative emotions during driving. The working principle of each step of Method 10 will be described in detail below.

[0025] Method 10 determines the identity of the object in step S102. Here, "object" refers to a person attempting to unlock or start the vehicle; therefore, identification can be performed on such individuals. For example, if an object is close enough to the vehicle (e.g., within 2 meters), it can be considered an individual attempting to unlock or start the vehicle, and thus its identity can be identified. It should be noted that "attempting" here is a judgment mechanism from the vehicle's perspective; the vehicle considers any object meeting a certain condition as an individual attempting to unlock or start the vehicle, but this does not imply whether the object close to the vehicle is consciously attempting to unlock it.

[0026] Although the following text describes acquiring the object's facial image in step S104 and using it for subsequent expression recognition, if the object's identity is determined by facial recognition in step S102, then in step S102, the object's face will first be captured by a camera configured on the vehicle that can capture the external environment, and the captured image can be reused in step S104. This reuse process is also considered a form of "acquisition" in step S104.

[0027] In some embodiments of this application, step S102 can also determine the identity of the object based on the radio frequency signals transmitted by the mobile device carried by the object and the unlocking device carried by the object. For example, the vehicle's infotainment system can be paired with a user's mobile device, allowing the user to operate it on the large screen of the infotainment system. Once the user's mobile device and the infotainment system are successfully paired wirelessly (e.g., via Bluetooth or WLAN), they will automatically establish a connection upon power-on. Therefore, the identity of the object can be determined based on the mobile device carried by the object. If the mobile device carried by the object can establish a wireless connection with the infotainment system, then the object can be considered to be someone capable of unlocking or opening the vehicle.

[0028] On the other hand, many vehicles are equipped with keyless entry / keyless start functionality. If the unlocking device (e.g., an electronic key) carried by an individual sends information indicating that it can unlock or open the vehicle, then the individual carrying the unlocking device is also considered to have passed identity authentication.

[0029] In step S104 of method 10, if the object's identity matches the preset identity information, the object's facial image is acquired. This can be done by capturing the facial image in real-time using, for example, an external camera of the vehicle, or by using the facial image generated in step S102. The preset identity information can be stored in the vehicle as owner information; this preset identity information can be a Bluetooth identifier, key identifier, facial image, etc. If the identity verification result of step S102 indicates that the object has passed identity verification, then the object's facial image will be acquired in step S104.

[0030] In some examples, there may be more than one object around the vehicle, making it difficult to determine which one or more are authenticated. For instance, one might be the driver, and the others passengers. Furthermore, there might be unrelated pedestrians around the vehicle. In such cases, a simple approach is to obtain the facial image of the object closest to the driver's seat. Specifically, the object with the shortest line connecting its facial image to the left rearview mirror (e.g., by counting pixels along the line) can be selected for further processing.

[0031] Alternatively, facial images of objects walking directly towards a vehicle can be used as the facial images to be acquired and processed. For example, the movement trajectories of various objects in multiple sequentially captured images can be used to determine which object is walking directly towards the vehicle, and its facial image can then be processed accordingly.

[0032] On the other hand, it can also acquire target images of multiple objects. For example, objects within a certain range (e.g., within 2 meters) around a vehicle can be used as processing targets, and their facial images can be processed subsequently.

[0033] In some examples, if the object is identified by a facial image in step S102, then in step S104 only the object identified by facial recognition can be examined.

[0034] In step S106, method 10 determines the emotional state of the object based on the facial image. This application does not limit how the emotional state of an object is obtained from its facial image. For example, a neural network model can be used to process the facial image and obtain its corresponding emotional state. In some embodiments of this application, the emotional state can be happiness, calmness, sadness, fatigue, intoxication, anger, etc.

[0035] Furthermore, if multiple facial images are acquired in step S104, their emotional states can be acquired separately in step S106. In some examples, the state with the most emotional states among multiple objects can be taken as the "emotional state of the object" and used in subsequent processing. For example, if there are three objects around a vehicle, two of which have an emotional state of "happy" and the other has an emotional state of "calm," then the emotional state of the object determined in step S106 is "happy."

[0036] In other examples, each object's emotional state can be treated as an "object's emotional state" and included in subsequent processing. In this case, a connection can be established between the object and its corresponding emotional state. For example, object A: happy; object B: tired.

[0037] In step S108 of method 10, facial mimicry adapted to an emotional state is presented through the vehicle's peripheral devices. Facial mimicry, as used herein, refers to utilizing the vehicle's peripheral devices, combined with the vehicle's shape, to allow the user to associate an expression similar to that of a person or animal when observing or experiencing it from outside the vehicle. Peripheral devices refer to devices that can be directly observed from outside the vehicle, and these devices are generally located on the vehicle's outer contour. Furthermore, peripheral devices are not limited to presenting visual sensations but may also present auditory sensations. Specifically, in some embodiments of this application, peripheral devices may be rearview mirrors, headlights, horns, windshield wipers, tires, etc.

[0038] For example, if the facial expression corresponding to "happiness" is "happy," then this expression can be represented by moving the front wheels of the vehicle from side to side, rapidly opening and closing the rearview mirrors, and alternating between high and low beams of the headlights. This echoes the "happy" emotional state presented on the subject's face, thereby enhancing the driving experience.

[0039] For example, if the facial expression mimicking "sadness" is "soothing," then this mimicry can be achieved by turning the front wheels of the vehicle to one side and fixing them, slowly tilting the rearview mirrors up and down, and turning on the low beams of the headlights. This echoes the "sad" emotional state displayed on the subject's face, thus soothing the driver and passengers. Furthermore, the mimicry of "soothing" can also be accompanied by sound; for example, a soothing rhythm can be played using the vehicle's horn.

[0040] As described above, emotional states can be associated with multiple different objects. In this case, facial mimicry can be presented for a specific object (e.g., the driver) or separately for different objects (e.g., the driver and passengers). For example, facial mimicry can be presented only for the driver. When the emotional state of the object near the driver's seat is "happy," a "happy" facial mimicry can be presented through the vehicle's peripheral devices. In other examples, facial mimicry can be presented separately for the driver and passengers. If the emotional state of object A near the driver's seat is "happy," while the emotional state of object B near the passenger seat is "fatigued," then "happy" and "awakened" facial mimicry can be presented respectively to these two objects. Specifically, a "happy" facial mimicry can be represented by swinging the vehicle's front wheels to the far left (pointing to object A), rapidly opening and closing the rearview mirrors, and alternating high and low beam flashing of the headlights. After mimicking the expression of "happiness", the expression of "awakening" can be expressed by swinging the front wheel of the vehicle to the far right (pointing to object B), flipping the rearview mirror upwards, flashing the low beam of the headlights, and playing inspiring music on the horn.

[0041] In some embodiments of this application, the vehicle can be automatically unlocked when it displays facial expressions via its peripheral devices. In some examples, method 10 may also include the following step (not shown in the figures): automatically unlocking the vehicle if the identity of the object matches preset identity information. With this setting, occupants can confirm that the vehicle has been unlocked after observing the facial expressions, and can then directly open the car door and enter the vehicle.

[0042] If a driver exhibits a negative state unsuitable for driving, such as extreme sadness or intoxication, a reminder can be issued. In some embodiments of this application, method 10 may further include the following step (not shown in the figure): playing a prompt voice when the emotional state indicates a tendency towards negative driving. Thus, while the vehicle displays facial expressions through its peripheral devices, prompt messages (e.g., "You need to rest, please do not drive while fatigued," "There are many roads, but safety is the first priority") can be played, thereby reducing the driver's driving risk. In some embodiments of this application, the emotional state indicating a tendency towards negative driving may be fatigue, intoxication, anger, or waiting. When a driver is in such a negative state, they should calmly consider whether to continue driving; method 10, by playing a prompt voice, gives the driver an opportunity to reassess their state.

[0043] In some embodiments of this application, the vehicle may not automatically unlock when it displays facial expressions via its peripheral devices. Since the intention of the occupants to approach the vehicle is not clear, automatic unlocking may cause inconvenience, especially in scenarios where the occupants need to pass the vehicle multiple times. On the other hand, the vehicle may not automatically unlock when the driver exhibits a negative driving tendency, thus providing the driver with an opportunity to reassess their state. To this end, method 10 may further include the following step (not shown in the figure): unlocking the vehicle in response to the object's unlocking command if the object's identity matches preset identity information. Specifically, if the occupant is carrying an electronic key, an unlocking command can be sent to the vehicle when the occupant touches the door handle, at which point the vehicle will automatically unlock. In other examples, the occupant can also send the unlocking command via a remote key.

[0044] In some embodiments of this application, method 10 further includes the step of (not shown in the figures): rejecting the subject's instruction to unlock the vehicle. In some examples, unlocking the vehicle may also be denied if the driver or passenger is in a negative driving mood, thus providing the driver with an opportunity to reassess their state. For example, the subject's instruction to unlock the vehicle may be rejected on the driver's first three attempts, with a voice prompt provided. When the driver sends the instruction to unlock the vehicle for the fourth time, unlocking the vehicle may be allowed, thereby avoiding misjudgment affecting the driver's use of the vehicle.

[0045] In some embodiments of this application, method 10 further includes the following step (not shown in the figures): restoring the peripheral device to its state before displaying the facial expression mimicry. To facilitate the driver starting the vehicle with a familiar vehicle body configuration, the vehicle can be restored to its original state after displaying the facial expression mimicry through its peripheral device; specifically, it can be restored to the state it was in when it was last parked. This configuration avoids the dangers of altering the vehicle's posture during vehicle start-up and reduces the driver's adaptation costs.

[0046] The foregoing describes how facial expressions can be mimicked using peripheral devices before starting the vehicle. In other examples, similar facial expressions can also be mimicked when parking as a farewell gesture. Specifically, in some embodiments of this application, method 10 further includes the following steps (not shown in the figures): acquiring a second facial image of the object in response to an object's locking command; determining a second emotional state of the object based on the second facial image; and presenting a second facial expression mimicking the second emotional state via a peripheral device. The terms "second facial image," "second emotional state," and "second facial expression mimicking" described herein are used only to distinguish between "facial images," "emotional states," and "facial expressions" formed at different times and occasions, and do not imply that they necessarily have numerical or categorical differences.

[0047] The above description of some embodiments of method 10 describes how the identity of the vehicle owner or other objects can be confirmed via methods such as Bluetooth. Subsequently, images of the vehicle owner or other objects can be captured by cameras located outside the vehicle. Big data analysis can then be used to assess the owner's current mood (e.g., happiness, calmness, sadness). Finally, the vehicle control system can control the vehicle to provide corresponding feedback. This process allows the vehicle to proactively perceive human emotions and respond accordingly, making the interaction more vivid and engaging.

[0048] According to another aspect of this application, a vehicle is provided. For example... Figure 2 As shown, vehicle 20 includes an identity authentication module 201, an image acquisition module 202, an emotion analysis module 203, and peripheral devices 204. Vehicle 20 can mimic the emotional state of the driver and passengers, allowing it to, for example, celebrate with them when they are happy or encourage them when they are frustrated. This enhances the customer's driving experience and mitigates or even prevents the dangers posed by negative emotions during driving. The working principles of each part of vehicle 20 will be described in detail below.

[0049] The vehicle 20's authentication module 201 is configured to determine the identity of an object. Here, "object" refers to a person attempting to unlock or start the vehicle; therefore, identification can be performed on such individuals. For example, if an object is close enough to the vehicle (e.g., within 2 meters), then that object can be considered as someone attempting to unlock or start the vehicle, and thus their identity can be verified. It should be noted that "attempting" here refers to the vehicle's perspective and does not imply whether the object approaching the vehicle is consciously attempting to unlock it.

[0050] Although the image acquisition module 202 can acquire the facial image of the object and use it for subsequent expression recognition, if the identity authentication module 201 determines the identity of the object through facial recognition, then the identity authentication module 201 will first take a picture of the object's face through the camera configured on the vehicle that can capture the external environment, and the captured image can be further called by the image acquisition module 202. This calling process is also regarded as a form of "acquisition" by the image acquisition module 202.

[0051] The image acquisition module 202 of vehicle 20 is configured to acquire the facial image of the object when the identity authentication module 201 determines that the object's identity matches the preset identity information. This can be achieved by capturing the facial image in real-time using, for example, an external camera of the vehicle, or by using a facial image generated by the identity authentication module 201. The preset identity information can be stored in the vehicle as owner information; this preset identity information can be a Bluetooth identifier, key identifier, facial image, etc. If the identity authentication module 201 determines that the object has passed authentication, then the image acquisition module 202 will acquire the object's facial image.

[0052] In some examples, there may be more than one object around the vehicle. For example, one might be the driver, and the others might be passengers. Additionally, there might be unrelated pedestrians around the vehicle. In such cases, as a simple approach, the facial image of the object closest to the driver's seat can be obtained. Specifically, the object whose facial image has the shortest line connecting it to the left rearview mirror (e.g., the line can be counted by pixels) can be selected as the processing target, and its facial image can be further processed.

[0053] In other examples, other methods can be used to identify the object and obtain its facial image. For example, the image acquisition module 202 can refer to the specific steps of method 10 above to carry out its work, and the relevant content is also cited here.

[0054] The emotion analysis module 203 of vehicle 20 is configured to determine the emotional state of an object based on a facial image. This application does not limit how the emotional state of an object is obtained from its facial image. For example, the emotion analysis module 203 can process the facial image using a neural network model and obtain its corresponding emotional state. In some embodiments of this application, the emotional state can be happiness, calmness, sadness, fatigue, intoxication, anger, etc.

[0055] The peripheral devices 204 of vehicle 20 are configured to mimic facial expressions adapted to emotional states. In this context, mimicry refers to using the vehicle's peripheral devices, combined with the vehicle's shape, to allow a user to associate an expression with a human, animal, or similar emotion when observing or experiencing the vehicle from the outside. Peripheral devices are those that can be directly observed from the outside of the vehicle, and these devices are generally located on the vehicle's outer contour. Furthermore, peripheral devices are not limited to providing visual sensations but may also provide auditory sensations. Specifically, in some embodiments of this application, peripheral devices may include rearview mirrors, headlights, horns, windshield wipers, tires, etc.

[0056] Other aspects of the working principles of the vehicle 20's identity authentication module 201, image acquisition module 202, emotion analysis module 203, and peripheral device 204 can be referred to the various steps of method 10 above. The relevant content is also cited here, but due to space limitations, it will not be elaborated on here.

[0057] According to another aspect of this application, a vehicle is provided, comprising: a memory configured to store instructions; and a processor configured to execute the instructions to perform any of the methods described above.

[0058] According to another aspect of this application, a computer-readable storage medium is provided, wherein instructions are stored that, when executed by a processor, cause the processor to perform any of the vehicle interaction methods described above. The computer-readable medium referred to in this application includes various types of computer storage media, and can be any available medium accessible by a general-purpose or special-purpose computer. For example, the computer-readable medium may include RAM, ROM, EPROM, E... 2PROM, registers, hard disks, removable disks, CD-ROMs or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other temporary or non-temporary medium capable of carrying or storing desired units of program code in the form of instructions or data structures and accessible by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. As used herein, disks typically magnetically copy data, while discs optically copy data using lasers. Combinations of the above should also be included within the scope of computer-readable media. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0059] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Those skilled in the art can conceive of other feasible variations or substitutions based on the technical scope disclosed in this application, and such variations or substitutions are all covered within the scope of protection of this application. Where there is no conflict, the embodiments and features described in the embodiments of this application can also be combined with each other. The scope of protection of this application is determined by the claims.

Claims

1. A vehicle interaction method, characterized in that, The method includes: Determine the identity of the object; Acquiring the facial image of the object when the object's identity matches the preset identity information, wherein acquiring the facial image of the object when the object's identity matches the preset identity information includes: when the number of objects is greater than one, acquiring the facial images of each of the objects within a given range from the vehicle; Determine the emotional state of the object based on the facial image; The vehicle's peripheral devices present facial expressions that are adapted to the emotional state, wherein when the number of objects is greater than one, facial expressions adapted to the respective emotional state of each object are presented for each object. In response to a lock command of the object, acquire a second facial image of the object; Determine the object's second emotional state based on the second facial image; and The peripheral device presents a second facial expression mimicking the second emotional state.

2. The method according to claim 1, wherein, The identity of the object is determined based on the radio frequency signals transmitted by the mobile device carried by the object and / or the unlocking device carried by the object.

3. The method according to claim 1, wherein, The peripheral equipment includes at least one of the following: rearview mirror, headlight, horn, windshield wipers, and tires.

4. The method according to claim 1, further comprising: The vehicle is unlocked if the identity of the object matches the preset identity information.

5. The method according to claim 1, further comprising: A prompt voice will be played when the emotional state indicates a tendency towards negative driving.

6. The method according to claim 5, wherein, Emotional states that indicate a tendency to drive negatively include at least one of the following: fatigue, intoxication, or anger.

7. The method according to claim 5, further comprising: If the identity of the object matches the preset identity information, the vehicle is unlocked in response to the object's unlock command.

8. The method according to claim 5, further comprising: The object's instruction to unlock the vehicle was rejected.

9. The method according to claim 1, wherein, The emotional state includes at least one of the following: happiness, calmness, sadness, fatigue, intoxication, and anger.

10. The method according to claim 1, further comprising: The peripheral device is restored to its state before it displayed the facial expression mimicry.

11. A vehicle, characterized in that, The vehicles include: The identity authentication module is configured to determine the identity of an object. An image acquisition module is configured to acquire a facial image of an object when the identity authentication module determines that the object's identity matches preset identity information. The image acquisition module is further configured to acquire facial images of each object within a given range from the vehicle when the number of objects is greater than one. An emotion analysis module configured to determine the emotional state of the object based on the facial image; and A peripheral device configured to present facial expressions adapted to the emotional state, wherein the peripheral device is further configured to present facial expressions adapted to the respective emotional state of each object when the number of objects is greater than one. The image acquisition module is further configured to acquire a second facial image of the object in response to the object's locking command; The emotion analysis module is further configured to determine a second emotional state of the object based on the second facial image; and The peripheral device is also configured to present a second facial expression mimicking the second emotional state.

12. A vehicle, characterized in that, The vehicles include: A memory configured to store instructions; and A processor configured to execute the instructions to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium storing instructions, characterized in that, When the instruction is executed by the processor, it causes the processor to perform the method as described in any one of claims 1-10.

Images