Vehicle control method and device based on emotion recognition, equipment and medium

Abstract

The invention discloses a vehicle control method and device based on emotion recognition, equipment and a medium. The method comprises the following steps: acquiring real-time image data in a vehicle; determining the current position information of each preset face key point on the face of each driver and passenger according to the current face information; wherein the face positions covered by the preset face key points on the faces of different drivers and passengers are the same; determining the offset of each preset face key point according to the current position information and the initial position information of each preset face key point; wherein the initial position information can be the position information of each preset face key point when the corresponding face is in a calm state; and determining an emotional state corresponding to the vehicle according to the offset of each preset face key point, and controlling to execute a corresponding vehicle function according to the emotional state. According to the technical scheme, automatic control execution of corresponding vehicle functions is achieved, and the intelligent degree of corresponding vehicle control is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a vehicle control method, device, equipment and medium based on emotion recognition. Background Art

[0002] With the development of intelligent driving technology, cars are not just a means of transportation, they are becoming an intelligent mobile space. Accordingly, modern cars are increasingly using intelligent systems to control vehicle functions to enhance users' driving experience.

[0003] When controlling vehicle functions, the existing technology usually requires people in the vehicle to actively issue corresponding instruction information. It is impossible to autonomously control the vehicle functions based on the emotional state of the drivers and passengers in the vehicle, and the degree of intelligence is low. Summary of the invention

[0004] The present invention provides a vehicle control method, device, equipment and medium based on emotion recognition, so as to realize automatic control execution of corresponding vehicle functions by integrating the emotional states of each driver and passenger in the vehicle, thereby improving the intelligence level of corresponding vehicle control.

[0005] According to one aspect of the present invention, a vehicle control method based on emotion recognition is provided, the method comprising:

[0006] Acquire real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle;

[0007] Determine the current position information of each preset facial key point on the face of each driver or passenger according to the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers or passengers are the same;

[0008] Determine the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state;

[0009] The emotional state corresponding to the vehicle is determined according to the offset of each preset facial key point, and the corresponding vehicle function is controlled to be executed according to the emotional state.

[0010] According to another aspect of the present invention, a vehicle control device based on emotion recognition is provided, the device comprising:

[0011] A real-time image data acquisition module, used to acquire real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle;

[0012] A current position information determination module, used to determine the current position information of each preset facial key point on the face of each driver and passenger according to the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers and passengers are the same;

[0013] An offset determination module, used to determine the offset of each preset facial key point according to the current position information and initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state;

[0014] The vehicle function control execution module is used to determine the emotional state corresponding to the vehicle according to the offset of each preset facial key point, and control the execution of the corresponding vehicle function according to the emotional state.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vehicle control method based on emotion recognition described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the vehicle control method based on emotion recognition described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the vehicle control method based on emotion recognition as described in any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention comprehensively determines the emotional state of the corresponding vehicle by acquiring the real-time image data inside the vehicle and integrating the offset of the face position corresponding to each preset facial key point on the face of each driver and passenger, and further realizes the control of the corresponding vehicle function based on the determined emotional state corresponding to the corresponding vehicle, so that the control execution of the vehicle function can be realized autonomously without the need for the occupants to actively send out directive information, thereby helping to improve the intelligence level of the corresponding vehicle control. In addition, since the present application comprehensively determines the emotional state of the corresponding vehicle based on the offset of each face position corresponding to each driver and passenger, it helps to avoid the situation where the determined emotional state of the vehicle is not accurate enough due to only performing emotional recognition on a single driver and passenger, and the vehicle emotional state determination process is more complicated due to the need to perform emotional recognition on each driver and passenger separately, thereby improving the accuracy and efficiency of the emotional state recognition for the corresponding vehicle, thereby also helping to improve the matching degree of the controlled vehicle function and enhance the user experience.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1A This is a flow chart of a vehicle control method based on emotion recognition provided according to Embodiment 1 of the present invention.

[0025] Figure 1B The present invention is a schematic diagram of the corresponding facial positions of preset facial key points on a human face according to the first embodiment of the present invention.

[0026] Figure 2 This is a flow chart of another vehicle control method based on emotion recognition provided according to the second embodiment of the present invention.

[0027] Figure 3 This is a flow chart of another vehicle control method based on emotion recognition provided according to Embodiment 3 of the present invention.

[0028] Figure 4 4 is a schematic diagram of the structure of a vehicle control device based on emotion recognition provided according to Embodiment 4 of the present invention.

[0029] Figure 5 It is a structural schematic diagram of an electronic device for implementing the vehicle control method based on emotion recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Figure 1A This is a flow chart of a vehicle control method based on emotion recognition provided in the first embodiment of the present invention. This embodiment can be applied to the situation where the corresponding emotional state of the vehicle is determined and the corresponding vehicle control function is controlled based on the facial information changes of each driver and passenger in the vehicle. The method can be executed by a vehicle control device based on emotion recognition. The device can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device. Figure 1A Said method comprises:

[0033] S110, acquiring real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle.

[0034] Among them, the real-time image data can be acquired in real time by an image acquisition device installed at a designated position inside the vehicle. The designated position can be a position located in the front of the vehicle such as a reading light inside the vehicle. The image acquisition device can be a high-resolution red, green, blue (Red, Green, Blue, RGB) camera, etc. The image acquisition device can be used to capture image data of the interior of the vehicle in real time, and can also be used to capture video data of the interior of the vehicle in real time. The driver and passenger can be a driver or a passenger. The drivers and passengers inside the vehicle can include the driver and all passengers in the vehicle. The current facial information can be the facial state information presented by each driver and passenger in the corresponding real-time image data.

[0035] Specifically, the image of the interior of the vehicle can be captured in real time by the corresponding image acquisition device, and the image can be used as the real-time image data of the interior of the corresponding vehicle. Optionally, the video data of the interior of the vehicle can be captured in real time by the corresponding image acquisition device, and the image data corresponding to each frame of the video data can be used as the corresponding real-time image data.

[0036] S120. Determine the current position information of each preset facial key point on the face of each driver or passenger based on the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers or passengers are the same.

[0037] Among them, each preset facial key point can be a position point corresponding to a different pre-selected facial position. The number of preset facial key points on the face of each driver and passenger can be set according to needs or experience, for example, it can be 68. The number of preset facial key points on the face of each driver and passenger and the facial position covered can be the same. The facial position can be a specific position associated with emotions in the facial area, for example, it can be the corner of the eye area, the tip of the nose area, and the corner of the mouth area. Correspondingly, the facial area can be the area where the organs and / or facial contours of the face are located, for example, it can be the eye area, the nose area, and the mouth area. The current position information can be the position information of the preset facial key point in the image corresponding to the corresponding real-time image data, and the position information can be coordinate information. The set position of the image corresponding to the real-time image data can be the coordinate origin, and the set position can be the center of the image or the lower left corner of the image.

[0038] Specifically, the current facial information of each driver and passenger in the real-time image data can be used as the input of a pre-trained image recognition model, and the current position information of each preset facial key point on the face of each driver and passenger can be output based on the image recognition model. The corresponding image recognition model can be a neural network model, a machine learning model, or a deep learning model.

[0039] For ease of understanding, refer to Figure 1BA specific example of the corresponding face positions of each preset facial key point on a face is provided. It should be noted that the positions and numbers of each preset facial key point involved in the figure should not be understood as a specific limitation of the present invention. Figure 1B A schematic diagram of the corresponding facial positions of various preset facial key points on a face. Figure 1B As shown, each infrared point in the corresponding face can be a preset facial key point, and the facial position corresponding to the point can be a position corresponding to the preset facial key point.

[0040] S130. Determine the offset of each preset facial key point according to the current position information and initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state.

[0041] The initial position information may be determined by facial information of each driver and passenger in a calm state in historical image data of the interior of the vehicle. The initial position information may be obtained by identification based on a corresponding pre-trained image recognition model. The historical image data may be real-time image data of the interior of the vehicle acquired before the current moment. The deviation may be the direction and distance in which the current position of each preset facial key point moves relative to the initial position. The offset may also be the distance in which the current position of each preset facial key point moves relative to the initial position in different coordinate directions. The different coordinate directions may include the x-axis direction, the y-axis direction, and the z-axis direction.

[0042] Specifically, a first mapping relationship database between different current position information, initial position information and offsets corresponding to each preset facial key point can be established in advance. Accordingly, after determining the current position information and initial position information of the corresponding preset facial key point, the corresponding offset can be matched by traversing the first mapping relationship database and the matched offset can be used as the offset of the preset facial key point.

[0043] S140, determining an emotional state corresponding to the vehicle according to the offset of each preset facial key point, and controlling execution of a corresponding vehicle function according to the emotional state.

[0044] The emotional state may include at least one of a calm emotional state, a happy emotional state, a nervous emotional state, and a sad emotional state. One emotional state may correspond to at least one vehicle function. The same vehicle function may correspond to different emotional states. At least one emotional state corresponding to the vehicle may exist at the same time.

[0045] Specifically, a second mapping relationship database between different offsets corresponding to each preset facial key point and each emotional state can be pre-established. Accordingly, when determining the offset of each preset facial key point, the corresponding emotional state can be matched by traversing the second mapping relationship database and the matched emotional state can be used as the emotional state corresponding to the corresponding vehicle. Furthermore, according to the emotional state corresponding to the vehicle, at least one vehicle function corresponding to the emotional state can be controlled and executed.

[0046] The technical solution of the embodiment of the present invention comprehensively determines the emotional state of the corresponding vehicle by acquiring the real-time image data inside the vehicle and integrating the offset of the face position corresponding to each preset facial key point on the face of each driver and passenger, and further realizes the control of the corresponding vehicle function based on the determined emotional state corresponding to the corresponding vehicle, so that the control execution of the vehicle function can be realized autonomously without the need for the occupants to actively send out directive information, thereby helping to improve the intelligence level of the corresponding vehicle control. In addition, since the present application comprehensively determines the emotional state of the corresponding vehicle based on the offset of each face position corresponding to each driver and passenger, it helps to avoid the situation where the determined emotional state of the vehicle is not accurate enough due to only performing emotional recognition on a single driver and passenger, and the vehicle emotional state determination process is more complicated due to the need to perform emotional recognition on each driver and passenger separately, thereby improving the accuracy and efficiency of the emotional state recognition for the corresponding vehicle, thereby also helping to improve the matching degree of the controlled vehicle function and enhance the user experience.

[0047] Figure 2 This is a flow chart of a vehicle control method based on emotion recognition provided in the second embodiment of the present invention. This embodiment is further optimized based on the above embodiments. It should be noted that for the parts not described in detail in this embodiment, please refer to the relevant descriptions in other embodiments.

[0048] Further, “determining the emotional state corresponding to the vehicle according to the offset of each preset facial key point” is refined into “determining the preset facial key points on the faces of each driver and passenger corresponding to the same facial position according to the facial position corresponding to each preset facial key point; taking the average of the offsets of the preset facial key points on the faces of each driver and passenger corresponding to the same facial position as the offset corresponding to the same facial position; determining the emotional state corresponding to the vehicle based on the offsets corresponding to different facial positions covered by each preset facial key point”, so as to improve the mechanism for determining the emotional state corresponding to the vehicle.

[0049] refer to Figure 2 , the method specifically comprises the following steps:

[0050] S210, acquiring real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle.

[0051] S220. Determine the current position information of each preset facial key point on the face of each driver or passenger based on the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers or passengers are the same.

[0052] S230. Determine the offset of each preset facial key point according to the current position information and initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state.

[0053] In an optional embodiment, before determining the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point, the method may further include:

[0054] When it is detected that a driver or passenger has entered the vehicle, it is determined whether this is the first time that the driver or passenger has entered the vehicle based on real-time image data from the interior of the vehicle; if so, the position information of the preset facial key points corresponding to the facial positions of each face of the driver or passenger in a calm state is obtained, and this position information is used as the initial position information.

[0055] Specifically, the entry and exit of the vehicle passengers can be monitored in real time based on a preset image acquisition device. Accordingly, when it is detected that a passenger has entered the vehicle, the facial information of the passenger entering the vehicle can be identified based on the real-time image data of the vehicle interior collected at the current moment, and based on the database for storing the facial information of each passenger who has entered the vehicle, it can be determined whether the passenger is entering the vehicle for the first time. Accordingly, when it is determined that the corresponding passenger is not entering the vehicle for the first time, the acquisition of the position information of the preset facial key points corresponding to each facial position of the passenger in a calm state can be skipped.

[0056] It can be understood that by determining whether the driver or passenger is entering the vehicle for the first time when detecting that the driver or passenger is entering the vehicle, and acquiring the corresponding initial position information when it is determined that the driver or passenger is entering the vehicle for the first time, the corresponding initial position information can be acquired in a timely manner while avoiding the situation where the corresponding initial position information acquisition operation is performed for each time a person entering the vehicle is detected, thereby helping to reduce repeated acquisition of the corresponding initial position information and reduce the waste of computing resources.

[0057] S240. Determine, based on the face positions corresponding to the preset facial key points, the preset facial key points on the faces of the drivers and passengers corresponding to the same face positions.

[0058] The preset facial key points on each driver's face may correspond to different facial positions. The number of preset facial key points on each driver's face corresponding to the same facial position may be the same as the number of drivers in the vehicle.

[0059] S250: Taking the average of the offsets of the preset facial key points on the faces of the driver and passenger corresponding to the same facial position as the offset corresponding to the same facial position.

[0060] Specifically, the average offset may be calculated based on the offsets of preset facial key points on the faces of the drivers and passengers corresponding to the facial positions, and the corresponding average offset may be used as the offset corresponding to the corresponding facial positions.

[0061] S260: Determine the emotional state corresponding to the vehicle based on the offsets corresponding to the different face positions covered by the preset facial key points, and control the execution of corresponding vehicle functions according to the emotional state.

[0062] Specifically, a third mapping relationship database can be established between the offset corresponding to each facial position and each emotional state. Accordingly, after determining the offset corresponding to each facial position, the corresponding emotional state can be matched by traversing the third mapping relationship database, and the matched emotional state can be used as the emotional state corresponding to the corresponding vehicle.

[0063] In an optional embodiment, before determining the offset of each preset facial key point based on the current position information and initial position information of each preset facial key point, the method may also include: obtaining a standard facial image in a calm state, and determining the standard initial position information of the preset facial key points corresponding to the different facial positions; accordingly, determining the emotional state corresponding to the vehicle based on the offsets corresponding to the different facial positions covered by the preset facial key points may include: determining the standard current position information corresponding to the different facial positions based on the offsets corresponding to the different facial positions and the standard initial position information, and updating the standard facial image in the calm state based on the standard current position information; inputting the updated standard facial image into a pre-trained emotion recognition model to obtain the emotional state corresponding to the vehicle.

[0064] The standard face image may be an image corresponding to a standard face in a calm state that is predetermined for determining an emotional state. The standard initial position information may be initial position information corresponding to each preset facial key point in the standard face image. The standard current position information may be position information after the standard current position information is offset by a corresponding offset. The pre-trained emotion recognition model may be a neural network model, a machine learning model, or a deep learning model, for example, a classification model such as a residual network ResNet.

[0065] Specifically, a fourth mapping relationship database can be pre-established between the offsets corresponding to each different face position, the standard initial position information and the standard current position information corresponding to each different face position. Accordingly, after determining the offsets and the corresponding standard initial position information corresponding to each different face position, the standard current position information corresponding to each different face position can be matched by traversing the fourth mapping relationship database, and the matched standard current position information corresponding to each different face position can be used as the standard current position information corresponding to the current different face position.

[0066] It can be understood that by introducing a standard facial image in a calm state and applying the determined offsets corresponding to the corresponding different facial positions to the standard facial image, an updated standard facial image is obtained, and then the emotional state is determined based on the updated standard facial image, thereby effectively avoiding the impact of personalized differences caused by different facial features of different drivers and passengers on emotion recognition, and thus helping to improve the accuracy of determining the emotional state of the corresponding vehicle.

[0067] In an embodiment of the present invention, by taking the average of the offsets of preset facial key points on the faces of each driver or occupant corresponding to the same facial position as the offset corresponding to the same facial position, and determining the emotional state of the corresponding vehicle based on the average, it helps to solve the problem of large amount of calculation and being affected by extreme situations caused by determining the corresponding emotional state based on the offsets of all preset facial key points corresponding to each driver or occupant, thereby helping to improve the efficiency and reliability of determining the corresponding emotional state.

[0068] Figure 3 This is a flow chart of a vehicle control method based on emotion recognition provided in Embodiment 3 of the present invention. This embodiment is further optimized based on the above embodiments. It should be noted that for the parts not described in detail in this embodiment, please refer to the relevant descriptions in other embodiments.

[0069] Further, “determine the offset of each preset facial key point based on the current position information and initial position information of each preset facial key point” is refined into “determine the candidate offset of each preset facial key point based on the current position information and initial position information of each preset facial key point; determine the distance between two set facial positions on the face of each driver and passenger, and determine the target offset of each preset facial key point based on the distance and the candidate offset of each preset facial key point on the corresponding driver and passenger's face; use the target offset of each preset facial key point as the offset of each preset facial key point” to improve the offset determination mechanism.

[0070] refer to Figure 3 , the method specifically comprises the following steps:

[0071] S310, acquiring real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle.

[0072] S320. Determine the current position information of each preset facial key point on the face of each driver or passenger based on the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers or passengers are the same.

[0073] S330. Determine a candidate offset for each preset facial key point according to the current position information and the initial position information of each preset facial key point.

[0074] S340: Determine the distance between two set face positions on each driver or passenger's face, and determine the target offset of each preset facial key point based on the distance and the candidate offset of each preset facial key point on the corresponding driver or passenger's face.

[0075] The candidate deviation may be the direction and distance of the current position of each preset facial key point relative to the initial position. The two set face positions on the face of the driver and passenger may be selected as needed, for example, the position of the inner corner of the left eye and the position of the inner corner of the right eye. The target offset may be the ratio of the corresponding candidate offset to the distance between the corresponding two set face positions.

[0076] Optionally, a fifth mapping relationship database can be established in advance, which includes the distance between two set facial positions on the driver's face, the candidate offsets corresponding to each preset facial key point on the driver's face, and each target offset. Accordingly, when determining the corresponding candidate offset and the corresponding distance, the fifth mapping relationship database can be traversed and matched, and the matched corresponding target offset can be used as the target offset of the corresponding preset facial key point.

[0077] S350: Using the target offset of each preset facial key point as the offset of each preset facial key point.

[0078] S360: Determine the emotional state corresponding to the vehicle according to the offset of each preset facial key point, and control the execution of the corresponding vehicle function according to the emotional state.

[0079] Exemplarily, according to the emotional state, controlling execution of corresponding vehicle functions may include:

[0080] According to the emotional state, control switching of the vehicle's driving mode and / or the type of music playlist; wherein the emotional state at least includes a calm state, a happy state and a nervous state, and the driving mode at least includes a comfort mode and a safety mode.

[0081] The driving mode may include a comfort mode and a safety mode, etc. The calm state and the happy state may correspond to the comfort mode first, and the nervous state may correspond to the safety mode first. The types of music playlists may include a soothing type and a high-pitched type, etc. The calm state and the nervous state may correspond to a soothing type of music playlist first, and the happy state may correspond to a high-pitched type of music playlist first.

[0082] Specifically, according to the pre-established correspondence between each emotional state and the corresponding driving mode and / or music play list, based on the determined emotional state of the corresponding vehicle, the corresponding driving mode and / or play list can be determined, and the switch to the driving mode and / or music play list can be controlled.

[0083] It can be understood that by determining and switching to the corresponding driving mode and / or music playlist based on the type of emotional state, when the emotional state inside the corresponding vehicle is identified, more appropriate vehicle functions can be automatically executed to provide the occupants with a more comfortable and safe driving experience.

[0084] In an optional embodiment, the position information of the preset facial key points may be obtained through a high-resolution network HRNet, wherein the input of HRNet may be an image corresponding to the corresponding real-time image data and / or a corresponding standard face image, and the output may be the position information of the corresponding preset facial key points.

[0085] In an embodiment of the present invention, by determining the distance between two set facial positions on the face of each driver and passenger, and based on the distance and the candidate offsets of each preset facial key point on the corresponding driver and passenger face, the target offset of each preset facial key point is determined, and then the target offset is used as the offset of the corresponding preset facial key point to achieve normalization of the offsets of each preset facial key point corresponding to different drivers and passengers, thereby helping to eliminate the influence of individual differences on the recognition of emotional states and improve the accuracy and reliability of the determination of the corresponding emotional states.

[0086] Figure 4 A vehicle control device based on emotion recognition is provided in the fourth embodiment of the present invention. This embodiment can be applied to the situation where the corresponding emotional state of the vehicle is determined and the corresponding vehicle control function is controlled based on the changes in the facial information of each driver and passenger in the vehicle. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 4 As shown, the device comprises:

[0087] The real-time image data acquisition module 410 is used to acquire real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle;

[0088] The current position information determination module 420 is used to determine the current position information of each preset facial key point on the face of each driver and passenger according to the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers and passengers are the same;

[0089] The offset determination module 430 is used to determine the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state;

[0090] The vehicle function control execution module 440 is used to determine the emotional state corresponding to the vehicle according to the offset of each preset facial key point, and control the execution of the corresponding vehicle function according to the emotional state.

[0091] The technical solution of the embodiment of the present invention comprehensively determines the emotional state of the corresponding vehicle by acquiring the real-time image data inside the vehicle and integrating the offset of the face position corresponding to each preset facial key point on the face of each driver and passenger, and further realizes the control of the corresponding vehicle function based on the determined emotional state corresponding to the corresponding vehicle, so that the control execution of the vehicle function can be realized autonomously without the need for the occupants to actively send out directive information, thereby helping to improve the intelligence level of the corresponding vehicle control. In addition, since the present application comprehensively determines the emotional state of the corresponding vehicle based on the offset of each face position corresponding to each driver and passenger, it helps to avoid the situation where the determined emotional state of the vehicle is not accurate enough due to only performing emotional recognition on a single driver and passenger, and the vehicle emotional state determination process is more complicated due to the need to perform emotional recognition on each driver and passenger separately, thereby improving the accuracy and efficiency of the emotional state recognition for the corresponding vehicle, thereby also helping to improve the matching degree of the controlled vehicle function and enhance the user experience.

[0092] Optionally, the vehicle function control execution module 440 may include:

[0093] A preset facial key point determination unit, used to determine the preset facial key points on the faces of the drivers and passengers corresponding to the same facial position according to the facial positions corresponding to the preset facial key points;

[0094] A same face position offset determination unit, used to take the average of the offsets of the preset facial key points on the faces of the drivers and passengers corresponding to the same face position as the offset corresponding to the same face position;

[0095] The emotional state determination unit is used to determine the emotional state corresponding to the vehicle based on the offsets corresponding to the different face positions covered by the preset facial key points.

[0096] Optionally, the corresponding device may further include:

[0097] a standard initial position information determining unit, configured to obtain a standard face image in a calm state and determine the standard initial position information of the preset facial key points corresponding to the different face positions before determining the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point;

[0098] Accordingly, the emotional state determination unit may include:

[0099] A standard face image updating subunit, configured to determine standard current position information corresponding to each different face position according to the offset and standard initial position information corresponding to each different face position, and update the standard face image in the calm state based on the standard current position information;

[0100] The emotional state acquisition subunit is used to input the updated standard face image into a pre-trained emotion recognition model to obtain the emotional state corresponding to the vehicle.

[0101] Optionally, the offset determination module 430 includes:

[0102] a candidate offset determination unit, configured to determine a candidate offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point;

[0103] a target offset determination unit, configured to determine a distance between two set face positions on each driver and passenger's face, and determine a target offset of each preset facial key point based on the distance and a candidate offset of each preset facial key point on the corresponding driver and passenger's face;

[0104] The offset determination unit is used to use the target offset of each preset facial key point as the offset of each preset facial key point.

[0105] Optionally, the corresponding device may further include:

[0106] a first entry determination unit, configured to determine whether the driver or passenger enters the vehicle for the first time based on the real-time image data of the interior of the vehicle when detecting that a driver or passenger enters the vehicle before determining the offset of each preset facial key point based on the current position information and the initial position information of each preset facial key point;

[0107] The initial position information determining unit is used to obtain the position information of the preset facial key points corresponding to the face positions of each face of the driver and passenger in a calm state, and use the position information as the initial position information.

[0108] Optionally, the vehicle function control execution module 440 may include:

[0109] A vehicle function control switching unit is used to control the switching of the vehicle's driving mode and / or the type of music playlist according to the emotional state; wherein the emotional state at least includes a calm state, a happy state and a tense state, and the driving mode at least includes a comfort mode and a safety mode.

[0110] Optionally, the position information of the preset facial key points may be obtained through a high-resolution network HRNet.

[0111] The vehicle control device based on emotion recognition provided in the embodiment of the present invention can execute any one of the vehicle control methods based on emotion recognition provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution of each vehicle control method based on emotion recognition. For the contents not described in detail in the embodiment of the present invention, reference can be made to the description in any one of the vehicle control methods based on emotion recognition in the embodiment of the present invention.

[0112] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0113] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0115] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a vehicle control method based on emotion recognition.

[0117] In some embodiments, the vehicle control method based on emotion recognition can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the vehicle control method based on emotion recognition described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the vehicle control method based on emotion recognition in any other appropriate manner (for example, by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0123] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0124] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0125] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0127] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A vehicle control method based on emotion recognition, characterized in that: include: Acquire real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle; Determine the current position information of each preset facial key point on the face of each driver or passenger according to the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers or passengers are the same; Determine the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state; The emotional state corresponding to the vehicle is determined according to the offset of each preset facial key point, and the corresponding vehicle function is controlled to be executed according to the emotional state.

2. The method according to claim 1, characterized in that Determining the emotional state corresponding to the vehicle according to the offset of each preset facial key point includes: According to the face positions corresponding to the preset facial key points, the preset facial key points on the faces of the drivers and passengers corresponding to the same face positions are determined; The average value of the offsets of the preset facial key points on the faces of the drivers and passengers corresponding to the same facial position is used as the offset corresponding to the same facial position; Based on the offsets corresponding to the different face positions covered by the preset facial key points, the emotional state corresponding to the vehicle is determined.

3. The method according to claim 2, characterized in that Before determining the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point, the method further includes: Acquire a standard human face image in a calm state, and determine standard initial position information of preset facial key points corresponding to each of the different human face positions; Accordingly, based on the offsets corresponding to the different face positions covered by the preset facial key points, determining the emotional state corresponding to the vehicle includes: Determining standard current position information corresponding to each different face position according to the offset and standard initial position information corresponding to each different face position, and updating the standard face image in the calm state based on the standard current position information; The updated standard face image is input into a pre-trained emotion recognition model to obtain the corresponding emotional state of the vehicle.

4. The method according to claim 1, characterized in that: Determining the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point includes: Determining a candidate offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point; Determine the distance between two set face positions on each driver and passenger's face, and determine the target offset of each preset facial key point based on the distance and the candidate offset of each preset facial key point on the corresponding driver and passenger's face; The target offset of each preset facial key point is used as the offset of each preset facial key point.

5. The method according to claim 1, characterized in that: Before determining the offset of each preset facial key point according to the current position information and the initial position information of each preset facial key point, the method further includes: When detecting that a driver or passenger has entered the vehicle, determining whether the driver or passenger is entering the vehicle for the first time based on real-time image data of the interior of the vehicle; If so, the position information of the preset facial key points corresponding to the face positions of each of the driver and passenger in a calm state is obtained, and the position information is used as the initial position information.

6. The method according to claim 1, characterized in that According to the emotional state, the corresponding vehicle functions are controlled and executed, including: According to the emotional state, control switching of the vehicle's driving mode and / or the type of music playlist; wherein the emotional state at least includes a calm state, a happy state and a nervous state, and the driving mode at least includes a comfort mode and a safety mode.

7. The method according to any one of claims 1 to 6, characterized in that: The position information of the preset facial key points is obtained through a high-resolution network HRNet.

8. A vehicle control device based on emotion recognition, characterized in that: include: A real-time image data acquisition module, used to acquire real-time image data of the interior of the vehicle; wherein the real-time image data includes current facial information of each driver and passenger in the vehicle; A current position information determination module, used to determine the current position information of each preset facial key point on the face of each driver and passenger according to the current facial information; wherein the facial positions covered by the preset facial key points on the faces of different drivers and passengers are the same; An offset determination module, used to determine the offset of each preset facial key point according to the current position information and initial position information of each preset facial key point; wherein the initial position information may be the position information of each preset facial key point when the corresponding face is in a calm state; The vehicle function control execution module is used to determine the emotional state corresponding to the vehicle according to the offset of each preset facial key point, and control the execution of the corresponding vehicle function according to the emotional state.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vehicle control method based on emotion recognition described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the vehicle control method based on emotion recognition according to any one of claims 1 to 7 when executed.

Images