Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, and the method comprises the steps: determining a face key point of a target object in a to-be-processed image in a driving process of a target vehicle; wherein the face key points at least comprise mouth corner key points corresponding to the mouth part; determining a current face datum line corresponding to the target object according to the rotation parameter of the steering wheel and the initial face datum line; based on the current face datum line and the mouth corner key points, the current state of the target object is determined, and according to the current state, a control instruction corresponding to the at least one component is determined, so that the corresponding component executes operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle. According to the technical scheme provided by the embodiment of the invention, the state information corresponding to the target object can be dynamically determined, so that the corresponding part is started, and the effects of driving safety and interestingness are achieved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle safety driving, and in particular to a data processing method, device, electronic equipment and storage medium. Background Art

[0002] With the widespread use of vehicles, driving safety has become an urgent issue to be solved. During the driving process, the driver's mood directly affects driving safety and vehicle operation efficiency.

[0003] Traditionally, driver emotion monitoring mainly relies on analyzing the driver's facial expressions to determine the driver's emotions, and then controls corresponding components based on the driver's emotions to achieve driving safety.

[0004] However, the above method has the main problem of: when only relying on the user's facial expression to analyze it, other factors affecting it are not taken into account, and there is a problem of low accuracy in emotion determination. Summary of the invention

[0005] The present invention provides a data processing method, device, electronic device and storage medium to effectively and accurately determine the current state corresponding to a target object, thereby performing an assisted driving function.

[0006] According to one aspect of the present invention, there is provided a data processing method, the method comprising:

[0007] During the driving process of the target vehicle, determining the facial key points of the target object in the image to be processed; wherein the facial key points at least include the mouth corner key points corresponding to the mouth part;

[0008] Determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line; wherein the initial facial reference line is determined based on the key points of the inner corners of the eyes and the key points of the geometric center of the mouth of the target object in an initial state;

[0009] Determining a current state of the target object based on the current facial baseline and the mouth corner key points;

[0010] According to the current state, a control instruction corresponding to at least one component is determined so that the corresponding component performs an operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle.

[0011] Further, the determining of facial key points of the target object in the image to be processed includes: collecting the image to be processed including the target object based on a camera device deployed in the target vehicle;

[0012] Inputting the image to be processed into a pre-trained key point recognition model to mark the facial key points on the target object of the image to be processed;

[0013] Among them, the eye key points in the facial key points include the inner eye corner key points, and the mouth key points in the facial key points include the mouth corner key points and the mouth geometric center key points.

[0014] Furthermore, the rotation parameters of the steering wheel are determined based on the following method:

[0015] When it is detected that the steering wheel has a steering angle, a final rotation angle of the steering wheel within a preset time period is acquired, and the final rotation angle is used as a rotation parameter of the steering wheel.

[0016] Further, determining a current facial reference line corresponding to the target object according to the rotation parameter of the steering wheel and the initial facial reference line includes:

[0017] When the rotation parameter of the steering wheel is within a first preset parameter range, controlling the initial reference line to move in a first direction to obtain a current facial reference line corresponding to the target object;

[0018] When the first turning parameter of the steering wheel is within a second preset parameter range, the initial reference line is controlled to move toward a second direction to obtain a current facial reference line corresponding to the target object.

[0019] Further, the distance information for controlling the initial reference line to move in the first direction or the second direction is determined based on the following method:

[0020] Obtaining the angle value in the rotation parameter;

[0021] The distance information is determined based on a preset coefficient and the angle value.

[0022] Further, the determining the current state of the target object based on the current facial baseline and the mouth corner key points includes:

[0023] Determining the relative position between the mouth corner key point and the current facial reference line;

[0024] If the relative position is the first direction of the mouth corner key point on the current facial reference line, determining that the current state of the target object is the first state;

[0025] If the relative position is that the mouth corner key point is in the second direction of the current facial reference line, it is determined that the current state of the target object is the second state.

[0026] Further, the current state includes a first state or a second state, and determining a control instruction corresponding to at least one component according to the current state includes:

[0027] When the current state is the first state, sending a control instruction to the target application so that the target application plays the audio and video data corresponding to the first state;

[0028] When the current state is the second state, a control instruction is sent to the assisted driving system to put the assisted driving system into a working state. According to another aspect of the present invention, a data processing device is provided, the device comprising:

[0029] A facial key point acquisition module is used to determine the facial key points of the target object in the image to be processed during the driving process of the target vehicle; wherein the facial key points at least include the mouth corner key points corresponding to the mouth part;

[0030] A facial reference line dynamic determination module, used to determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line; wherein the initial facial reference line is determined based on the key points of the inner corners of the eyes and the key points of the geometric center of the mouth of the target object in an initial state;

[0031] A user state determination module, used to determine the current state of the target object based on the current facial baseline and the mouth corner key points;

[0032] A control module is used to determine a control instruction corresponding to at least one component according to the current state, so that the corresponding component performs an operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle.

[0033] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0034] at least one processor; and

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method described in any embodiment of the present invention.

[0037] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present invention when executed.

[0038] According to another aspect of the present invention, a computer program product is provided, characterized in that the computer program product comprises a computer program, and when the computer program is executed by a processor, the data processing method described in any embodiment of the present invention is implemented.

[0039] The technical solution of the embodiment of the present invention can identify facial key points of the acquired image to be processed during the driving of the target vehicle to obtain facial key points corresponding to the target object. At the same time, the initial facial baseline of the target object in the initial state can be adjusted according to the rotation parameters of the steering wheel to obtain the current facial baseline of the target object at the current moment. The current state of the target object can be determined according to the current facial baseline and the identified facial key points. According to the current state, the target instruction is determined to make the target component corresponding to the target instruction work, so as to achieve the effect of assisted driving, solve the problem that when the driver's emotions are determined only by relying on facial features in the prior art, there is a single factor to consider and it is impossible to effectively and accurately judge it, and realizes the comprehensive consideration of the user's current state in combination with the steering wheel rotation parameters and the user's facial features, improves the accuracy of state determination, and thus improves the effect of assisted driving.

[0040] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0044] Figure 3 is a flow chart of a data processing method provided according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of facial key points provided according to an embodiment of the present invention;

[0046] Figure 5is a schematic diagram of an initial facial baseline provided according to an embodiment of the present invention;

[0047] Figure 6 is a structural schematic diagram of a data processing device provided according to an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of the structure of an electronic device for implementing the data processing method according to an embodiment of the present invention; DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] Figure 1 The present invention provides a flow chart of a data processing method according to an embodiment of the present invention. The present embodiment can be applied to any scenario where the user status needs to be determined, and can be particularly applied to determining the current status of the driver in a vehicle and then performing assisted driving. The method can be executed by a data processing device, which can be implemented in the form of hardware and / or software, and can be configured in any target vehicle that needs assisted driving.

[0052] like Figure 1 As shown, the method includes:

[0053] S110 , during the driving process of the target vehicle, determining the facial key points of the target object in the image to be processed.

[0054] In the process of driving the target vehicle, in order to achieve the effect of assisted driving, that is, to ensure the safety or fun of the target object during driving, the image to be processed including the target object can be collected in real time. After the image to be processed is collected, the facial key points of the target object in the image to be processed can be recognized to determine the facial key points of the target object.

[0055] In this embodiment, the facial key points include at least the mouth corner key points corresponding to the mouth area.

[0056] In this embodiment, determining the facial key points of the target object in the image to be processed can be: based on a camera device deployed in the target vehicle, collecting the image to be processed including the target object; inputting the image to be processed into a pre-trained key point recognition model to mark the facial key points on the target object in the image to be processed; wherein the eye key points among the facial key points include inner eye corner key points, and the mouth key points among the facial key points include mouth corner key points and mouth geometric center key points.

[0057] Among them, a camera device can be deployed in the target vehicle so that the camera device can collect images to be processed including the target object. The user in the main driving position of the target vehicle is taken as the target object. The method of determining the facial key points can be based on facial recognition technology to identify, or based on a deep learning algorithm to identify and track the facial key points of the target object. Optionally, if a deep algorithm is used to identify facial key points, it can be identified based on a neural network. For example, the above-mentioned neural network is used as a key point recognition model. The facial key points include the corner of the mouth key points and the mouth geometric center key points. The corner of the mouth key points can be the key points corresponding to the corners of the mouth part. The mouth geometric center key points can be the key points corresponding to the center of the entire mouth part.

[0058] Specifically, an image to be processed including a target object can be collected based on a camera device deployed in a target vehicle. The image to be processed is input into a key point recognition model to mark facial key points on the target object in the image to be processed. That is, the key points of the corners of the mouth and the key points of the geometric center of the mouth are marked on the target object in the image to be processed.

[0059] S120: Determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line.

[0060] Wherein, the rotation parameter of the steering wheel may include the rotation angle of the steering wheel, and optionally, the rotation angle may be a positive value or a negative value, and its specific positive and negative angles are determined according to the rotation direction of the steering wheel. The initial facial reference line is determined based on the inner eye corner key points and the mouth geometric center key points of the target object in the initial state. The initial state may be a state in which the target object is facing the steering wheel. The initial facial reference line may be determined based on the facial key points of the target object in this state. Optionally, the facial key points of the target object in the initial state are the inner eye corner key points and the mouth center key points. Based on the inner eye corner key points and the mouth center key points, the initial facial reference line may be determined. The current facial reference line is determined based on the rotation parameter and the initial facial reference line.

[0061] Specifically, the rotation parameters of the steering wheel may be acquired in real time, so as to determine the current facial reference line that matches the rotation parameters according to the rotation parameters and the initial facial reference line of the target object in the initial state.

[0062] In this embodiment, the rotation parameter of the steering wheel is determined based on the following method: when a steering angle of the steering wheel is detected, a final rotation angle of the steering wheel within a preset time period is obtained, and the final rotation angle is used as the rotation parameter of the steering wheel.

[0063] It can be understood that: the final rotation angle of the steering wheel within the preset time window can be obtained, and the final rotation angle is used as the rotation parameter of the steering wheel.

[0064] That is to say, when a steering angle of the steering wheel is detected, the final rotation angle of the steering wheel within a certain period of time can be obtained, and the final rotation angle is used as the rotation parameter of the steering wheel.

[0065] In this embodiment, determining the current facial baseline corresponding to the target object based on the rotation parameters of the steering wheel and the initial facial baseline includes: when the rotation parameters of the steering wheel are within a first preset parameter range, controlling the initial facial baseline to move in a first direction to obtain the current facial baseline corresponding to the target object; when the rotation parameters of the steering wheel are within a second preset parameter range, controlling the initial facial baseline to move in a second direction to obtain the current facial baseline corresponding to the target object.

[0066] The first preset parameter range is pre-set and used to determine the parameter range of the initial facial reference line moving in the first direction. The second preset parameter range is pre-set and used to determine the parameter range of the initial facial reference line moving in the second direction. The first direction may be a direction perpendicular to the horizontal plane and away from the horizontal plane, and the second direction is a direction perpendicular to the horizontal plane and toward the horizontal plane.

[0067] Exemplarily, the first preset parameter range is [-30°, 30°], and the second preset parameter range is the rotation angle excluding the first preset parameter range. In the process of the target object controlling the vehicle driving, not only the facial image including the target object can be collected, but also the rotation parameter of the steering wheel can be obtained. If the rotation parameter is within the first preset parameter range, the initial facial reference line is moved in the first direction; if the rotation parameter is within the second preset parameter range, the initial facial reference line is moved in the second direction.

[0068] Based on the above technical solution, no matter the initial facial reference line moves in the first direction or in the second direction, the moving distance is dynamically related to the rotation parameter.

[0069] Optionally, the distance information for controlling the initial baseline to move in the first direction or the second direction is determined based on the following method: obtaining an angle value in the rotation parameter; and determining the distance information based on a preset coefficient and the angle value.

[0070] Specifically, the angle value in the rotation parameter, that is, the final rotation angle of the steering wheel, is obtained, and the angle is multiplied by a preset coefficient to obtain the moving distance of the initial facial reference line. Based on the moving distance, the initial facial reference line is controlled to move along the first direction or the second direction to obtain the current facial reference line.

[0071] S130: Determine the current state of the target object based on the current facial baseline and the mouth corner key points.

[0072] The current state mainly refers to whether the target object is in a happy state or a sad state. The current state of the target object can be determined based on the relative position of the mouth corner key point and the current facial baseline.

[0073] S140. Determine a control instruction corresponding to at least one component according to the current state, so that the corresponding component performs an operation corresponding to the control instruction.

[0074] Wherein, the at least one component is a component in the target vehicle.

[0075] Specifically, after determining the current state of the target object, a control instruction consistent with the current state can be determined according to a pre-established correspondence, and the control instruction can be sent to a corresponding component to put the component into a working state.

[0076] The technical solution of the embodiment of the present invention can identify facial key points of the acquired image to be processed during the driving of the target vehicle to obtain facial key points corresponding to the target object. At the same time, the initial facial baseline of the target object in the initial state can be adjusted according to the rotation parameters of the steering wheel to obtain the current facial baseline of the target object at the current moment. The current state of the target object can be determined according to the current facial baseline and the identified facial key points. According to the current state, the target instruction is determined to make the target component corresponding to the target instruction work, so as to achieve the effect of assisted driving, solve the problem that when the driver's emotions are determined only by relying on facial features in the prior art, there is a single factor to consider and it is impossible to effectively and accurately judge it, and realizes the comprehensive consideration of the user's current state in combination with the steering wheel rotation parameters and the user's facial features, improves the accuracy of state determination, and thus improves the effect of assisted driving.

[0077] Figure 2 It is a flow chart of a data processing method provided by an embodiment of the present invention. On the basis of the above-mentioned embodiment, the current state of the target object determined based on the current facial baseline and the key points of the corners of the mouth can be further optimized. The specific implementation method can refer to the detailed description of the present technical solution, wherein the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated in this embodiment.

[0078] like Figure 2 As shown, the method includes:

[0079] S210. During the driving process of the target vehicle, determine the facial key points of the target object in the image to be processed; wherein the facial key points at least include the mouth corner key points corresponding to the mouth area.

[0080] For example, during the driving process of the target vehicle, as long as the image to be processed including the target object is collected, the image to be processed can be analyzed and processed to identify the facial key points of the target object in the image to be processed. At this time, the facial key points at least include the mouth corner key points corresponding to the mouth part.

[0081] S220: Determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line.

[0082] Exemplarily, when a steering angle of the steering wheel is detected, a rotation parameter of the steering wheel is obtained. According to the rotation parameter, a moving direction and a moving distance of the initial facial reference line are determined, so as to adjust the current facial reference line adapted to the steering wheel rotation parameter based on the moving direction and the moving distance.

[0083] It should be noted that the image to be processed including the target object may be acquired in real time, so that when the rotation parameter of the steering wheel is detected, the facial key points of the target object in the image to be processed are determined, and the offset distance information of the initial facial baseline is determined, so as to determine the current facial baseline. Alternatively, when the rotation parameter of the steering wheel is detected, the image to be processed including the target object is acquired and analyzed and processed to determine the current facial baseline.

[0084] S230: Determine the relative position between the mouth corner key point and the current facial reference line.

[0085] Among them, the relative position is used to characterize whether the mouth corner key point is located above or below the current facial baseline.

[0086] Exemplarily, the first line in the initial facial baseline can be determined based on the two inner eye corner key points of the target object in the initial state, and a second line parallel to the first line can be made based on the mouth center key point. The initial facial baseline is determined based on the first line and the second line. Accordingly, the shape of the current facial baseline is the same as the shape of the initial facial baseline, and the only difference is that the relative height to the horizontal plane is different. Accordingly, the second line in the current facial baseline can be used as a reference line to determine the relative position of the mouth corner key point and the second line, that is, whether the mouth corner key point is above the second line or below the second line. The above and below are used as the relative position between the mouth corner key point and the current facial baseline.

[0087] S240: If the relative position is that the mouth corner key point is in the first direction of the current facial baseline, determine that the current state of the target object is the first state.

[0088] It can be understood that if the relative position is that the mouth corner key point is in the first direction of the current facial baseline, that is, above the second line in the current facial baseline, then it can be determined that the current state of the target object is the first state. That is, the mouth corner of the target object is in an upturned state, and at this time, the first state is a happy state.

[0089] S250: If the relative position is that the mouth corner key point is in the second direction of the current facial baseline, determine that the current state of the target object is the second state.

[0090] It can be understood that if the relative position is that the mouth corner key point is in the second direction of the current facial baseline, that is, below the second line in the current facial baseline, then it can be determined that the current state of the target object is the second state. That is, the mouth corner of the target object is in a pulled-down state, and at this time, the second state is an unhappy state.

[0091] S260: When the current state is the first state, send a control instruction to a target application so that the target application plays audio and video data corresponding to the first state.

[0092] The target application may be an application for playing audio or video.

[0093] It can be understood that: if the current state is the first state, a control instruction can be sent to the target application so that the target application can play the audio and video data corresponding to the first state.

[0094] S270: When the current state is the second state, send a control instruction to the assisted driving system to put the assisted driving system into a working state.

[0095] It can be understood that if the current state is the second state, a control instruction can be sent to the assisted driving system so that the assisted driving system can be in a working state to assist the target object in driving, thereby ensuring the safety of the target object driving the vehicle in a sad state.

[0096] The technical solution of the embodiment of the present invention can identify facial key points of the acquired image to be processed during the driving of the target vehicle to obtain facial key points corresponding to the target object. At the same time, the initial facial baseline of the target object in the initial state can be adjusted according to the rotation parameters of the steering wheel to obtain the current facial baseline of the target object at the current moment. The current state of the target object can be determined according to the current facial baseline and the identified facial key points. According to the current state, the target instruction is determined to make the target component corresponding to the target instruction work, so as to achieve the effect of assisted driving, solve the problem that when the driver's emotions are determined only by relying on facial features in the prior art, there is a single factor to consider and it is impossible to effectively and accurately judge it, and realizes the comprehensive consideration of the user's current state in combination with the steering wheel rotation parameters and the user's facial features, improves the accuracy of state determination, and thus improves the effect of assisted driving.

[0097] As an alternative embodiment of the above embodiment, Figure 3 A data processing method flow chart provided by an embodiment of the present invention is based on the above embodiment and combined with Figure 3 as well as Figure 4 The corresponding key points are used to explain the solution provided by the embodiment of the present invention.

[0098] Data collection,Data collection mainly includes facial data collection and steering wheel angle data collection.

[0099] Among them, facial data collection can be: using a high-resolution camera on the vehicle to capture the driver's (target object) facial image in real time. The camera can have a sufficient frame rate (such as above 30fps) to capture subtle changes in expression. Steering wheel angle data collection can be: using the vehicle's built-in sensor to record the angle changes of the steering wheel in real time. These data should be collected at a high frequency to ensure that fast operating actions are captured.

[0100] Keypoint detection and tracking.

[0101] Use facial landmark recognition technology (e.g. deep learning-based algorithms) to identify and track facial landmarks, especially the corners of the mouth and the inner corners of the eyes. Figure 4 , the green key points are the left and right inner corners of the eyes, the blue key points are the left and right corners of the mouth, and the yellow key point is the midpoint of the lips.

[0102] Determine the initial facial baseline.

[0103] Determine the initial facial baseline. The initial facial baseline is translated from the line connecting the inner corner key points to the key point passing through the center of the lips.

[0104] In this embodiment, the above method is used to determine the initial facial baseline mainly because:

[0105] 1. Facial symmetry

[0106] The inner corner of the eye key point is a relatively stable feature point in the facial structure, located on the symmetry axis of the face. Using the line connecting these two points as a baseline can take advantage of the symmetry of the face and provide a natural and stable reference standard, which helps to accurately measure and analyze the position changes of other facial feature points (such as the corners of the mouth).

[0107] 2. Reduce errors

[0108] Head tilt or different viewing angles may affect the absolute position measurement of facial landmarks, especially compared to a horizontal or vertical reference line. By defining the reference line as a line connecting the inner corners of the eyes, this line naturally adjusts with head movement, thus reducing measurement errors caused by changes in head posture.

[0109] 3. Strong adaptability

[0110] The inner canthus line not only reflects the natural state of facial expression, but also naturally adapts to differences between individuals. Everyone's facial structure is slightly different, and using the inner canthus keypoint as a reference point can achieve better standardization and comparability between individuals.

[0111] During the driving process of the target vehicle, the facial key points of the target object are determined, and the position of the initial facial baseline is adjusted according to the rotation angle of the steering wheel to obtain the current facial baseline.

[0112] The rotation parameters of the steering wheel may be detected by: detecting the angle change of the steering wheel, especially the rapid change in a short time, and defining a time window Δt (for example, 1S or 2S) to measure the maximum angle change Δθ of the steering wheel within this time.

[0113] Next, the offset of the initial facial baseline is determined.

[0114] Based on the change in the steering wheel angle, the adjustment amount of the initial facial baseline is calculated. A scaling factor k can be set, which determines how the change in the steering wheel angle affects the adjustment of the initial facial baseline. The following formula can be used:

[0115] Δy=k×Δθ.

[0116] Δy is the offset of the initial facial baseline, k is the scaling factor, which determines the influence of the steering wheel rotation parameters on the position of the initial facial baseline, and Δθ is the angle change of the steering wheel within Δt.

[0117] Dynamically adjust the initial facial baseline.

[0118] Based on the offset of the initial facial baseline calculated above, the position of the initial facial baseline is adjusted. If the rapid change of the steering wheel angle indicates that there may be an abnormality in the external environment, the initial facial baseline should be moved upward (making the corners of the mouth more likely to be judged as being below the baseline, tending to show pessimism). Otherwise, the baseline is adjusted downward to make it tend to be optimistic, such as Figure 5 That is to say, the moving direction of the initial facial reference line can be determined not only according to the change angle of the steering wheel within the preset time length, but also according to the change rate of the steering wheel within the preset time length.

[0119] Emotional state judgment

[0120] The driver's emotional state is judged using the adjusted current facial baseline. This can be done by comparing the positions of the mouth corner keypoints relative to the adjusted current facial baseline.

[0121] Through this method, the emotion judgment system can be made more sensitive and adaptable to changes in the driving environment, thereby providing more accurate driver emotion monitoring and more timely and effective driving assistance. This dynamic adjustment mechanism can greatly improve the practicality and effectiveness of the system.

[0122] Car-machine function linkage

[0123] The vehicle computer functions are adjusted according to the driver's current emotional state. For example, when the driver is in an optimistic mood, cheerful music and ambient lighting are switched; when the driver is in a pessimistic mood, the access priority of the autonomous driving emotion is adjusted to respond to possible accidents at any time and avoid unexpected events.

[0124] The solution provided by the embodiment of the present invention is, first, to improve accuracy, that is, by incorporating the angle change of the steering wheel into the emotion analysis, the system can more accurately reflect the driver's emotions in the actual driving environment. Second, enhance adaptability: the system can automatically adjust the parameters of emotion assessment to adapt to sudden driving situations, improving the ability to respond to emergencies. Third, improve driving safety: accurate emotion monitoring can prevent driving errors caused by emotional problems, thereby improving overall driving safety.

[0125] The technical solution of the embodiment of the present invention can identify facial key points of the acquired image to be processed during the driving of the target vehicle to obtain facial key points corresponding to the target object. At the same time, the initial facial baseline of the target object in the initial state can be adjusted according to the rotation parameters of the steering wheel to obtain the current facial baseline of the target object at the current moment. The current state of the target object can be determined according to the current facial baseline and the identified facial key points. According to the current state, the target instruction is determined to make the target component corresponding to the target instruction work, so as to achieve the effect of assisted driving, solve the problem that when the driver's emotions are determined only by relying on facial features in the prior art, there is a single factor to consider and it is impossible to effectively and accurately judge it, and realizes the comprehensive consideration of the user's current state in combination with the steering wheel rotation parameters and the user's facial features, improves the accuracy of state determination, and thus improves the effect of assisted driving.

[0126] Figure 6 A structural schematic diagram of a data processing device provided by an embodiment of the present invention, the device comprises: a facial key point acquisition module 310, a facial baseline dynamic determination module 320, a user state determination module 330 and a control module 340.

[0127] Among them, the facial key point acquisition module 310 is used to determine the facial key points of the target object in the image to be processed during the driving process of the target vehicle; wherein the facial key points include at least the corner of mouth key points corresponding to the mouth part; the facial baseline dynamic determination module 320 is used to determine the current facial baseline corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial baseline; wherein the initial facial baseline is determined based on the inner eye corner key points and the mouth geometric center key points of the target object in the initial state; the user state determination module 330 is used to determine the current state of the target object based on the current facial baseline and the corner of mouth key points; the control module 340 is used to determine the control instruction corresponding to at least one component according to the current state, so that the corresponding component performs the operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle.

[0128] The technical solution of the embodiment of the present invention can identify facial key points of the acquired image to be processed during the driving of the target vehicle to obtain facial key points corresponding to the target object. At the same time, the initial facial baseline of the target object in the initial state can be adjusted according to the rotation parameters of the steering wheel to obtain the current facial baseline of the target object at the current moment. The current state of the target object can be determined according to the current facial baseline and the identified facial key points. According to the current state, the target instruction is determined to make the target component corresponding to the target instruction work, so as to achieve the effect of assisted driving, solve the problem that when the driver's emotions are determined only by relying on facial features in the prior art, there is a single factor to consider and it is impossible to effectively and accurately judge it, and realizes the comprehensive consideration of the user's current state in combination with the steering wheel rotation parameters and the user's facial features, improves the accuracy of state determination, and thus improves the effect of assisted driving.

[0129] Based on the above technical solution, the facial key point acquisition module includes:

[0130] An image acquisition unit is used to acquire an image to be processed including a target object based on a camera device deployed in a target vehicle; a facial key point recognition unit is used to input the image to be processed into a pre-trained key point recognition model to mark the facial key points on the target object in the image to be processed; wherein the eye key points in the facial key points include inner eye corner key points, and the mouth key points in the facial key points include mouth corner key points and mouth geometric center key points.

[0131] Based on the above technical solutions, the rotation parameter of the steering wheel is determined in the following way: when it is detected that the steering wheel has a steering angle, the final rotation angle of the steering wheel within a preset time length is obtained, and the final rotation angle is used as the rotation parameter of the steering wheel.

[0132] On the basis of the above technical solutions, the facial baseline dynamic determination module includes:

[0133] A first adjustment unit, configured to control the initial reference line to move in a first direction to obtain a current facial reference line corresponding to the target object when the rotation parameter of the steering wheel is within a first preset parameter range;

[0134] The second adjustment unit is used to control the initial reference line to move in a second direction when the first turning parameter of the steering wheel is within a second preset parameter range, so as to obtain a current facial reference line corresponding to the target object.

[0135] On the basis of the above technical solutions, the distance information for controlling the initial reference line to move in the first direction or the second direction is determined based on the following method:

[0136] Obtaining the angle value in the rotation parameter;

[0137] The distance information is determined based on a preset coefficient and the angle value.

[0138] On the basis of the above technical solutions, the user status determination module includes:

[0139] A relative position determination unit, used to determine the relative position between the mouth corner key point and the current facial reference line;

[0140] A first state determining unit, configured to determine that the current state of the target object is a first state if the relative position is that the mouth corner key point is in a first direction of the current facial reference line;

[0141] The second state determining unit is used to determine that the current state of the target object is the second state if the relative position is that the mouth corner key point is in the second direction of the current facial reference line.

[0142] On the basis of the above technical solutions, the current state includes the first state or the second state, and the control module includes:

[0143] A first control unit, configured to send a control instruction to a target application when the current state is a first state, so that the target application plays audio and video data corresponding to the first state;

[0144] The second control unit is used to send a control instruction to the assisted driving system when the current state is the second state, so that the assisted driving system is in a working state.

[0145] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0146] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0147] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0148] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0149] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.

[0150] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0151] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0156] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: During the driving process of the target vehicle, determining the facial key points of the target object in the image to be processed; wherein the facial key points at least include the mouth corner key points corresponding to the mouth part; Determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line; wherein the initial facial reference line is determined based on the key points of the inner corners of the eyes and the key points of the geometric center of the mouth of the target object in an initial state; Determining a current state of the target object based on the current facial baseline and the mouth corner key points; According to the current state, a control instruction corresponding to at least one component is determined so that the corresponding component performs an operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle.

2. The method according to claim 1, characterized in that: The step of determining facial key points of a target object in an image to be processed comprises: Collecting an image to be processed including a target object based on a camera device deployed in the target vehicle; Inputting the image to be processed into a pre-trained key point recognition model to mark the facial key points on the target object of the image to be processed; Among them, the eye key points in the facial key points include the inner eye corner key points, and the mouth key points in the facial key points include the mouth corner key points and the mouth geometric center key points.

3. The method according to claim 1, characterized in that The rotation parameters of the steering wheel are determined based on the following method: When it is detected that the steering wheel has a steering angle, a final rotation angle of the steering wheel within a preset time period is acquired, and the final rotation angle is used as a rotation parameter of the steering wheel.

4. The method according to claim 2, characterized in that: The step of determining a current facial reference line corresponding to the target object according to the rotation parameter of the steering wheel and the initial facial reference line comprises: When the rotation parameter of the steering wheel is within a first preset parameter range, controlling the initial facial reference line to move in a first direction to obtain a current facial reference line corresponding to the target object; When the first turning parameter of the steering wheel is within a second preset parameter range, the initial facial reference line is controlled to move in a second direction to obtain a current facial reference line corresponding to the target object.

5. The method according to claim 4, characterized in that The distance information for controlling the initial facial reference line to move toward the first direction or the second direction is determined based on the following method: Obtaining the angle value in the rotation parameter; The distance information is determined based on a preset coefficient and the angle value.

6. The method according to claim 1, characterized in that The determining the current state of the target object based on the current facial reference line and the mouth corner key point comprises: Determining the relative position between the mouth corner key point and the current facial reference line; If the relative position is the first direction of the mouth corner key point on the current facial reference line, determining that the current state of the target object is the first state; If the relative position is that the mouth corner key point is in the second direction of the current facial reference line, it is determined that the current state of the target object is the second state.

7. The method according to claim 1, characterized in that The current state includes a first state or a second state, and determining a control instruction corresponding to at least one component according to the current state includes: When the current state is the first state, sending a control instruction to the target application so that the target application plays the audio and video data corresponding to the first state; When the current state is the second state, a control instruction is sent to the assisted driving system to put the assisted driving system into a working state.

8. A data processing device, characterized in that: include: A facial key point acquisition module is used to determine the facial key points of the target object in the image to be processed during the driving process of the target vehicle; wherein the facial key points at least include the mouth corner key points corresponding to the mouth part; A facial reference line dynamic determination module, used to determine a current facial reference line corresponding to the target object according to the rotation parameters of the steering wheel and the initial facial reference line; wherein the initial facial reference line is determined based on the key points of the inner corners of the eyes and the key points of the geometric center of the mouth of the target object in an initial state; A user state determination module, used to determine the current state of the target object based on the current facial baseline and the mouth corner key points; A control module is used to determine a control instruction corresponding to at least one component according to the current state, so that the corresponding component performs an operation corresponding to the control instruction; wherein the at least one component is a component in the target vehicle.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 7 when executed.

Images