Expression recognition method, device, equipment and medium based on motion information correction

Abstract

The present invention discloses an expression recognition method, device, equipment and medium based on action information correction. The method comprises: convolution processing of the standardized video corresponding to the target vehicle to generate a target video frame sequence containing each target video frame; based on the first key point detection model, action recognition processing of each target video frame determines the target action information, and based on the second key point detection model, detection processing of each target video frame determines the basic facial key points; based on the target action information and the preset action standard, the basic facial key points are positionally corrected to generate the target facial key points; based on the fully connected expression classification network, the target facial key points are classified and processed, the expression category corresponding to the target facial key points is determined, and the expression category is used as the target expression recognition information corresponding to the standardized video. Through the technical solution of the present invention, the facial expression of the driver in the car can be recognized in real time, and the accuracy and robustness of expression recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an expression recognition method, device, equipment and medium based on motion information correction. Background Art

[0002] With the development of intelligent transportation and autonomous driving technology, monitoring the driver's emotional state has become particularly important in vehicle safety systems. Existing technologies generally use in-vehicle monitoring systems to identify the driver's facial expressions to assess their emotional state and predict potential driving risks.

[0003] However, the existing facial expression recognition methods mainly rely on static facial expression analysis, which leads to a decrease in recognition accuracy when the face is partially blocked or the expression information is insufficient. Therefore, how to recognize the facial expression of the driver in the car in real time and improve the accuracy of expression recognition is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides an expression recognition method, device, equipment and medium based on motion information correction, which can solve the problem of low accuracy of facial expression recognition results of drivers in vehicles.

[0005] According to one aspect of the present invention, there is provided a method for facial expression recognition based on motion information correction, comprising:

[0006] Obtaining a standardized video corresponding to the target vehicle, and convolutionally processing the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame;

[0007] Processing each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and processing each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video;

[0008] Based on the target action information and the preset action standard, the basic facial key points are positionally corrected to generate target facial key points corresponding to the basic facial key points;

[0009] The target facial key points are classified and processed based on a fully connected expression classification network, the expression category corresponding to the target facial key points is determined, and the expression category is used as the target expression recognition information corresponding to the standardized video.

[0010] According to another aspect of the present invention, there is provided an expression recognition device based on motion information correction, comprising:

[0011] A data acquisition module is used to acquire a standardized video corresponding to a target vehicle, and convolute the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame;

[0012] A key point determination module, configured to process each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and to process each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video;

[0013] A position correction module, used to perform position correction on the basic facial key points based on the target action information and a preset action standard, and generate target facial key points corresponding to the basic facial key points;

[0014] The expression recognition module is used to classify and process the target facial key points based on a fully connected expression classification network, determine the expression category corresponding to the target facial key points, and use the expression category as the target expression recognition information corresponding to the standardized video.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the expression recognition method based on motion information correction described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the expression recognition method based on motion information correction described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention generates a target video frame sequence including each target video frame by convolution processing the standardized video corresponding to the target vehicle, then determines the target action information by processing each target video frame based on the first key point detection model action recognition, and determines the basic facial key points by detecting and processing each target video frame based on the second key point detection model, further, performs position correction on the basic facial key points based on the target action information and preset action standards to generate target facial key points, and finally, processes the target facial key points based on the fully connected expression classification network classification, determines the expression category corresponding to the target facial key points, and uses the expression category as the target expression recognition information corresponding to the standardized video, thereby solving the problem of low accuracy of facial expression recognition results of the driver in the car, being able to perform real-time recognition of the facial expression of the driver in the car, and improving the accuracy and robustness of expression recognition.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of an expression recognition method based on motion information correction provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of an expression recognition method based on motion information correction provided according to Embodiment 2 of the present invention;

[0025] Figure 3 is a schematic diagram of facial key points to be selected according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a flowchart of a mapping process provided according to Embodiment 2 of the present invention;

[0027] Figure 5 is a flowchart of an optional expression recognition method based on motion information correction provided according to Embodiment 2 of the present invention;

[0028] Figure 6 is a structural schematic diagram of an expression recognition device based on motion information correction provided according to Embodiment 3 of the present invention;

[0029] Figure 7 The present invention is a schematic diagram of the structure of an electronic device for implementing the expression recognition method based on motion information correction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", "target", "base", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0033] Embodiment 1

[0034] Figure 1 This is a flowchart of an expression recognition method based on motion information correction provided by the first embodiment of the present invention. This embodiment can be applied to the situation of real-time recognition of the facial expression of the driver in the vehicle. The method can be executed by an expression recognition device based on motion information correction. The expression recognition device based on motion information correction can be implemented in the form of hardware and / or software. The expression recognition device based on motion information correction can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110, obtaining a standardized video corresponding to the target vehicle, and performing convolution processing on the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame.

[0036] The target vehicle may refer to a vehicle whose facial expression needs to be recognized. The standardized video may refer to a video generated after standardization.

[0037] In an optional embodiment, the step of obtaining a standardized video corresponding to the target vehicle includes: receiving a data acquisition permission instruction issued by the target vehicle, and obtaining a real-time video inside the target vehicle based on the data acquisition permission instruction; and processing the real-time video in a standardized manner to generate a standardized video corresponding to the target vehicle. The data acquisition permission instruction may refer to a data authorization instruction issued by the driver. Exemplarily, when it is necessary to obtain data corresponding to the target vehicle, a data acquisition request may be sent to the target vehicle, and then the driver may review the data acquisition request and send a data acquisition permission instruction to the data that is allowed to be accessed, so as to implement data authorization and ensure the security of user data. The real-time video may refer to a dynamic video collected in real time inside the target vehicle. Exemplarily, a high-resolution camera installed on the dashboard of the target vehicle or the connecting pillars (A-pillar, A-pillar) connecting the roof and the front cabin on the left and right front sides may be used to collect real-time video of the driver inside the target vehicle. Standardization processing may refer to related processing operations for optimizing and adjusting the real-time video. Exemplarily, operations such as brightness adjustment, contrast optimization, and denoising may be performed. Therefore, by standardizing the real-time video of the target vehicle with authorized access, the real-time video collected in real time can be optimized and adjusted to adapt to subsequent video analysis processing.

[0038] The convolution processing may refer to convolution calculations implemented using a convolution layer. For example, a convolution layer of size , a convolution layer with a step size of 2 completes the convolution process. In this way, both the initial feature extraction can be achieved and the image size can be reduced by half, reducing the overall computational complexity of subsequent steps. The target video frame may refer to an image frame generated after the standardized video is convoluted. The target video frame sequence may refer to a set of target video frames corresponding to the standardized video.

[0039] S120, processing each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and processing each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video.

[0040] The first key point detection model may refer to a key point detection model trained using a limb key point data set. The action recognition process may refer to an operation of recognizing action information in a target video frame. The target action information may refer to an action behavior contained in a target video frame. For example, the target action information may include clapping, hugging arms, patting thighs, or touching the face.

[0041] The second key point detection model may refer to a key point detection model trained using a facial key point data set. The detection process may refer to an operation of identifying facial key points in a target video frame. The basic facial key points may refer to facial key points initially obtained after detection process of a target video frame.

[0042] S130, performing position correction on the basic facial key points based on the target action information and a preset action standard, and generating target facial key points corresponding to the basic facial key points.

[0043] The preset action standard may refer to a preset expression and action association standard. For example, a sad expression may be associated with a hand covering the face action, an angry expression may be associated with a fist clenching action, a happy expression may be associated with a hand clapping action, a surprised expression may be associated with a hand raising action, a confused expression may be associated with a hand touching the chin, and a bored expression may be associated with a chin supporting action.

[0044] The position correction may refer to an operation of correcting the coordinates of key points in the basic facial key points, and the target facial key points may refer to the coordinates of the facial key points obtained after the position correction.

[0045] S140, classifying and processing the target facial key points based on a fully connected expression classification network, determining the expression category corresponding to the target facial key points, and using the expression category as target expression recognition information corresponding to the standardized video.

[0046] Among them, the fully connected expression classification network may refer to a neural network based on a multilayer perceptron (MLP) for expression classification. Generally, the fully connected expression classification network may include three layers: an input layer, a hidden layer, and an output layer. Classification processing may refer to an operation of determining the expression category corresponding to the target facial key points. The expression category may refer to the expression information corresponding to the target facial key points. Exemplarily, it may be happy, sad, etc. The target expression recognition information may refer to the expression category corresponding to the standardized video.

[0047] The technical solution of the embodiment of the present invention generates a target video frame sequence including each target video frame by convolution processing the standardized video corresponding to the target vehicle, then determines the target action information by processing each target video frame based on the first key point detection model action recognition, and determines the basic facial key points by detecting and processing each target video frame based on the second key point detection model, further, performs position correction on the basic facial key points based on the target action information and preset action standards to generate target facial key points, and finally, processes the target facial key points based on the fully connected expression classification network classification, determines the expression category corresponding to the target facial key points, and uses the expression category as the target expression recognition information corresponding to the standardized video, thereby solving the problem of low accuracy of facial expression recognition results of the driver in the car, being able to perform real-time recognition of the facial expression of the driver in the car, and improving the accuracy and robustness of expression recognition.

[0048] Embodiment 2

[0049] Figure 2 A flowchart of an expression recognition method based on action information correction is provided in the second embodiment of the present invention. This embodiment is refined based on the above-mentioned embodiment. In this embodiment, the operation of performing position correction on the basic facial key points based on the target action information and preset action standards to generate target facial key points corresponding to the basic facial key points is specifically refined. It may specifically include: judging whether the target action information meets the target action category based on the preset action standard; if the target action information meets the target action category, obtaining the upper limb key points corresponding to the target action information; performing position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points. Figure 2 As shown, the method includes:

[0050] S210, obtaining a standardized video corresponding to the target vehicle, and performing convolution processing on the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame.

[0051] S220, detecting and processing each of the target video frames based on the first key point detection model to generate upper limb key points corresponding to the standardized video.

[0052] Among them, the upper limb key points may refer to the limb nodes related to the upper body movements of the driver in the target vehicle. Exemplarily, the upper limb key points may include trunk key points and finger key points. The trunk key points may include different positions such as the center of the neck, the junction of the neck and the left shoulder, the junction of the neck and the right shoulder, the left shoulder, the right shoulder, the left elbow, the right elbow, the left abdomen, the right abdomen, the center of the left and right abdomens and the heart. The finger key points may include the wrist and the joints of the five fingers, for a total of 21 positions. Usually, the upper limb key points can be represented by the coordinate values ​​corresponding to each key point.

[0053] Specifically, the first key point detection model can extract the coordinates of all limb nodes related to the driver's upper body movements in the target video frame, thereby providing an effective basis for subsequent operations.

[0054] S230, classifying and processing the upper limb key points based on a preset action classification model to determine target action information corresponding to the standardized video.

[0055] The preset action classification model may refer to a pre-trained action classification model. The action category corresponding to the standardized video can be determined by the preset action classification model.

[0056] Therefore, by utilizing the key points of the upper limbs to perform action recognition based on a preset action classification model and generating target action information, it is possible to perform expression recognition without relying entirely on facial information when the driver's face is blocked or the monitoring angle of view changes. At the same time, background interference in non-interested areas in the car can be eliminated, thereby improving the accuracy of expression recognition.

[0057] In addition, since expressions often do not appear in isolation when facing complex environments and situations, they are closely related to the individual's actions and the situation they are in. For example, a person may clap their hands or pat their thighs when laughing, and may hug their arms or touch their face when sad. Therefore, by performing action recognition, more contextual information can be provided for expressions, so as to more accurately understand and identify the true emotions expressed by expressions. Moreover, some complex emotional states may not be fully expressed only through facial expressions, they may need to be more accurately conveyed through a combination of body language. For example, tension or anxiety may be expressed by constant movement of feet and shaking of fingers. Therefore, the introduction of action information can help identify such complex emotions.

[0058] It is worth noting that when determining the target action information corresponding to the standardized video, the upper limb key points corresponding to all target video frames in the target video frame sequence need to be simultaneously sent to the preset action classification model for action recognition.

[0059] S240 , uniformly sample and process the target video frames to generate sampled video frames corresponding to the standardized video.

[0060] The uniform sampling process may refer to a sampling operation of acquiring a continuous time signal at a fixed sampling interval. For example, the fixed sampling interval may be set to 3 frames or 5 frames, etc. The sampled video frame may refer to each image frame obtained by uniformly sampling the target video frame.

[0061] S250, detecting and processing each of the sampled video frames based on the second key point detection model to generate candidate facial key points corresponding to the sampled video frames.

[0062] The facial key points to be selected may refer to the facial key point coordinates generated after key point extraction is performed on the sampled video frame based on the second key point detection model.

[0063] Specifically, the face image in each sampled video frame can be selected by using the face coordinate detection model. Usually, a rectangular frame can be used for selection, and the face coordinates are the coordinates of the upper left corner and the lower right corner of the rectangular frame. Then, the face coordinates are used to extract the image to obtain a partial image containing the face image. Finally, the second key point detection model is used to detect the key points of the partial image to obtain the facial key points to be selected, which provides an effective basis for subsequent operations.

[0064] Figure 3 FIG. 1 is a schematic diagram of a facial key point to be selected provided by an embodiment of the present invention, wherein 68 position points related to facial expressions are included.

[0065] S260: average each candidate facial key point to generate basic facial key points corresponding to the standardized video.

[0066] The mean processing may refer to an operation of taking the mean of the coordinates of each candidate facial key point. The mean processing may be used to fuse each candidate facial key point in the standardized video, thereby ensuring the accuracy of the basic facial key points.

[0067] It is worth noting that the order of the upper limb key point detection process and the basic facial key point detection process is not limited in the embodiment of the present invention. Exemplarily, the upper limb key point detection process can be performed first, and then the basic facial key point detection process. The basic facial key point detection process can also be performed first, and then the upper limb key point detection process. The upper limb key point detection process and the basic facial key point detection process can also be completed in parallel.

[0068] S270: Determine whether the target action information meets the target action category based on a preset action standard.

[0069] The target action category may refer to a pre-selected action that is strongly related to a specific emotion. For example, the target action category may be an associated action in a preset action standard, such as covering the face, clenching a fist, clapping hands, raising hands, touching the chin with hands, or supporting the chin.

[0070] In the in-car monitoring system, the driver may present a variety of postures and movements, which vary widely and are not all directly related to the driver's emotional state. If the motion information is introduced indiscriminately for expression recognition, the recognition accuracy may be reduced due to incorrect motion information, leading to misjudgment. For example, the driver's actions of adjusting the seat posture or operating the vehicle system may have nothing to do with emotional expression. If these actions are mistakenly interpreted as emotion-related actions, it may interfere with the expression recognition results. Therefore, it is necessary to clearly define which actions are directly related to specific emotional expressions. In this way, we can focus on those action information that have a clear contribution to expression recognition, thereby avoiding the introduction of irrelevant action information that has a negative impact on the recognition results.

[0071] S280: If the target action information satisfies the target action category, obtain upper limb key points corresponding to the target action information.

[0072] Specifically, if the target action information obtained based on the first key point detection model is included in the target action category, the coordinates of the upper limb key points corresponding to the target action information can be obtained for use in subsequent operations.

[0073] S290. Perform position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points.

[0074] In an optional embodiment, the position correction of the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points includes: mapping the upper limb key points to generate position offsets corresponding to the basic facial key points; adding and processing the position offsets and the basic facial key points to generate target facial key points corresponding to the basic facial key points.

[0075] The mapping process may refer to the operation of converting the upper limb key point information into the position offset of the facial key point. The position offset may refer to the coordinate amount of the basic facial key point coordinates that need to be offset and corrected. The summing process may refer to the operation of converting the mapped upper limb key point information into the position offset of the facial key point. The position offset is added to the corresponding coordinates of the 68 basic facial key points detected.

[0076] Figure 4 FIG. 1 is a flow chart of a mapping process provided by an embodiment of the present invention. Specifically, first, the coordinates of N upper limb key points are expanded into The vector is a vector of 2, where 2 can represent the two values ​​of the horizontal coordinate x and the vertical coordinate y. After that, the vector is input to the fully connected layer and output after passing through the hidden layer. Position offset, where 2 can represent the corresponding horizontal and vertical coordinate offset.

[0077] Therefore, the mapping process can fuse the upper limb key point coordinates with the basic facial key point coordinates without increasing excessive computing resources, thereby improving the accuracy of the facial key point coordinates. At the same time, by adding the position offset and the basic facial key point coordinates, the basic facial key point coordinates can be corrected and enhanced.

[0078] S2100, classifying and processing the target facial key points based on a fully connected expression classification network, determining the expression category corresponding to the target facial key points, and using the expression category as target expression recognition information corresponding to the standardized video.

[0079] Specifically, based on the fully connected expression classification network, the expression recognition probability of each expression corresponding to the target facial key point can be obtained, and the expression with the highest probability is used as the expression category corresponding to the target facial key point to achieve fully connected classification.

[0080] S2110. Based on a pre-set function linkage rule, determine a target linkage function that matches the target expression recognition information, and trigger execution of the target linkage function.

[0081] Among them, the function linkage rule may refer to a pre-set rule for matching facial expressions with vehicle functions. For example, the function linkage rule may be to switch to cheerful music when the driver is happy, or to adjust the interior lights to a dark tone when the driver is sad. The target linkage function may refer to a function in the function linkage rule that matches the target expression recognition information.

[0082] Specifically, if the target expression recognition information is sadness, and the function linkage rule is to match sadness to adjust the interior lights of the car to dark tones, then the corresponding target linkage function is to adjust the interior lights of the car to dark tones, thereby calling the light adjustment function.

[0083] The technical solution of the embodiment of the present invention generates a target video frame sequence including each target video frame by convolution processing the standardized video corresponding to the target vehicle, then, based on the first key point detection model, detects and processes each target video frame to generate the upper limb key points corresponding to the standardized video, classifies and processes the upper limb key points based on a preset action classification model, determines the target action information corresponding to the standardized video, and uniformly samples and processes each target video frame to generate a sampled video frame corresponding to the standardized video, detects and processes each sampled video frame based on a second key point detection model, generates candidate facial key points corresponding to the sampled video frame, and averages each candidate facial key point to generate basic facial key points corresponding to the standardized video, and further, determines whether the target action information meets the target action standard based on the preset action standard. If the target action information meets the target action category, the upper limb key points corresponding to the target action information are obtained, and the basic facial key points are positionally corrected based on the upper limb key points to generate target facial key points corresponding to the basic facial key points. Finally, the target facial key points are classified and processed based on the fully connected expression classification network to determine the expression category corresponding to the target facial key points. The expression category is used as the target expression recognition information corresponding to the standardized video, and based on the pre-set function linkage rules, the target linkage function matching the target expression recognition information is determined to trigger the execution of the target linkage function, which solves the problem of low accuracy of facial expression recognition results of the driver in the car, can perform real-time recognition of the facial expression of the driver in the car, and improves the accuracy and robustness of expression recognition.

[0084] Figure 5The figure shows a flow chart of an optional expression recognition method based on motion information correction provided by an embodiment of the present invention. Specifically, first, a real-time video of the interior of the target vehicle is acquired, the real-time video is standardized, a standardized video corresponding to the target vehicle is generated, and the standardized video is convoluted to generate a target video frame sequence corresponding to the target vehicle, which includes each target video frame. Then, based on the first key point detection model, each target video frame is detected and processed to generate upper limb key points corresponding to the standardized video to achieve upper limb key point detection, and the upper limb key points are classified and processed based on a preset action classification model to determine the target action information corresponding to the standardized video to achieve action recognition, and each target video frame is uniformly sampled to generate a sampled video frame corresponding to the standardized video, and the face image in each sampled video frame is selected based on the face coordinate detection model to achieve face detection, and each face image is detected and processed based on the second key point detection model to generate the candidate facial key points corresponding to the sampled video frame, and the average Process each candidate facial key point, generate the basic facial key points corresponding to the standardized video, and realize facial key point detection. Further, determine whether the target action information meets the target action category based on the preset action standard. If so, map and process the upper limb key points to generate the position offset corresponding to the basic facial key points, realize offset mapping, and add and process the position offset and the basic facial key points to generate the target facial key points corresponding to the basic facial key points, and realize key point relocation. Finally, classify and process the target facial key points based on the fully connected expression classification network, determine the expression category corresponding to the target facial key points, use the expression category as the target expression recognition information corresponding to the standardized video, realize expression classification, and, based on the pre-set function linkage rules, determine the target linkage function that matches the target expression recognition information, and trigger the execution of the target linkage function to realize expression linkage.

[0085] Embodiment 3

[0086] Figure 6 FIG. 1 is a schematic diagram of a structure of an expression recognition device based on motion information correction provided by Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a data acquisition module 310, a key point determination module 320, a position correction module 330 and an expression recognition module 340;

[0087] The data acquisition module 310 is used to acquire a standardized video corresponding to the target vehicle, and convolute the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame;

[0088] A key point determination module 320 is used to process each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and to process each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video;

[0089] A position correction module 330 is used to perform position correction on the basic facial key points based on the target action information and a preset action standard to generate target facial key points corresponding to the basic facial key points;

[0090] The expression recognition module 340 is used to classify and process the target facial key points based on a fully connected expression classification network, determine the expression category corresponding to the target facial key points, and use the expression category as the target expression recognition information corresponding to the standardized video.

[0091] The technical solution of the embodiment of the present invention generates a target video frame sequence including each target video frame by convolution processing the standardized video corresponding to the target vehicle, then determines the target action information by processing each target video frame based on the first key point detection model action recognition, and determines the basic facial key points by detecting and processing each target video frame based on the second key point detection model, further, performs position correction on the basic facial key points based on the target action information and preset action standards to generate target facial key points, and finally, processes the target facial key points based on the fully connected expression classification network classification, determines the expression category corresponding to the target facial key points, and uses the expression category as the target expression recognition information corresponding to the standardized video, thereby solving the problem of low accuracy of facial expression recognition results of the driver in the car, being able to perform real-time recognition of the facial expression of the driver in the car, and improving the accuracy and robustness of expression recognition.

[0092] Optionally, the data acquisition module 310 may be specifically used for:

[0093] Receiving a data acquisition permission instruction issued by a target vehicle, and acquiring a real-time video inside the target vehicle based on the data acquisition permission instruction;

[0094] The real-time video is processed by standardization to generate a standardized video corresponding to the target vehicle.

[0095] Optionally, the key point determination module 320 may be specifically used for:

[0096] Detect and process each of the target video frames based on the first key point detection model to generate upper limb key points corresponding to the standardized video;

[0097] The upper limb key points are classified and processed based on a preset action classification model to determine the target action information corresponding to the standardized video.

[0098] Optionally, the key point determination module 320 may be specifically used for:

[0099] Uniformly sampling and processing each of the target video frames to generate sampled video frames corresponding to the standardized video;

[0100] Detect and process each of the sampled video frames based on the second key point detection model to generate candidate facial key points corresponding to the sampled video frames;

[0101] The facial key points to be selected are processed by averaging to generate the basic facial key points corresponding to the standardized video.

[0102] Optionally, the position correction module 330 may specifically include: an action judgment unit, a key point acquisition unit and a position correction unit;

[0103] The action judgment unit is used to judge whether the target action information meets the target action category based on a preset action standard;

[0104] A key point acquisition unit, configured to acquire key points of an upper limb corresponding to the target action information if the target action information satisfies the target action category;

[0105] A position correction unit is used to perform position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points.

[0106] Optionally, the position correction unit may be specifically used for:

[0107] Mapping and processing the upper limb key points to generate position offsets corresponding to the basic facial key points;

[0108] The position offset and the basic facial key points are added and processed to generate target facial key points corresponding to the basic facial key points.

[0109] Optionally, the expression recognition device based on motion information correction may also include: a functional linkage module, which is used to classify and process the target facial key points based on the fully-connected expression classification network, determine the expression category corresponding to the target facial key points, and use the expression category as the target expression recognition information corresponding to the standardized video. Then, based on pre-set functional linkage rules, determine the target linkage function that matches the target expression recognition information, and trigger the execution of the target linkage function.

[0110] The expression recognition device based on motion information correction provided in the embodiment of the present invention can execute the expression recognition method based on motion information correction provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment 4

[0112] Figure 7 A schematic diagram of the structure of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0113] like Figure 7 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other through the bus 450. The input / output (I / O) interface 460 is also connected to the bus 450.

[0114] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 420 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 420 performs the various methods and processes described above, such as the expression recognition method based on motion information correction.

[0116] The method includes:

[0117] Obtaining a standardized video corresponding to the target vehicle, and convolutionally processing the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame;

[0118] Processing each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and processing each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video;

[0119] Based on the target action information and the preset action standard, the basic facial key points are positionally corrected to generate target facial key points corresponding to the basic facial key points;

[0120] The target facial key points are classified and processed based on a fully connected expression classification network, the expression category corresponding to the target facial key points is determined, and the expression category is used as the target expression recognition information corresponding to the standardized video.

[0121] In some embodiments, the method for facial expression recognition based on motion information correction may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the method for facial expression recognition based on motion information correction described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the method for facial expression recognition based on motion information correction in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for facial expression recognition based on motion information correction, characterized in that: include: Obtaining a standardized video corresponding to the target vehicle, and convolutionally processing the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame; Processing each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and processing each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video; Based on the target action information and the preset action standard, the basic facial key points are positionally corrected to generate target facial key points corresponding to the basic facial key points; Classify and process the target facial key points based on a fully connected expression classification network, determine the expression category corresponding to the target facial key points, and use the expression category as the target expression recognition information corresponding to the standardized video; The step of correcting the positions of the basic facial key points based on the target action information and the preset action standard to generate target facial key points corresponding to the basic facial key points includes: Determining whether the target action information meets the target action category based on a preset action standard; If the target action information satisfies the target action category, obtaining the upper limb key points corresponding to the target action information; Performing position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points; The method of performing position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points includes: mapping the upper limb key points to generate position offsets corresponding to the basic facial key points; and adding and processing the position offsets and the basic facial key points to generate target facial key points corresponding to the basic facial key points.

2. The method according to claim 1, characterized in that The step of obtaining a standardized video corresponding to the target vehicle includes: Receiving a data acquisition permission instruction issued by a target vehicle, and acquiring a real-time video inside the target vehicle based on the data acquisition permission instruction; The real-time video is processed by standardization to generate a standardized video corresponding to the target vehicle.

3. The method according to claim 1, characterized in that The step of processing each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video includes: Detect and process each of the target video frames based on the first key point detection model to generate upper limb key points corresponding to the standardized video; The upper limb key points are classified and processed based on a preset action classification model to determine the target action information corresponding to the standardized video.

4. The method according to claim 1, characterized in that: The detecting and processing each of the target video frames based on the second key point detection model to determine the basic facial key points corresponding to the standardized video includes: Uniformly sampling and processing each of the target video frames to generate sampled video frames corresponding to the standardized video; Detect and process each of the sampled video frames based on the second key point detection model to generate facial key points to be selected corresponding to the sampled video frames; The facial key points to be selected are processed by averaging to generate the basic facial key points corresponding to the standardized video.

5. The method according to claim 1, characterized in that After classifying and processing the target facial key points based on the fully connected expression classification network, determining the expression category corresponding to the target facial key points, and using the expression category as the target expression recognition information corresponding to the standardized video, the method further includes: Based on a preset function linkage rule, a target linkage function matching the target expression recognition information is determined, and the target linkage function is triggered to be executed.

6. An expression recognition device based on motion information correction, characterized in that: include: A data acquisition module is used to acquire a standardized video corresponding to a target vehicle, and convolute the standardized video to generate a target video frame sequence corresponding to the target vehicle; wherein the target video frame sequence includes each target video frame; A key point determination module, configured to process each of the target video frames based on the first key point detection model action recognition to determine the target action information corresponding to the standardized video, and to process each of the target video frames based on the second key point detection model detection to determine the basic facial key points corresponding to the standardized video; A position correction module, used to perform position correction on the basic facial key points based on the target action information and a preset action standard, and generate target facial key points corresponding to the basic facial key points; An expression recognition module is used to classify and process the target facial key points based on a fully connected expression classification network, determine the expression category corresponding to the target facial key points, and use the expression category as the target expression recognition information corresponding to the standardized video; Among them, the position correction module specifically includes: an action judgment unit, a key point acquisition unit and a position correction unit; An action judgment unit, used to judge whether the target action information meets the target action category based on a preset action standard; A key point acquisition unit, configured to acquire key points of an upper limb corresponding to the target action information if the target action information satisfies the target action category; A position correction unit, used for performing position correction on the basic facial key points based on the upper limb key points to generate target facial key points corresponding to the basic facial key points; The position correction unit is specifically used for: mapping and processing the upper limb key points to generate the position offset corresponding to the basic facial key points; adding and processing the position offset and the basic facial key points to generate the target facial key points corresponding to the basic facial key points.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the expression recognition method based on motion information correction as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the expression recognition method based on motion information correction according to any one of claims 1 to 5 when executed.

Images