Expression recognition method, device, equipment and medium based on time decay mechanism

Abstract

The present invention discloses an expression recognition method, device, equipment and medium based on a time decay mechanism. The method comprises: classifying and processing the current standardized video frame based on a preset area detection model and a preset expression recognition model to determine the expression information, and identifying and processing the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model to determine the action information; determining the first strong information degree frame or the second strong information degree frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information; performing confidence fusion processing on the current standardized video frame based on the time decay mechanism and the first strong information degree frame or the second strong information degree frame to obtain a fused expression confidence value, and determining the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value. Through the technical solution of the present invention, the expression category in the video sequence can be identified, and the accuracy and robustness of expression recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an expression recognition method, device, equipment and medium based on a time decay mechanism. Background Art

[0002] With the development of artificial intelligence technology, expression recognition plays an important role in security monitoring, sentiment analysis, human-computer interaction and other fields. Existing expression recognition technology generally recognizes expressions through static images or short video clips.

[0003] However, existing expression recognition technology ignores the continuity and evolution of emotional states in the temporal dimension, resulting in low accuracy and robustness of expression recognition results in long video sequences. Therefore, how to improve the accuracy and robustness of expression recognition results in long video sequences is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides an expression recognition method, device, equipment and medium based on a time decay mechanism, which can solve the problem of low accuracy and robustness of expression recognition results in long video sequences.

[0005] According to one aspect of the present invention, there is provided a method for facial expression recognition based on a time decay mechanism, comprising:

[0006] Acquire a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes each standardized video frame;

[0007] Based on the preset area detection model and the preset expression recognition model, the current standardized video frame is classified and processed to determine the expression information corresponding to the current standardized video frame, and based on the preset upper limb key point detection model and the preset action recognition model, the current standardized video frame is recognized and processed to determine the action information corresponding to the current standardized video frame; wherein the expression information includes the expression category and the expression confidence score, and the action information includes the action category and the action confidence score;

[0008] Based on the consistency relationship between the action information and the corresponding expression information, determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information;

[0009] Based on the time decay mechanism and the first strong information frame or the second strong information frame, the current standardized video frame is subjected to confidence fusion processing to obtain a fused expression confidence value, and the expression recognition result corresponding to the current standardized video frame is determined based on the fused expression confidence value.

[0010] According to another aspect of the present invention, there is provided an expression recognition device based on a time decay mechanism, comprising:

[0011] A data acquisition module, used to acquire a standardized video frame sequence corresponding to a target vehicle; wherein the standardized video frame sequence includes each standardized video frame;

[0012] An information determination module is used to classify and process the current standardized video frame based on a preset area detection model and a preset expression recognition model, determine the expression information corresponding to the current standardized video frame, and recognize and process the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, and determine the action information corresponding to the current standardized video frame; wherein the expression information includes an expression category and an expression confidence score, and the action information includes an action category and an action confidence score;

[0013] A strong information frame determination module is used to determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information;

[0014] The recognition result determination module is used to perform confidence fusion processing on the current standardized video frame based on the time attenuation mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value, and determine the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the expression recognition method based on the time decay mechanism described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the expression recognition method based on the time decay mechanism described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention classifies and processes the current standardized video frame corresponding to the target vehicle through a preset area detection model and a preset expression recognition model, determines the expression information corresponding to the current standardized video frame, and recognizes and processes the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, determines the action information corresponding to the current standardized video frame, and then, based on the consistency relationship between the action information and the corresponding expression information, determines the first strong information frame or the second strong information frame corresponding to the current standardized video frame, further, based on the time attenuation mechanism and the first strong information frame or the second strong information frame, performs confidence fusion processing on the current standardized video frame to obtain a fused expression confidence value, and determines the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value, thereby solving the problem of low accuracy and robustness of expression recognition results in long video sequences and improving the accuracy and robustness of expression recognition results in long video sequences.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of an expression recognition method based on a time decay mechanism provided according to Embodiment 1 of the present invention;

[0024] Figure 2 is a schematic diagram of facial key points provided according to Embodiment 1 of the present invention;

[0025] Figure 3 is a flow chart of an expression recognition method based on a time decay mechanism provided according to Embodiment 2 of the present invention;

[0026] Figure 4 is a flow chart of an optional expression recognition method based on a time decay mechanism provided according to Embodiment 2 of the present invention;

[0027] Figure 5 is a structural schematic diagram of an expression recognition device based on a time decay mechanism provided according to Embodiment 3 of the present invention;

[0028] Figure 6It is a structural schematic diagram of an electronic device for implementing the expression recognition method based on the time decay mechanism of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a method for facial expression recognition based on a time decay mechanism provided in the first embodiment of the present invention. This embodiment is applicable to the case of facial expression recognition in a long video sequence. The method can be performed by a facial expression recognition device based on a time decay mechanism. The facial expression recognition device based on a time decay mechanism can be implemented in the form of hardware and / or software. The facial expression recognition device based on a time decay mechanism can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110, obtaining a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes standardized video frames.

[0035] The target vehicle may refer to a vehicle for which facial expressions of drivers in the vehicle need to be recognized. The standardized video may refer to a video generated after standardization processing. For example, it may be a video generated after brightness adjustment, contrast optimization or denoising.

[0036] In an optional embodiment, the step of acquiring a standardized video frame sequence corresponding to the target vehicle may include: receiving a data acquisition permission instruction issued by the target vehicle, and acquiring a real-time video inside the target vehicle based on the data acquisition permission instruction; and processing the real-time video in a standardized manner to generate a standardized video frame sequence corresponding to the target vehicle. The data acquisition permission instruction may refer to a data authorization instruction issued by the driver. Exemplarily, when it is necessary to acquire data corresponding to the target vehicle, a data acquisition request may be sent to the target vehicle, and then the driver may review the data acquisition request and send a data acquisition permission instruction to the data that is allowed to be accessed, thereby implementing data authorization and ensuring the security of user data. The real-time video may refer to a dynamic long video captured in real time inside the target vehicle. Exemplarily, a high-resolution camera installed on the dashboard of the target vehicle or the connecting pillars (A-pillar, A-pillar) connecting the roof and the front cabin on the left and right front sides may be used to capture real-time video of the driver inside the target vehicle. Standardization processing may refer to related processing operations for optimizing and adjusting the real-time video. Exemplarily, operations such as brightness adjustment, contrast optimization, and denoising may be performed. Therefore, by standardizing the real-time video of the target vehicle with authorized access, the real-time video collected in real time can be optimized and adjusted to adapt to subsequent video analysis processing.

[0037] S120. Classify and process the current standardized video frame based on a preset area detection model and a preset expression recognition model to determine the expression information corresponding to the current standardized video frame, and recognize and process the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model to determine the action information corresponding to the current standardized video frame; wherein the expression information includes expression category and expression confidence score, and the action information includes action category and action confidence score.

[0038] Among them, the preset area detection model may refer to a pre-selected face detection model, which is used to detect the face area of ​​the driver in the target vehicle. The preset expression recognition model may refer to a pre-selected expression model, which is used to identify the current expression category of the driver in the target vehicle. Expression information may refer to associated information related to expression. Expression information may include expression categories and expression confidence scores. Expression categories may refer to the current classification of facial expressions. Exemplarily, expression categories may be sadness, anger, happiness, surprise, confusion, boredom or calmness, etc. The expression confidence score may refer to the confidence coefficient of the corresponding expression category. The current standardized video frame may refer to the standardized video frame that currently needs to perform expression recognition.

[0039] Specifically, the face detection can be performed on the current standardized video frame through the preset area detection model to obtain the driver's face area, and then the expression recognition can be performed on the driver's face area through the preset expression recognition model. Usually, the preset expression recognition model can output the confidence corresponding to the above seven expressions, and use the expression corresponding to the maximum confidence value as the expression category corresponding to the current standardized video frame. Exemplarily, if the output confidence value is: 0.1, 0.2, 0.3, 0.1, 0.1, 0.1, 0.1, then the expression category corresponding to the current standardized video frame is happy, and the expression confidence score is 0.3.

[0040] Among them, the preset upper limb key point detection model may refer to a key point detection model trained using a limb key point data set. Upper limb key points may refer to limb nodes related to the upper body movements of the driver in the target vehicle. Exemplarily, upper limb key points may include torso key points, facial key points, and finger key points. The torso key points may include different positions such as the center of the neck, the junction of the neck and the left shoulder, the junction of the neck and the right shoulder, the left shoulder, the right shoulder, the left elbow, the right elbow, the left abdomen, the right abdomen, the center of the left and right abdomens, and the heart. The finger key points may include the joints of the wrist and the five fingers, a total of 21 positions. The facial key points may include 68 positions such as the eyes and the mouth, such as Figure 2 FIG. 1 is a schematic diagram of a facial key point provided by an embodiment of the present invention. Generally, the upper limb key points can be represented by the coordinate values ​​corresponding to each key point.

[0041] Among them, the preset action recognition model may refer to a preset action recognition network model, which is used to identify the current body movements of the driver in the target vehicle. Action information may refer to associated information related to body movements. Action information may include action categories and action confidence scores. Action categories may refer to the current classification of body movements. Exemplarily, the action categories may be covering the face with hands and slightly drooping mouth corners, clenching fists and slightly opening the mouth to show teeth, clapping hands and raising mouth corners, raising hands and opening mouth, touching chin with hands and scratching head or supporting chin and yawning, etc. The action confidence score may refer to the confidence coefficient of the corresponding action category.

[0042] Specifically, the upper limb key point detection model can be used to detect the upper limb key points of the current standardized video frame to obtain the upper limb key points, and then the preset action recognition model can be used to perform action recognition on the upper limb key points. Usually, the preset action recognition model can output the confidence corresponding to the above six actions, and use the action corresponding to the maximum confidence value as the action category corresponding to the current standardized video frame. Exemplarily, if the output confidence value is: 0.1, 0.2, 0.4, 0.1, 0.1, 0.1, then the action category corresponding to the standardized video frame is clapping with the corners of the mouth raised, and the action confidence score is 0.4.

[0043] S130. Based on the consistency relationship between the action information and the corresponding expression information, determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame; wherein the first strong information frame is obtained based on the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained based on the indirect matching relationship between the action information and the corresponding expression information.

[0044] Among them, a strong information frame may refer to a standardized video frame in which action information and expression information are strongly correlated. The first strong information frame may refer to a strong information frame determined after directly matching action information and expression information. Usually, the first strong information frame in a standardized video frame sequence is determined once during the expression recognition process. The second strong information frame may refer to a strong information frame determined by processing the expression information and then matching it with the action information. Usually, during the expression recognition process, the second strong information frame will change over time, that is, the current standardized video frame may also serve as the second strong information frame of a subsequent standardized video frame.

[0045] S140, performing confidence fusion processing on the current standardized video frame based on the time attenuation mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value, and determining an expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value.

[0046] Among them, the time decay mechanism may refer to a mechanism for calculating the decay score of the current standardized video frame by utilizing the continuity and evolution of the emotional state over a longer time span. The confidence fusion processing may refer to the operation of fusing the expression confidence score and the action confidence score corresponding to the current standardized video frame. The fused expression confidence value may refer to the expression confidence score corresponding to the current standardized video frame after the confidence fusion processing. The expression recognition result may refer to the expression category corresponding to the current standardized video frame that is finally determined.

[0047] The technical solution of the embodiment of the present invention classifies and processes the current standardized video frame corresponding to the target vehicle through a preset area detection model and a preset expression recognition model, determines the expression information corresponding to the current standardized video frame, and recognizes and processes the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, determines the action information corresponding to the current standardized video frame, and then, based on the consistency relationship between the action information and the corresponding expression information, determines the first strong information frame or the second strong information frame corresponding to the current standardized video frame, further, based on the time attenuation mechanism and the first strong information frame or the second strong information frame, performs confidence fusion processing on the current standardized video frame to obtain a fused expression confidence value, and determines the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value, thereby solving the problem of low accuracy and robustness of expression recognition results in long video sequences and improving the accuracy and robustness of expression recognition results in long video sequences.

[0048] Embodiment 2

[0049] Figure 3 A flow chart of an expression recognition method based on a time decay mechanism is provided in the second embodiment of the present invention. This embodiment is refined based on the above-mentioned embodiment. In this embodiment, the operation of performing confidence fusion processing on the current standardized video frame based on the time decay mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value is specifically refined. Specifically, it may include: determining the attenuation score of the current standardized video frame based on the expression confidence score and the action confidence score corresponding to the first strong information frame or the second strong information frame, and the time decay mechanism; fusing the full expression confidence score, the full action confidence score and the attenuation score corresponding to the current standardized video frame to obtain a fused expression confidence value. Figure 3 As shown, the method includes:

[0050] S210, obtaining a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes each standardized video frame.

[0051] Specifically, after obtaining the real-time long video authorized by the driver, the real-time long video can be subjected to standardized processing operations to obtain a standardized video frame sequence corresponding to the target vehicle, providing an effective basis for subsequent operations.

[0052] S220, classify and process the current standardized video frame based on the preset area detection model and the preset expression recognition model, determine the expression information corresponding to the current standardized video frame, and recognize and process the current standardized video frame based on the preset upper limb key point detection model and the preset action recognition model, and determine the action information corresponding to the current standardized video frame.

[0053] The expression information includes the expression category and the expression confidence score, and the action information includes the action category and the action confidence score.

[0054] It is worth noting that in the embodiment of the present invention, the upper limb key points corresponding to the standardized video frames can be transmitted to the preset action recognition model every 30 frames, or the sliding window method can be used to transmit the upper limb key points corresponding to each standardized video frame after 30 frames to the preset action recognition model. The embodiment of the present invention is not limited to this.

[0055] S230: Determine the first high-information frame or the second high-information frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information.

[0056] In an optional implementation, determining the first high-information frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information can include: judging whether the first action category corresponding to the first standardized video frame meets the target action category based on a preset action set; wherein the frame number of the first standardized video frame is less than the frame number of the current standardized video frame; if the first action category meets the target action category, judging whether the first action category and the first expression category corresponding to the first standardized video frame meet the target association relationship based on a preset expression-action association relationship; if the first action category and the first expression category meet the target association relationship, using the first standardized video frame as the first high-information frame.

[0057] Among them, the preset action set may refer to a set of pre-set action categories. Exemplarily, the preset action set may include covering the face with hands and slightly drooping mouth corners, clenching fists and slightly opening the mouth to show teeth, clapping hands with raised mouth corners, raising hands and opening mouth, touching the chin with hands and scratching the head, and supporting the chin and yawning. The first standardized video frame may refer to the standardized video frame currently selected in the action category judgment stage. The first action category may refer to the action category corresponding to the first standardized video frame. The target action category may refer to the action category included in the preset action set.

[0058] Among them, the preset expression-action association relationship may refer to a preset expression-action association standard. For example, the expression of sadness may be associated with the action of covering the face with the hands and the corners of the mouth slightly drooping, the expression of anger may be associated with the action of clenching a fist and slightly opening the mouth to show the teeth, the expression of happiness may be associated with the action of clapping hands and raising the corners of the mouth, the expression of surprise may be associated with the action of raising the hands and opening the mouth, the expression of confusion may be associated with the action of touching the chin with the hands and scratching the head, and the expression of boredom may be associated with the action of supporting the chin with the hands and yawning, etc. The first expression category may refer to the expression category corresponding to the first standardized video frame. The target association relationship may refer to the association relationship contained in the preset expression-action association relationship.

[0059] Specifically, after generating the action information corresponding to the first standardized video frame, it is necessary to use the preset action set to match and judge the action information corresponding to the first standardized video frame. After the first action category corresponding to the first standardized video frame meets the target action category, the first action category can be matched with the action in the preset expression-action association relationship, and then the expression corresponding to the action is used as the expression corresponding to the first action category. Further, the expression is compared with the first expression category. If the expression is consistent with the first expression category, it indicates that the first action category and the first expression category meet the target association relationship. Therefore, by introducing action information, more contextual information can be provided for expression recognition, so as to more accurately understand and identify the real emotions expressed by the expression, and the first standardized video frame can be used as a high-information frame to provide an effective basis for subsequent operations.

[0060] It is worth noting that if there is no first strong information frame before the current standardized video frame, the current standardized video frame needs to be used as the first standardized video frame to determine whether the current standardized video frame is the first strong information frame. If the current standardized video frame is the first strong information frame, the expression category corresponding to the current standardized video frame can be directly used as the expression recognition result, and the first strong information frame is introduced in the process of expression recognition of subsequent standardized video frames. At the same time, if the current standardized video frame is not the first strong information frame, the expression category corresponding to the current standardized video frame is directly used as the expression recognition result, and the judgment of whether it is the first strong information frame is repeated in the process of expression recognition of subsequent standardized video frames until the first strong information frame appears.

[0061] S240, determining the attenuation score of the current standardized video frame based on the expression confidence score and the action confidence score corresponding to the first strong information frame or the second strong information frame, and a time attenuation mechanism.

[0062] The attenuation score may refer to a current standardized video frame score calculated based on the attenuation of the previous first strong information frame or the second strong information frame over a time span.

[0063] Specifically, if the serial number of the current standardized video frame is i, the serial number of the first strong information frame or the second strong information frame is m, and the expression confidence score corresponding to the first strong information frame or the second strong information frame is , the action confidence score is , then the attenuation score of the current standardized video frame can be: .in, It can represent the attenuation coefficient, and its value range is (0,1). Usually, The larger the value is, the greater the influence of the strong information frame on the expression recognition results of the subsequent standardized video frames. As time goes by, the influence of the strong information frame on the subsequent standardized video frames becomes smaller and smaller.

[0064] It is worth noting that in the embodiment of the present invention, the first strong information frame is usually only judged once. After the subsequent standardized video frame is judged as a strong information frame, the expression recognition process no longer uses the first strong information frame. That is, when there is only the first strong information frame, the expression confidence score and action confidence score corresponding to the first strong information frame, as well as the time decay mechanism, are used to determine the decay score of the current standardized video frame; when there is a second strong information frame, the expression confidence score and action confidence score corresponding to the second strong information frame, as well as the time decay mechanism, are used to determine the decay score of the current standardized video frame.

[0065] S250, fusing the full expression confidence score, the full action confidence score, and the attenuation score corresponding to the current standardized video frame to obtain a fused expression confidence value.

[0066] The full expression confidence score may refer to the confidence scores corresponding to all expressions corresponding to the current standardized video frame, and the full action confidence score may refer to the confidence scores corresponding to all actions corresponding to the current standardized video frame.

[0067] Specifically, if the expression confidence score corresponding to the current standardized video frame is , action confidence score , then the fusion expression confidence value corresponding to the current standardized video frame can be: .in, , as well as It can represent the weighting coefficient, which usually needs to be adjusted according to the actual application scenario to ensure the rationality of each part's contribution. It is worth noting that if the expression category and action category of the current standardized video frame are inconsistent, the action confidence score does not participate in the calculation, that is, . The action category can be expressed as an adjustment coefficient of the expression confidence score, thereby reflecting the impact of body movements on expression recognition results. The value is If the expression category of the current standardized video frame is consistent with the expression category of the first strong information frame or the second strong information frame, then +1 is taken, otherwise -1 is taken.

[0068] S260: Determine and process the confidence value of the fused expression based on a preset confidence threshold.

[0069] The preset confidence threshold may refer to a preset value for evaluating the confidence value of the fused expression. For example, the preset confidence threshold may be set to 0.3 or 0.5. The embodiment of the present invention does not limit this.

[0070] S270: If the fused expression confidence value includes a target expression confidence score that exceeds a preset confidence threshold, the expression category corresponding to the target expression confidence score is used as the expression recognition result corresponding to the current standardized video frame.

[0071] The target expression confidence score may refer to the expression confidence score of the fused expression confidence value that exceeds the preset confidence threshold. For example, if the fused expression confidence value is 0.1, 0.2, 0.3, 0.2, 0.2, 0.1, 0.1, and the preset confidence threshold is 0.3, then the target expression confidence score is 0.3, and the expression recognition result corresponding to the current standardized video frame is happy.

[0072] Specifically, after obtaining the fused expression confidence value corresponding to the current standardized video frame, the expression recognition result corresponding to the current standardized video frame can be determined based on a preset confidence threshold to achieve determination of the expression recognition result.

[0073] In an optional embodiment, after taking the expression category corresponding to the target expression confidence score as the expression recognition result corresponding to the current standardized video frame, it can also include: judging whether the expression recognition result and the action category corresponding to the current standardized video frame satisfy the target association relationship based on a preset expression-action association relationship; if the expression recognition result and the action category corresponding to the current standardized video frame satisfy the target association relationship, marking the current standardized video frame as the second strongest information frame.

[0074] Specifically, after determining the expression recognition result corresponding to the current standardized video frame, the preset expression-action association relationship can be used to judge and process the expression recognition result and action category corresponding to the current standardized video frame. If the expression recognition result and action category corresponding to the current standardized video frame meet the target association relationship, the current standardized video frame can be marked as the second strongest information frame, providing an effective basis for expression recognition in subsequent standardized video frames.

[0075] S280, determining the majority recognition result among the expression recognition results within a set time interval.

[0076] The set time interval may refer to a preset recognition result acquisition period, which may be 2 seconds or 3 seconds, etc. The majority recognition result may refer to the expression recognition result that appears the most times within the set time interval.

[0077] Therefore, by screening the expression recognition results within the set time interval, the efficiency of data transmission can be improved, providing an effective basis for functional linkage.

[0078] S290: Based on a preset function linkage rule, determine a target linkage function that matches the majority recognition result, and trigger execution of the target linkage function.

[0079] Among them, the function linkage rule may refer to a pre-set rule for matching facial expressions with vehicle functions. For example, the function linkage rule may be to switch to cheerful music when the driver is happy, or to adjust the interior lights to a dark tone when the driver is sad. The target linkage function may refer to a function in the function linkage rule that matches the target expression recognition information.

[0080] Specifically, if the target expression recognition information is sadness, and the function linkage rule is to match sadness to adjust the interior lights of the car to dark tones, then the corresponding target linkage function is to adjust the interior lights of the car to dark tones, thereby calling the light adjustment function.

[0081] The technical solution of the embodiment of the present invention classifies and processes the current standardized video frame corresponding to the target vehicle through a preset area detection model and a preset expression recognition model, determines the expression information corresponding to the current standardized video frame, and recognizes and processes the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, determines the action information corresponding to the current standardized video frame, and then, based on the consistency relationship between the action information and the corresponding expression information, determines the first strong information frame or the second strong information frame corresponding to the current standardized video frame, further, based on the expression confidence score and action confidence score corresponding to the first strong information frame or the second strong information frame and the time attenuation mechanism, determines the attenuation score of the current standardized video frame, and fuses and processes the corresponding The full expression confidence score, the full action confidence score and the attenuation score are used to obtain the fused expression confidence value. Finally, the fused expression confidence value is judged and processed based on the preset confidence threshold. If the fused expression confidence value contains a target expression confidence score that exceeds the preset confidence threshold, the expression category corresponding to the target expression confidence score is used as the expression recognition result corresponding to the current standardized video frame, and the majority recognition result among the expression recognition results within the set time interval is determined. Based on the pre-set function linkage rules, the target linkage function matching the majority recognition result is determined, and the target linkage function is triggered to execute, which solves the problem of low accuracy and robustness of expression recognition results in long video sequences and improves the accuracy and robustness of expression recognition results in long video sequences.

[0082] Figure 4The figure shows a flow chart of an optional expression recognition method based on a time decay mechanism provided by an embodiment of the present invention. The method is applicable to the situation where there is no second high-information frame before the current standardized video frame. Specifically, first, a data acquisition permission instruction issued by the target vehicle is received, and the real-time video inside the target vehicle is acquired based on the data acquisition permission instruction, and the real-time video is processed in a standardized manner to generate a standardized video frame sequence corresponding to the target vehicle. Then, the current standardized video frame is classified and processed based on a preset area detection model and a preset expression recognition model to achieve face detection and preliminary expression recognition to determine the expression information corresponding to the current standardized video frame, and the current standardized video frame is identified and processed based on a preset upper limb key point detection model and a preset action recognition model to achieve limb key point detection and action recognition to determine the action information corresponding to the current standardized video frame. Furthermore, based on the preset action set, it is determined whether the first action category corresponding to the first standardized video frame meets the target action category. If the first action category corresponds to the first standardized video frame, the first action category corresponds to the first standardized video frame. An action category satisfies the target action category. Based on the preset expression-action association relationship, it is determined whether the first action category and the first expression category corresponding to the first standardized video frame meet the target association relationship. If the first action category and the first expression category meet the target association relationship, the first standardized video frame is used as the first strong information frame. Further, based on the time attenuation mechanism and the first strong information frame, the current standardized video frame is subjected to confidence fusion processing to obtain the fused expression confidence value, and the expression recognition result corresponding to the current standardized video frame is determined based on the fused expression confidence value. Finally, the majority recognition result among the expression recognition results within the set time interval is determined, and based on the pre-set function linkage rules, the target linkage function matching the majority recognition result is determined, and the target linkage function is triggered to execute. In this way, the expression recognition of the driver in the target vehicle is completed.

[0083] Embodiment 3

[0084] Figure 5 This is a schematic diagram of the structure of an expression recognition device based on a time decay mechanism provided by the third embodiment of the present invention. Figure 5 As shown, the device includes: a data acquisition module 310, an information determination module 320, a strong information frame determination module 330 and a recognition result determination module 340;

[0085] The data acquisition module 310 is used to acquire a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes each standardized video frame;

[0086] The information determination module 320 is used to classify and process the current standardized video frame based on the preset area detection model and the preset expression recognition model, determine the expression information corresponding to the current standardized video frame, and recognize and process the current standardized video frame based on the preset upper limb key point detection model and the preset action recognition model, and determine the action information corresponding to the current standardized video frame; wherein the expression information includes the expression category and the expression confidence score, and the action information includes the action category and the action confidence score;

[0087] A strong information frame determination module 330 is used to determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information;

[0088] The recognition result determination module 340 is used to perform confidence fusion processing on the current standardized video frame based on the time attenuation mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value, and determine the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value.

[0089] The technical solution of the embodiment of the present invention classifies and processes the current standardized video frame corresponding to the target vehicle through a preset area detection model and a preset expression recognition model, determines the expression information corresponding to the current standardized video frame, and recognizes and processes the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, determines the action information corresponding to the current standardized video frame, and then, based on the consistency relationship between the action information and the corresponding expression information, determines the first strong information frame or the second strong information frame corresponding to the current standardized video frame, further, based on the time attenuation mechanism and the first strong information frame or the second strong information frame, performs confidence fusion processing on the current standardized video frame to obtain a fused expression confidence value, and determines the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value, thereby solving the problem of low accuracy and robustness of expression recognition results in long video sequences and improving the accuracy and robustness of expression recognition results in long video sequences.

[0090] Optionally, the data acquisition module 310 may be specifically used for:

[0091] Receiving a data acquisition permission instruction issued by a target vehicle, and acquiring a real-time video inside the target vehicle based on the data acquisition permission instruction;

[0092] The real-time video is processed by standardization to generate a standardized video frame sequence corresponding to the target vehicle.

[0093] Optionally, the high-information frame determination module 330 may be specifically used for:

[0094] Determining whether a first action category corresponding to a first standardized video frame satisfies a target action category based on a preset action set; wherein a frame number of the first standardized video frame is smaller than a frame number of the current standardized video frame;

[0095] If the first action category satisfies the target action category, determining whether the first action category and the first expression category corresponding to the first standardized video frame satisfy the target association relationship based on a preset expression-action association relationship;

[0096] If the first action category and the first expression category satisfy a target association relationship, the first standardized video frame is used as a first high-information frame.

[0097] Optionally, the recognition result determination module 340 may be specifically used for:

[0098] Determine the attenuation score of the current standardized video frame based on the expression confidence score and the action confidence score corresponding to the first strong information frame or the second strong information frame, and the time attenuation mechanism;

[0099] The full expression confidence score, full action confidence score and attenuation score corresponding to the current standardized video frame are fused to obtain a fused expression confidence value.

[0100] Optionally, the recognition result determination module 340 may be specifically used for:

[0101] Determine and process the confidence value of the fused expression based on a preset confidence threshold;

[0102] If the fused expression confidence value includes a target expression confidence score that exceeds a preset confidence threshold, the expression category corresponding to the target expression confidence score is used as the expression recognition result corresponding to the current standardized video frame.

[0103] Optionally, the expression recognition device based on the time decay mechanism may further include: a strong information degree frame marking module, which is used to determine whether the expression recognition result corresponding to the current standardized video frame and the action category satisfy the target association relationship based on a preset expression-action association relationship after taking the expression category corresponding to the target expression confidence score as the expression recognition result corresponding to the current standardized video frame;

[0104] If the expression recognition result corresponding to the current standardized video frame and the action category meet the target association relationship, the current standardized video frame is marked as the second strongest information frame.

[0105] Optionally, the expression recognition device based on the time decay mechanism may also include: a function linkage module, which is used to perform confidence fusion processing on the target weak information frames in the standardized video frame sequence based on the time decay mechanism and each strong information frame to obtain a fused expression confidence value, and determine the expression recognition results corresponding to each standardized video frame based on the fused expression confidence value, and then determine the majority recognition result among the expression recognition results within a set time interval; based on a pre-set function linkage rule, determine the target linkage function that matches the majority recognition result, and trigger the execution of the target linkage function.

[0106] The expression recognition device based on the time decay mechanism provided in the embodiment of the present invention can execute the expression recognition method based on the time decay mechanism provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0107] Embodiment 4

[0108] Figure 6 A schematic diagram of the structure of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0109] like Figure 6 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other through the bus 450. The input / output (I / O) interface 460 is also connected to the bus 450.

[0110] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The processor 420 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 420 performs the various methods and processes described above, such as the expression recognition method based on the time decay mechanism.

[0112] The method includes:

[0113] Acquire a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes each standardized video frame;

[0114] Based on the preset area detection model and the preset expression recognition model, the current standardized video frame is classified and processed to determine the expression information corresponding to the current standardized video frame, and based on the preset upper limb key point detection model and the preset action recognition model, the current standardized video frame is recognized and processed to determine the action information corresponding to the current standardized video frame; wherein the expression information includes the expression category and the expression confidence score, and the action information includes the action category and the action confidence score;

[0115] Based on the consistency relationship between the action information and the corresponding expression information, determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information;

[0116] Based on the time decay mechanism and the first strong information frame or the second strong information frame, the current standardized video frame is subjected to confidence fusion processing to obtain a fused expression confidence value, and the expression recognition result corresponding to the current standardized video frame is determined based on the fused expression confidence value.

[0117] In some embodiments, the expression recognition method based on the time decay mechanism may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the expression recognition method based on the time decay mechanism described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the expression recognition method based on the time decay mechanism in any other appropriate manner (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0123] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0124] The embodiment of the present application also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the expression recognition method based on the time decay mechanism provided in any embodiment of the present application is implemented. The program product and the expression recognition method based on the time decay mechanism disclosed in each embodiment of the present application belong to the same inventive concept, so it will not be repeated here.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for facial expression recognition based on a time decay mechanism, characterized in that: include: Acquire a standardized video frame sequence corresponding to the target vehicle; wherein the standardized video frame sequence includes each standardized video frame; Based on the preset area detection model and the preset expression recognition model, the current standardized video frame is classified and processed to determine the expression information corresponding to the current standardized video frame, and based on the preset upper limb key point detection model and the preset action recognition model, the current standardized video frame is recognized and processed to determine the action information corresponding to the current standardized video frame; wherein the expression information includes the expression category and the expression confidence score, and the action information includes the action category and the action confidence score; Based on the consistency relationship between the action information and the corresponding expression information, determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information; Based on the time decay mechanism and the first strong information frame or the second strong information frame, a confidence fusion process is performed on the current standardized video frame to obtain a fused expression confidence value, and based on the fused expression confidence value, an expression recognition result corresponding to the current standardized video frame is determined; The step of performing confidence fusion processing on the current standardized video frame based on the time decay mechanism and the first strong information frame or the second strong information frame to obtain the fused expression confidence value includes: Determine the attenuation score of the current standardized video frame based on the expression confidence score and the action confidence score corresponding to the first strong information frame or the second strong information frame, and the time attenuation mechanism; The full expression confidence score, full action confidence score and attenuation score corresponding to the current standardized video frame are fused to obtain a fused expression confidence value.

2. The method according to claim 1, characterized in that The step of obtaining a standardized video frame sequence corresponding to the target vehicle includes: Receiving a data acquisition permission instruction issued by a target vehicle, and acquiring a real-time video inside the target vehicle based on the data acquisition permission instruction; The real-time video is processed by standardization to generate a standardized video frame sequence corresponding to the target vehicle.

3. The method according to claim 1, characterized in that Determining a first high-information frame corresponding to the current standardized video frame based on a consistency relationship between the action information and the corresponding expression information includes: Determining whether a first action category corresponding to a first standardized video frame satisfies a target action category based on a preset action set; wherein a frame number of the first standardized video frame is smaller than a frame number of the current standardized video frame; If the first action category satisfies the target action category, determining whether the first action category and the first expression category corresponding to the first standardized video frame satisfy the target association relationship based on a preset expression-action association relationship; If the first action category and the first expression category satisfy a target association relationship, the first standardized video frame is used as a first high-information frame.

4. The method according to claim 1, characterized in that: The step of determining the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value includes: Determine and process the confidence value of the fused expression based on a preset confidence threshold; If the fused expression confidence value includes a target expression confidence score that exceeds a preset confidence threshold, the expression category corresponding to the target expression confidence score is used as the expression recognition result corresponding to the current standardized video frame.

5. The method according to claim 4, characterized in that After taking the expression category corresponding to the target expression confidence score as the expression recognition result corresponding to the current standardized video frame, the method further includes: Based on the preset expression-action association relationship, it is determined whether the expression recognition result and the action category corresponding to the current standardized video frame meet the target association relationship; If the expression recognition result corresponding to the current standardized video frame and the action category meet the target association relationship, the current standardized video frame is marked as the second strongest information frame.

6. The method according to claim 1, characterized in that After the confidence fusion processing is performed on the current standardized video frame based on the time decay mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value, and the expression recognition result corresponding to the current standardized video frame is determined based on the fused expression confidence value, the method further includes: Determine the majority recognition result among the expression recognition results within a set time interval; Based on a preset function linkage rule, a target linkage function matching the majority recognition result is determined, and the target linkage function is triggered to be executed.

7. An expression recognition device based on a time decay mechanism, characterized in that: include: A data acquisition module, used to acquire a standardized video frame sequence corresponding to a target vehicle; wherein the standardized video frame sequence includes each standardized video frame; An information determination module is used to classify and process the current standardized video frame based on a preset area detection model and a preset expression recognition model, determine the expression information corresponding to the current standardized video frame, and recognize and process the current standardized video frame based on a preset upper limb key point detection model and a preset action recognition model, and determine the action information corresponding to the current standardized video frame; wherein the expression information includes an expression category and an expression confidence score, and the action information includes an action category and an action confidence score; A strong information frame determination module is used to determine the first strong information frame or the second strong information frame corresponding to the current standardized video frame based on the consistency relationship between the action information and the corresponding expression information; wherein the first strong information frame is obtained according to the direct matching relationship between the action information and the corresponding expression information; and the second strong information frame is obtained according to the indirect matching relationship between the action information and the corresponding expression information; A recognition result determination module, used to perform confidence fusion processing on the current standardized video frame based on the time decay mechanism and the first strong information frame or the second strong information frame to obtain a fused expression confidence value, and determine the expression recognition result corresponding to the current standardized video frame based on the fused expression confidence value; The recognition result determination module is specifically used for: Determine the attenuation score of the current standardized video frame based on the expression confidence score and the action confidence score corresponding to the first strong information frame or the second strong information frame, and the time attenuation mechanism; The full expression confidence score, full action confidence score and attenuation score corresponding to the current standardized video frame are fused to obtain a fused expression confidence value.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the expression recognition method based on the time decay mechanism described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the expression recognition method based on a time decay mechanism as described in any one of claims 1 to 6 when executed.

Images