Face action recognition method and device, electronic equipment and storage medium

Abstract

This invention provides a face action recognition method, comprising: acquiring the face key point recognition results of a local face parsing service for a sequence of images to be recognized, wherein the sequence of images to be recognized is obtained by a face image acquisition service based on face action recognition instructions; determining the organ key point sequence corresponding to each target facial organ based on the face key point recognition results, wherein the target facial organs are determined according to the face action recognition instructions; determining the action recognition results corresponding to each target facial organ based on the organ key point sequence; and determining the face action recognition results of the front-end browser based on the action recognition results. Face action recognition can be directly performed in the front-end browser based on the organ key point sequence, reducing server-side computing power. Furthermore, using the organ key point sequence for face action recognition eliminates the need to directly capture temporal dependencies in the sequence of images to be recognized, reducing the parameter requirements of the local face parsing service, thereby reducing development and deployment costs.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for facial motion recognition. Background Technology

[0002] With the increasing demand for information security and the continuous development of optical technology, the need for facial motion recognition functionality in front-end systems is growing stronger. Without motion recognition in facial recognition systems, individuals could easily steal faces, with common attack methods including photos, videos, and 3D models. Current facial motion recognition methods in front-end browsers typically upload the aforementioned video stream to the server for facial motion recognition and then return the results to the front-end browser. This significantly increases the server's computing power requirements. Integrating the video stream into the local client's SDK could reduce server load, but it would increase the development and deployment costs of the local SDK. Summary of the Invention

[0003] This invention provides a facial action recognition method aimed at addressing the high development and deployment costs associated with facial action recognition in existing front-end browsers. By directly transmitting the image to be recognized obtained from a facial image acquisition service to a local facial parsing service via a peer-to-peer connection, the method identifies facial key points. Based on the facial key point recognition results, it determines the organ key point sequences corresponding to each target facial organ. Furthermore, it determines the action recognition results corresponding to each target facial organ based on the organ key point sequences. This allows facial action recognition to be completed directly on the front-end browser side, reducing server-side computing power. Communication via peer-to-peer connection eliminates the need for additional SDK development to access the image sequence. Simultaneously, utilizing organ key point sequences for facial action recognition avoids directly capturing temporal dependencies in the image sequence, reducing the parameter requirements of the local facial parsing service and thus lowering development and deployment costs.

[0004] In a first aspect, embodiments of the present invention provide a facial action recognition method, the method comprising:

[0005] The local face parsing service obtains the facial key point recognition results of the image sequence to be recognized, wherein the image sequence to be recognized is obtained by the face image acquisition service based on the face action recognition instruction;

[0006] Based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined, and the target facial organ is determined according to the facial action recognition instruction;

[0007] Based on the organ key point sequence, determine the action recognition result corresponding to each of the target facial organs;

[0008] The facial action recognition result of the front-end browser is determined based on the action recognition result.

[0009] Optionally, before obtaining the facial landmark recognition results of the local face parsing service for the image sequence to be recognized, the method further includes:

[0010] Based on the face action recognition request from the front-end browser, determine the corresponding face action recognition instruction;

[0011] The face image acquisition service collects the corresponding image sequence to be recognized in relation to the face action recognition command, and transmits the image sequence to be recognized from the face image acquisition service to the local face parsing service through the point-to-point connection.

[0012] The local face analysis service is used to perform facial landmark recognition on the image sequence to be identified, and the facial landmark recognition result corresponding to the image sequence to be identified is obtained.

[0013] Optionally, the facial landmark recognition result includes multiple facial landmark frames, and the step of determining the organ landmark sequence corresponding to each target facial organ based on the facial landmark recognition result includes:

[0014] Each facial key point frame in the facial key point recognition result is classified into key points, and the key points corresponding to each target facial organ in each facial key point frame are determined according to the classification results.

[0015] Based on the key points corresponding to each target facial organ in each facial key point frame, determine the organ key point sequence corresponding to each target facial organ.

[0016] Optionally, determining the action recognition result corresponding to each of the target facial organs based on the organ key point sequence includes:

[0017] Determine the number of motion frames corresponding to each of the target facial organs;

[0018] The key point array corresponding to each of the target facial organs is determined based on the number of action frames corresponding to each target facial organ;

[0019] Based on the key point array corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

[0020] Optionally, determining the action recognition result corresponding to each of the target facial organs based on the key point arrays corresponding to each target facial organ includes:

[0021] For each target facial organ, the aspect ratio array corresponding to the target facial organ is determined based on the key point array;

[0022] Based on the aspect ratio arrays corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

[0023] Optionally, the method for determining the action recognition result corresponding to each of the target facial organs based on the aspect ratio arrays corresponding to each of the target facial organs further includes:

[0024] For each target facial organ, the aspect ratio array is determined to be a valid array based on a preset aspect ratio threshold.

[0025] If the aspect ratio array is a valid array, then determine the standard deviation of the aspect ratio array;

[0026] The action recognition result corresponding to the target facial organ is determined based on the standard deviation of the aspect ratio array.

[0027] Optionally, the target facial organ is both eyes, the action recognition result includes blinking, and the step of determining the action recognition result corresponding to the target facial organ based on the standard deviation of the aspect ratio array includes:

[0028] Determine whether the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold.

[0029] If the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold, then the action recognition result corresponding to both eyes is determined to be blinking.

[0030] Secondly, embodiments of the present invention also provide a face motion recognition device, the face motion recognition device comprising:

[0031] The first acquisition module acquires the facial key point recognition results of the local face parsing service for the image sequence to be recognized;

[0032] The first determining module is used to determine the organ key point sequence corresponding to each target facial organ based on the facial key point recognition result.

[0033] The second determining module is used to determine the action recognition result corresponding to each of the target facial organs based on the organ key point sequence;

[0034] The third determining module determines the face action recognition result of the front-end browser based on the action recognition result.

[0035] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the face action recognition method provided in embodiments of the present invention.

[0036] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the face action recognition method provided in the embodiments of the present invention.

[0037] In this embodiment of the invention, the facial key point recognition results of the local face parsing service for the image sequence to be recognized are obtained, wherein the image sequence to be recognized is obtained by the face image acquisition service based on the face action recognition instruction; based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined, wherein the target facial organ is determined according to the face action recognition instruction; based on the organ key point sequence, the action recognition result corresponding to each target facial organ is determined; and based on the action recognition result, the face action recognition result of the front-end browser is determined. The image to be recognized, acquired by the face image acquisition service, is directly transmitted to the local face parsing service for facial key point recognition via a point-to-point connection. Based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined. Then, the action recognition result corresponding to each target facial organ can be determined based on the organ key point sequence. This enables face action recognition to be completed directly on the front-end browser side, reducing server-side computing power. Communication via point-to-point connection eliminates the need for additional SDK development to call the image sequence to be recognized. At the same time, using the organ key point sequence for face action recognition eliminates the need to directly capture temporal dependencies in the image sequence to be recognized, reducing the parameter requirements of the local face parsing service, thereby reducing development and deployment costs. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a face action recognition method provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a face motion recognition device provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown, Figure 1 This is a flowchart of a face action recognition method provided in an embodiment of the present invention. The face action recognition method includes the following steps:

[0044] 101. Obtain the facial landmark recognition results of the image sequence to be recognized from the local face parsing service.

[0045] In this embodiment of the invention, the facial action recognition method described above can be deployed on any terminal or server capable of running a front-end browser. The front-end browser includes a local face parsing service and a face image acquisition service, which communicate via a peer-to-peer connection. The terminal can be a PC or mobile device capable of running a front-end browser. The face image acquisition service acquires the image sequence to be recognized and transmits it to the local face parsing service via a peer-to-peer connection for processing to obtain the facial key point recognition result. Similarly, the server can be any server or server cluster capable of running a front-end browser. The face image acquisition service acquires the image sequence to be recognized and transmits it to the local face parsing service via a peer-to-peer connection for processing to obtain the facial key point recognition result.

[0046] The aforementioned image sequence to be identified may include multiple facial action capture sequences or a single facial action capture sequence. These multiple or single facial action capture sequences are consecutive captured image frames, and each facial action capture sequence corresponds to one facial landmark recognition result. The facial landmark recognition results of the aforementioned image sequence to be identified may include facial landmark recognition results for each facial action, and these facial landmark recognition results for each facial action may include a sequence of landmark recognition images for each facial action, where the landmark recognition image sequence is a consecutive captured image frame.

[0047] The aforementioned local face parsing service can be a parsing service that processes the image sequence to be recognized based on the face-landmarks-detection library of the front-end browser to obtain the key point recognition result. Alternatively, it can be a parsing service that processes the image sequence to be recognized based on a face parsing model to obtain the key point recognition result. The aforementioned face parsing model can be a face parsing model that has been trained and then encapsulated using hypertext language. This encapsulated face parsing model can then be placed within the aforementioned front-end browser.

[0048] The aforementioned front-end browser calls the API (Application Programming Interface) of the aforementioned face-landmarks-detection library or the aforementioned face parsing model to parse the image sequence to be recognized and obtain the key point recognition results.

[0049] The aforementioned face image acquisition service can be based on an image capturing device installed on any terminal or server capable of running a front-end browser, or it can be based on an image storage device installed on any terminal or server capable of running a front-end browser. It is an acquisition and processing service that obtains a sequence of images to be recognized based on a face action recognition command. The image capturing device is used to execute the face action recognition command and capture the corresponding image sequence, and the image storage device is used to store the image sequence corresponding to the face action recognition command. The sequence of images to be recognized can be continuous face action images captured by the image capturing device when executing the face action recognition command, or continuous face action images read from the image storage device.

[0050] The aforementioned peer-to-peer connection can be established using WebRTC (Web Real-Time Communications) technology, without requiring any plugins or third-party software, to connect the face image acquisition service of the front-end browser with the local face parsing service, enabling the transmission of data such as the image sequence to be recognized. Alternatively, the peer-to-peer connection can utilize WebSockets real-time bidirectional communication technology, establishing an efficient channel between the face image acquisition service of the front-end browser and the local face parsing service. This allows the transmission of the image sequence to be recognized from the face image acquisition service to the local face parsing service, and can also enable bidirectional transmission when needed, allowing the transmission of the image sequence parsed by the local face parsing service to the face image acquisition service in special circumstances.

[0051] The aforementioned facial motion recognition instructions can be determined based on business rules. For example, when performing liveness detection, it is necessary to determine whether the user is alive based on their blinking or mouth-opening movements. Furthermore, the aforementioned facial motion recognition instructions include at least one of the following: blinking motion recognition instructions, mouth-opening motion recognition instructions, etc.

[0052] Furthermore, when the liveness detection service of the front-end browser is started, the corresponding face action recognition instruction is determined according to the business rules of the liveness detection service. When the above-mentioned image capturing device receives the face action recognition instruction, it will capture the user's face. After successful capture, the image sequence to be recognized corresponding to the user's face is obtained. The image sequence to be recognized is transmitted to the face-landmarks-detection library through the point-to-point connection established by WebRTC real-time communication technology to obtain the face key point recognition result.

[0053] 102. Based on the facial key point recognition results, determine the organ key point sequence corresponding to each target facial organ.

[0054] In this embodiment of the invention, the target facial organ may include facial organs such as eyes and mouth, which are determined according to facial action recognition instructions. Different facial organs correspond to different facial action recognition instructions. Eyes may correspond to blinking action recognition instructions, and mouths may correspond to opening mouth action recognition instructions or closing mouth action recognition instructions. When the facial action recognition instruction is a blinking action recognition instruction, the target facial organ is the eyes; when the facial action recognition instruction is an opening mouth action recognition instruction, the target facial organ is the mouth. The organ key point sequence may be at least one of eye key point sequence and mouth key point sequence. The organ key point sequence corresponds to the target facial organ. Specifically, the eyes correspond to the eye key point sequence, and the mouth corresponds to the mouth key point sequence. The facial key point recognition result contains organ key point sequences corresponding to each facial organ. After obtaining the facial action recognition instruction, the target facial organ is determined according to the facial action recognition instruction, and the organ key point sequence corresponding to the target facial organ is extracted from the facial key point recognition result.

[0055] In one possible embodiment, when the facial key point recognition result obtained by the local face parsing service from the image sequence to be recognized is obtained, a facial action recognition instruction can be obtained, and the key point sequence of the target facial organ corresponding to the facial action recognition instruction in the facial key point recognition result can be determined through the facial action recognition instruction.

[0056] 103. Based on the organ key point sequence, determine the action recognition results corresponding to each target facial organ.

[0057] In this embodiment of the invention, the above-mentioned action recognition result may include at least one face recognition result such as eye action recognition result and mouth action recognition result, wherein the eye action recognition result corresponds to the eye key point sequence, and the mouth action recognition result corresponds to the mouth key point sequence.

[0058] The above action recognition results can be determined based on the key point sequence of the corresponding facial organs. For example, the above eye action recognition results can be determined based on the eye key point sequence, and the above mouth action recognition results can be determined based on the mouth key point sequence.

[0059] The action recognition result described above can be the recognition result of a single action or the recognition result of multiple actions. The specific number of recognition results depends on how many actions are contained in the organ keypoint sequence. For example, when the organ keypoint sequence contains only the eye keypoint sequence, the action recognition result is the recognition result of a single action. When the organ keypoint sequence contains only the mouth keypoint sequence, the action recognition result is the recognition result of a single action. When the organ keypoint sequence contains both the mouth keypoint sequence and the eye keypoint sequence, the action recognition result is the recognition result of multiple actions.

[0060] The aforementioned organ keypoint sequence can be used by a processor in any terminal or server capable of running a front-end browser to determine the positional or area changes of each organ keypoint, and then determine the action recognition result corresponding to each target facial organ based on these positional or area changes. For example, the eye action recognition result can be determined based on the positional or area changes of each eye keypoint in the eye keypoint sequence, and the mouth action recognition result can be determined based on the positional or area changes of each mouth keypoint in the mouth keypoint sequence.

[0061] The aforementioned action recognition results can include action recognition image sequences and action recognition attributes. The action recognition image sequence is a subsequence of the image sequence to be recognized. Each action recognition image sequence corresponds to an action recognition attribute, which can be an action type and a result type. The action type refers to the action type corresponding to a facial feature; for example, blinking corresponds to the eye action type, and opening the mouth corresponds to the mouth action type. The result type can be recognition successful or recognition failed. The action type and result type can be represented by label values ​​to facilitate the final facial action recognition judgment. For example, the label value corresponding to the action type can be 00 or 01. Here, 00 represents the action type as blinking, and 01 represents the action type as opening the mouth. The label value corresponding to the result type can be represented by 0 or 1, where 1 represents recognition successful and 0 represents recognition failed. For each action recognition image sequence, its corresponding output value includes the label value corresponding to the action type and the label value corresponding to the result type. If the output value is 000, it indicates that blinking recognition failed; if the output value is 001, it indicates that blinking recognition succeeded; if the output value is 010, it indicates that opening the mouth recognition failed; and if the output value is 011, it indicates that opening the mouth recognition succeeded. The number of action types that a label value can represent can be controlled by adjusting the length of the label value corresponding to the action type.

[0062] Specifically, the action recognition image sequence, action type, and result type corresponding to each facial organ can be determined based on the positional and area changes of the key points of each organ. The action recognition attribute can be determined based on the label value corresponding to the action type and the label value corresponding to the main result type. Finally, the action recognition result can be determined based on the action recognition attribute and the action recognition image sequence.

[0063] Sampling can be performed on the organ keypoint sequences corresponding to each target facial organ at a preset step size and preset length to obtain sampling results. The preset step size can be manually set, for example, to n frames, where n is an integer greater than or equal to 1. Alternatively, the preset step size can be determined by the frame rate of the image to be recognized; a higher frame rate indicates denser frames, allowing for a larger step size, while a lower frame rate indicates sparser frames, allowing for a smaller step size. The preset length can be determined based on the number of action frames corresponding to the target facial organ. The corresponding frame images in the image sequence to be recognized can be determined based on the frame positions of the organ keypoints in the sampling results to serve as the action recognition image sequence. The action type and result type corresponding to each facial organ can be determined based on the changes in the position and area of ​​the organ keypoints in the sampling results.

[0064] 104. Determine the face action recognition result of the front-end browser based on the action recognition result.

[0065] In this embodiment of the invention, the facial motion recognition result is the final output result, which is output to the user interface corresponding to the front-end browser through the front-end browser. After the motion recognition result is determined, the corresponding facial motion recognition result is determined based on the motion recognition result. The facial motion recognition result can be used as the identification basis of the front-end browser to perform liveness authentication on the user. When the facial motion recognition result is successful, the front-end browser performs liveness authentication on the user and the authentication is passed. When the facial motion recognition result is unsuccessful, the front-end browser performs liveness authentication on the user and the authentication is not passed.

[0066] The above facial motion recognition result is determined based on the motion recognition result. If the motion recognition result is "motion recognition successful", then the facial motion recognition result is "recognition successful". If the motion recognition result is "motion recognition failed", then the facial motion recognition result is "recognition failed".

[0067] The facial motion recognition results can be output to the corresponding user interface on the browser.

[0068] For example, when the label value corresponding to the action type of the above-mentioned action recognition attribute is 00 and the label value corresponding to the result type is 0, the processor in any terminal or server that can be equipped with a front-end browser will determine the output value of the above-mentioned action recognition result as 000. At this time, the corresponding face recognition result is recognition failure. The above-mentioned output value can be output to the above-mentioned user interface and converted into text content for display. The above-mentioned text content can be Chinese text content, and the above-mentioned Chinese text content can be "eye action recognition failure". When the label value corresponding to the action type of the above-mentioned action recognition attribute is 00 and the label value corresponding to the result type is 1, the processor in any terminal or server that can be equipped with a front-end browser will determine the output value of the above-mentioned action recognition result as 001. At this time, the corresponding face recognition result is recognition success. The above-mentioned output value can be output to the above-mentioned user interface and converted into text content for display. The above-mentioned text content can be Chinese text content, and the above-mentioned Chinese text content can be "eye action recognition success". When the label value corresponding to the action type of the above-mentioned action recognition attribute is 01 and the label value corresponding to the result type is 0, the processor in any terminal or server that can be equipped with a front-end browser determines the output value of the above-mentioned action recognition result to be 010. At this time, the corresponding face recognition result is recognition failure. The above-mentioned output value can be output to the above-mentioned user interface and converted into text content for display. The above-mentioned text content can be Chinese text content, and the above-mentioned Chinese text content can be "mouth action recognition failure". When the label value corresponding to the action type of the above-mentioned action recognition attribute is 01 and the label value corresponding to the result type is 1, the processor in any terminal or server that can be equipped with a front-end browser determines the output value of the above-mentioned action recognition result to be 011. At this time, the corresponding face recognition result is recognition success. The above-mentioned output value can be output to the above-mentioned user interface and converted into text content for display. The above-mentioned text content can be Chinese text content, and the above-mentioned Chinese text content can be "mouth action recognition success".

[0069] In this embodiment of the invention, the facial key point recognition results of the local face parsing service for the image sequence to be recognized are obtained, wherein the image sequence to be recognized is obtained by the face image acquisition service based on the face action recognition instruction; based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined, wherein the target facial organ is determined according to the face action recognition instruction; based on the organ key point sequence, the action recognition result corresponding to each target facial organ is determined; and based on the action recognition result, the face action recognition result of the front-end browser is determined. The image to be recognized, acquired by the face image acquisition service, is directly transmitted to the local face parsing service for facial key point recognition via a point-to-point connection. Based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined. Then, the action recognition result corresponding to each target facial organ can be determined based on the organ key point sequence. This enables face action recognition to be completed directly on the front-end browser side, reducing server-side computing power. Communication via point-to-point connection eliminates the need for additional SDK development to call the image sequence to be recognized. At the same time, using the organ key point sequence for face action recognition eliminates the need to directly capture temporal dependencies in the image sequence to be recognized, reducing the parameter requirements of the local face parsing service, thereby reducing development and deployment costs.

[0070] Optionally, before obtaining the facial landmark recognition results of the image sequence to be recognized from the local face resolution service, the corresponding facial action recognition instruction can be determined based on the facial action recognition request from the front-end browser; the image sequence to be recognized corresponding to the facial action recognition instruction can be collected through the face image acquisition service, and the image sequence to be recognized can be transmitted from the face image acquisition service to the local face resolution service through a point-to-point connection; the local face resolution service can then perform facial landmark recognition on the image sequence to be recognized to obtain the facial landmark recognition results corresponding to the image sequence to be recognized.

[0071] In this embodiment of the invention, the aforementioned face action recognition request corresponds to the aforementioned face action recognition instruction. The aforementioned face action recognition request may be a request selected by the user through the user interface corresponding to the aforementioned front-end browser. When the user confirms the face action recognition request through the aforementioned user interface, the face image acquisition service corresponding to the front-end browser responds and obtains the corresponding face action recognition instruction. Alternatively, the aforementioned face action recognition request may be a face action recognition request applied for by any authorized plugin or any authorized webpage in the aforementioned front-end browser. When any authorized plugin or any authorized webpage applies for a face action recognition request, the face image acquisition service corresponding to the front-end browser responds and obtains the corresponding face action recognition instruction.

[0072] The above-mentioned acquisition of the image sequence corresponding to the face action recognition command through the face image acquisition service can be achieved by calling the aforementioned front-end browser API interface (programming interface) to obtain the video stream of the image capturing device corresponding to the face action recognition command, and transmitting the aforementioned video stream of the image capturing device to the local face parsing service through a point-to-point connection, and the aforementioned local face parsing service to perform face key point recognition to obtain the face key point recognition result corresponding to the image capturing device video stream.

[0073] The aforementioned image sequence to be identified is transmitted to a local face analysis service. The local face analysis service analyzes the facial key points of each frame in the image sequence to obtain the facial key point frame corresponding to each frame. Each facial key point frame corresponds to one frame in the image sequence to be identified. Each facial key point frame includes key points corresponding to each facial feature.

[0074] Optionally, the facial landmark recognition result includes multiple facial landmark frames. Based on the facial landmark recognition result, the organ landmark sequence corresponding to each target facial organ is determined, including: classifying the landmarks of each facial landmark frame in the facial landmark recognition result; determining the landmarks corresponding to each target facial organ in each facial landmark frame based on the classification result; and determining the organ landmark sequence corresponding to each target facial organ based on the landmarks corresponding to each target facial organ in each facial landmark frame.

[0075] In this embodiment of the invention, the facial key point recognition result includes a facial key point sequence, which comprises multiple facial key point frames. Each facial key point frame corresponds to a frame image in the image to be recognized, and each facial key point frame includes key points corresponding to multiple facial organs. Specifically, the facial key point recognition result includes consecutive facial key point frames, and the target facial organs correspond to different facial organs. Since the facial organs correspond to facial key points, it is necessary to classify the facial key points according to the facial organs in order to obtain the organ key point sequence corresponding to each target facial organ. The aforementioned facial key points can be classified and distinguished by color or by number. For example, the color classification can be as follows: if the facial key point corresponding to the mouth is marked in red, then the above key point sequence is the set of all facial key points marked in red corresponding to the mouth in the target facial organ; if the facial key point corresponding to the nose is marked in green, then the above key point sequence is the set of all facial key points marked in green corresponding to the nose in the target facial organ; and if the facial key point corresponding to the eyes is marked in yellow, then the above key point sequence is the set of all facial key points marked in yellow corresponding to the eyes in the target facial organ. The above-mentioned numerical classification can be as follows: facial key points corresponding to the mouth are numbered according to numbers, with the facial key points corresponding to the mouth marked as numbers 1-15. Then, the above-mentioned organ key point sequence is the set of facial key points with numbers 1-15 corresponding to the mouth in the target facial organ. Facial key points corresponding to the nose are numbered according to numbers, with the facial key points corresponding to the nose marked as numbers 16-35. Then, the above-mentioned organ key point sequence is the set of facial key point sequences with numbers 16-35 corresponding to the nose in the target facial organ. Facial key points corresponding to the eyes are marked as numbers 36-50. Then, the above-mentioned organ key point sequence is the set of facial key point sequences with numbers 36-50 corresponding to the eyes in the target facial organ.

[0076] Specifically, after the local face parsing service completes the parsing and obtains the face key point recognition results, key points are classified according to each face key point frame in the face key point recognition results. The key points can be classified to the corresponding specific face actions, and the target face organs corresponding to the key points in each consecutive face key point frame are determined according to the classification results. Based on the key points corresponding to each target face organ in each face key point frame, the organ key point sequence corresponding to each target face organ is determined.

[0077] Optionally, based on the organ key point sequence, the action recognition result corresponding to each target facial organ is determined, including: determining the number of action frames corresponding to each target facial organ; determining the key point array corresponding to each target facial organ based on the number of action frames corresponding to each target facial organ; and determining the action recognition result corresponding to each target facial organ based on the key point array corresponding to each target facial organ.

[0078] In this embodiment of the invention, since the time required for a user to perform different facial actions varies, the number of action frames required for each facial action can be determined based on the time required for each action. Furthermore, the user performs corresponding actions based on facial action recognition commands. Different facial action recognition commands correspond to different facial organs; therefore, different facial organs can correspond to different command actions. Because the time required for the user to control different facial organs to complete the corresponding command actions varies, the number of action frames required to complete the corresponding command actions also varies. More action frames indicate a longer time required for the user to complete the corresponding command action, and fewer action frames indicate a shorter time required. The association between different facial organs and their corresponding number of action frames can be maintained through a mapping table. After identifying the target facial organ, the number of action frames corresponding to the target facial organ can be determined based on the association between each facial organ and its corresponding number of action frames in the mapping table, thus obtaining the number of action frames corresponding to each target facial organ. Specifically, the number of action frames required for the command action corresponding to the eyes can be five action frames, and the number of action frames required for the command action corresponding to the mouth can be eight action frames. The aforementioned key point array includes the key points corresponding to each frame in the aforementioned number of action frames. After determining the number of action frames corresponding to each target facial organ, the key point array corresponding to each target facial organ is determined based on the number of action frames corresponding to each target facial organ, and the action recognition result corresponding to each target facial organ is determined based on the key point array corresponding to each target facial organ.

[0079] Optionally, based on the key point array corresponding to each target facial organ, the action recognition result corresponding to each target facial organ is determined, including: for each target facial organ, determining the aspect ratio array corresponding to the target facial organ based on the key point array; and determining the action recognition result corresponding to each target facial organ based on the aspect ratio array corresponding to each target facial organ.

[0080] In this embodiment of the invention, sampling can be performed on the organ keypoint sequence corresponding to each target facial organ according to a preset step size and a preset length. The preset length is the number of action frames corresponding to the target facial organ, resulting in a keypoint array corresponding to each target facial organ. For a target organ, the keypoint array includes facial keypoint frames, the number of which is the number of action frames corresponding to the target facial organ. Each facial keypoint frame includes keypoints corresponding to the target facial organ. Based on the image coordinates of the keypoints corresponding to the target facial organ, the aspect ratio array corresponding to the target facial organ is calculated. The image coordinates of the keypoints can be represented by (x, y, z), where (x, y) are the horizontal and vertical coordinates of the keypoint in the image, and z is the frame coordinate of the keypoint. Each aspect ratio in the aspect ratio array corresponds to the aspect ratio of a facial keypoint frame in the keypoint array. For each target facial organ, the corresponding action recognition result can be determined using the corresponding aspect ratio array.

[0081] Specifically, the aspect ratio mentioned above can be calculated using the following formula:

[0082]

[0083] Among them, the above S i Let P1 be the aspect ratio of the i-th face keypoint frame. i P2 represents the minimum x-coordinate of the facial landmark in the i-th facial landmark frame. i P3 represents the maximum x-coordinate of the facial landmark in the i-th facial landmark frame. i P4 represents the minimum y-coordinate of the facial landmark in the i-th facial landmark frame. i P5 represents the second minimum value of the y-coordinate of the facial landmark in the i-th facial landmark frame. i P6 represents the maximum value of the y-coordinate of the facial landmark in the i-th facial landmark frame. i It represents the second-highest value of the y-coordinate of the face key point in the i-th face key point frame.

[0084] After obtaining the aspect ratio array corresponding to the target facial features, the action recognition result corresponding to the target facial features can be determined based on the changes in the aspect ratio in the aspect ratio array.

[0085] Optionally, the action recognition result corresponding to each target facial organ is determined based on the aspect ratio array corresponding to each target facial organ. The method further includes: for each target facial organ, determining whether the aspect ratio array is a valid array based on a preset aspect ratio threshold; if the aspect ratio array is a valid array, determining the standard deviation of the aspect ratio array; and determining the action recognition result corresponding to the target facial organ based on the standard deviation of the aspect ratio array.

[0086] In this embodiment of the invention, for a target facial organ, there can be multiple aspect ratio arrays. The number of aspect ratio arrays is related to the preset step size and the length of the organ keypoint sequence corresponding to the target facial organ. The longer the preset step size, the fewer the number of aspect ratio arrays, and vice versa. The longer the organ keypoint sequence, the more aspect ratio arrays, and vice versa. Among the multiple aspect ratio arrays, some aspect ratio arrays can represent the corresponding facial actions, while others cannot.

[0087] The aforementioned aspect ratio threshold can be set according to the command action corresponding to the target facial organ. Comparing the aspect ratio array with the preset aspect ratio threshold determines whether the aspect ratio array is valid. The aspect ratio array corresponding to the valid array indicates the existence of a corresponding facial action, while the aspect ratio array corresponding to the invalid array indicates the absence of a corresponding facial action. The standard deviation can be the magnitude of the action change in the aspect ratio array corresponding to the target facial organ, and the action recognition result corresponding to the target facial organ can be determined based on this magnitude of change.

[0088] A threshold for aspect ratio can be used to determine whether a corresponding aspect ratio array includes the specific state corresponding to a facial action. The standard deviation is used to determine whether the magnitude of action change corresponding to the aspect ratio array meets the corresponding state change requirement. If an aspect ratio array contains the corresponding specific state, it is considered a valid array; otherwise, it is invalid. If the magnitude of action change corresponding to a valid array meets the corresponding state change requirement, then the action recognition is considered successful.

[0089] Specifically, the above standard deviation can be calculated using the following formula:

[0090]

[0091]

[0092] Where A is the average value of the aspect ratio array, n is the number of motion frames corresponding to the aspect ratio array, and P1 i P2 represents the minimum x-coordinate of the facial landmark in the i-th facial landmark frame. i P3 represents the maximum x-coordinate of the facial landmark in the i-th facial landmark frame. i P4 represents the minimum y-coordinate of the facial landmark in the i-th facial landmark frame. i P5 represents the second minimum value of the y-coordinate of the facial landmark in the i-th facial landmark frame. i In the i-th face landmark frame, the face landmark is the maximum value of the y-coordinate, P6 i Z is the second-highest value of the y-coordinate of the facial landmark in the i-th facial landmark frame, Z is the standard deviation, and S is the second-highest value. iLet be the aspect ratio of the i-th face keypoint frame.

[0093] For example, the target facial organ mentioned above can be the mouth. Based on the mouth, a preset mouth aspect ratio threshold is determined. When any mouth aspect ratio in the mouth aspect ratio array is less than the preset mouth aspect ratio threshold, the facial key point frame corresponding to that mouth aspect ratio is determined to be in a closed mouth state. When any mouth aspect ratio in the mouth aspect ratio array is greater than or equal to the preset mouth aspect ratio threshold, the facial key point frame corresponding to that mouth aspect ratio is determined to be in an open mouth state. All mouth aspect ratios in the mouth aspect ratio array are compared one by one with the preset mouth aspect ratio threshold. If all mouth aspect ratios in the mouth aspect ratio array are greater than or less than the preset mouth aspect ratio threshold, the mouth aspect ratio array is considered an invalid array. If the mouth aspect ratio array contains both mouth aspect ratios greater than and less than the preset mouth aspect ratio threshold, the mouth aspect ratio array is considered a valid array. When the standard deviation of the mouth aspect ratio array is greater than or equal to the preset standard deviation threshold, it can be determined that the change range of the mouth aspect ratio array meets the corresponding change state, and the corresponding mouth opening action can be determined to be successfully recognized.

[0094] The target facial organ can be the eyes. A preset eye aspect ratio threshold is determined based on the eyes. When any eye aspect ratio in the eye aspect ratio array is less than the preset threshold, the facial key point frame corresponding to that eye aspect ratio is determined to be in a closed-eye state. When any eye aspect ratio in the eye aspect ratio array is greater than or equal to the preset threshold, the facial key point frame corresponding to that eye aspect ratio is determined to be in an open-eye state. All eye aspect ratios in the eye aspect ratio array are compared one by one with the preset threshold. If all eye aspect ratios in the eye aspect ratio array are greater than or less than the preset threshold, the eye aspect ratio array is considered invalid. If the eye aspect ratio array contains both eye aspect ratios greater than and less than the preset threshold, the eye aspect ratio array is considered valid. When the standard deviation of the above eye aspect ratio array is greater than or equal to the preset standard deviation threshold, it can be determined that the amplitude of the movement change of the eye aspect ratio array meets the corresponding change state, and the corresponding blinking action can be determined to be successfully recognized.

[0095] Optionally, the target facial organs are both eyes, and the action recognition result includes blinking. The action recognition result corresponding to the target facial organs is determined based on the standard deviation of the aspect ratio array, including: determining whether the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold; if the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than the preset standard deviation threshold, then the action recognition result corresponding to both eyes is determined to be blinking.

[0096] In this embodiment of the invention, the target facial organs can be both eyes, and the action recognition result can be blinking. Since blinking involves both eyes closing and opening simultaneously, the determination of whether a blink has occurred requires simultaneous determination of both eyes. The valid array corresponding to the left eye is defined as the standard deviation corresponding to the left eye, and the valid array corresponding to the right eye is defined as the standard deviation corresponding to the right eye. It can be determined whether the standard deviations corresponding to the left and right eyes are simultaneously greater than a preset standard deviation threshold. If both the standard deviations corresponding to the left and right eyes are simultaneously greater than the preset standard deviation threshold, the action recognition result for both eyes is determined to be a successful blinking action recognition. If neither the standard deviations corresponding to the left and right eyes are simultaneously greater than the preset standard deviation threshold, the action recognition result for both eyes is determined to be a failed blinking action recognition.

[0097] Furthermore, before determining whether blinking has occurred, it can be determined whether the effective arrays corresponding to the left and right eyes are within the effective time frame. If the effective arrays corresponding to the left and right eyes contain the same aspect ratio with the same frame number, then it can be determined that the effective arrays corresponding to the left and right eyes are within the effective time frame; otherwise, it can be determined that the effective arrays corresponding to the left and right eyes are within the effective time frame. The standard deviation of the effective array corresponding to the left eye within the effective time frame is used as the standard deviation of the left eye, and the standard deviation of the effective array corresponding to the right eye within the effective time frame is used as the standard deviation of the right eye.

[0098] It should be noted that the facial motion recognition method provided in this embodiment of the invention can be applied to devices such as shooting devices, smartphones, computers, and servers that can perform facial motion recognition.

[0099] like Figure 2 As shown, an embodiment of the present invention provides a face motion recognition device, which includes:

[0100] The first acquisition module 201 acquires the facial key point recognition results of the local face parsing service for the image sequence to be recognized;

[0101] The first determining module 202 is used to determine the organ key point sequence corresponding to each target facial organ based on the facial key point recognition result;

[0102] The second determining module 203 is used to determine the action recognition result corresponding to each of the target facial organs based on the organ key point sequence;

[0103] The third determining module 204 determines the face action recognition result of the front-end browser based on the action recognition result.

[0104] Optionally, the first acquisition module 201 further includes:

[0105] The fourth determining submodule is used to determine the corresponding face action recognition instruction based on the face action recognition request from the front-end browser.

[0106] The first acquisition submodule is used to acquire the corresponding image sequence to be recognized and the face action recognition command through the face image acquisition service, and to transmit the image sequence to be recognized from the face image acquisition service to the local face parsing service through the point-to-point connection.

[0107] The first recognition submodule performs facial landmark recognition on the image sequence to be recognized through the local face parsing service to obtain the facial landmark recognition result corresponding to the image sequence to be recognized.

[0108] Optionally, the first determining module 202 includes:

[0109] The first classification submodule is used to classify key points in each face key point frame in the face key point recognition result, and determine the key points corresponding to each target face organ in each face key point frame according to the classification result.

[0110] The fifth determining submodule is used to determine the organ key point sequence corresponding to each of the target facial organs based on the key points corresponding to each of the target facial organs in each facial key point frame.

[0111] Optionally, the first determining module 202 further includes:

[0112] The sixth determining submodule is used to determine the number of action frames corresponding to each of the target facial organs;

[0113] The seventh determining submodule is used to determine the key point array corresponding to each of the target facial organs based on the number of action frames corresponding to each of the target facial organs;

[0114] The eighth determining submodule is used to determine the action recognition result corresponding to each of the target facial organs based on the key point array corresponding to each of the target facial organs.

[0115] Optionally, the eighth determining submodule includes:

[0116] The first determining unit is used to determine the aspect ratio array corresponding to each target facial organ based on the key point array.

[0117] The second determining unit is used to determine the action recognition result corresponding to each of the target facial organs based on the aspect ratio array corresponding to each of the target facial organs.

[0118] Optionally, the second determining unit includes:

[0119] The first determining subunit is used to determine whether the aspect ratio array is a valid array for each target facial organ according to a preset aspect ratio threshold; if the aspect ratio array is a valid array, then the standard deviation of the aspect ratio array is determined.

[0120] The second determining subunit is used to determine the action recognition result corresponding to the target facial organ based on the standard deviation of the aspect ratio array.

[0121] Optionally, the second determining unit further includes:

[0122] The third determining subunit is used to determine whether the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are simultaneously greater than a preset standard deviation threshold.

[0123] The fourth determining subunit is used to determine that if the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold, the action recognition result corresponding to the two eyes is blinking.

[0124] It should be noted that the facial motion recognition device provided in this embodiment of the invention can be applied to devices such as shooting devices, smartphones, computers, and servers that can perform facial motion recognition.

[0125] The facial motion recognition device provided in this embodiment of the invention can implement all the processes of the facial motion recognition method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, further details are omitted here.

[0126] See Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program for a face action recognition method stored in the memory 302 and executable on the processor 301, wherein:

[0127] The processor 301 is used to call the computer program stored in the memory 302 and perform the following steps:

[0128] The local face parsing service obtains the facial key point recognition results of the image sequence to be recognized, wherein the image sequence to be recognized is obtained by the face image acquisition service based on the face action recognition instruction;

[0129] Based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined, and the target facial organ is determined according to the facial action recognition instruction;

[0130] Based on the organ key point sequence, determine the action recognition result corresponding to each of the target facial organs;

[0131] The facial action recognition result of the front-end browser is determined based on the action recognition result.

[0132] Optionally, before obtaining the facial landmark recognition results of the local face parsing service for the image sequence to be recognized, the method executed by the processor 301 further includes:

[0133] Based on the face action recognition request from the front-end browser, determine the corresponding face action recognition instruction;

[0134] The face image acquisition service collects the corresponding image sequence to be recognized in relation to the face action recognition command, and transmits the image sequence to be recognized from the face image acquisition service to the local face parsing service through the point-to-point connection.

[0135] The local face analysis service is used to perform facial landmark recognition on the image sequence to be identified, and the facial landmark recognition result corresponding to the image sequence to be identified is obtained.

[0136] Optionally, the facial key point recognition result executed by the processor 301 includes multiple facial key point frames, and the step of determining the organ key point sequence corresponding to each target facial organ based on the facial key point recognition result includes:

[0137] Each facial key point frame in the facial key point recognition result is classified into key points, and the key points corresponding to each target facial organ in each facial key point frame are determined according to the classification results.

[0138] Based on the key points corresponding to each target facial organ in each facial key point frame, determine the organ key point sequence corresponding to each target facial organ.

[0139] Optionally, the step of determining the action recognition result corresponding to each of the target facial organs based on the organ key point sequence executed by the processor 301 includes:

[0140] Determine the number of motion frames corresponding to each of the target facial organs;

[0141] The key point array corresponding to each of the target facial organs is determined based on the number of action frames corresponding to each target facial organ;

[0142] Based on the key point array corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

[0143] Optionally, the step of determining the action recognition result corresponding to each of the target facial organs based on the key point array corresponding to each of the target facial organs, executed by the processor 301, includes:

[0144] For each target facial organ, the aspect ratio array corresponding to the target facial organ is determined based on the key point array;

[0145] Based on the aspect ratio arrays corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

[0146] Optionally, the method executed by the processor 301, which determines the action recognition result corresponding to each of the target facial organs based on the aspect ratio arrays corresponding to each of the target facial organs, further includes:

[0147] For each target facial organ, the aspect ratio array is determined to be a valid array based on a preset aspect ratio threshold.

[0148] If the aspect ratio array is a valid array, then determine the standard deviation of the aspect ratio array;

[0149] The action recognition result corresponding to the target facial organ is determined based on the standard deviation of the aspect ratio array.

[0150] Optionally, the target facial organ executed by the processor 301 is both eyes, and the action recognition result includes blinking. Determining the action recognition result corresponding to the target facial organ based on the standard deviation of the aspect ratio array includes:

[0151] Determine whether the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold.

[0152] If the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold, then the action recognition result corresponding to both eyes is determined to be blinking.

[0153] It should be noted that the electronic device provided in the embodiments of the present invention can be applied to devices such as smartphones, computers, and servers that can perform facial motion recognition methods.

[0154] The electronic device provided in this embodiment of the invention can implement all the processes of the face action recognition method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0155] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the face action recognition method or the application-side face action recognition method provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0156] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0157] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A facial motion recognition method, characterized in that, The facial action recognition method is used by a front-end browser to recognize facial actions. The front-end browser is equipped with a local face parsing service and a face image acquisition service. The local face parsing service and the face image acquisition service communicate via a peer-to-peer connection. The method includes the following steps: The local face parsing service obtains the facial key point recognition results of the image sequence to be recognized, wherein the image sequence to be recognized is obtained by the face image acquisition service based on the face action recognition instruction; Based on the facial key point recognition results, the organ key point sequence corresponding to each target facial organ is determined, and the target facial organ is determined according to the facial action recognition instruction; Based on the organ key point sequence, determine the action recognition result corresponding to each of the target facial organs; The face action recognition result of the front-end browser is determined based on the action recognition result. The facial landmark recognition result includes multiple facial landmark frames. The step of determining the organ landmark sequence corresponding to each target facial organ based on the facial landmark recognition result includes: Each facial key point frame in the facial key point recognition result is classified into key points, and the key points corresponding to each target facial organ in each facial key point frame are determined according to the classification results. Based on the key points corresponding to each of the target facial organs in each facial key point frame, determine the organ key point sequence corresponding to each of the target facial organs; The step of determining the action recognition result corresponding to each of the target facial organs based on the organ key point sequence includes: The relationship between different facial organs and their corresponding action frames is maintained through a mapping table. After the target facial organ is determined, the action frame number corresponding to the target facial organ is determined according to the relationship between each facial organ and its corresponding action frame number in the mapping table. The time required for the user to perform different facial actions is different. The key point array corresponding to each of the target facial organs is determined based on the number of action frames corresponding to each target facial organ; Based on the key point array corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

2. The facial motion recognition method as described in claim 1, characterized in that, Before obtaining the facial landmark recognition results of the local face parsing service for the image sequence to be recognized, the method further includes: Based on the face action recognition request from the front-end browser, determine the corresponding face action recognition instruction; The face image acquisition service collects the corresponding image sequence to be recognized in relation to the face action recognition command, and transmits the image sequence to be recognized from the face image acquisition service to the local face parsing service through the point-to-point connection. The local face analysis service is used to perform facial landmark recognition on the image sequence to be identified, and the facial landmark recognition result corresponding to the image sequence to be identified is obtained.

3. The facial motion recognition method as described in claim 1, characterized in that, The step of determining the action recognition result corresponding to each of the target facial organs based on the key point array corresponding to each of the target facial organs includes: For each target facial organ, the aspect ratio array corresponding to the target facial organ is determined based on the key point array; Based on the aspect ratio arrays corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

4. The facial motion recognition method as described in claim 3, characterized in that, The method further includes determining the action recognition result corresponding to each of the target facial organs based on the aspect ratio array corresponding to each of the target facial organs. For each target facial organ, the aspect ratio array is determined to be a valid array based on a preset aspect ratio threshold. If the aspect ratio array is a valid array, then determine the standard deviation of the aspect ratio array; The action recognition result corresponding to the target facial organ is determined based on the standard deviation of the aspect ratio array.

5. The facial motion recognition method as described in claim 4, characterized in that, The target facial features are the eyes, the action recognition result includes blinking, and the step of determining the action recognition result corresponding to the target facial features based on the standard deviation of the aspect ratio array includes: Determine whether the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold. If the standard deviation corresponding to the left eye and the standard deviation corresponding to the right eye are both greater than a preset standard deviation threshold, then the action recognition result corresponding to both eyes is determined to be blinking.

6. A facial motion recognition device, characterized in that, The facial motion recognition device is used by a front-end browser to recognize facial movements. The front-end browser includes a local facial parsing service and a facial image acquisition service, which communicate via a peer-to-peer connection. The facial motion recognition device comprises: The first acquisition module is used to acquire the facial key point recognition results of the local face parsing service for the image sequence to be recognized; The first determining module is used to determine the organ key point sequence corresponding to each target facial organ based on the facial key point recognition result. The second determining module is used to determine the action recognition result corresponding to each of the target facial organs based on the organ key point sequence; The third determining module is used to determine the face action recognition result of the front-end browser based on the action recognition result; The facial landmark recognition result includes multiple facial landmark frames. The step of determining the organ landmark sequence corresponding to each target facial organ based on the facial landmark recognition result includes: Each facial key point frame in the facial key point recognition result is classified into key points, and the key points corresponding to each target facial organ in each facial key point frame are determined according to the classification results. Based on the key points corresponding to each of the target facial organs in each facial key point frame, determine the organ key point sequence corresponding to each of the target facial organs; The step of determining the action recognition result corresponding to each of the target facial organs based on the organ key point sequence includes: The relationship between different facial organs and their corresponding action frames is maintained through a mapping table. After the target facial organ is determined, the action frame number corresponding to the target facial organ is determined according to the relationship between each facial organ and its corresponding action frame number in the mapping table. The time required for the user to perform different facial actions is different. The key point array corresponding to each of the target facial organs is determined based on the number of action frames corresponding to each target facial organ; Based on the key point array corresponding to each of the target facial organs, the action recognition result corresponding to each of the target facial organs is determined.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the face action recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the face action recognition method as described in any one of claims 1 to 5.

Images