Facial symmetry assessment method, device, equipment, storage medium and program product

Abstract

The present application relates to a facial symmetry assessment method, apparatus, device, storage medium, and program product. The method, which can be used in the field of financial technology or other related fields, comprises: obtaining a facial image to be assessed, obtaining first facial key points from the facial image for forming a preset facial feature region trajectory, and second facial key points for forming a preset facial key point connection line; obtaining the facial feature region trajectory corresponding to the facial image based on the first facial key points, and obtaining a first symmetry for the facial image based on the facial feature region trajectory; obtaining a facial key point connection line corresponding to the facial image based on the second facial key points, and obtaining a second symmetry for the facial image based on the facial key point connection line; and obtaining a symmetry for the facial image based on the first symmetry and the second symmetry. This method can improve the accuracy of facial symmetry assessment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a facial symmetry assessment method, apparatus, device, storage medium, and program product. Background Art

[0002] With the development of artificial intelligence technology, a facial expression recognition technology has emerged that recognizes facial expressions based on evaluating facial symmetry.

[0003] Traditional approaches to assessing facial symmetry rely on connecting lines with key facial points and using the deflection angle of these lines to assess facial symmetry. However, using only these lines can easily lead to inaccurate assessments, resulting in low accuracy in existing facial symmetry assessment methods. Summary of the Invention

[0004] Based on this, it is necessary to provide a facial symmetry assessment method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of facial symmetry assessment in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for evaluating facial symmetry. The method comprises:

[0006] Acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory and second facial key points for forming a line connecting the preset facial key points;

[0007] Obtaining a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtaining a first degree of symmetry for the facial image based on the facial feature region trajectory;

[0008] Obtaining a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtaining a second degree of symmetry for the facial image based on the line connecting facial key points;

[0009] A symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

[0010] In one embodiment, the facial feature area trajectory includes: a left facial feature area trajectory, and a right facial feature area trajectory; obtaining the facial feature area trajectory corresponding to the facial image based on the first facial key points includes: obtaining the midpoint of the line connecting the facial key points; based on the midpoint, obtaining the central axis of the line connecting the facial key points; based on the central axis, obtaining the left facial feature area trajectory and the right facial feature area trajectory.

[0011] In one embodiment, obtaining the first degree of symmetry of the facial image based on the facial feature area trajectory includes: folding the facial feature area trajectory in half based on the central axis; obtaining a first one-way distance from the left facial feature area trajectory to the right facial feature area trajectory, and a second one-way distance from the right facial feature area trajectory to the left facial feature area trajectory based on the left facial feature area trajectory and the right facial feature area trajectory; obtaining an area enclosed by the left facial feature area trajectory and the right facial feature area trajectory based on the first one-way distance and the second one-way distance; when the area is less than a preset area threshold, obtaining the first degree of symmetry based on the ratio of the area to the preset area threshold.

[0012] In one embodiment, the symmetry degree for the facial image is obtained based on the first symmetry degree and the second symmetry degree, including: when the area is less than the preset area threshold and the second symmetry degree is zero, setting the first symmetry degree to the symmetry degree; when the area is less than the preset area threshold and the second symmetry degree is not zero, setting the second symmetry degree to the symmetry degree; when the area is greater than or equal to the preset area threshold, the symmetry degree is a preset value.

[0013] In one embodiment, there are multiple facial key point lines; obtaining the second symmetry for the facial image based on the facial key point lines includes: obtaining a baseline and obtaining multiple angles formed between each facial key point line and the baseline; obtaining an angle difference between the angles based on the angles; and obtaining the second symmetry based on the angle difference.

[0014] In one embodiment, obtaining first facial key points for forming a preset facial feature area trajectory and second facial key points for forming a preset facial key point connection line from the facial image includes: obtaining a sample facial image; marking the facial key points of the sample facial image to obtain a marked image of the sample facial image; inputting the sample facial image into a facial key point recognition model to be trained, and using the marked image to train the facial key point recognition model to be trained to obtain the trained facial key point recognition model; inputting the facial image to be evaluated into the trained facial key point recognition model to obtain the first facial key points and the second facial key points.

[0015] In one embodiment, obtaining the facial image to be evaluated includes: obtaining the facial image to be evaluated in response to a use request for a smart device; after obtaining the symmetry of the facial image, further including: obtaining an evaluation result for enabling a corresponding function of the smart device based on the symmetry of the facial image; and enabling the corresponding function of the smart device based on the evaluation result.

[0016] In a second aspect, the present application further provides a facial symmetry assessment device. The device comprises:

[0017] A facial key point acquisition module is used to acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory, and second facial key points for forming a line connecting the preset facial key points;

[0018] a first symmetry acquisition module, configured to obtain a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtain a first symmetry degree for the facial image based on the facial feature region trajectory;

[0019] a second symmetry acquisition module, configured to obtain a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtain a second symmetry for the facial image based on the line connecting facial key points;

[0020] The facial image symmetry acquisition module is configured to obtain the symmetry of the facial image according to the first symmetry and the second symmetry.

[0021] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0022] Acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory and second facial key points for forming a line connecting the preset facial key points;

[0023] Obtaining a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtaining a first degree of symmetry for the facial image based on the facial feature region trajectory;

[0024] Obtaining a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtaining a second degree of symmetry for the facial image based on the line connecting facial key points;

[0025] A symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0027] Acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory and second facial key points for forming a line connecting the preset facial key points;

[0028] Obtaining a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtaining a first degree of symmetry for the facial image based on the facial feature region trajectory;

[0029] Obtaining a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtaining a second degree of symmetry for the facial image based on the line connecting facial key points;

[0030] A symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

[0031] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0032] Acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory and second facial key points for forming a line connecting the preset facial key points;

[0033] Obtaining a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtaining a first degree of symmetry for the facial image based on the facial feature region trajectory;

[0034] Obtaining a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtaining a second degree of symmetry for the facial image based on the line connecting facial key points;

[0035] A symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

[0036] The above-mentioned facial symmetry assessment method, apparatus, computer device, storage medium, and computer program product obtain a facial image to be assessed, obtain first facial key points for forming a preset facial feature region trajectory, and second facial key points for forming a preset facial key point connection line from the facial image; obtain the facial feature region trajectory corresponding to the facial image based on the first facial key points, and obtain a first symmetry for the facial image based on the facial feature region trajectory; obtain the facial key point connection line corresponding to the facial image based on the second facial key points, and obtain a second symmetry for the facial image based on the facial key point connection line; and obtain the symmetry for the facial image based on the first symmetry and the second symmetry. By using the first symmetry based on the facial feature region trajectory and the second symmetry based on the facial key point connection line, the present application provides a more comprehensive assessment of facial symmetry, thereby improving the accuracy of facial symmetry assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic flow chart of a facial symmetry assessment method according to one embodiment;

[0038] Figure 2 1 is a schematic diagram of a process for obtaining a first degree of symmetry of a facial image in one embodiment;

[0039] Figure 3 1 is a schematic diagram of a process for obtaining the symmetry of a facial image in one embodiment;

[0040] Figure 4 1 is a schematic diagram of a process for obtaining a second degree of symmetry of a facial image in one embodiment;

[0041] Figure 5 is a structural block diagram of a facial symmetry assessment device in one embodiment;

[0042] Figure 6 is a diagram of the internal structure of a computer device in one embodiment;

[0043] Figure 7 is a schematic diagram of an asymmetric convolution module in one embodiment;

[0044] Figure 8 A schematic diagram of a process for establishing a personalized cognitive profile in one embodiment;

[0045] Figure 9 A schematic diagram of a flow chart for generating an auxiliary verification method in an embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] It should be noted that the terms "first" and "second" as used in the embodiments of the present invention are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the terms "first" and "second" may interchangeably represent a specific order or precedence, where permitted. It should be understood that the objects distinguished by "first" and "second" may be interchangeable, where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0048] In one embodiment, Figure 1 As shown, a method for assessing facial symmetry is provided. This embodiment uses the method applied to a terminal as an example. It is understood that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0049] Step S110 , obtaining a facial image to be evaluated, and obtaining from the facial image first facial key points for forming a preset facial feature region trajectory, and second facial key points for forming a line connecting the preset facial key points.

[0050] Among them, the facial image to be evaluated is a facial image for evaluating facial symmetry; the preset facial feature area is a pre-set facial area, such as: eyebrow area, mouth area; the preset facial feature area trajectory is a curved trajectory connected by key points in the facial feature area; the first facial key point is a key point in the facial feature area, used to form the preset facial feature area trajectory; the second facial key point is a preset facial feature point, such as: left and right inner corners of the eye, left and right outer corners of the eye, left and right nose wings, left and right corners of the mouth, left and right eyebrows, and left and right eyebrow tips; the preset facial key point connection line is a straight line formed by connecting the two corresponding second facial key points on the left and right.

[0051] Specifically, a facial image to be evaluated is acquired through a camera of the smart device, and then a first facial key point and a second facial key point are identified from the facial image.

[0052] Step S120 : obtaining a facial feature region trajectory corresponding to the facial image according to the first facial key point, and obtaining a first degree of symmetry for the facial image based on the facial feature region trajectory.

[0053] The first symmetry is obtained based on the first facial key point and is used to represent the symmetry of the facial image.

[0054] Specifically, the first facial key points are connected to obtain a facial feature area trajectory corresponding to the facial image, the facial feature area trajectory is folded in half, and the first symmetry of the facial image is obtained by the size of the area enclosed by the left facial feature area trajectory and the right facial feature area trajectory.

[0055] Step S130 : Obtaining facial key point lines corresponding to the facial image according to the second facial key points, and obtaining a second degree of symmetry for the facial image based on the facial key point lines.

[0056] The second symmetry is obtained based on the second facial key point and is used to represent the symmetry of the facial image.

[0057] Specifically, the two second facial key points corresponding to the left and right features are connected to obtain a facial key point line corresponding to the facial image. Based on the facial key point line, a second degree of symmetry for the facial image is obtained. For example, the second degree of symmetry for the facial image can be obtained by comparing the angles between the facial key point lines, or by the degree of overlap of the facial key point lines after folding them in half, without limitation.

[0058] Step S140: Obtaining a symmetry degree for the facial image according to the first symmetry degree and the second symmetry degree.

[0059] The symmetry of the facial image is used to evaluate the symmetry of the facial image.

[0060] Specifically, the symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

[0061] The above-mentioned facial symmetry assessment method, apparatus, computer device, storage medium, and computer program product obtain a facial image to be assessed, obtain first facial key points for forming a preset facial feature region trajectory, and second facial key points for forming a preset facial key point connection line from the facial image; obtain the facial feature region trajectory corresponding to the facial image based on the first facial key points, and obtain a first symmetry for the facial image based on the facial feature region trajectory; obtain the facial key point connection line corresponding to the facial image based on the second facial key points, and obtain a second symmetry for the facial image based on the facial key point connection line; and obtain the symmetry for the facial image based on the first symmetry and the second symmetry. By using the first symmetry based on the facial feature region trajectory and the second symmetry based on the facial key point connection line, the present application provides a more comprehensive assessment of facial symmetry, thereby improving the accuracy of facial symmetry assessment.

[0062] In one embodiment, the facial feature region trajectory includes: a left facial feature region trajectory and a right facial feature region trajectory; obtaining the facial feature region trajectory corresponding to the facial image according to the first facial key point includes the following steps:

[0063] Obtain the midpoint of the line connecting the facial key points; based on the midpoint, obtain the central axis of the line connecting the facial key points; based on the central axis, obtain the trajectory of the left facial feature area and the trajectory of the right facial feature area.

[0064] The line connecting the facial key points may be a line connecting the left and right outer canthus key points, the midpoint being the midpoint of the line connecting the left and right outer canthus key points; the central axis of the line connecting the left and right outer canthus key points being the median perpendicular line connecting the left and right outer canthus key points;

[0065] Specifically, a central axis is established vertically with the middle point of the line connecting the key points of the left and right inner corners of the eyes, and the trajectory of the left face feature area and the trajectory of the right face feature area are obtained.

[0066] In this embodiment, by selecting the left and right inner corner key points that are relatively symmetrical on the face and obtaining the central axis of the line connecting the corresponding key points, the left face feature area trajectory and the right face feature area trajectory are obtained more accurately.

[0067] In one embodiment, Figure 2 As shown, obtaining the first symmetry of the facial image based on the facial feature area trajectory includes the following steps:

[0068] Step S210: fold the facial feature region trajectory in half based on the central axis.

[0069] Specifically, the facial feature region trajectory is folded in half along the central axis so that the left facial region trajectory fits together with the right facial feature region trajectory.

[0070] Step S220: Based on the left face area trajectory and the right face feature area trajectory, obtain a first one-way distance from the left face feature area trajectory to the right face feature area trajectory, and a second one-way distance from the right face feature area trajectory to the left face feature area trajectory.

[0071] The first one-way distance and the second one-way distance are used to calculate the area enclosed by the left face feature region trajectory and the right face feature region trajectory.

[0072] Specifically, based on the following formula:

[0073]

[0074]

[0075] in, is the first one-way distance, For the left face trajectory, For the right face trajectory, represents the length of the left face trajectory, express Point p on distance; is the second one-way distance, represents the length of the right face trajectory, express Point p on Based on the above formula, the first one-way distance and the second one-way distance are obtained.

[0076] Step S230 : Obtaining the area enclosed by the left face feature region trajectory and the right face feature region trajectory based on the first one-way distance and the second one-way distance.

[0077] The area enclosed by the left facial feature region trajectory and the right facial feature region trajectory is the area enclosed by the curves of the left and right facial feature region trajectories after the facial feature region trajectory is folded in half.

[0078] Specifically, based on the following formula:

[0079]

[0080] in is the area enclosed by the above-mentioned left and right facial feature area trajectories, is the first one-way distance, is the second one-way distance. According to the first one-way distance and the second one-way distance, the area enclosed by the left face feature region trajectory and the right face feature region trajectory can be obtained.

[0081] Step S240: When the area is smaller than the preset area threshold, a first degree of symmetry is obtained based on a ratio of the area to the preset area threshold.

[0082] The preset area threshold is used to compare with the area enclosed by the trajectories of the left and right facial feature regions; and the ratio of the area to the preset area threshold is used to obtain a first degree of symmetry.

[0083] Specifically, based on the following formula:

[0084]

[0085] in, is the first degree of symmetry, is the preset area threshold, is the second symmetry. When the area is smaller than the preset area threshold and the second symmetry is zero, the first symmetry is obtained based on the ratio of the area to the preset area threshold.

[0086] In this embodiment, the area enclosed by the two trajectories is calculated by using a segmented method for calculating the one-way distance. The larger the area, the greater the distance between the two trajectories, and the more asymmetrical the face. Then, based on a preset area threshold, the first degree of symmetry is accurately obtained.

[0087] In one embodiment, Figure 3 As shown, step S140 includes the following steps:

[0088] Step S310, when the area is less than the preset area threshold and the second symmetry is zero, the first symmetry is set to the symmetry; Step S320, when the area is less than the preset area threshold and the second symmetry is not zero, the second symmetry is set to the symmetry; Step S330, when the area is greater than or equal to the preset area threshold, the symmetry is the preset value.

[0089] The preset value is the set value of the symmetry, which means that when the area enclosed by the left and right tracks is greater than or equal to the preset area threshold, the set value of the symmetry can be zero.

[0090] Specifically, based on the following formula:

[0091]

[0092] in is the symmetry of the facial image, is the first degree of symmetry, is the second symmetry; when the area is less than the preset area threshold and the second symmetry is zero, the first symmetry is set to the symmetry; when the area is less than the preset area threshold and the second symmetry is not zero, the second symmetry is set to the symmetry; when the area is greater than or equal to the preset area threshold, the symmetry is zero.

[0093] In this embodiment, different symmetry representation methods are used to represent the symmetry of the facial image in different situations, so that the symmetry of the facial image can be obtained more accurately.

[0094] In one embodiment, Figure 4 As shown, there are multiple facial key point connection lines; based on the facial key point connection lines, obtaining the second symmetry for the facial image includes the following steps:

[0095] Step S410: Acquire a baseline and obtain multiple angles formed between the lines connecting each facial key point and the baseline.

[0096] The baseline is a preset horizontal straight line; the angles are multiple angles formed between the lines connecting each facial key point and the baseline.

[0097] Specifically, a baseline is obtained, and multiple angles formed between the lines connecting each facial key point and the baseline are calculated.

[0098] Step S420: Obtain an angle difference of the included angle according to the included angle; and obtain the second degree of symmetry based on the angle difference.

[0099] The angle difference is the difference between one of the included angles and the remaining included angles.

[0100] Specifically, based on the following formula:

[0101]

[0102] in, The weighted angle difference ratio, also known as the second degree of symmetry, is calculated by calculating the straight line formed by the left and right inner corners of the eyes. , the straight line formed by the left and right outer corners of the eyes , the straight line formed by the left and right nose wings , the straight line formed by the left and right corners of the mouth , the straight line formed by the left and right eyebrows and the straight line formed by the left and right eyebrows The angle between each and the horizontal line , for Calculate the angle between it and the horizontal line. The ratio of the weighted angle difference with the other five angles is the second degree of symmetry.

[0103] In this embodiment, the second symmetry degree for the facial image is accurately obtained based on the above formula by calculating the difference between one included angle and the other included angles.

[0104] In one embodiment, step S110 includes the following steps:

[0105] A sample facial image is obtained; facial key points of the sample facial image are marked to obtain a marked image of the sample facial image; the sample facial image is input into a facial key point recognition model to be trained, and the facial key point recognition model to be trained is trained using the marked image to obtain a trained facial key point recognition model; the facial image to be evaluated is input into the trained facial key point recognition model to obtain a first facial key point and a second facial key point.

[0106] The sample facial image is used to input the facial key point recognition model to be trained; the marked image is the image after the facial key points of the sample facial image are marked, which is used to train the facial key point recognition model; the facial key point recognition model is used to identify the key points of the facial image;

[0107] Specifically, facial images of this set are sampled through a camera device; all sample facial images are processed into the same size, key points of the samples are labeled, and saved in the form of horizontal and vertical coordinates; the sample facial images are input into a facial key point recognition model to be trained, and the labeled images are used to train the facial key point recognition model to obtain a trained facial key point recognition model; the facial image to be evaluated is input into the trained facial key point recognition model to identify the first facial key point and the second facial key point.

[0108] In this embodiment, the first facial key point and the second facial key point are accurately identified by using a facial key point recognition model trained based on a large number of sample facial images.

[0109] In one embodiment, obtaining a facial image to be evaluated includes the following steps:

[0110] In response to a request for use of a smart device, a facial image to be evaluated is obtained; after obtaining the symmetry of the facial image, an evaluation result for enabling a corresponding function of the smart device is obtained based on the symmetry of the facial image; and based on the evaluation result, the corresponding function of the smart device is enabled.

[0111] Among them, smart devices can be but are not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices; Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc.; portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.; the usage request is a request to enable the corresponding function; the corresponding function of the smart device is a function that requires an evaluation result to be enabled; the evaluation result is an evaluation result obtained through a comprehensive evaluation, and the comprehensive evaluation includes facial symmetry evaluation.

[0112] Specifically, in response to a request for use of a smart device, a facial image to be evaluated is obtained, a facial symmetry evaluation is performed on the facial image to be evaluated, a facial symmetry evaluation result is obtained, and then a comprehensive evaluation result is obtained. Based on the comprehensive evaluation result, it is determined whether to enable the corresponding function of the smart device.

[0113] In this embodiment, by performing facial symmetry evaluation on the facial image to be evaluated, a facial symmetry evaluation result is obtained, and then a comprehensive evaluation result is obtained. Based on the comprehensive evaluation result, it can be more accurately determined whether to enable the corresponding function of the smart device.

[0114] In one embodiment, the present invention proposes an auxiliary verification method for automatically identifying the reaction ability of the elderly.

[0115] 1. Establishing a Responsiveness Assessment Model

[0116] This model can assess the behavioral and cognitive abilities of older adults. Based on the assessment results, it creates a personalized profile for each individual and determines their ability to continue conducting business. Because cognitive abilities don't vary significantly over short periods of time, the initial assessment can be repeated based on the personalized profile. Technically, this invention can be implemented using the OpenCV open-source library.

[0117] 1. Neurological examination

[0118] Neurological examinations primarily assess the behavioral abilities of older adults. This study uses computer image recognition and processing technology to extract features of facial expressions and body movements, calculate facial symmetry and limb limitations, and define thresholds for abnormal expressions and limbs, which serve as the basis for subsequent comprehensive scoring.

[0119] Step (1) Recognize facial expressions: The computer prompts the user to smile and extracts facial features such as mouth, eyes, nose, and eyebrows. After calibrating and aligning the facial image, it uses layer convolution to identify whether the user's expression is a smile.

[0120] Step (2): Ask the user to memorize three words in advance as the basis for calculating the memory in step (11). The computer randomly plays three words and asks the user to memorize them.

[0121] Step (3) Calculate the facial constraint F: Facial constraint is expressed based on facial symmetry. In this application, facial constraint is facial symmetry, and the value of F is in the range of [0,10]. If the expression recognized in step (1) is not a smile, it means that the user has not correctly understood the machine instruction and there is a cognitive barrier, and F is assigned a value of 0. If the expression recognized in step (1) is a smile, it means that the user has correctly understood the meaning of the instruction, and steps (3.1)-(3.2) are executed to further calculate the degree of facial symmetry.

[0122] Step (3.1) Facial image processing: Detect key points in the face using step (1), including 12 key points: the left and right inner corners of the eyes, the left and right outer corners of the eyes, the left and right nose wings, the left and right corners of the mouth, the left and right eyebrows, and the left and right eyebrows. In order to strengthen network spatial learning and channel learning, the present invention adds an asymmetric convolution structure based on the residual network ResNet structure. At the same time, in order to accurately fit the face in a complex environment, the network model is deepened to 101 layers. When only one network is used in a single stage, the false detection rate is reduced while also improving the robustness of the algorithm. The specific steps are as follows:

[0123] Step 3.1.1: Sample processing. Collect training samples, process all samples to the same size, annotate the samples with 12 key points, and save them in the form of horizontal and vertical coordinates;

[0124] Step 3.1.2: Model construction. The training samples are used as the input of the model network; the model is mainly composed of multiple residual modules, each residual module contains three convolutional layers and three activation layers. In order to strengthen network space learning and channel learning, the present invention adds an asymmetric convolution structure to each convolution layer. The feature map calculated by the three activation layers and the input feature map are superimposed through the residual layer; the positional relationship between the weights in the convolution kernel and the weight space is used to strengthen the ordinary convolution layer, and the robustness of the model is improved by adding horizontal and vertical convolution operations, where the convolution operation The size of the convolution kernel is , the number of input channels is 64.

[0125] Single path Convolution is split into 、 and The three-way convolution is combined by linear addition of skeleton weights , strengthen the features extracted by the convolution layer while keeping the original network structure unchanged, the asymmetric convolution module such as Figure 7 shown.

[0126] Step 3.1.3: Output key point prediction coordinates. Based on the prediction model trained in step 3.1.2, predict 12 key points of the face information captured by the camera and output the corresponding 12 key point coordinates.

[0127] Step 3.1.4: Using the 12 key points predicted in step 3.1.3 as input, calculate the relative symmetry of the left and right faces. Specific steps: Calculate the straight lines formed by the left and right inner corners of the eyes respectively. , the straight line formed by the left and right outer corners of the eyes , the straight line formed by the left and right nose wings , the straight line formed by the left and right corners of the mouth , the straight line formed by the left and right eyebrows and the straight line formed by the left and right eyebrows The angles of each with the horizontal ,calculate Weighted angle difference ratio with the other 5 angles , which is the first degree of symmetry.

[0128] Preferably, a central axis is established vertically with the middle point of the left and right outer corners of the eyes to extract the mouth and eyebrow trajectories on both sides of the central axis, and the overlap degree of the two trajectories after being flipped along the central axis is compared.

[0129] Step (3.2) F and the degree of overlap, i.e., the mapping function of the second degree of facial symmetry: the degree of overlap is the degree of similarity between the two left and right trajectories after flipping. The present invention adopts a method based on segmented calculation of one-way distance to calculate the area enclosed by the two trajectories. The larger the area, the greater the distance between the trajectories, and the more asymmetrical the expression.

[0130] Step (3.2.1) Definition of one-way distance OWD:

[0131]

[0132] in, For the left face trajectory, For the right face trajectory. represents the length of the left face trajectory, express Point p on For symmetry, modify the above formula and get the final formula as follows:

[0133]

[0134] Step (3.2.2) defines the area threshold f. When the area enclosed by the left and right trajectories is greater than or equal to f, F=0; when When it is less than f:

[0135]

[0136] Step (4): Define the expression abnormality threshold a,

[0137] Step (5) Identify body movements: The computer prompts the user to make the "left hand touches the right ear, right hand touches the left ear" movement, extracts the arm, hand, and ear features, calibrates and aligns the images, and then identifies whether the user's movements are accurate.

[0138] Step (6) Process the limb image and calculate the limb limitation Z: the value range of Z is [0,1]. Take the key points detected in step (5): arm, hand and ear as input. If there is no intersection or more than one intersection in the arm trajectory, it means that the user has not completed the specified action, and Z is assigned a value of 0. If there is one intersection in the arm trajectory, determine the distance D between the ear and the finger. The larger the distance, the more restricted it is. Define the distance threshold z. When D exceeds z, Z=0.

[0139] When D is less than z,

[0140] Step (7): Define the limb abnormality threshold b,

[0141] 2. Mental status examination

[0142] The mental status examination is mainly assessed by whether the voice answers are correct answers, and the scores of the elderly in time orientation, spatial orientation, judgment and memory are calculated.

[0143] Step (8) Calculate the time orientation score T: The computer asks the user "What year and season is it now?", if the user answers correctly T=10, if the user answers incorrectly T=0.

[0144] Step (9) Calculate the spatial orientation score S: The computer asks the user "What province and city are you in?" If the user answers correctly, S=10; if the user answers incorrectly, S=0.

[0145] Step (10) Calculate the judgment score D: The computer asks the user "Which is heavier, a pound of cotton or a pound of iron?" If the user answers correctly, D=10, and if the user answers incorrectly, D=0.

[0146] Step (11) Calculate the memory score M: The computer asks the user what the three words played by the computer in step (2) are. If the user answers 0 words, M=0; if the user answers 1 word, M=5; if the user answers 2 words, M=8; if the user answers all 3 words correctly, M=10.

[0147] 2. Comprehensive Assessment and Establishment of Personalized Cognitive Profiles

[0148] Based on the above-mentioned reaction ability model, the present invention scores in four aspects: behavioral ability, memory, orientation, and judgment. The comprehensive evaluation score divides users into normal, mildly impaired, moderately impaired, and severely impaired, and establishes personalized cognitive profiles for the elderly. Users with mild disabilities can reduce the difficulty or increase the time of auxiliary verification based on their weaknesses in a certain ability. Users with moderate disabilities will be prompted to require assistance from others. Users with severe disabilities no longer have independent awareness and are not suitable for handling business, so the transaction will be directly rejected and a warning signal will be issued.

[0149] Step (13) Calculate the comprehensive score A: Medical research has found that the time orientation, spatial orientation, and memory scores of Alzheimer's patients are significantly lower than those of patients with other types of dementia. Therefore, the comprehensive score A amplifies the proportion of these three items.

[0150]

[0151] Step (14) defines the normal, mild disorder, moderate disorder and severe disorder intervals: the comprehensive score A in the range of [55,60] is normal, in the range of [50,55) is mild disorder, in the range of [40,50) is moderate disorder, and below 40 points is severe disorder.

[0152] Step (15) Create a personalized cognitive profile: The cognitive profile is stored in vector form as follows:

[0153]

[0154] The values in the vector are the individual's comprehensive score and the scores of various response abilities, which serve as the input for subsequent adaptive auxiliary verification. For the overall process established, please refer to Figure 8 .

[0155] III. Adaptive Auxiliary Verification

[0156] According to medical diagnosis indicators, the facial limitation F of behavioral ability represents signs of stroke, the limb limitation Z represents signs of Parkinson's disease, and the orientation and memory of mental state represent signs of Alzheimer's disease.

[0157] Steps (16) to (18) are all generated on the premise that A is in the normal and mild impairment range, indicating that there are pathological signs for a specific item, and an adaptive auxiliary method needs to be generated.

[0158] Auxiliary method for stroke signs in step (16): F = 0 means the user cannot complete verification or operation through facial dynamic changes. 0 < F < 10 means the user has a certain degree of facial symmetry deviation, and the system automatically adjusts according to the deviation degree.

[0159] Auxiliary method for Parkinson's disease signs in step (17): Z = 0 means the user cannot complete verification or operation through limb movements, gestures, etc. 0 < Z < 10 means the user has a certain degree of movement limitation, and the system automatically adjusts to voice recognition or other methods.

[0160] Auxiliary method for Alzheimer's disease signs in step (18):

[0161] Step (18.1): T = 0 indicates that the user has a disorder in time orientation. The system increases the verification duration and highlights transaction information related to dates and times.

[0162] Step (18.2): S = 0 indicates that the user has a disorder in spatial orientation. The system increases the verification duration and highlights transaction information related to transaction locations and positions.

[0163] Step (18.3): D = 0 indicates that the user has a disorder in judgment. The system increases the verification duration and highlights important information (amount, counterparty information, etc.).

[0164] Step (18.4): M = 0 indicates that the user has a serious memory defect. The system issues a warning and prompts for manual processing.

[0165] Step (18.5): M < 10 indicates that the user has a certain memory disorder. The transaction process needs to be simplified, the number of jump steps reduced, and the operation should be completed within one page as much as possible.

[0166] Step (19): A is in the moderate impairment range, indicating that the user shows multiple comprehensive case signs. The system issues a warning and prompts for manual processing.

[0167] Step (20): A is in the severe obstacle range, indicating that the user has a serious cognitive impairment and is not capable of handling business. The transaction is directly rejected and an early warning is issued.

[0168] The overall process of generating adaptive auxiliary methods is detailed in Figure 9 . It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be executed in other orders. Moreover, at least part of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0169] Based on the same inventive concept, embodiments of the present application also provide a facial symmetry assessment device for implementing the aforementioned facial symmetry assessment method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more facial symmetry assessment device embodiments provided herein can be found in the limitations of the facial symmetry assessment method described above and will not be further elaborated here.

[0170] In one embodiment, Figure 5 As shown, a facial symmetry evaluation device 500 is provided, comprising: a facial key point acquisition module 501, a first symmetry acquisition module 502, a second symmetry acquisition module 503 and a facial image symmetry acquisition module 504, wherein:

[0171] A facial key point acquisition module 501 is configured to acquire a facial image to be evaluated, and to acquire, from the facial image, first facial key points for forming a predetermined facial feature region trajectory, and second facial key points for forming a predetermined facial key point connection line;

[0172] A first symmetry acquisition module 502 is configured to obtain a facial feature region trajectory corresponding to the facial image based on the first facial key points, and obtain a first symmetry degree for the facial image based on the facial feature region trajectory;

[0173] A second symmetry acquisition module 503 is configured to obtain a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtain a second symmetry for the facial image based on the line connecting facial key points;

[0174] The facial image symmetry acquisition module 504 is configured to obtain the symmetry of the facial image according to the first symmetry and the second symmetry.

[0175] In one embodiment, the first symmetry acquisition module 502 is further used to obtain the midpoint of the line connecting the facial key points; based on the midpoint, obtain the central axis of the line connecting the facial key points; based on the central axis, obtain the left face feature area trajectory and the right face feature area trajectory; wherein the facial feature area trajectory includes: the left face feature area trajectory and the right face feature area trajectory.

[0176] In one embodiment, the first symmetry acquisition module 502 is further used to fold the facial feature area trajectory in half based on the central axis; based on the left facial area trajectory and the right facial feature area trajectory, obtain a first one-way distance from the left facial feature area trajectory to the right facial feature area trajectory, and a second one-way distance from the right facial feature area trajectory to the left facial feature area trajectory; based on the first one-way distance and the second one-way distance, obtain the area enclosed by the left facial feature area trajectory and the right facial feature area trajectory; when the area is less than a preset area threshold, obtain the first symmetry based on the ratio of the area to the preset area threshold.

[0177] In one embodiment, the facial image symmetry acquisition module 504 is further configured to set the first symmetry as the symmetry when the area is less than a preset area threshold and the second symmetry is zero; set the second symmetry as the symmetry when the area is less than the preset area threshold and the second symmetry is not zero; and set the symmetry to the preset value when the area is greater than or equal to the preset area threshold.

[0178] In one embodiment, the second symmetry acquisition module 503 is further used to obtain a baseline and obtain multiple angles formed between the lines connecting each facial key point and the baseline; obtain an angle difference of the angles according to the angles; and obtain the second symmetry based on the angle difference.

[0179] In one embodiment, the facial key point acquisition module 501 is further used to obtain a sample facial image; mark the facial key points of the sample facial image to obtain a marked image of the sample facial image; input the sample facial image into a facial key point recognition model to be trained, and use the marked image to train the facial key point recognition model to be trained to obtain a trained facial key point recognition model; input the facial image to be evaluated into the trained facial key point recognition model to obtain a first facial key point and a second facial key point.

[0180] In one embodiment, the facial image symmetry acquisition module 504 is further used to obtain a facial image to be evaluated in response to a usage request for a smart device; after obtaining the symmetry of the facial image, it is also used to: obtain an evaluation result for enabling a corresponding function of the smart device based on the symmetry of the facial image; and enable the corresponding function of the smart device based on the evaluation result.

[0181] Each module in the facial symmetry assessment device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0182] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a method for assessing facial symmetry. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, or keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0183] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0184] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0185] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0186] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0188] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0189] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A facial symmetry assessment method, characterized in that: The method comprises: Acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory and second facial key points for forming a line connecting the preset facial key points; Obtaining, based on the first facial key points, a facial feature region trajectory corresponding to the facial image, and obtaining, based on the facial feature region trajectory, an area enclosed by the facial feature region trajectory; and when the area is less than a preset area threshold, obtaining a first degree of symmetry for the facial image based on a ratio of the area to the preset area threshold; Obtaining a line connecting the facial key points corresponding to the facial image based on the second facial key points, obtaining a baseline, and obtaining a second degree of symmetry for the facial image based on a difference between an angle between the line connecting the facial key points and the baseline; A symmetry degree for the facial image is obtained according to the first symmetry degree and the second symmetry degree.

2. The method according to claim 1, characterized in that The facial feature area trajectory includes: a left facial feature area trajectory and a right facial feature area trajectory; The step of obtaining a facial feature area trajectory corresponding to the facial image based on the first facial key points includes: Obtaining the midpoint of the line connecting the facial key points; Based on the midpoint, obtaining the central axis of the line connecting the facial key points; Based on the central axis, the left face feature area trajectory and the right face feature area trajectory are obtained.

3. The method according to claim 2, characterized in that The step of obtaining an area enclosed by the facial feature region trajectory based on the facial feature region trajectory, and obtaining a first degree of symmetry of the facial image based on a ratio of the area to the preset area threshold when the area is smaller than a preset area threshold, includes: Based on the central axis, folding the facial feature area trajectory in half; Based on the left face feature area trajectory and the right face feature area trajectory, obtaining a first one-way distance from the left face feature area trajectory to the right face feature area trajectory, and a second one-way distance from the right face feature area trajectory to the left face feature area trajectory; Obtaining an area enclosed by the left face feature region trajectory and the right face feature region trajectory based on the first one-way distance and the second one-way distance; When the area is smaller than a preset area threshold, the first degree of symmetry is obtained based on a ratio of the area to the preset area threshold.

4. The method according to claim 3, characterized in that Obtaining a symmetry degree for the facial image based on the first symmetry degree and the second symmetry degree includes: When the area is smaller than the preset area threshold and the second symmetry is zero, setting the first symmetry as the symmetry; When the area is smaller than the preset area threshold and the second symmetry is not zero, setting the second symmetry as the symmetry; When the area is greater than or equal to the preset area threshold, the degree of symmetry is a preset value.

5. The method according to claim 1, wherein There are multiple facial key point connection lines; The obtaining of the reference line, and obtaining a second degree of symmetry for the facial image based on a difference between an angle between a line connecting the facial key points and the reference line, includes: Obtaining a baseline, which is a horizontal straight line, and obtaining a plurality of angles formed between lines connecting each facial key point and the baseline; According to the included angle, an angle difference of the included angle is obtained; based on the angle difference, the second degree of symmetry is obtained.

6. The method according to claim 1, characterized in that The step of acquiring, from the facial image, first facial key points for forming a preset facial feature area trajectory and second facial key points for forming a line connecting the preset facial key points comprises: Get a sample face image; Marking facial key points of the sample facial image to obtain a marked image of the sample facial image; Inputting the sample facial image into a facial key point recognition model to be trained, and training the facial key point recognition model to be trained using the labeled image to obtain a trained facial key point recognition model; The facial image to be evaluated is input into the trained facial key point recognition model to obtain the first facial key points and the second facial key points.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of a facial image to be evaluated comprises: Responding to a usage request for the smart device, obtaining a facial image to be evaluated; After obtaining the degree of symmetry of the facial image, the method further includes: Obtaining, based on the symmetry of the facial image, an evaluation result for enabling a corresponding function of the smart device; Based on the evaluation result, the corresponding function of the smart device is enabled.

8. A facial symmetry assessment device, characterized in that: The device comprises: A facial key point acquisition module is used to acquire a facial image to be evaluated, and acquire, from the facial image, first facial key points for forming a preset facial feature region trajectory, and second facial key points for forming a line connecting the preset facial key points; a first symmetry acquisition module, configured to obtain a facial feature region trajectory corresponding to the facial image according to the first facial key points, and obtain a first symmetry degree for the facial image based on the facial feature region trajectory; a second symmetry acquisition module, configured to obtain a line connecting facial key points corresponding to the facial image based on the second facial key points, and obtain a second symmetry for the facial image based on the line connecting facial key points; The facial image symmetry acquisition module is configured to obtain the symmetry of the facial image according to the first symmetry and the second symmetry.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images