Facial expression recognition optimization method and device based on deep learning and electronic equipment

Abstract

The invention provides a facial expression recognition optimization method and device based on deep learning and electronic equipment. The method comprises the following steps: acquiring facial expression data, and constructing a facial three-dimensional model by adopting a 3D facial reconstruction technology according to the facial expression data; extracting facial features from the facial three-dimensional model by adopting an SIFT algorithm, determining expression tags corresponding to the facial features, constructing a deep learning model, and training the deep learning model by adopting the facial features and the expression tags as training samples to obtain a facial expression recognition model; and at least adopting the facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result. By integrating 3D face reconstruction and SIFT algorithm technologies, the accuracy, robustness, fine expression recognition capability, dynamic expression processing capability and anomaly detection and processing capability of facial expression recognition are improved. The problem that an existing facial feature extraction algorithm is limited in fine expression recognition capacity is solved.

Description

Technical Field

[0001] The present application relates to the technical field of facial expression recognition optimization, and in particular to a facial expression recognition optimization method, device, computer-readable storage medium and electronic device based on deep learning. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of deep learning in the field of image recognition has become increasingly mature, especially in facial expression recognition technology. In recent years, through continuous technological innovation, researchers have achieved facial expression recognition from single modality to multimodal data fusion, significantly improving the accuracy and robustness of recognition. Traditional facial expression recognition methods mainly rely on two-dimensional image analysis. Although some results have been achieved, they still have obvious limitations in dealing with complex lighting conditions, facial occlusion, and subtle expression recognition.

[0003] However, the existing related technologies still have shortcomings in the following aspects: first, traditional preprocessing methods often ignore the depth information of facial data, resulting in poor performance of recognition models in complex environments; second, existing facial feature extraction algorithms have limited recognition capabilities for subtle expressions and have difficulty capturing subtle changes in expressions; third, the training process of deep learning models often requires a lot of manual parameter adjustment and lacks effective adaptive learning rate adjustment strategies; finally, the anomaly detection and processing strategies for facial expression recognition results are still immature, making it difficult to achieve efficient and stable recognition effects in practical applications. Summary of the invention

[0004] The main purpose of the present application is to provide a facial expression recognition optimization method, device, computer-readable storage medium and electronic device based on deep learning, so as to at least solve the problem that the existing facial feature extraction algorithm has limited recognition ability for subtle expressions.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a facial expression recognition optimization method based on deep learning is provided, including: obtaining facial expression data, and constructing a facial three-dimensional model according to the facial expression data using 3D facial reconstruction technology; extracting facial features from the facial three-dimensional model using a SIFT algorithm, and determining expression labels corresponding to the facial features, constructing a deep learning model, and using the facial features and the expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; and at least using the facial expression recognition model to recognize the facial expression to be recognized to generate a recognition result.

[0006] Optionally, acquiring facial expression data includes: acquiring image data of facial expressions, expression labels, facial action units, biometric data, environmental information, audio data, physiological data and metadata; wherein the image data includes static photos and continuous video frames, the expression labels include labeling each facial image and the video frame with an expression category, the biometric data includes facial key points and head posture, the facial key points include position information of eyes, nose, mouth and eyebrows, the head posture includes posture information of head tilt and rotation, the environmental information includes lighting conditions and background, the audio data includes tone, volume and speaking speed, the physiological data includes heart rate, skin conductance and respiratory rate, and the metadata includes time, place and device type of data collection.

[0007] Optionally, at least the facial expression recognition model is used to recognize the facial expression to be recognized and generate a recognition result, including: using a continuous-time Markov model to process the dynamic changes of the facial expression to generate a dynamic change sequence, and using a micro-expression magnification recognition method to magnify the facial expression to generate an amplification processing result; based on the dynamic change sequence and the amplification processing result, the facial expression recognition model is used to recognize the facial expression to be recognized and generate the recognition result.

[0008] Optionally, in the process of training the deep learning model using the facial features and the expression labels as training samples, the method further includes: calculating a loss function of the deep learning model, and optimizing the deep learning model using an optimization algorithm based on the loss function.

[0009] Optionally, calculating the loss function of the deep learning model, and optimizing the deep learning model using an optimization algorithm based on the loss function, includes: according to a first formula: Calculating the loss function of the deep learning model Among them, h i is the expression label of the i-th training sample, is the model prediction label of the i-th training sample, N is the number of samples of the training sample; according to the second formula: The model parameters of the deep learning model are optimized, where β j is the model parameter of the deep learning model at the jth iteration, β j-1 is the model parameter of the deep learning model at the j-1th iteration, α is the learning rate, is the partial derivative of the loss function with respect to the model parameters.

[0010] Optionally, the SIFT algorithm is used to extract facial features from the facial three-dimensional model, including: extracting coordinate information of the facial key points from the facial three-dimensional model using the SIFT algorithm, and extracting emotional features from the facial three-dimensional model; fusing the coordinate information of the facial key points with the emotional features to generate the facial features.

[0011] Optionally, after acquiring the facial expression data, the method further comprises: preprocessing the facial expression data, wherein the preprocessing comprises resizing, cropping and aligning, color adjustment, denoising and facial key point detection of the image data.

[0012] According to another aspect of the present application, a facial expression recognition optimization device based on deep learning is provided, comprising: an acquisition unit, used to acquire facial expression data, and construct a facial three-dimensional model according to the facial expression data using 3D facial reconstruction technology; a training unit, used to extract facial features from the facial three-dimensional model using a SIFT algorithm, and determine expression labels corresponding to the facial features, and construct a deep learning model, and use the facial features and the expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; and a recognition unit, used to at least use the facial expression recognition model to recognize the facial expression to be recognized and generate a recognition result.

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the facial expression recognition optimization methods based on deep learning.

[0014] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the deep learning-based facial expression recognition optimization methods.

[0015] By applying the technical solution of the present application, facial expression data is acquired, and a facial three-dimensional model is constructed based on the facial expression data using 3D facial reconstruction technology; facial features are extracted from the facial three-dimensional model using the SIFT algorithm, and expression labels corresponding to the facial features are determined, a deep learning model is constructed, and facial features and expression labels are used as training samples to train the deep learning model to obtain a facial expression recognition model; at least the facial expression recognition model is used to recognize the facial expression to be recognized and generate a recognition result. By integrating 3D facial reconstruction and SIFT algorithm technology, the accuracy, robustness, subtle expression recognition capability, dynamic expression processing capability, and anomaly detection and processing capability of facial expression recognition are improved. The problem of the limited recognition capability of existing facial feature extraction algorithms for subtle expressions is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a facial expression recognition optimization method based on deep learning provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a facial expression recognition optimization method based on deep learning provided according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a process flow of a facial expression recognition optimization method based on deep learning provided according to an embodiment of the present application is shown;

[0020] Figure 4 A structural block diagram of a facial expression recognition optimization device based on deep learning provided according to an embodiment of the present application is shown.

[0021] The above drawings include the following reference numerals:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background technology, the existing facial feature extraction algorithms have limited recognition capabilities for subtle expressions. In order to solve the problem of limited recognition capabilities of existing facial feature extraction algorithms for subtle expressions, the embodiments of the present application provide a facial expression recognition optimization method, device, computer-readable storage medium and electronic device based on deep learning.

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a facial expression recognition optimization method based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the facial expression recognition optimization method based on deep learning in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific network example may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a deep learning-based facial expression recognition optimization method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0031] Figure 2 is a flow chart of a facial expression recognition optimization method based on deep learning according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, acquiring facial expression data, and constructing a facial three-dimensional model using 3D facial reconstruction technology based on the facial expression data.

[0033] Specifically, facial expression data is collected and preprocessed, a facial three-dimensional model is constructed using 3D facial reconstruction technology, and facial expression data and voice emotion data are integrated using multimodal data fusion technology. 3D facial reconstruction technology includes capturing facial shape and structure using a three-dimensional scanner, capturing facial three-dimensional information using structured light technology, capturing facial images using multiple cameras, and reconstructing a facial three-dimensional model using triangulation.

[0034] Step S202, using the SIFT algorithm to extract facial features from the above-mentioned three-dimensional facial model, and determine the expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model.

[0035] Specifically, the SIFT algorithm is used to extract the coordinates of facial key points from the 3D facial model and the emotional features from the audio; the feature fusion includes fusing the facial key point coordinates and the emotional features to form a multimodal feature representation. The facial feature extraction using the SIFT algorithm includes the construction of a scale space, the detection of key points, the allocation of key point directions and the generation of key point descriptors; the construction of the scale space includes the construction of a Gaussian pyramid using a Gaussian kernel function.

[0036] Step S203, at least using the above facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result.

[0037] Through this embodiment, facial expression data is obtained, and a facial three-dimensional model is constructed based on the facial expression data using 3D facial reconstruction technology; facial features are extracted from the facial three-dimensional model using the SIFT algorithm, and expression labels corresponding to the facial features are determined, a deep learning model is constructed, and facial features and expression labels are used as training samples to train the deep learning model to obtain a facial expression recognition model; at least the facial expression recognition model is used to recognize the facial expression to be recognized, and a recognition result is generated. By integrating 3D facial reconstruction and SIFT algorithm technology, the accuracy, robustness, subtle expression recognition capability, dynamic expression processing capability, and abnormality detection and processing capability of facial expression recognition are improved. The problem that the existing facial feature extraction algorithm has limited recognition capability for subtle expressions is solved.

[0038] In the specific implementation process, facial expression data is obtained, including: obtaining image data of facial expressions, expression labels, facial action units, biometric data, environmental information, audio data, physiological data and metadata; wherein the above-mentioned image data includes static photos and continuous video frames, the above-mentioned expression labels include marking each facial image and the above-mentioned video frames with expression categories, the above-mentioned biometric data includes facial key points and head postures, the above-mentioned facial key points include position information of eyes, nose, mouth and eyebrows, the above-mentioned head posture includes posture information of head tilt and rotation, the above-mentioned environmental information includes lighting conditions and background, the above-mentioned audio data includes tone, volume and speaking speed, the above-mentioned physiological data includes heart rate, skin conductance and respiratory rate, and the above-mentioned metadata includes the time, place and device type of data collection.

[0039] The image data of this method includes static photos and continuous video frames; expression labeling includes marking each facial image and video frame with an expression category, such as basic emotions such as happiness, sadness, anger, surprise, fear, disgust, or more subtle expression states; biometric data includes facial key points and head posture; facial key points include the position information of eyes, nose, mouth and eyebrows, and are expressed in the form of coordinates; head posture includes posture information of head tilt and rotation, which may affect the interpretation of expressions; environmental information includes lighting conditions and background. Facial images under different lighting conditions may have different appearances, so recording lighting information is crucial for data preprocessing and model training. Although the background is usually removed or ignored when performing expression recognition, background information is sometimes used to analyze the context in which the expression occurs; audio data includes intonation, volume and speaking speed. In multimodal data fusion, audio data can provide additional emotional information; physiological data includes heart rate, skin conductance and respiratory rate; metadata includes the time, location and device type of data collection.

[0040] Specifically, at least the above-mentioned facial expression recognition model is used to recognize the facial expression to be recognized and generate a recognition result, including: using a continuous-time Markov model to process the dynamic changes of the above-mentioned facial expressions to generate a dynamic change sequence, and using a micro-expression magnification recognition method to magnify the above-mentioned facial expressions to generate a magnification processing result; based on the above-mentioned dynamic change sequence and the above-mentioned magnification processing result, the above-mentioned facial expression recognition model is used to recognize the above-mentioned facial expression to be recognized and generate the above-mentioned recognition result.

[0041] The method uses the trained facial expression recognition model to recognize facial expressions, and uses the continuous-time Markov model to process the dynamic changes of facial expressions. The micro-expression amplification recognition method is used to process subtle expressions, improve the recognition ability of subtle expression changes, and output the facial expression recognition results. The continuous-time Markov model includes defining the facial expression state transition probability matrix and predicting the future state of facial expressions.

[0042] More specifically, in the process of training the above-mentioned deep learning model using the above-mentioned facial features and the above-mentioned expression labels as training samples, the above-mentioned method also includes: calculating the loss function of the above-mentioned deep learning model, and optimizing the above-mentioned deep learning model using an optimization algorithm based on the above-mentioned loss function.

[0043] This method uses loss function and optimization algorithm to train deep learning models, and uses adaptive learning rate adjustment strategy to optimize the model training process.

[0044] Further, the loss function of the deep learning model is calculated, and the deep learning model is optimized by using an optimization algorithm based on the loss function, including: according to the first formula: Calculate the above loss function for the above deep learning model Among them, h i is the above expression label of the i-th training sample, is the model prediction label of the i-th training sample, N is the number of samples of the training sample; according to the second formula: The model parameters of the above deep learning model are optimized, where β j is the model parameter of the above deep learning model at the jth iteration, β j-1 is the model parameter of the above deep learning model at the j-1th iteration, α is the learning rate, is the partial derivative of the above loss function with respect to the above model parameters.

[0045] Furthermore, the SIFT algorithm is used to extract facial features from the above-mentioned facial three-dimensional model, including: extracting the coordinate information of the above-mentioned facial key points from the above-mentioned facial three-dimensional model using the above-mentioned SIFT algorithm, and extracting emotional features from the above-mentioned facial three-dimensional model; fusing the above-mentioned coordinate information of the above-mentioned facial key points with the above-mentioned emotional features to generate the above-mentioned facial features.

[0046] The feature fusion of this method includes fusing the coordinates of facial key points and emotional features to form a multimodal feature representation.

[0047] Specifically, after acquiring the facial expression data, the method further includes: preprocessing the facial expression data, wherein the preprocessing includes resizing, cropping and alignment, color adjustment, denoising and facial key point detection of the image data.

[0048] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the deep learning-based facial expression recognition optimization method of the present application will be described in detail below in combination with specific embodiments.

[0049] This embodiment relates to a specific facial expression recognition optimization method based on deep learning, such as Figure 3 As shown, the following steps are included:

[0050] Step S1: Collect facial expression data, pre-process the collected facial expression data, build a facial three-dimensional model using 3D facial reconstruction technology, and integrate facial expression data and voice emotion data using multimodal data fusion technology.

[0051] Collecting facial expression data includes collecting image data of facial expressions, expression labels, facial action units, biometric data, environmental information, audio data, physiological data and metadata.

[0052] Image data includes static photos and continuous video frames; expression labeling includes marking each facial image and video frame with an expression category, such as basic emotions such as happiness, sadness, anger, surprise, fear, disgust, or more subtle expression states; biometric data includes facial key points and head posture; facial key points include the location information of the eyes, nose, mouth and eyebrows, and are expressed in the form of coordinates; head posture includes posture information of head tilt and rotation, which may affect the interpretation of expressions; environmental information includes lighting conditions and background. Facial images under different lighting conditions may have different appearances, so recording lighting information is crucial for data preprocessing and model training. Although the background is usually removed or ignored when performing expression recognition, background information is sometimes used to analyze the context in which the expression occurs; audio data includes tone, volume and speaking speed. In multimodal data fusion, audio data can provide additional emotional information; physiological data includes heart rate, skin conductance and respiratory rate; metadata includes the time, location and device type of data collection.

[0053] Preprocessing includes resizing, cropping and alignment, color adjustment, denoising, and facial key point detection; resizing includes resizing the image using bilinear interpolation. The formula is:

[0054] I resized (x,y)=I original (f x (x),f y(y));

[0055] Among them, I resized (x, y) is the adjusted image, I original is the original image, f x and f y is the interpolation function along the x and y axes, and (x, y) is the original image coordinate.

[0056] Cropping and alignment include setting facial key points, calculating facial center points and facial angles, and using affine transformation for cropping and alignment; the affine transformation formula is:

[0057]

[0058] Among them, (x ′ ,y ′ ) is the transformed coordinate, (x,y) is the original coordinate, (x c ,y c ) is the facial center point, and θ is the facial angle.

[0059] Color adjustment includes normalizing the image brightness. The formula is:

[0060]

[0061] Among them, I normalized is the normalized image brightness, I is the original image brightness, μ is the average value of the image pixel values, and σ is the standard deviation of the image pixel values.

[0062] Denoising includes using a Gaussian filter for denoising, and the formula is expressed as:

[0063]

[0064] Among them, I filtered (x, y) is the denoised image data, w(a, b) is the Gaussian weight, k is the filter radius, and I(x, y) is the original image data.

[0065] 3D facial reconstruction technology includes using a 3D scanner to capture facial shape and structure, capturing facial 3D information through structured light technology, capturing facial images using multiple cameras, and reconstructing a 3D facial model through triangulation.

[0066] Multimodal data fusion technology includes facial expression data extraction and voice emotion data extraction; facial expression data extraction includes extracting facial key points from 3D facial models for expression recognition; voice emotion data extraction includes extracting emotional features such as tone, speaking speed and volume through audio analysis.

[0067] Integration includes feature extraction and feature fusion; feature extraction includes using the SIFT algorithm to extract facial key point coordinates from the 3D facial model and extract emotional features from the audio; feature fusion includes fusing the facial key point coordinates and emotional features to form a multimodal feature representation.

[0068] Step S2: Use SIFT algorithm to extract facial features to build a deep learning model, use loss function and optimization algorithm to train the deep learning model, use adaptive learning rate adjustment strategy to optimize the model training process, and use the trained deep learning model to perform facial expression recognition.

[0069] The extraction of facial features using the SIFT algorithm includes the construction of scale space, the detection of key points, the allocation of key point directions, and the generation of key point descriptors; the construction of scale space includes the construction of a Gaussian pyramid using a Gaussian kernel function, which is expressed as follows:

[0070] Among them, G(x,y,φ) is the Gaussian kernel function, (x,y) is the original image coordinate, and φ is the scale parameter.

[0071] The detection of key points includes calculating the gradient value in the neighborhood for each pixel point in each layer of the image in the scale space, comparing the gradient value of each pixel point with the gradient value of the remaining pixels in the neighborhood, if the gradient value of the pixel point is greater than the gradient value of the remaining pixels in the neighborhood, then the pixel point is a potential key point, for each potential key point, calculate the position of the potential key point in the scale space to improve the positioning accuracy of the key point, which is achieved by calculating the weighted average of the pixels in the neighborhood of the key point, in each scale layer, the non-maximum suppression method is used to remove key points that are not local extreme values, which means that in each scale layer, only those key points with the largest gradient values ​​in their neighborhood are retained, for each potential key point, calculate the local extreme value in the scale space, including comparison in the current scale layer and in adjacent scale layers.

[0072] The assignment of key point directions includes calculating a main direction for each key point. The main direction is consistent with the gradient direction of the key point neighborhood. The formula is expressed as:

[0073] Among them, θ z is the main direction of the key point, arctan is the inverse tangent function, g y (x, y, φ) is the component of the gradient in the vertical direction, g x (x, y, φ) is the horizontal component of the gradient.

[0074] The formula for feature fusion is:

[0075] Among them, F 3Dis the extracted facial key point coordinates, F audio is the emotional feature extracted from the audio. It is a feature fusion operation; the feature fusion operation includes splicing, feature scaling and feature conversion after splicing.

[0076] Building a deep learning model includes extracting facial features using the SIFT algorithm and building a deep learning model using the extracted facial features; the construction steps include determining the facial expression recognition task requirements, the output type is multi-classification output, selecting a convolutional neural network as the deep learning architecture to capture the features of facial expressions, and designing the network structure.

[0077] Designing the network structure includes determining the number of network layers based on task requirements and computing resources, designing the number of neurons in each layer based on the depth of the network and task requirements, and using activation functions for activation after each convolution.

[0078] The activation function formula is expressed as: ReLU(d)=max(0,d);

[0079] Among them, ReLU(d) is the output of the activation function and d is the input of the neuron.

[0080] Training a deep learning model involves loss functions and optimization algorithms, defining the model's input and output, and optimizing the model training process with an adaptive learning rate adjustment strategy.

[0081] The loss function can effectively measure the difference between the model prediction results and the actual results. The formula is:

[0082]

[0083] Among them, h i is the true label of the i-th sample, is the probability value predicted by the i-th sample model, and N is the number of samples in the training face dataset.

[0084] The optimization algorithm formula is:

[0085] Among them, β j is the model parameter at the jth iteration, β j-1 is the model parameter at the j-1th iteration, α is the learning rate, is the partial derivative of the loss function with respect to the model parameters, and L is the loss function.

[0086] The adaptive learning rate adjustment strategy includes learning rate decay and learning rate warm-up; the learning rate decay formula is:

[0087] α j =α j-1 ×r;

[0088] Among them, α j is the learning rate at the jth iteration, α j-1 is the learning rate at the j-1th iteration, and r is the learning rate decay rate.

[0089] Step S3: Use the continuous-time Markov model to process the dynamic changes of facial expressions, use the micro-expression amplification recognition method to process subtle expressions, output the facial expression recognition results, perform facial recognition result anomaly detection and match the anomaly processing strategy for the detection results.

[0090] The output facial expression recognition results include using the trained deep learning model for facial expression recognition, using the continuous-time Markov model to process the dynamic changes of facial expressions, using the micro-expression amplification recognition method to process subtle expressions, improving the recognition ability of subtle expression changes, and outputting the facial expression recognition results.

[0091] Anomaly detection includes calculating the anomaly detection score output by the model, setting the anomaly threshold and anomaly handling strategy function, and matching different anomaly handling strategies in different anomaly intervals.

[0092] The formula for calculating anomaly detection score is:

[0093] Among them, A(E,γ,δ) is the anomaly detection score, E is the facial expression feature vector, γ is the feature vector standard deviation, δ is the anomaly detection threshold, W(E,γ) is the comprehensive evaluation function of the normalized feature vector and the standard deviation, and V(E,γ) is the complex information filtering function.

[0094] The comprehensive evaluation function formula is:

[0095] Among them, n is the dimension of the feature vector, W(E,γ) is the comprehensive evaluation function of the normalized feature vector and the standard deviation, γ l is the standard deviation of the lth element of the eigenvector, E l is the lth element in the feature vector.

[0096] The formula for complex information filtering function is:

[0097] Among them, V(E,γ) is the complex information filtering function, n is the dimension of the feature vector, E l is the lth element in the eigenvector, ε is the average value of the eigenvector, γ l is the standard deviation of the lth element of the eigenvector.

[0098] Among them, the value range of A(E,γ,δ) is (0,1]. According to the analysis of historical anomaly data, the anomaly detection threshold of δ is 0.75. When 0.75 < A(E,γ,δ) ≤ 1, it indicates that the facial expression recognition result is normal. The current state of the system is maintained, and facial expression recognition continues. The facial expression feature library is updated regularly to optimize the recognition accuracy and provide user-friendly feedback. When 0 < A(E,γ,δ) ≤ 0.75, it indicates that the facial expression recognition result is abnormal. Data collection and model training are restarted, system parameters are updated, and anomaly detection is restarted after adjustment until A reaches the normal level.

[0099] In this embodiment, facial expression data is collected, and the collected facial expression data is preprocessed. A 3D facial model is constructed using 3D facial reconstruction technology, and facial expression data and speech emotion data are integrated using multimodal data fusion technology. The Scale-Invariant Feature Transform (SIFT) algorithm is used to extract facial features to construct a deep learning model. The deep learning model is trained using a loss function and an optimization algorithm, and the model training process is optimized using an adaptive learning rate adjustment strategy. The continuous-time Markov model is used to process the dynamic changes of facial expressions. The micro-expression amplification recognition method is used to process subtle expressions, and anomaly detection of facial recognition results is performed and an anomaly handling strategy is matched according to the detection results. This method can eliminate the influence of seismic wavelets and solve the problems of dealing with complex lighting conditions, facial occlusion, and limitations in subtle expression recognition.

[0100] An embodiment of the present application also provides an optimization device for facial expression recognition based on deep learning. It should be noted that the optimization device for facial expression recognition based on deep learning in the embodiment of the present application can be used to execute the method for optimizing facial expression recognition based on deep learning provided in the embodiment of the present application. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0101] The following introduces the optimization device for facial expression recognition based on deep learning provided in the embodiment of the present application.

[0102] Figure 4 is a schematic diagram of the optimization device for facial expression recognition based on deep learning according to the embodiment of the present application. As Figure 4 shown, the device includes:

[0103] An acquisition unit 41, configured to acquire facial expression data and construct a 3D facial model according to the above facial expression data using 3D facial reconstruction technology;

[0104] A training unit 42 is used to extract facial features from the above-mentioned three-dimensional facial model using a SIFT algorithm, and determine expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model;

[0105] The recognition unit 43 is used to recognize the facial expression to be recognized by at least using the above-mentioned facial expression recognition model to generate a recognition result.

[0106] In this embodiment, the acquisition unit is used to acquire facial expression data, and construct a facial three-dimensional model based on the facial expression data using 3D facial reconstruction technology; the training unit is used to extract facial features from the facial three-dimensional model using the SIFT algorithm, and determine the expression labels corresponding to the facial features, to construct a deep learning model, and use the facial features and the expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; the recognition unit is used to at least use the facial expression recognition model to recognize the facial expression to be recognized, and generate a recognition result. By integrating 3D facial reconstruction and SIFT algorithm technology, the accuracy, robustness, subtle expression recognition ability, dynamic expression processing ability, and abnormality detection and processing ability of facial expression recognition are improved. The problem that the existing facial feature extraction algorithm has limited recognition ability for subtle expressions is solved.

[0107] As an optional solution, the acquisition unit includes an acquisition module for acquiring image data, expression labels, facial action units, biometric data, environmental information, audio data, physiological data and metadata of facial expressions; wherein the above-mentioned image data includes static photos and continuous video frames, the above-mentioned expression labels include marking each facial image and the above-mentioned video frames with expression categories, the above-mentioned biometric data includes facial key points and head postures, the above-mentioned facial key points include position information of eyes, nose, mouth and eyebrows, the above-mentioned head posture includes posture information of head tilt and rotation, the above-mentioned environmental information includes lighting conditions and background, the above-mentioned audio data includes tone, volume and speaking speed, the above-mentioned physiological data includes heart rate, skin conductance and respiratory rate, and the above-mentioned metadata includes the time, place and device type of data collection.

[0108] An optional solution, the recognition unit includes a processing module and a recognition module, the processing module is used to use a continuous-time Markov model to process the dynamic changes of the above facial expressions, generate a dynamic change sequence, and use a micro-expression magnification recognition method to magnify the above facial expressions to generate magnification processing results; the recognition module is used to use the above facial expression recognition model to recognize the above facial expressions to be recognized based on the above dynamic change sequence and the above magnification processing results, and generate the above recognition results.

[0109] In an optional solution, the device also includes an optimization processing unit, which is used to calculate the loss function of the deep learning model in the process of training the deep learning model using the facial features and the expression labels as training samples, and optimize the deep learning model using an optimization algorithm based on the loss function.

[0110] In an optional solution, the optimization processing unit includes a calculation module and an optimization processing module, and the calculation module is used to calculate the value according to the first formula: Calculate the above loss function for the above deep learning model Among them, h i is the above expression label of the i-th training sample, is the model prediction label of the i-th training sample, N is the number of samples of the training sample; the optimization processing module is used to calculate the label according to the second formula: The model parameters of the above deep learning model are optimized, where β j is the model parameter of the above deep learning model at the jth iteration, β j-1 is the model parameter of the above deep learning model at the j-1th iteration, α is the learning rate, is the partial derivative of the above loss function with respect to the above model parameters.

[0111] An optional solution, the training unit includes an extraction module and a fusion processing module, the extraction module is used to extract the coordinate information of the above-mentioned facial key points from the above-mentioned facial three-dimensional model using the above-mentioned SIFT algorithm, and to extract emotional features from the above-mentioned facial three-dimensional model; the fusion processing module is used to fuse the above-mentioned coordinate information of the above-mentioned facial key points with the above-mentioned emotional features to generate the above-mentioned facial features.

[0112] In an optional solution, the device also includes a preprocessing unit, which is used to preprocess the facial expression data after acquiring the facial expression data, wherein the preprocessing includes resizing, cropping and alignment, color adjustment, denoising and facial key point detection of the image data.

[0113] The above-mentioned facial expression recognition optimization device based on deep learning includes a processor and a memory. The above-mentioned acquisition unit, training unit, recognition unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in the form of any combination.

[0114] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of the limited recognition ability of the existing facial feature extraction algorithm for subtle expressions can be solved by adjusting the kernel parameters.

[0115] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0116] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the facial expression recognition optimization method based on deep learning.

[0117] Specifically, the facial expression recognition optimization method based on deep learning includes:

[0118] Step S201, acquiring facial expression data, and constructing a facial three-dimensional model using 3D facial reconstruction technology based on the facial expression data.

[0119] Step S202, using the SIFT algorithm to extract facial features from the above-mentioned three-dimensional facial model, and determine the expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model.

[0120] Step S203, at least using the above facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result.

[0121] An embodiment of the present invention provides a processor, which is used to run a program, wherein the facial expression recognition optimization method based on deep learning is executed when the program is running.

[0122] Specifically, the facial expression recognition optimization method based on deep learning includes:

[0123] Step S201, acquiring facial expression data, and constructing a facial three-dimensional model using 3D facial reconstruction technology based on the facial expression data.

[0124] Step S202, using the SIFT algorithm to extract facial features from the above-mentioned three-dimensional facial model, and determine the expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model.

[0125] Step S203, at least using the above facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result.

[0126] An embodiment of the present invention provides an electronic device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0127] Step S201, acquiring facial expression data, and constructing a facial three-dimensional model using 3D facial reconstruction technology based on the facial expression data.

[0128] Step S202, using the SIFT algorithm to extract facial features from the above-mentioned three-dimensional facial model, and determine the expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model.

[0129] Step S203, at least using the above facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result.

[0130] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0131] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0132] Step S201, acquiring facial expression data, and constructing a facial three-dimensional model using 3D facial reconstruction technology based on the facial expression data.

[0133] Step S202, using the SIFT algorithm to extract facial features from the above-mentioned three-dimensional facial model, and determine the expression labels corresponding to the above-mentioned facial features, construct a deep learning model, and use the above-mentioned facial features and the above-mentioned expression labels as training samples to train the above-mentioned deep learning model to obtain a facial expression recognition model.

[0134] Step S203, at least using the above facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result.

[0135] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0142] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0144] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0145] 1) The present application discloses a facial expression recognition optimization method based on deep learning, comprising: obtaining facial expression data, and constructing a facial three-dimensional model based on the facial expression data using 3D facial reconstruction technology; extracting facial features from the facial three-dimensional model using the SIFT algorithm, and determining the expression labels corresponding to the facial features, constructing a deep learning model, and using facial features and expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; and at least using the facial expression recognition model to recognize the facial expression to be recognized, and generating a recognition result. By integrating 3D facial reconstruction and SIFT algorithm technology, the accuracy, robustness, subtle expression recognition capability, dynamic expression processing capability, and anomaly detection and processing capability of facial expression recognition are improved. This solves the problem that the existing facial feature extraction algorithm has limited recognition capability for subtle expressions.

[0146] 2) A facial expression recognition optimization device based on deep learning of the present application includes: an acquisition unit, which is used to acquire facial expression data, and construct a facial three-dimensional model based on the facial expression data using 3D facial reconstruction technology; a training unit, which is used to extract facial features from the facial three-dimensional model using the SIFT algorithm, and determine the expression labels corresponding to the facial features, and construct a deep learning model, and use facial features and expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; a recognition unit, which is used to at least use the facial expression recognition model to recognize the facial expression to be recognized and generate a recognition result. By integrating 3D facial reconstruction and SIFT algorithm technology, the accuracy, robustness, subtle expression recognition ability, dynamic expression processing ability, and abnormality detection and processing ability of facial expression recognition are improved. The problem that the existing facial feature extraction algorithm has limited recognition ability for subtle expressions is solved.

[0147] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A facial expression recognition optimization method based on deep learning, characterized in that: include: Acquire facial expression data, and construct a three-dimensional facial model using 3D facial reconstruction technology according to the facial expression data; Extract facial features from the three-dimensional facial model using the SIFT algorithm, determine expression labels corresponding to the facial features, construct a deep learning model, and use the facial features and the expression labels as training samples to train the deep learning model to obtain a facial expression recognition model; At least the facial expression recognition model is used to recognize the facial expression to be recognized and generate a recognition result.

2. The method according to claim 1, characterized in that Get facial expression data, including: Obtain image data of facial expressions, expression labels, facial action units, biometric data, environmental information, audio data, physiological data, and metadata; The image data includes static photos and continuous video frames, the expression tags include marking each facial image and the video frame with an expression category, the biometric data includes facial key points and head posture, the facial key points include position information of eyes, nose, mouth and eyebrows, the head posture includes posture information of head tilt and rotation, the environmental information includes lighting conditions and background, the audio data includes tone, volume and speaking speed, the physiological data includes heart rate, skin conductance and respiratory rate, and the metadata includes the time, place and device type of data collection.

3. The method according to claim 1, characterized in that At least using the facial expression recognition model to recognize the facial expression to be recognized and generating a recognition result, including: Using a continuous-time Markov model to process the dynamic changes of the facial expression to generate a dynamic change sequence, and using a micro-expression magnification recognition method to perform expression magnification processing on the facial expression to generate a magnification processing result; Based on the dynamic change sequence and the amplification processing result, the facial expression to be recognized is recognized using the facial expression recognition model to generate the recognition result.

4. The method according to claim 1, characterized in that In the process of using the facial features and the expression labels as training samples to train the deep learning model, the method further includes: The loss function of the deep learning model is calculated, and based on the loss function, an optimization algorithm is used to optimize the deep learning model.

5. The method according to claim 4, characterized in that Calculating the loss function of the deep learning model, And based on the loss function, an optimization algorithm is used to optimize the deep learning model, including: According to the first formula: Calculating the loss function of the deep learning model Among them, h i is the expression label of the i-th training sample, is the model prediction label of the i-th training sample, where N is the number of samples of the training sample; According to the second formula: The model parameters of the deep learning model are optimized, where β j is the model parameter of the deep learning model at the jth iteration, β j-1 is the model parameter of the deep learning model at the j-1th iteration, α is the learning rate, is the partial derivative of the loss function with respect to the model parameters.

6. The method according to claim 2, characterized in that The facial features are extracted from the three-dimensional facial model using the SIFT algorithm, including: The SIFT algorithm is used to extract the coordinate information of the facial key points from the facial three-dimensional model, and the emotional features are extracted from the facial three-dimensional model; The coordinate information of the facial key points is fused with the emotional features to generate the facial features.

7. The method according to claim 1, characterized in that After acquiring the facial expression data, the method further includes: The facial expression data is preprocessed, wherein the preprocessing includes resizing, cropping and alignment, color adjustment, denoising and facial key point detection of the image data.

8. A facial expression recognition optimization device based on deep learning, characterized in that: include: An acquisition unit, used for acquiring facial expression data, and constructing a three-dimensional facial model using 3D facial reconstruction technology according to the facial expression data; A training unit, configured to extract facial features from the three-dimensional facial model using a SIFT algorithm, determine expression labels corresponding to the facial features, construct a deep learning model, and train the deep learning model using the facial features and the expression labels as training samples to obtain a facial expression recognition model; The recognition unit is used to recognize the facial expression to be recognized by at least using the facial expression recognition model to generate a recognition result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the deep learning-based facial expression recognition optimization method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the deep learning-based facial expression recognition optimization method described in any one of claims 1 to 7.

Images