Facial expression-based behavior analysis method and apparatus, device, medium, and product

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a facial expression-based behavior analysis method and apparatus, a device, a medium, and a product. The method comprises: acquiring a plurality of expression weight vectors generated by a user in a virtual scene; on the basis of the plurality of expression weight vectors, performing facial expression feature extraction to obtain a facial expression feature matrix; inputting the plurality of expression weight vectors into a time series feature extraction model to obtain an output expression time series feature vector; and on the basis of the facial expression feature matrix and the expression time series feature vector, performing behavior type analysis to obtain a target behavior type of the user in the virtual scene. In the facial expression-based behavior analysis method provided by the present disclosure, a static feature is reflected by means of facial expression feature extraction, a dynamic feature is reflected by means of time series feature extraction, and behavior type analysis is performed on the basis of the combination of the static feature and the dynamic feature, so that a user behavior can be automatically recognized, and the accuracy and objectiveness of facial expression-based behavior analysis can be improved.

Description

Facial expression-based behavior analysis methods, devices, equipment, media, and products

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410935871.6, filed on July 12, 2024, entitled "Method, Apparatus, Device, Medium and Product for Behavior Analysis Based on Facial Expressions", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium and product for behavior analysis based on facial expressions. Background Technology

[0004] When judging user behavior, nonverbal information (such as facial expressions, posture, and eye contact) often carries important information. Facial data, as a non-invasive form of biometric information, contains rich emotional and cognitive information. When users exhibit abnormal behavior, anomalies will appear in facial expression recognition, eye contact, and facial movements. These anomalies can be objectively measured and assessed by collecting and analyzing facial data.

[0005] Existing technologies utilize facial data to analyze user behavior through subjective consciousness. However, these methods are often limited to the analysis of specific facial expressions or movements, lacking comprehensiveness and systematicity. They also present certain challenges in handling the complexity and variability of facial data, resulting in low accuracy in behavior analysis.

[0006] Summary of the Invention

[0007] This disclosure provides a method, apparatus, device, medium, and product for behavior analysis based on facial expressions, in order to solve the problem of low accuracy in behavior analysis based on facial expressions in the prior art.

[0008] In a first aspect, this disclosure provides a behavior analysis method based on facial expressions, including:

[0009] Obtain multiple facial expression weight vectors generated by the user in a virtual scene; each of the facial expression weight vectors represents a facial expression; each of the facial expression weight vectors is composed of weights of multiple types of basic micro-expressions;

[0010] Facial expression features are extracted based on the multiple expression weight vectors to obtain a facial expression feature matrix;

[0011] The multiple facial expression weight vectors are input into the temporal feature extraction model to obtain the facial expression temporal feature vector output by the temporal feature extraction model; the temporal feature extraction model is obtained by training the model based on facial expression weight vector samples;

[0012] Based on the facial expression feature matrix and the expression temporal feature vector, behavior type analysis is performed to obtain the user's target behavior type in the virtual scene.

[0013] In one embodiment, the facial expression feature matrix is ​​composed of facial expression feature vectors of multiple types of basic micro-expressions. When extracting facial features based on the multiple expression weight vectors, the following steps are performed for multiple basic micro-expressions of the same type:

[0014] The sampling time of each of the multiple basic micro-expressions is obtained;

[0015] The basic micro-expressions with weights greater than zero among the multiple basic micro-expressions are identified as the target basic micro-expressions;

[0016] The total duration of occurrence of the basic micro-expression of the type is obtained by summing the sampling times corresponding to multiple target basic micro-expressions;

[0017] The consecutive sampling times corresponding to the sampling times of the multiple target basic micro-expressions are divided into target time periods;

[0018] The earliest target time period is determined as the duration of the first appearance of the basic micro-expression of the aforementioned type;

[0019] The number of multiple target time periods is determined as the total number of occurrences of the basic micro-expression of the aforementioned type;

[0020] Divide the total occurrence duration by the total number of occurrences to obtain the average occurrence duration of the basic micro-expression of the type;

[0021] The variance of the weights corresponding to the multiple target basic micro-expressions is calculated to obtain the weight variance of the type of basic micro-expression.

[0022] The average weights of the various target basic micro-expressions are calculated to obtain the average weight of the basic micro-expressions of the type.

[0023] The weights corresponding to the multiple target basic micro-expressions are summed to obtain the first weight sum of the basic micro-expressions of the type;

[0024] Determine the second weighted sum for each other type of basic micro-expression; the other types of basic micro-expressions refer to basic micro-expressions of other types besides the stated type among the multiple types of basic micro-expressions;

[0025] The first weighted sum is summed with multiple second weighted sums to obtain the third weighted sum;

[0026] Divide the first weighted sum by the third weighted sum to obtain the weight percentage of the basic micro-expression type;

[0027] Based on the total occurrence duration, first occurrence duration, average occurrence duration, weight variance, weight mean, and weight percentage of the basic micro-expression of the type, a facial expression feature vector of the basic micro-expression of the type is constructed.

[0028] In one embodiment, the temporal feature extraction model includes a first hidden layer, a second hidden layer, a concatenation layer, an attention layer, and an output layer; the step of inputting the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model includes:

[0029] The multiple facial expression weight vectors are input into the first hidden layer to obtain the first temporal feature vector output by the first hidden layer;

[0030] The first temporal feature vector is input into the second hidden layer to obtain the second temporal feature vector output by the second hidden layer.

[0031] The first temporal feature vector and the second temporal feature vector are input into the splicing layer to obtain the splicing feature matrix output by the splicing layer;

[0032] The concatenated feature matrix is ​​input into the attention layer to obtain the attention vector output by the attention layer;

[0033] The first temporal feature vector and the attention vector are multiplied by the attention layer to obtain the output result, and the output result is determined as the second temporal feature vector containing attention information output by the second hidden layer.

[0034] The second temporal feature vector containing attention information is input into the output layer to obtain the facial expression temporal feature vector output by the output layer.

[0035] In one embodiment, the temporal feature extraction model is trained in the following manner:

[0036] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0037] Based on the expression weight vector samples, unsupervised learning is performed on the first initial model to obtain the temporal feature extraction model.

[0038] In one embodiment, the step of performing behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene includes:

[0039] The facial expression feature matrix and the expression temporal feature vector are concatenated to obtain a fused feature matrix;

[0040] The fused feature matrix is ​​input into the behavior classification model to obtain the probability vector output by the behavior classification model; each component in the probability vector represents the probability of a behavior type; the behavior classification model is obtained by training the model based on the fused feature matrix samples and behavior type labels.

[0041] The behavior type corresponding to the largest component in the probability vector is determined as the user's target behavior type in the virtual scene.

[0042] In one embodiment, the behavior classification model is trained as follows:

[0043] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0044] Based on the expression weight vector samples, determine the facial expression feature matrix samples and expression temporal feature vector samples;

[0045] The facial expression feature matrix sample and the expression temporal feature vector sample are concatenated to obtain a fused feature matrix sample;

[0046] The fused feature matrix samples are labeled with behavior types to obtain behavior type labels;

[0047] Based on the fused feature matrix samples and the behavior type labels, the second initial model is trained to obtain the behavior classification model.

[0048] Secondly, this disclosure also provides a facial expression-based behavior analysis device, comprising:

[0049] The acquisition module is used to acquire multiple facial expression weight vectors generated by the user in a virtual scene; each of the facial expression weight vectors represents a facial expression; each of the facial expression weight vectors is composed of weights of multiple types of basic micro-expressions;

[0050] The first feature extraction module is used to extract facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix.

[0051] The second feature extraction module is used to input the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model; the temporal feature extraction model is obtained by training the model based on expression weight vector samples;

[0052] The behavior analysis module is used to perform behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene.

[0053] Thirdly, this disclosure provides an apparatus comprising an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described facial expression-based behavior analysis methods.

[0054] Fourthly, this disclosure also provides a medium comprising a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the above-described facial expression-based behavior analysis methods.

[0055] Fifthly, this disclosure also provides a product comprising a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and the computer program, when executed by the processor, implementing the steps of any of the above-described facial expression-based behavior analysis methods.

[0056] The facial expression-based behavior analysis method, apparatus, device, medium, and product disclosed herein can more comprehensively understand user behavior in virtual scenes by extracting static features from multiple facial expression weight vectors generated by users in virtual scenes and extracting dynamic features from temporal features. By combining static and dynamic features for behavior type analysis, the target behavior type of users in virtual scenes can be obtained, enabling automatic identification of user behavior and improving the accuracy and objectivity of facial expression-based behavior analysis. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 is a flowchart illustrating the facial expression-based behavior analysis method provided in this disclosure.

[0059] Figure 2 is a schematic diagram of the structure of the facial expression-based behavior analysis device provided in this disclosure.

[0060] Figure 3 is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0062] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein.

[0063] The following description, in conjunction with Figures 1-3, describes the facial expression-based behavior analysis method, apparatus, device, medium, and product provided by this disclosure.

[0064] Figure 1 is a flowchart of one of the facial expression-based behavior analysis methods provided in this disclosure. As shown in Figure 1, the method includes the following steps 101, 102, 103 and 104.

[0065] Step 101: Obtain the weight vectors of multiple facial expressions generated by the user in the virtual scene.

[0066] Step 102: Extract facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix.

[0067] Step 103: Input the multiple facial expression weight vectors into the temporal feature extraction model to obtain the facial expression temporal feature vector output by the temporal feature extraction model.

[0068] Step 104: Based on the facial expression feature matrix and the expression temporal feature vector, perform behavior type analysis to obtain the user's target behavior type in the virtual scene.

[0069] It should be noted that the facial expression-based behavior analysis method provided in this disclosure is implemented based on a facial expression-based behavior analysis device, which will be referred to as the behavior analysis device below. Therefore, this disclosure describes the facial expression-based behavior analysis method using the behavior analysis device as the execution subject as an example.

[0070] Before conducting facial expression-based behavioral analysis, the user needs to wear a virtual reality (VR) device. After wearing the device, a five-point eye-tracking calibration is performed. Specifically, five fixed points are displayed in the virtual environment, and the user is required to look at these fixed points in sequence to complete the eye-tracking calibration, which can accurately capture the user's gaze position in the virtual environment.

[0071] After calibration, the virtual scene is launched, and users perform interactive levels with objects in the virtual scene in sequence. Each level is introduced by a virtual host, and then the user performs the interactive task.

[0072] In one embodiment, the duration of a single level is 2 minutes, and there are a total of 5 levels, so the total duration of the virtual scene is 10 minutes.

[0073] In a virtual environment, users will exhibit different emotions and expressions as the plot unfolds. During this process, facial expression data can be collected from users using VR devices at a sampling frequency. It should be noted that each facial expression data includes the sampling time and a single expression weight vector.

[0074] In one embodiment, a Pico 4 Enterprise VR device can be used, with a facial data acquisition sampling frequency of 30Hz. The virtual scene is rendered on the Pico 4 Enterprise VR device through a C++ program. The virtual scene image is dynamically updated according to the user's head orientation and spatial position, thereby providing a highly immersive interactive virtual reality environment, allowing the user to obtain a realistic and vivid experience. The virtual character plays a mediating role in the virtual scene. During the virtual plot display, the virtual character uses voice to show the patient the plot development and task instructions, helping the user complete interactive tasks in the virtual environment.

[0075] In one embodiment, a level in the virtual scene can be: the virtual character waves to the user, then suddenly discovers that its own items are missing, showing anxiety and sadness, then asks the user to help find the items and finds them, showing joy.

[0076] Specifically, the behavior analysis device maps each facial expression data to a basic micro-expression, transforms it into a basic micro-expression array, and obtains multiple expression weight vectors generated by the user in the virtual scene. Therefore, any expression weight vector represents a facial expression, and any expression weight vector is composed of the weights of multiple types of basic micro-expressions.

[0077] It should be noted that expression blending technology is a technique used to simulate facial expressions. It achieves various facial expressions by adjusting the vertex positions of a 3D model. A facial expression can be represented by multiple different basic micro-expressions, each with a weight that determines the degree or intensity of that basic micro-expression in the final presentation.

[0078] In one embodiment, an expression weight vector contains 52 basic micro-expression elements, and the expression weight vector can be represented as [weight of basic micro-expression 1, weight of basic micro-expression 2, ..., weight of basic micro-expression 52].

[0079] The matrix formed by the five expression weight vectors is shown below:

[0080] Furthermore, the behavior analysis device extracts facial expression features based on multiple expression weight vectors and the same type of basic micro-expressions, obtaining facial expression feature vectors for each type of basic micro-expression.

[0081] Furthermore, the behavior analysis device concatenates the facial expression feature vectors of each type of basic micro-expression to obtain a facial expression feature matrix.

[0082] Furthermore, the behavior analysis device inputs the expression weight vector into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model. Therefore, the expression temporal feature vector contains the temporal information of the expression weight vector.

[0083] In one embodiment, a Long Short-Term Memory (LSTM) network can be used. LSTM is a commonly used recurrent neural network architecture used to solve long-term dependency problems and can effectively capture and utilize long-term dependencies in the input sequence.

[0084] Furthermore, the behavior analysis device fuses the facial expression feature matrix and the expression temporal feature vector to obtain a fused feature matrix.

[0085] Furthermore, the behavior analysis device performs behavior type analysis based on the fused feature matrix to obtain the user's target behavior type in the virtual scene.

[0086] It should be noted that the facial expression-based behavioral analysis method can be applied to assist in the diagnosis or treatment decision-making of diseases, such as autism and other diseases related to user behavior patterns. It can also be applied to product application evaluation, advertising effectiveness evaluation, etc., without limitation.

[0087] The facial expression-based behavior analysis method disclosed herein uses facial expression feature extraction to reflect static features and temporal feature extraction to reflect dynamic features for multiple expression weight vectors generated by users in a virtual scene. This allows for a more comprehensive understanding of user behavior in the virtual scene. By combining static and dynamic features for behavior type analysis, the target behavior type of the user in the virtual scene can be obtained, enabling automatic identification of user behavior and improving the accuracy and objectivity of facial expression-based behavior analysis.

[0088] Furthermore, when extracting facial features based on the multiple expression weight vectors, the following steps are performed for multiple basic micro-expressions of the same type:

[0089] The sampling time of each of the multiple basic micro-expressions is obtained;

[0090] The basic micro-expressions with weights greater than zero among the multiple basic micro-expressions are identified as the target basic micro-expressions;

[0091] The total duration of occurrence of the basic micro-expression of the type is obtained by summing the sampling times corresponding to multiple target basic micro-expressions;

[0092] The consecutive sampling times corresponding to the sampling times of the multiple target basic micro-expressions are divided into target time periods;

[0093] The earliest target time period is determined as the duration of the first appearance of the basic micro-expression of the aforementioned type;

[0094] The number of multiple target time periods is determined as the total number of occurrences of the basic micro-expression of the aforementioned type;

[0095] Divide the total occurrence duration by the total number of occurrences to obtain the average occurrence duration of the basic micro-expression of the type;

[0096] The variance of the weights corresponding to the multiple target basic micro-expressions is calculated to obtain the weight variance of the type of basic micro-expression.

[0097] The average weights of the various target basic micro-expressions are calculated to obtain the average weight of the basic micro-expressions of the type.

[0098] The weights corresponding to the multiple target basic micro-expressions are summed to obtain the first weight sum of the basic micro-expressions of the type;

[0099] Determine the second weighted sum for each other type of basic micro-expression; the other types of basic micro-expressions refer to basic micro-expressions of other types besides the stated type among the multiple types of basic micro-expressions;

[0100] The first weighted sum is summed with multiple second weighted sums to obtain the third weighted sum;

[0101] Divide the first weighted sum by the third weighted sum to obtain the weight percentage of the basic micro-expression type;

[0102] Based on the total occurrence duration, first occurrence duration, average occurrence duration, weight variance, weight mean, and weight percentage of the basic micro-expression of the type, a facial expression feature vector of the basic micro-expression of the type is constructed.

[0103] It should be noted that each facial expression data point has a corresponding sampling time. Therefore, the expression weight vector obtained by converting facial expression data also has the same sampling time, and the various types of basic micro-expressions in the expression weight vector also have the same sampling time.

[0104] In addition, all facial expression data collected during the sampling process is temporal, therefore, all expression weight vectors are also temporal, and all basic micro-expressions of the same type are also temporal.

[0105] The following provides a detailed explanation of the basic micro-expressions of the same type for multiple expression weight vectors.

[0106] Specifically, the behavior analysis device acquires the sampling time of multiple basic micro-expressions.

[0107] Furthermore, the behavior analysis device identifies the basic micro-expressions with weights greater than 0 among multiple basic micro-expressions as target basic micro-expressions. It should be noted that when the weight of a basic micro-expression is greater than 0, it means that the basic micro-expression can be used to form facial expressions.

[0108] Furthermore, the behavior analysis device sums the sampling times corresponding to multiple target basic micro-expressions to obtain the total occurrence duration of the type of basic micro-expression.

[0109] Furthermore, the behavior analysis device divides multiple consecutive sampling times corresponding to the sampling times of multiple target basic micro-expressions into target time periods, thus obtaining multiple target time periods.

[0110] Furthermore, the behavior analysis device determines the first occurrence duration of the target time period, which is the first time period of the current type of basic micro-expression, as the duration of the first occurrence of the current type of basic micro-expression.

[0111] Furthermore, the target time period during which the target basic micro-expression appears continuously is assumed to be the appearance of the target basic micro-expression once. Therefore, the behavior analysis device determines the number of multiple target time periods as the total number of appearances of the current type of basic micro-expression.

[0112] Furthermore, the behavior analysis device divides the total occurrence duration of the current type of basic micro-expression by the total occurrence frequency of the current type of basic micro-expression to obtain the average occurrence duration of the current type of basic micro-expression.

[0113] Furthermore, the behavior analysis device calculates the variance of the weights corresponding to multiple target basic micro-expressions to obtain the weight variance of the current type of basic micro-expression.

[0114] Furthermore, the behavior analysis device calculates the average weight of the basic micro-expressions of multiple targets to obtain the average weight of the basic micro-expressions of the current type.

[0115] Furthermore, the behavior analysis device sums the weights corresponding to multiple target basic micro-expressions to obtain the first weight sum of the basic micro-expressions of the current type.

[0116] Furthermore, the behavior analysis device calculates a second weighted sum for each other type of basic micro-expression, where other types of basic micro-expressions refer to basic micro-expressions of other types besides the current type among multiple types of basic micro-expressions.

[0117] Furthermore, the behavior analysis device sums the first weighted sum and the second weighted sum to obtain the third weighted sum.

[0118] Furthermore, the behavior analysis device divides the sum of the first weights and the sum of the third weights to obtain the weight percentage of the basic micro-expressions of the current type.

[0119] Furthermore, the behavior analysis device constructs a facial expression feature vector for the current type of basic micro-expression based on the total occurrence duration, first occurrence duration, average occurrence duration, weight variance, weight mean, and weight percentage. Therefore, the facial expression feature vector for the current type of basic micro-expression is as follows:

[0120] Therefore, after obtaining the facial expression feature vectors of each type of basic micro-expression using the above method, the behavior analysis device concatenates the facial expression feature vectors of multiple types of basic micro-expressions to obtain the facial expression feature matrix.

[0121] Based on the analysis and calculation of duration, frequency, weight, etc., the embodiments of this disclosure construct facial expression feature vectors, which can comprehensively capture all aspects of basic micro-expressions, effectively express the details and features of facial expressions, and provide more detailed data support for subsequent behavior analysis.

[0122] Further, based on step 103, the step of inputting the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model includes:

[0123] The multiple facial expression weight vectors are input into the first hidden layer to obtain the first temporal feature vector output by the first hidden layer;

[0124] The first temporal feature vector is input into the second hidden layer to obtain the second temporal feature vector output by the second hidden layer.

[0125] The first temporal feature vector and the second temporal feature vector are input into the splicing layer to obtain the splicing feature matrix output by the splicing layer;

[0126] The concatenated feature matrix is ​​input into the attention layer to obtain the attention vector output by the attention layer;

[0127] The first temporal feature vector and the attention vector are multiplied by the attention layer to obtain the output result, and the output result is determined as the second temporal feature vector containing attention information output by the second hidden layer.

[0128] The second temporal feature vector containing attention information is input into the output layer to obtain the facial expression temporal feature vector output by the output layer.

[0129] It should be noted that, in one embodiment, several frames of facial expression data collected by the user in the virtual scene are formed into a time-series vector E = {x0, x1, ..., xk}. Considering that the duration of each virtual task is different, the number of frames of facial expression data collected is also different. In order to ensure the output quality, facial expression data collected in all levels and time periods can be sampled, and N fixed frames of data, i.e. N one-dimensional time-series vectors, can be sampled and input into the LSTM network to meet the input requirements of the LSTM network.

[0130] Specifically, the behavior analysis device inputs multiple expression weight vectors into the first hidden layer, which learns to capture time-series patterns in the input data, resulting in multiple first temporal feature vectors output by the first hidden layer.

[0131] Furthermore, the behavior analysis device inputs multiple first time-series feature vectors into the second hidden layer, which then learns and extracts higher-level time-series features to obtain multiple second time-series feature vectors output by the second hidden layer.

[0132] Furthermore, the behavior analysis device inputs multiple first temporal feature vectors and multiple second temporal feature vectors into the splicing layer, which splices the multiple first temporal feature vectors and multiple second temporal feature vectors to obtain the spliced ​​feature matrix output by the splicing layer.

[0133] Furthermore, the behavior analysis device inputs the spliced ​​feature matrix into the attention layer, which then weights the spliced ​​feature matrix to obtain the attention vector output by the attention layer. This attention vector will contain the time series patterns and time series features learned in the previous layers.

[0134] It should be noted that the attention layer mainly consists of a fully connected layer, a normalized softmax calculation layer, and a dot product calculation layer. The attention layer weights the concatenated feature matrix as follows: the concatenated feature matrix is ​​input into the fully connected layer, and the output of the fully connected layer is processed by the softmax calculation layer to normalize the output of the fully connected layer to between 0 and 1. The sum of all elements of the output vector of the softmax calculation layer is 1. The above properties meet the requirements of the weight values. The output vector of the softmax calculation layer and the first temporal feature vector are then processed by the dot product calculation layer to obtain the output result. The output result is determined as the second temporal feature vector containing attention information output by the second hidden layer.

[0135] Furthermore, the behavior analysis device inputs a second temporal feature vector containing attention information into the output layer, which then performs further processing and mapping to obtain the facial expression temporal feature vector output by the output layer.

[0136] The embodiments of this disclosure can effectively capture the temporal information in the expression weight vector through the temporal feature extraction model, and enhance the expression of key features through the attention mechanism, thereby improving the accuracy of facial expression feature depiction and providing more detailed data support for subsequent behavior analysis.

[0137] Furthermore, the temporal feature extraction model is trained in the following manner:

[0138] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0139] Based on the expression weight vector samples, unsupervised learning is performed on the first initial model to obtain the temporal feature extraction model.

[0140] Specifically, the behavior analysis device acquires facial expression weight vector samples generated by multiple subjects in a virtual scene.

[0141] Furthermore, the behavior analysis device performs unsupervised learning on the first initial model based on the facial expression weight vector samples to obtain a trained time feature extraction model.

[0142] It should be noted that unsupervised learning is a machine learning method in which the model attempts to discover patterns and structures in data without labels. In this case, the model automatically adjusts its parameters to extract temporal features to the greatest extent possible by observing the patterns and changes in expression weight vector samples.

[0143] The embodiments disclosed herein utilize a large number of facial expression weight vector samples and effectively capture the temporal information of the facial expression weight vector samples through unsupervised learning, thereby enabling the model to have better generalization ability and accuracy in temporal feature extraction tasks.

[0144] Further, based on step 104, the behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene includes:

[0145] The facial expression feature matrix and the expression temporal feature vector are concatenated to obtain a fused feature matrix;

[0146] The fused feature matrix is ​​input into the behavior classification model to obtain the probability vector output by the behavior classification model; each component in the probability vector represents the probability of a behavior type; the behavior classification model is obtained by training the model based on the fused feature matrix samples and behavior type labels.

[0147] The behavior type corresponding to the largest component in the probability vector is determined as the user's target behavior type in the virtual scene.

[0148] Specifically, the behavior analysis device concatenates the facial expression feature matrix and the expression temporal feature vector to obtain a more comprehensive and richer fusion feature matrix.

[0149] Furthermore, the behavior analysis device inputs the fused feature matrix into the behavior classification model to obtain the probability vector output by the behavior classification model, where each component in the probability vector represents the probability of a behavior type.

[0150] It should be noted that the behavior classification model can use a multilayer perceptron network, which mainly includes a fully connected layer and a softmax computation layer. The fused feature matrix is ​​mapped to a probability vector P = {p1, p2, ..., pc} after passing through the fully connected layer and the softmax computation layer. The probability vector P has c components, each component representing the probability that the input sample belongs to one of the c categories. The calculation formula for its mapping is as follows:

[0151] Where X represents the fusion feature matrix, e represents the natural logarithm, fi represents the i-th element in the fully connected layer output vector, and c represents that there are c elements in the fully connected layer output vector.

[0152] Furthermore, the behavior analysis device compares the numerical values ​​of each component in the probability vector to obtain the numerical comparison result.

[0153] Furthermore, based on the numerical comparison results, the behavior analysis device determines the behavior type corresponding to the largest component in the probability vector as the user's target behavior type in the virtual scene.

[0154] This embodiment combines the static features reflected by the facial expression feature matrix and the dynamic features reflected by the expression temporal feature vector to perform behavior type analysis, thereby obtaining the user's target behavior type in a virtual scene, realizing automatic recognition of user behavior, and improving the accuracy and objectivity of facial expression-based behavior analysis.

[0155] Furthermore, the behavior classification model is trained in the following manner:

[0156] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0157] Based on the expression weight vector samples, determine the facial expression feature matrix samples and expression temporal feature vector samples;

[0158] The facial expression feature matrix sample and the expression temporal feature vector sample are concatenated to obtain a fused feature matrix sample;

[0159] The fused feature matrix samples are labeled with behavior types to obtain behavior type labels;

[0160] Based on the fused feature matrix samples and the behavior type labels, the second initial model is trained to obtain the behavior classification model.

[0161] Specifically, the behavior analysis device acquires facial expression weight vector samples generated by multiple subjects in a virtual scene.

[0162] Furthermore, the behavior analysis device extracts facial expression features based on the expression weight vector samples to obtain facial expression feature matrix samples.

[0163] Furthermore, the behavior analysis device inputs the facial expression weight vector samples into the temporal feature extraction model to obtain the facial expression temporal feature vector samples output by the temporal feature extraction model.

[0164] Furthermore, the behavior analysis device concatenates the facial expression feature matrix samples and the expression temporal feature vector samples to obtain a fused feature matrix sample.

[0165] Furthermore, the behavior analysis device annotates the behavior types of the fused feature matrix samples to obtain behavior type labels.

[0166] Furthermore, the behavior analysis device trains the second initial model based on the fused feature matrix samples and behavior type labels to obtain a behavior classification model.

[0167] The embodiments disclosed herein combine facial expression features and temporal information to effectively improve the accuracy and robustness of behavior classification, providing strong technical support for behavior analysis.

[0168] The facial expression-based behavior analysis device provided in this disclosure is described below. The facial expression-based behavior analysis device described below can be referred to in correspondence with the facial expression-based behavior analysis method described above.

[0169] Referring to Figure 2, which is a schematic diagram of the structure of the facial expression-based behavior analysis device provided in this disclosure.

[0170] The facial expression-based behavior analysis device includes:

[0171] The acquisition module 210 is used to acquire multiple facial expression weight vectors generated by the user in a virtual scene; each of the facial expression weight vectors represents a facial expression; each of the facial expression weight vectors is composed of weights of multiple types of basic micro-expressions.

[0172] The first feature extraction module 220 is used to extract facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix.

[0173] The second feature extraction module 230 is used to input the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model; the temporal feature extraction model is obtained by training the model based on the expression weight vector samples.

[0174] The behavior analysis module 240 is used to perform behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene.

[0175] The facial expression-based behavior analysis device disclosed herein can reflect static features through facial expression feature extraction and dynamic features through temporal feature extraction for multiple expression weight vectors generated by users in a virtual scene. This allows for a more comprehensive understanding of user behavior in the virtual scene. By combining static and dynamic features for behavior type analysis, the target behavior type of the user in the virtual scene can be obtained, enabling automatic recognition of user behavior and improving the accuracy and objectivity of facial expression-based behavior analysis.

[0176] Furthermore, the first feature extraction module 220 is also used for:

[0177] The sampling time of each of the multiple basic micro-expressions is obtained;

[0178] The basic micro-expressions with weights greater than zero among the multiple basic micro-expressions are identified as the target basic micro-expressions;

[0179] The total duration of occurrence of the basic micro-expression of the type is obtained by summing the sampling times corresponding to multiple target basic micro-expressions;

[0180] The consecutive sampling times corresponding to the sampling times of the multiple target basic micro-expressions are divided into target time periods;

[0181] The earliest target time period is determined as the duration of the first appearance of the basic micro-expression of the aforementioned type;

[0182] The number of multiple target time periods is determined as the total number of occurrences of the basic micro-expression of the aforementioned type;

[0183] Divide the total occurrence duration by the total number of occurrences to obtain the average occurrence duration of the basic micro-expression of the type;

[0184] The variance of the weights corresponding to the multiple target basic micro-expressions is calculated to obtain the weight variance of the type of basic micro-expression.

[0185] The average weights of the various target basic micro-expressions are calculated to obtain the average weight of the basic micro-expressions of the type.

[0186] The weights corresponding to the multiple target basic micro-expressions are summed to obtain the first weight sum of the basic micro-expressions of the type;

[0187] Determine the second weighted sum for each other type of basic micro-expression; the other types of basic micro-expressions refer to basic micro-expressions of other types besides the stated type among the multiple types of basic micro-expressions;

[0188] The first weighted sum is summed with multiple second weighted sums to obtain the third weighted sum;

[0189] Divide the first weighted sum by the third weighted sum to obtain the weight percentage of the basic micro-expression type;

[0190] Based on the total occurrence duration, first occurrence duration, average occurrence duration, weight variance, weight mean, and weight percentage of the basic micro-expression of the type, a facial expression feature vector of the basic micro-expression of the type is constructed.

[0191] Furthermore, the second feature extraction module 230 is also used for:

[0192] The multiple facial expression weight vectors are input into the first hidden layer to obtain the first temporal feature vector output by the first hidden layer;

[0193] The first temporal feature vector is input into the second hidden layer to obtain the second temporal feature vector output by the second hidden layer.

[0194] The first temporal feature vector and the second temporal feature vector are input into the splicing layer to obtain the splicing feature matrix output by the splicing layer;

[0195] The concatenated feature matrix is ​​input into the attention layer to obtain the attention vector output by the attention layer;

[0196] The first temporal feature vector and the attention vector are multiplied by the attention layer to obtain the output result, and the output result is determined as the second temporal feature vector containing attention information output by the second hidden layer.

[0197] The second temporal feature vector containing attention information is input into the output layer to obtain the facial expression temporal feature vector output by the output layer.

[0198] Furthermore, the facial expression-based behavior analysis device is also used for:

[0199] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0200] Based on the expression weight vector samples, unsupervised learning is performed on the first initial model to obtain the temporal feature extraction model.

[0201] Furthermore, the behavior analysis module 240 is also used for:

[0202] The facial expression feature matrix and the expression temporal feature vector are concatenated to obtain a fused feature matrix;

[0203] The fused feature matrix is ​​input into the behavior classification model to obtain the probability vector output by the behavior classification model; each component in the probability vector represents the probability of a behavior type; the behavior classification model is obtained by training the model based on the fused feature matrix samples and behavior type labels.

[0204] The behavior type corresponding to the largest component in the probability vector is determined as the user's target behavior type in the virtual scene.

[0205] Furthermore, the facial expression-based behavior analysis device is also used for:

[0206] Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene;

[0207] Based on the expression weight vector samples, determine the facial expression feature matrix samples and expression temporal feature vector samples;

[0208] The facial expression feature matrix sample and the expression temporal feature vector sample are concatenated to obtain a fused feature matrix sample;

[0209] The fused feature matrix samples are labeled with behavior types to obtain behavior type labels;

[0210] Based on the fused feature matrix samples and the behavior type labels, the second initial model is trained to obtain the behavior classification model.

[0211] Figure 3 is a schematic diagram of the structure of the electronic device provided in this disclosure. As shown in Figure 3, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a facial expression-based behavior analysis method. This method includes: acquiring multiple expression weight vectors generated by the user in a virtual scene; each expression weight vector represents a facial expression; each expression weight vector is composed of weights of multiple types of basic micro-expressions; extracting facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix; inputting the multiple expression weight vectors into a temporal feature extraction model to obtain a temporal feature vector of expressions output by the temporal feature extraction model; the temporal feature extraction model is obtained by training a model based on expression weight vector samples; and performing behavior type analysis based on the facial expression feature matrix and the temporal feature vector of expressions to obtain the target behavior type of the user in the virtual scene.

[0212] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0213] On the other hand, this disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the facial expression-based behavior analysis method provided in the above embodiments. The method includes: acquiring multiple expression weight vectors generated by a user in a virtual scene; each expression weight vector represents a facial expression; each expression weight vector is composed of weights of multiple types of basic micro-expressions; extracting facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix; inputting the multiple expression weight vectors into a temporal feature extraction model to obtain an expression temporal feature vector output by the temporal feature extraction model; the temporal feature extraction model is obtained by training a model based on expression weight vector samples; and performing behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene.

[0214] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the facial expression-based behavior analysis method provided in the above embodiments. The method includes: acquiring multiple expression weight vectors generated by a user in a virtual scene; each expression weight vector representing a facial expression; each expression weight vector consisting of weights of multiple types of basic micro-expressions; extracting facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix; inputting the multiple expression weight vectors into a temporal feature extraction model to obtain a temporal feature vector of expressions output by the temporal feature extraction model; the temporal feature extraction model being obtained through model training based on expression weight vector samples; and performing behavior type analysis based on the facial expression feature matrix and the temporal feature vector of expressions to obtain the target behavior type of the user in the virtual scene.

[0215] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure. Industrial applicability

[0218] This disclosure provides a method, apparatus, device, medium, and product for behavior analysis based on facial expressions, enabling a more comprehensive understanding of user behavior in virtual scenarios. The method combines static and dynamic features for automatic user behavior recognition, improving the accuracy and objectivity of facial expression-based behavior analysis. The feature vectors in this method comprehensively capture all aspects of basic micro-expressions, effectively expressing the details and features of facial expressions. The extraction model in this method strengthens the expression of key features, thereby improving the accuracy of facial expression feature depiction and providing more detailed data support for behavior analysis. The unsupervised learning model used in this method has better generalization ability and accuracy. Furthermore, the facial expression-based behavior analysis method provided in this disclosure can be subsequently applied to assist in disease diagnosis or treatment decisions, such as autism and other diseases related to user behavior patterns, and can also be applied to product application evaluation, advertising effectiveness evaluation, etc.

Claims

1. A behavior analysis method based on facial expressions, characterized in that, include: Obtain the weight vectors of multiple facial expressions generated by the user in a virtual scene; Each of the aforementioned expression weight vectors represents a facial expression; Each of the aforementioned expression weight vectors consists of weights for multiple types of basic micro-expressions; Facial expression features are extracted based on the multiple expression weight vectors to obtain a facial expression feature matrix; The multiple facial expression weight vectors are input into the temporal feature extraction model to obtain the facial expression temporal feature vector output by the temporal feature extraction model; The temporal feature extraction model is obtained by training the model based on facial expression weight vector samples; Based on the facial expression feature matrix and the expression temporal feature vector, behavior type analysis is performed to obtain the user's target behavior type in the virtual scene.

2. The facial expression-based behavior analysis method according to claim 1, characterized in that, The facial expression feature matrix is ​​composed of facial expression feature vectors of multiple types of basic micro-expressions. When extracting facial features based on the multiple expression weight vectors, the following steps are performed for multiple basic micro-expressions of the same type: The sampling time of each of the multiple basic micro-expressions is obtained; The basic micro-expressions with weights greater than zero among the multiple basic micro-expressions are identified as the target basic micro-expressions; The total duration of occurrence of the basic micro-expression of the type is obtained by summing the sampling times corresponding to multiple target basic micro-expressions; The consecutive sampling times corresponding to the sampling times of the multiple target basic micro-expressions are divided into target time periods; The earliest target time period is determined as the duration of the first appearance of the basic micro-expression of the aforementioned type; The number of multiple target time periods is determined as the total number of occurrences of the basic micro-expression of the aforementioned type; Divide the total occurrence duration by the total number of occurrences to obtain the average occurrence duration of the basic micro-expression of the type; The variance of the weights corresponding to the multiple target basic micro-expressions is calculated to obtain the weight variance of the type of basic micro-expression. The average weights of the various target basic micro-expressions are calculated to obtain the average weight of the basic micro-expressions of the type. The weights corresponding to the multiple target basic micro-expressions are summed to obtain the first weight sum of the basic micro-expressions of the type; Determine the second weighted sum for each other type of basic micro-expression; the other types of basic micro-expressions refer to basic micro-expressions of other types besides the stated type among the multiple types of basic micro-expressions; The first weighted sum is summed with multiple second weighted sums to obtain the third weighted sum; Divide the first weighted sum by the third weighted sum to obtain the weight percentage of the basic micro-expression type; Based on the total occurrence duration, first occurrence duration, average occurrence duration, weight variance, weight mean, and weight percentage of the basic micro-expression of the type, a facial expression feature vector of the basic micro-expression of the type is constructed.

3. The facial expression-based behavior analysis method according to claim 1, characterized in that, The temporal feature extraction model includes a first hidden layer, a second hidden layer, a splicing layer, an attention layer, and an output layer; The step of inputting the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model includes: The multiple facial expression weight vectors are input into the first hidden layer to obtain the first temporal feature vector output by the first hidden layer; The first temporal feature vector is input into the second hidden layer to obtain the second temporal feature vector output by the second hidden layer. The first temporal feature vector and the second temporal feature vector are input into the splicing layer to obtain the splicing feature matrix output by the splicing layer; The concatenated feature matrix is ​​input into the attention layer to obtain the attention vector output by the attention layer; The first temporal feature vector and the attention vector are multiplied by the attention layer to obtain the output result, and the output result is determined as the second temporal feature vector containing attention information output by the second hidden layer. The second temporal feature vector containing attention information is input into the output layer to obtain the facial expression temporal feature vector output by the output layer.

4. The facial expression-based behavior analysis method according to claim 3, characterized in that, The temporal feature extraction model was trained in the following manner: Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene; Based on the expression weight vector samples, unsupervised learning is performed on the first initial model to obtain the temporal feature extraction model.

5. The facial expression-based behavior analysis method according to claim 1, characterized in that, The behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene includes: The facial expression feature matrix and the expression temporal feature vector are concatenated to obtain a fused feature matrix; The fused feature matrix is ​​input into the behavior classification model to obtain the probability vector output by the behavior classification model; each component in the probability vector represents the probability of a behavior type; the behavior classification model is obtained by training the model based on the fused feature matrix samples and behavior type labels. The behavior type corresponding to the largest component in the probability vector is determined as the user's target behavior type in the virtual scene.

6. The facial expression-based behavior analysis method according to claim 5, characterized in that, The behavior classification model was trained in the following way: Obtain facial expression weight vector samples generated by multiple subjects in the virtual scene; Based on the expression weight vector samples, determine the facial expression feature matrix samples and expression temporal feature vector samples; The facial expression feature matrix sample and the expression temporal feature vector sample are concatenated to obtain a fused feature matrix sample; The fused feature matrix samples are labeled with behavior types to obtain behavior type labels; Based on the fused feature matrix samples and the behavior type labels, the second initial model is trained to obtain the behavior classification model.

7. A behavior analysis device based on facial expressions, characterized in that, include: The acquisition module is used to acquire multiple facial expression weight vectors generated by the user in a virtual scene; Any of the aforementioned rights to expression A repetitive vector represents a facial expression; Each of the aforementioned expression weight vectors consists of weights for multiple types of basic micro-expressions; The first feature extraction module is used to extract facial expression features based on the multiple expression weight vectors to obtain a facial expression feature matrix. The second feature extraction module is used to input the multiple expression weight vectors into the temporal feature extraction model to obtain the expression temporal feature vector output by the temporal feature extraction model; The temporal feature extraction model is obtained by training the model based on facial expression weight vector samples; The behavior analysis module is used to perform behavior type analysis based on the facial expression feature matrix and the expression temporal feature vector to obtain the user's target behavior type in the virtual scene.

8. An apparatus comprising an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the facial expression-based behavior analysis method as described in any one of claims 1 to 6.

9. A medium comprising a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the facial expression-based behavior analysis method as described in any one of claims 1 to 6.

10. A product comprising a computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the facial expression-based behavior analysis method as described in any one of claims 1 to 6.

Images