Video data processing method, device and computer equipment based on micro-expression

Abstract

The present application relates to artificial intelligence, and provides a video data processing method, device, computer equipment and storage medium based on micro-expressions. The method includes: receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image; inputting the first face image and the second face image into a trained micro-expression recognition model respectively, obtaining a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image; determining a target satisfaction level according to the first micro-expression, determining a target service level according to the second micro-expression, sending the target satisfaction level and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal. The use of this method can save video traffic resources and improve efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video data processing, device, computer equipment and storage medium based on micro-expression. Background Art

[0002] With the development of the Internet, more and more people are enjoying the convenience brought by the Internet. For example, video calls enable people in different places to communicate face to face. For example, making friends and trading through video. At present, when trading or promoting through video calls, it usually takes a lot of time to communicate through video, and even offline visits are required to satisfy customers and complete transactions or promotions. However, a large number of video calls will waste a lot of video traffic resources and be inefficient. Summary of the invention

[0003] Based on this, it is necessary to provide a video data processing method, device, computer equipment and storage medium based on micro-expressions that can save video traffic resources and improve efficiency in response to the above technical problems.

[0004] A method for processing video data based on micro-expressions, the method comprising:

[0005] Receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image;

[0006] Inputting the first facial image and the second facial image into a trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image;

[0007] A target satisfaction is determined according to the first micro-expression, a target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal.

[0008] In one of the embodiments, the receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image includes:

[0009] capturing video pictures from the first video image and the second video image at a preset time interval;

[0010] The face portion of the video image is extracted to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0011] In one of the embodiments, after receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image, the method further includes:

[0012] Acquire a target facial image, and respectively calculate similarities between the first facial image and the second facial image and the target facial image;

[0013] When the similarity between the first face image and the target face image is greater than a first preset threshold, obtaining a first target face image corresponding to the first face image;

[0014] When the similarity between the second facial image and the target facial image is greater than a second preset threshold, a second target facial image corresponding to the second facial image is obtained.

[0015] In one embodiment, respectively calculating the similarity between the first face image and the second face image and the target face image includes:

[0016] Determine facial skin color feature information of the first facial image and the second facial image;

[0017] Dividing the first face image and the second face image respectively to obtain local face blocks of the first face image and the second face image, and determining contour feature information, texture feature information, and area feature information of the local face blocks;

[0018] The similarities between the first face image and the second face image and the target face image are calculated based on the face skin color feature information, the contour feature information, the texture feature information and the area feature information of the first face image and the second face image.

[0019] In one embodiment, the training step of the micro-expression recognition model includes:

[0020] Obtain historical video images and micro-expression labels corresponding to the historical video images to obtain an initial sample set;

[0021] Randomly sampling with replacement from the initial sample set to obtain a target training set;

[0022] Obtaining a corresponding historical micro-expression feature set according to the target training set, and randomly selecting micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set;

[0023] Calculate segmentation expression features according to the target micro-expression feature set, use the segmentation expression features to binary divide the target training set to obtain sub-training sets, and use the sub-training sets as the target training set;

[0024] Returning the step of obtaining a corresponding historical micro-expression feature set according to the target training set, randomly selecting micro-expression features from the historical micro-expression feature set, and obtaining a target micro-expression feature set, and when a preset condition is met, obtaining a target decision tree;

[0025] Return to the step of randomly sampling with replacement from the initial sample set to obtain a target training set, and when a preset number of target decision trees is reached, the micro-expression recognition model is obtained.

[0026] In one embodiment, determining a target satisfaction level according to the first micro-expression and determining a target service level according to the second micro-expression includes:

[0027] Receiving first voice information and second voice information, obtaining first text information corresponding to the first voice information, and obtaining second text information according to the second voice information;

[0028] Extracting keywords from the first text information and the second file information to obtain a first keyword corresponding to the first text information and a second keyword corresponding to the second file information;

[0029] When the first keyword is consistent with the first preset keyword, the target satisfaction is determined according to the first keyword and the first micro-expression; when the second keyword is consistent with the second preset keyword, the target service level is determined according to the second keyword and the second micro-expression.

[0030] In one embodiment, sending the target satisfaction level and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal includes:

[0031] Determine a loading position of the target satisfaction level in the first video image, load the target satisfaction level into the first video image according to the loading position, obtain a first target video image, and send the first target video image to the second terminal;

[0032] Determine a loading position of the target service level in the second video image, load the target service level into the second video image according to the loading position, obtain a second target video image, and send the second target video image to the first terminal.

[0033] A video data processing device based on micro-expressions, the device comprising:

[0034] A face detection module, configured to receive a first video image sent by a first terminal and a second video image sent by a second terminal, perform face detection on the first video image and the second video image, and obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image;

[0035] A micro-expression recognition module, used to input the first facial image and the second facial image into a trained micro-expression recognition model, respectively, to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image;

[0036] The recognition result processing module is used to determine the target satisfaction according to the first micro-expression, determine the target service level according to the second micro-expression, send the target satisfaction and the first video image to the second terminal, and send the target service level and the second video image to the first terminal.

[0037] In one embodiment, the device further comprises:

[0038] A similarity calculation module, used to calculate the similarity between the first face image and the second face image and the target face image respectively;

[0039] A first target facial image determination module, configured to obtain a first target facial image corresponding to the first facial image when the similarity between the first facial image and the target facial image is greater than a first preset threshold;

[0040] A second target facial image determination module is used to obtain a second target facial image corresponding to the second facial image when the similarity between the second facial image and the target facial image is greater than a second preset threshold. A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0041] Receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image;

[0042] Inputting the first facial image and the second facial image into a trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image;

[0043] A target satisfaction is determined according to the first micro-expression, a target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0045] Receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image;

[0046] Inputting the first facial image and the second facial image into a trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image;

[0047] A target satisfaction is determined according to the first micro-expression, a target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal.

[0048] The above-mentioned video data processing method, device, computer equipment and storage medium based on micro-expressions, by receiving a first video image sent by a first terminal and a second video image sent by a second terminal, perform face detection on the first video image and the second video image, and obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image. The first face image and the second face image are respectively input into the trained micro-expression recognition model to obtain a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image. The target satisfaction is determined according to the first micro-expression, the target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal, so that both parties of the video can adjust the communication in time according to the content displayed by the terminal, reduce the video call time, save video traffic resources, and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is an application scenario diagram of a video data processing method based on micro-expressions in one embodiment;

[0050] Figure 2 1 is a flow chart of a method for processing video data based on micro-expressions in one embodiment;

[0051] Figure 3 A schematic diagram of a process for obtaining a face image in one embodiment;

[0052] Figure 4 A schematic diagram of a process for determining a target face image in one embodiment;

[0053] Figure 5 A schematic diagram of a process for calculating similarity in one embodiment;

[0054] Figure 6 A schematic diagram of a process for obtaining a micro-expression recognition model in one embodiment;

[0055] Figure 7 A schematic diagram of a process for determining a target satisfaction level and a target service level in one embodiment;

[0056] Figure 8 A schematic diagram of a process for obtaining a target video image in one embodiment;

[0057] Fig. 9 is a structural block diagram of a video data processing device based on micro-expressions in one embodiment;

[0058] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The video data processing method based on micro-expression provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network, and the terminal 106 communicates with the server 104 through the network. The server 104 receives the first video image sent by the first terminal 102 and the second video image sent by the second terminal 106, performs face detection on the first video image and the second video image, and obtains the first face image corresponding to the first video image and the second face image corresponding to the second video image; the server 104 inputs the first face image and the second face image into the trained micro-expression recognition model respectively, and obtains the first micro-expression corresponding to the first face image and the second micro-expression corresponding to the second face image; the server 104 determines the target satisfaction according to the first micro-expression, determines the target service level according to the second micro-expression, sends the target satisfaction and the first video image to the second terminal 106, and sends the target service level and the second video image to the first terminal 102. Among them, the terminal 102 and the terminal 106 can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0061] In one embodiment, Figure 2 As shown, a method for processing video data based on micro-expressions is provided, and the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0062] S202, receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0063] Among them, the first terminal refers to the user terminal, and the second terminal refers to the salesperson terminal. The user terminal establishes a video call with the salesperson terminal through the server. The user terminal collects the user video image through the video acquisition device to obtain the first video image, and the salesperson terminal collects the salesperson video image through the video acquisition device to obtain the second video image.

[0064] Specifically, after the server establishes a video call between the first terminal and the second terminal, it receives a first video image sent by the first terminal and a second video image sent by the second terminal, and can perform face detection on the received first video image and second video image at a preset time interval to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0065] S204, inputting the first face image and the second face image into the trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image.

[0066] Among them, the micro-expression recognition model is obtained by a large number of trainings based on historical face data and corresponding micro-expressions using machine learning algorithms, where the machine learning algorithms can be deep neural network algorithms, random forest algorithms, etc. Micro-expressions are short and involuntary rapid facial expressions that trigger hidden real emotions. The standard duration of micro-expressions is between 1 / 5 and 1 / 25.

[0067] Specifically, the server inputs the first facial image into the trained micro-expression recognition model to calculate a first micro-expression corresponding to the first facial image, and inputs the second facial image into the trained micro-expression recognition model to calculate a second micro-expression corresponding to the second facial image.

[0068] S206, determining a target satisfaction level according to the first micro-expression, determining a target service level according to the second micro-expression, sending the target satisfaction level and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal.

[0069] Among them, the target satisfaction refers to the satisfaction of the first facial image obtained according to the first micro-expression, which is used to reflect the satisfaction of the user corresponding to the first facial image. The correspondence between micro-expressions and satisfaction is preset. For example, the satisfaction corresponding to the happy micro-expression is 80%. The satisfaction of the disgusted micro-expression is 20%. The target service level refers to the service level of the second facial image obtained according to the second micro-expression, which is used to reflect the service attitude of the salesperson corresponding to the second facial image. The correspondence between micro-expressions and service levels is preset, wherein the service levels can include poor, general, good, and excellent. For example, the service level corresponding to a sincere micro-expression is excellent, and the service level corresponding to a negative micro-expression is poor.

[0070] Specifically, the server determines the corresponding target satisfaction level according to the first micro-expression with a preset corresponding relationship, determines the corresponding target service level according to the second micro-expression, sends the target satisfaction level and the first video image to the second terminal, and the second terminal displays the target satisfaction level and the first video image in association after receiving them. The server sends the target service level and the second video image to the first terminal, and the first terminal displays the target service level and the second video image in association after receiving them.

[0071] The above-mentioned video data processing method based on micro-expressions receives a first video image sent by a first terminal and a second video image sent by a second terminal, performs face detection on the first video image and the second video image, and obtains a first face image corresponding to the first video image and a second face image corresponding to the second video image. The first face image and the second face image are respectively input into the trained micro-expression recognition model to obtain a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image. The target satisfaction is determined according to the first micro-expression, the target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal. The user can adjust the communication or stop the video communication according to the service level displayed on the first terminal, and the business personnel can adjust the communication according to the target satisfaction displayed on the second terminal, so that both parties of the video can adjust the communication in time according to the content displayed on the terminal, reduce the video call time, save video traffic resources, and improve efficiency.

[0072] In one embodiment, when the video call ends, the time proportions of the target satisfaction and target service level during the video call are counted, and the target satisfaction and target service level with the highest time proportions are used as the final satisfaction and service level of the video call. The final satisfaction and service level are sent to the corresponding terminals for display, and the final satisfaction and service level are saved for subsequent use.

[0073] In one embodiment, Figure 3 As shown, step S202, i.e., receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image, includes the steps of:

[0074] S302: capturing video pictures from the first video image and the second video image at a preset time interval.

[0075] Specifically, the server can intercept video frames from the first video image to obtain a video picture corresponding to the first video image, and intercept video frames from the second video image to obtain a video picture corresponding to the second video image according to a preset time interval, wherein the preset time interval can be 5 seconds.

[0076] S304, extracting the face part in the video image to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0077] Specifically, the server can use a face detection algorithm to extract the face part of the video picture corresponding to the first video image, and obtain a first face image corresponding to the first video image. Use the face detection algorithm to extract the face part of the video picture corresponding to the second video image, and obtain a second face image corresponding to the second video image. Among them, the face detection algorithm can use the AdaBoost algorithm. Adaboost is an iterative algorithm. Its core idea is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier). When performing detection, the face haar integral map features are calculated, and then the trained final classifier is used for recognition to obtain the recognition result.

[0078] In the above embodiment, video pictures are captured from the first video image and the second video image at preset time intervals, and the facial part in the video picture is extracted to obtain a first facial image corresponding to the first video image and a second facial image corresponding to the second video image. The facial part of the video image is obtained through face detection, which facilitates subsequent facial micro-expression recognition and improves efficiency.

[0079] In one embodiment, Figure 4 As shown, step S202, i.e., receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image, further includes the steps of:

[0080] S402, obtaining a target facial image, and respectively calculating similarities between the first facial image and the second facial image and the target facial image.

[0081] The target face image refers to a face image that is collected in advance and has a face identifier, and the target face image is stored in the database. For example, the target face image can be a face image of a person named "Zhang San" and so on.

[0082] Specifically, the server sequentially obtains the stored target face images from the face database, calculates the similarity between the target face image and the first face image, and the similarity between the target face image and the second face image, and performs face recognition.

[0083] S404: When the similarity between the first facial image and the target facial image is greater than a first preset threshold, a first target facial image corresponding to the first facial image is obtained.

[0084] S406: When the similarity between the second facial image and the target facial image is greater than a second preset threshold, a second target facial image corresponding to the second facial image is obtained.

[0085] The first preset threshold is a preset threshold of similarity between the first face image and the target face image. The second preset threshold is a preset threshold of similarity between the second face image and the target face image. The second target face image refers to the recognition result image corresponding to the first face image. The second target face image is the recognition result image corresponding to the second face image.

[0086] Specifically, when the similarity between the target face image and the first face image is greater than a first preset threshold, a recognition result corresponding to the first face image is obtained, and the target face image with a similarity greater than a second preset threshold is used as the first target face image. When the similarity between the target face image and the second face image is greater than a preset threshold, a recognition result corresponding to the second face image is obtained, and the target face image with a similarity greater than the second preset threshold is used as the second target face image. For example, when making a video call, facial recognition can be performed on both parties in the video to determine the corresponding facial images of both parties in the video in the facial database, and the corresponding identifiers of both parties in the video can be obtained based on the recognized facial images.

[0087] In the above embodiment, by acquiring the target facial image, the similarities between the first facial image and the second facial image and the target facial image are calculated respectively. When the similarity between the first facial image and the target facial image is greater than a first preset threshold, a first target facial image corresponding to the first facial image is obtained; when the similarity between the second facial image and the target facial image is greater than a second preset threshold, a second target facial image corresponding to the second facial image is obtained. Facial recognition can be performed on facial images to obtain facial recognition results, thereby improving convenience of use.

[0088] In one embodiment, Figure 5 As shown, step S402, respectively calculating the similarity between the first face image and the second face image and the target face image, includes the steps of:

[0089] S502: Determine facial skin color feature information of the first facial image and the second facial image.

[0090] Specifically, facial skin color feature information corresponding to the first facial image and facial skin color feature information corresponding to the second facial image are determined based on the pixels of the first facial image and the pixels of the second facial image. The facial skin color feature information of the first facial image and the second facial image may also be determined using a skin color model, such as a Gaussian model, a mixed Gaussian model, and a histogram model.

[0091] S504, dividing the first face image and the second face image to obtain local face blocks of the first face image and the second face image, and determining contour feature information, texture feature information and area feature information of the local face blocks.

[0092] Specifically, the server divides the first face image and the second face image into a plurality of interconnected local areas, and determines the contour feature information, texture feature information, and area feature information of each local area. For example, the face image can be divided into different local blocks such as the forehead area, the eyebrow area, the eye area, the nose area, and the mouth area, and the contour feature information, texture feature information, and area feature information of each local block are calculated respectively.

[0093] S506, calculating the similarity between the first face image and the second face image and the target face image based on the face skin color feature information, contour feature information, texture feature information and area feature information of the first face image and the second face image.

[0094] Specifically, the server pre-calculates and obtains the skin color feature information, contour feature information, texture feature information, and area feature information of each local block of the target face image. The feature information of each local area in the target face image is compared with the facial skin color feature information of the first face image and the second face image, and the contour feature information, texture feature information, and area feature information of each local block to obtain the similarity of each local block. Based on the similarity of each local block, the similarity between the first face image and the second face image and the target face image is obtained.

[0095] In the above embodiment, by determining the facial skin color feature information of the first facial image and the second facial image, the first facial image and the second facial image are divided respectively to obtain local facial blocks of the first facial image and the second facial image, and the contour feature information, texture feature information and area feature information of the local facial blocks are determined. Based on the facial skin color feature information, contour feature information, texture feature information and area feature information of the first facial image and the second facial image, the similarity between the first facial image and the second facial image and the target facial image is calculated respectively. By dividing the facial image and calculating the similarity of each local block to obtain the final similarity, the accuracy of the obtained similarity can be improved.

[0096] In one embodiment, Figure 6 As shown, the training steps of the micro-expression recognition model include the following steps:

[0097] S602, obtaining historical video images and micro-expression labels corresponding to the historical video images to obtain an initial sample set.

[0098] Specifically, the server obtains historical video images, uses micro-expressions corresponding to video frames at preset time intervals in the historical video images as labels, obtains the micro-expression labels, and obtains an initial sample set based on the video frames in the historical video images and the corresponding micro-expression labels. The sample set includes the video frames of the historical video images and the micro-expression labels corresponding to the video frames.

[0099] S604, randomly sample with replacement from the initial sample set to obtain a target training set.

[0100] S606, obtaining a corresponding historical micro-expression feature set according to the target training set, and randomly selecting micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set.

[0101] Among them, the target training set is a sample set in which a preset number of samples are randomly selected from the initial sample set with replacement, or a random number of samples can be selected. The historical micro-expression feature set refers to the micro-expression feature set of historical video frames in the target training set. Among them, the micro-expression features include upper facial features, lower facial features, mouth features and other features, etc. In a specific embodiment, the micro-expression features may include AU (basic deformation unit), such as: inner corner of the eyebrow raised, outer corner of the eyebrow raised, upper eyelid raised, upper lip raised, chin sunken and nostril contracted, etc.

[0102] Specifically, the server randomly samples with replacement from the initial sample set, takes the randomly selected video frames of the historical video images as the target training set, randomly selects micro-expression features from the target training set to obtain a target micro-expression feature set, and randomly selects some micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set.

[0103] S608, calculating the partitioning expression features according to the target micro-expression feature set, using the partitioning expression features to binary-partition the target training set to obtain sub-training sets, and using each sub-training set as the target training set.

[0104] Among them, dividing the expression features refers to using the Gini impurity algorithm to calculate the Gini impurity of each micro-expression feature in the target micro-expression feature set to obtain the optimal micro-expression feature for dividing the target training set.

[0105] Specifically, the server uses the Gini index to select the optimal partitioning expression feature according to the target micro-expression feature set, and uses the partitioning expression feature to binary-partition the video frames in the target training set to obtain left and right training sets, and the left and right sub-training sets are respectively used as target training sets. When the target training set is the first node, the target training set is the root node, and the information corresponding to the partitioning expression feature is obtained. The target training set is binary-partitioned according to the partitioning expression feature information to obtain left and right training sets. For example, if the partitioning feature obtained is the inner corner of the eyebrow, the video frame with the inner corner of the eyebrow raised is divided into the left node to obtain the left node training set, and the video frame with the inner corner of the eyebrow not raised is divided into the right node to obtain the right node training set, and the left node training set and the right node training set are respectively used as the target training set.

[0106] S610, determine whether the preset condition is met. If the preset condition is not met, return to step S606 to continue execution.

[0107] S612, when the preset conditions are met, a target decision tree is obtained.

[0108] The preset condition refers to that the micro-expression labels of the samples in the target training set are the same, that is, the micro-expression labels corresponding to each video frame in the target training set are the same, for example, the micro-expression labels corresponding to all video frames are happy.

[0109] Specifically, when the divided target training set is obtained, the server determines whether each target training set has met the preset conditions, that is, whether the micro-expression labels corresponding to the video frames in each divided target training set are the same. When the preset conditions are not met, that is, the micro-expression labels corresponding to the video frames in the target training set are not the same, then return to step S606 for execution, that is, continue to divide. Until the micro-expression labels in the target training set are the same, that is, the micro-expression labels in each node training set are the same, the target decision tree is obtained. For example, the micro-expression labels in the obtained left node training set are all surprised, and at this time the left node training set meets the preset conditions. If the micro-expression labels in the right node training set are different, then the right node training set does not meet the preset conditions. At this time, the right node training set is used as the target training set, and returns to step S606 for execution until the right node training set meets the preset conditions. An expression decision tree is obtained. When the micro-expression labels in the left node training set and the right node training set are different, the left node training set and the right node training set are respectively used as the target training set, and the process returns to step S606 for execution until each sub-training set after division meets the preset conditions, and a decision tree is obtained. When the micro-expression labels in the left node training set are different and the micro-expression labels in the right node training set are the same, the left node training set is used as the target training set for further division until each sub-training set after division meets the preset conditions, and a decision tree is obtained.

[0110] S614, determining whether the preset number of target decision trees has been reached. If the preset number of target decision trees has not been reached, the process returns to step S604 and continues to execute.

[0111] S616, when the preset number of target decision trees is reached, a micro-expression recognition model is obtained.

[0112] Specifically, when a target decision tree is obtained, at this time, the server determines whether the number of target decision trees reaches a preset number, such as 100. When the number of target decision trees reaches the preset number, a micro-expression random forest is obtained, and the micro-expression random forest is used as a micro-expression recognition model. When the number of target decision trees does not reach the preset number, return to step S604 for execution until the number of target decision trees obtained reaches the preset number.

[0113] In the above embodiment, the initial sample set is obtained by obtaining the historical video image and the micro-expression label corresponding to the historical video image, and the target training set is obtained by randomly sampling with replacement from the initial sample set. The corresponding historical micro-expression feature set is obtained according to the target training set, and the micro-expression feature is randomly selected from the historical micro-expression feature set to obtain the target micro-expression feature set. The partitioning expression feature is calculated according to the target micro-expression feature set, and the target training set is binary divided using the partitioning expression feature to obtain each sub-training set, and each sub-training set is used as the target training set. Return to the step of obtaining the corresponding historical micro-expression feature set according to the target training set, and obtain the target micro-expression feature set according to the historical micro-expression feature set. When the preset conditions are met, the target decision tree is obtained. Return to the step of randomly sampling with replacement from the initial sample set to obtain the target training set. When the preset number of target decision trees is reached, a random forest is formed to obtain a micro-expression recognition model. The micro-expression recognition model can be trained in advance. When performing micro-expression recognition, the trained micro-expression recognition model can be directly used to improve the efficiency of micro-expression recognition.

[0114] In one embodiment, Figure 7 As shown, step S206, i.e. determining the target satisfaction level according to the first micro-expression and determining the target service level according to the second micro-expression, includes the following steps:

[0115] S702, receiving first voice information and second voice information, obtaining corresponding first text information according to the first voice information, and obtaining second text information according to the second voice information.

[0116] The first voice information is the voice information collected by the first terminal, and the second voice information is the voice information collected by the second terminal.

[0117] Specifically, during the video call, the server simultaneously receives a first voice message sent by the first terminal and a second voice message sent by the second terminal, converts the first voice message into a text message, and converts the first voice message into a second text message.

[0118] S704: extract keywords from the first text information and the second file information to obtain a first keyword corresponding to the first text information and a second keyword corresponding to the second file information.

[0119] Specifically, a keyword extraction algorithm is used to extract keywords from the first text information and the second file information to obtain a first keyword corresponding to the first text information and a second keyword corresponding to the second file information. The keyword extraction algorithm may use LDA (Latent Dirichlet Allocation, document topic generation model), TextRank (a graph-based sorting algorithm for text), etc.

[0120] In one embodiment, the server segments the obtained text information and filters the segmentation results, establishes a candidate keyword graph based on the words obtained after filtering, and obtains preset initial weights of word nodes in the candidate keyword graph; it iterates the candidate keyword graph in a loop until a preset condition is met, obtains word node weights, sorts the word node weights in reverse order, and obtains a preset number of words as keywords based on the sorting results.

[0121] In one embodiment, the server performs word segmentation on the obtained text information and filters the word segmentation results, calculates the probability of each preset topic based on the words obtained after filtering, calculates the word classification corresponding to each preset topic based on the probability of each preset topic; calculates the probability of the word classification for each preset topic, and obtains keywords based on the probability of the word classification for each preset topic.

[0122] S706, when the first keyword is consistent with the first preset keyword, determine the target satisfaction level according to the first keyword and the first micro-expression; when the second keyword is consistent with the second preset keyword, determine the target service level according to the second keyword and the second micro-expression.

[0123] The first preset keyword refers to a keyword preset for a user, and the second preset keyword refers to a keyword preset for a salesperson.

[0124] Specifically, the server can match the first keyword in the pre-set first preset keyword, and when a consistent preset keyword is matched, the target satisfaction is determined according to the first keyword and the obtained first micro-expression. When the second keyword is consistent with the second preset keyword, the target service level is determined according to the second keyword and the second micro-expression.

[0125] In the above embodiment, by receiving the first voice information and the second voice information, the corresponding first text information is obtained according to the first voice information, the second text information is obtained according to the second voice information, and the keywords in the first text information and the second file information are extracted to obtain the first keyword corresponding to the first text information and the second keyword corresponding to the second file information. When the first keyword is consistent with the first preset keyword, the target satisfaction is determined according to the first keyword and the first micro-expression. When the second keyword is consistent with the second preset keyword, the target service level is determined according to the second keyword and the second micro-expression. By determining the satisfaction and service level according to the keywords and grievances obtained from the voice information, the accuracy of the determined satisfaction and service level is further improved.

[0126] In one embodiment, Figure 8 As shown, step SS206, that is, sending the target satisfaction and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal, includes the steps of:

[0127] S802, determining a loading position of the target satisfaction level in the first video image, loading the target satisfaction level into the first video image according to the loading position, obtaining a first target video image, and sending the first target video image to a second terminal.

[0128] Specifically, the server obtains the preset loading position of the target satisfaction in the first video image, synthesizes the target satisfaction and the first video image according to the loading file, obtains the first target video image, and sends the first target video image to the second terminal for display. For example, the loading position may be the upper left corner of the video image, etc.

[0129] S804, determining a loading position of the target service level in the second video image, loading the target service level into the second video image according to the loading position, obtaining a second target video image, and sending the second target video image to the first terminal.

[0130] Specifically, the server obtains a preset loading position of the target service level in the second video image, synthesizes the target service level and the second video image according to the loading position to obtain a second target video image, and sends the second target video image to the first terminal for display.

[0131] In the above embodiment, by determining the loading position of the target satisfaction in the first video image, loading the target satisfaction into the first video image according to the loading position to obtain the first target video image, sending the first target video image to the second terminal, determining the loading position of the target service level in the second video image, loading the target service level into the second video image according to the loading position to obtain the second target video image, and sending the second target video image to the first terminal, the satisfaction and service level can be loaded into the video image according to the loading position for display, thereby improving the user experience.

[0132] In a specific embodiment, an insurance salesperson promotes insurance products to a customer through video. At this time, the insurance salesperson initiates a video call request to the customer terminal corresponding to the customer through the service terminal. The customer accepts the video call request through the customer terminal, and the service terminal establishes a video connection with the customer terminal through the server. Then, the insurance business source and the customer make a video call. The server receives the customer video image and the salesperson video image, captures the video frame every 5 seconds, performs face detection on the captured video frame, obtains the customer face image and the salesperson face image, inputs the customer face image and the salesperson face image into the trained micro-expression recognition model, obtains the customer micro-expression and the salesperson micro-expression, obtains the corresponding customer satisfaction according to the customer micro-expression, obtains the salesperson service level according to the salesperson micro-expression, displays the customer satisfaction and the customer video image to the service terminal to which the customer video image is sent, and the salesperson can decide to promote the product or change the product according to the displayed customer satisfaction. The salesperson service level and the salesperson video image are sent to the customer terminal for display, and the customer can decide whether to continue communication or change the salesperson according to the displayed salesperson service level, which can save video traffic resources, improve the efficiency of business achievement, and enhance the user experience.

[0133] It should be understood that although Figure 2-8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-8 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0134] In one embodiment, Fig. 9As shown, a video data device 900 based on micro-expressions is provided, comprising: a face detection module 902, a micro-expression recognition module 904 and a recognition result processing module 906, wherein:

[0135] The face detection module 902 is used to receive a first video image sent by a first terminal and a second video image sent by a second terminal, perform face detection on the first video image and the second video image, and obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0136] The micro-expression recognition module 904 is used to input the first face image and the second face image into the trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image.

[0137] The recognition result processing module 906 is used to determine the target satisfaction according to the first micro-expression, determine the target service level according to the second micro-expression, send the target satisfaction and the first video image to the second terminal, and send the target service level and the second video image to the first terminal.

[0138] In one embodiment, the face detection module 902 includes:

[0139] A video capture module, used to capture video images from the first video image and the second video image at a preset time interval;

[0140] The face extraction module is used to extract the face part in the video image to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0141] In one embodiment, the micro-expression based video data device 900 further includes:

[0142] A similarity calculation module is used to obtain a target face image and calculate the similarity between the first face image and the second face image and the target face image respectively;

[0143] A first target facial image determination module, configured to obtain a first target facial image corresponding to the first facial image when the similarity between the first facial image and the target facial image is greater than a first preset threshold;

[0144] The second target facial image determination module is used to obtain a second target facial image corresponding to the second facial image when the similarity between the second facial image and the target facial image is greater than a second preset threshold.

[0145] In one embodiment, the similarity calculation module includes:

[0146] A skin color feature determination module, used to determine facial skin color feature information of the first facial image and the second facial image;

[0147] A segmentation module, used for segmenting the first face image and the second face image to obtain partial face blocks, and determining contour feature information, texture feature information and area feature information of the partial face blocks;

[0148] The calculation module is used to calculate the similarity with the target face image based on the face skin color feature information, contour feature information, texture feature information and area feature information.

[0149] In one embodiment, the video data device 900 based on micro-expressions includes:

[0150] A sample acquisition module is used to obtain historical video images and micro-expression labels corresponding to the historical video images to obtain an initial sample set;

[0151] The training set acquisition module is used to randomly sample with replacement from the initial sample set to obtain the target training set;

[0152] A feature set acquisition module is used to obtain a corresponding historical micro-expression feature set according to a target training set, and randomly select micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set;

[0153] A training set partitioning module is used to calculate partitioning expression features according to a target micro-expression feature set, and use the partitioning expression features to binary partition the target training set to obtain sub-training sets, and use each sub-training set as a target training set;

[0154] A decision tree obtaining module is used to return the historical micro-expression feature set corresponding to the target training set, randomly select micro-expression features from the historical micro-expression feature set, and obtain the step of obtaining the target micro-expression feature set. When the preset conditions are met, a target decision tree is obtained;

[0155] The model acquisition module is used to return to the step of obtaining the target training set by randomly sampling with replacement from the initial sample set. When a preset number of target decision trees is reached, a micro-expression recognition model is obtained.

[0156] In one embodiment, the recognition result processing module 906 includes:

[0157] A voice receiving module, used for receiving a first voice message and a second voice message, obtaining a corresponding first text message according to the first voice message, and obtaining a second text message according to the second voice message;

[0158] A keyword extraction module, used to extract keywords from the first text information and the second file information, and obtain a first keyword corresponding to the first text information and a second keyword corresponding to the second file information;

[0159] The data processing module is used to determine the target satisfaction level according to the first keyword and the first micro-expression when the first keyword is consistent with the first preset keyword, and to determine the target service level according to the second keyword and the second micro-expression when the second keyword is consistent with the second preset keyword.

[0160] In one embodiment, the recognition result processing module 906 includes:

[0161] A first target video image obtaining module, used for determining a loading position of the target satisfaction in the first video image, loading the target satisfaction into the first video image according to the loading position, obtaining the first target video image, and sending the first target video image to the second terminal;

[0162] The second target video image obtaining module is used to determine the loading position of the target service level in the second video image, load the target service level into the second video image according to the loading position, obtain the second target video image, and send the second target video image to the first terminal.

[0163] For the specific definition of the video data device based on micro-expressions, please refer to the definition of the video data method based on micro-expressions above, which will not be repeated here. Each module in the above-mentioned video data device based on micro-expressions can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0164] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video-related data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video data processing method based on micro-expressions is implemented.

[0165] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image. The first face image and the second face image are respectively input into a trained micro-expression recognition model to obtain a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image. The target satisfaction is determined according to the first micro-expression, the target service level is determined according to the second micro-expression, the target satisfaction and the first video image are sent to the second terminal, and the target service level and the second video image are sent to the first terminal.

[0167] In one embodiment, when the processor executes the computer program, the following steps are also implemented: capturing video pictures from the first video image and the second video image at preset time intervals; extracting the face part in the video picture to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0168] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a target facial image, and respectively calculating the similarity between the first facial image and the second facial image and the target facial image; when the similarity between the first facial image and the target facial image is greater than a first preset threshold, obtaining a first target facial image corresponding to the first facial image; when the similarity between the second facial image and the target facial image is greater than a second preset threshold, obtaining a second target facial image corresponding to the second facial image.

[0169] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining facial skin color feature information of the first facial image and the second facial image; dividing the first facial image and the second facial image to obtain local facial blocks, and determining contour feature information, texture feature information, and area feature information of the local facial blocks; and calculating the similarity with the target facial image based on the facial skin color feature information, contour feature information, texture feature information, and area feature information.

[0170] In one embodiment, the processor further implements the following steps when executing the computer program: obtaining historical video images and micro-expression labels corresponding to the historical video images to obtain an initial sample set; randomly sampling with replacement from the initial sample set to obtain a target training set; obtaining a corresponding historical micro-expression feature set based on the target training set, randomly selecting micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set; calculating partitioning expression features based on the target micro-expression feature set, using the partitioning expression features to binary divide the target training set to obtain each sub-training set, and using each sub-training set as the target training set; returning to the step of obtaining the corresponding historical micro-expression feature set based on the target training set, randomly selecting micro-expression features from the historical micro-expression feature set to obtain the target micro-expression feature set, and when a preset condition is met, a target decision tree is obtained; returning to the step of randomly sampling with replacement from the initial sample set to obtain the target training set, and when a preset number of target decision trees is reached, a micro-expression recognition model is obtained.

[0171] In one embodiment, when the processor executes the computer program, the following steps are also implemented: receiving a first voice message and a second voice message, obtaining corresponding first text message according to the first voice message, and obtaining second text message according to the second voice message; extracting keywords in the first text message and the second file information to obtain a first keyword corresponding to the first text message and a second keyword corresponding to the second file information; when the first keyword is consistent with the first preset keyword, determining the target satisfaction according to the first keyword and the first micro-expression, and when the second keyword is consistent with the second preset keyword, determining the target service level according to the second keyword and the second micro-expression.

[0172] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the loading position of the target satisfaction in the first video image, loading the target satisfaction into the first video image according to the loading position, obtaining the first target video image, and sending the first target video image to the second terminal; determining the loading position of the target service level in the second video image, loading the target service level into the second video image according to the loading position, obtaining the second target video image, and sending the second target video image to the first terminal

[0173] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: receiving a first video image sent by a first terminal and a second video image sent by a second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image. Inputting the first face image and the second face image into a trained micro-expression recognition model respectively, obtaining a first micro-expression corresponding to the first face image and a second micro-expression corresponding to the second face image. Determining a target satisfaction level based on the first micro-expression, determining a target service level based on the second micro-expression, sending the target satisfaction level and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal.

[0174] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: capturing video pictures from the first video image and the second video image at preset time intervals; extracting the face part in the video picture to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining a target facial image, and respectively calculating the similarity between the first facial image and the second facial image and the target facial image; when the similarity between the first facial image and the target facial image is greater than a first preset threshold, obtaining a first target facial image corresponding to the first facial image; when the similarity between the second facial image and the target facial image is greater than a second preset threshold, obtaining a second target facial image corresponding to the second facial image.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: determining facial skin color feature information of the first facial image and the second facial image; dividing the first facial image and the second facial image to obtain local facial blocks, and determining contour feature information, texture feature information, and area feature information of the local facial blocks; and calculating the similarity with the target facial image based on the facial skin color feature information, contour feature information, texture feature information, and area feature information.

[0177] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining historical video images and micro-expression labels corresponding to the historical video images to obtain an initial sample set; randomly sampling with replacement from the initial sample set to obtain a target training set; obtaining the corresponding historical micro-expression feature set based on the target training set, randomly selecting micro-expression features from the historical micro-expression feature set to obtain a target micro-expression feature set; calculating the partitioning expression features based on the target micro-expression feature set, using the partitioning expression features to binary divide the target training set to obtain each sub-training set, and using each sub-training set as the target training set; returning to the step of obtaining the corresponding historical micro-expression feature set based on the target training set, randomly selecting micro-expression features from the historical micro-expression feature set to obtain the target micro-expression feature set, and when a preset condition is met, a target decision tree is obtained; returning to the step of randomly sampling with replacement from the initial sample set to obtain the target training set, and when a preset number of target decision trees is reached, a micro-expression recognition model is obtained.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: receiving a first voice message and a second voice message, obtaining corresponding first text message according to the first voice message, and obtaining second text message according to the second voice message; extracting keywords from the first text message and the second file information to obtain a first keyword corresponding to the first text message and a second keyword corresponding to the second file information; when the first keyword is consistent with the first preset keyword, determining the target satisfaction according to the first keyword and the first micro-expression, and when the second keyword is consistent with the second preset keyword, determining the target service level according to the second keyword and the second micro-expression.

[0179] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the loading position of the target satisfaction in the first video image, loading the target satisfaction into the first video image according to the loading position, obtaining the first target video image, and sending the first target video image to the second terminal; determining the loading position of the target service level in the second video image, loading the target service level into the second video image according to the loading position, obtaining the second target video image, and sending the second target video image to the first terminal.

[0180] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing video data based on micro-expressions, the method comprising: After a video call is established between a first terminal and a second terminal, a first video image sent by the first terminal and a second video image sent by the second terminal are received, and face detection is performed on the first video image and the second video image to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image; Inputting the first facial image and the second facial image into a trained micro-expression recognition model respectively to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image; Receive a first voice message and a second voice message, obtain corresponding first text message according to the first voice message, and obtain second text message according to the second voice message; extract keywords in the first text message and the second text message to obtain a first keyword corresponding to the first text message and a second keyword corresponding to the second text message; when the first keyword is consistent with a first preset keyword, determine a target satisfaction level according to the first keyword and the first micro-expression, when the second keyword is consistent with a second preset keyword, determine a target service level according to the second keyword and the second micro-expression, send the target satisfaction level and the first video image to the second terminal, and send the target service level and the second video image to the first terminal; When the video call ends, the time proportions of the target satisfaction and target service level during the video call are counted, and the target satisfaction and target service level with the highest time proportions are taken as the final satisfaction and final service level of the video call. The final satisfaction and final service level are sent to the corresponding terminals for display.

2. The method according to claim 1, characterized in that The receiving a first video image sent by the first terminal and a second video image sent by the second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image, includes: capturing video pictures from the first video image and the second video image at a preset time interval; The face portion of the video image is extracted to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

3. The method according to claim 1, characterized in that After receiving the first video image sent by the first terminal and the second video image sent by the second terminal, performing face detection on the first video image and the second video image, and obtaining a first face image corresponding to the first video image and a second face image corresponding to the second video image, the method further includes: Acquire a target facial image, and respectively calculate similarities between the first facial image and the second facial image and the target facial image; When the similarity between the first facial image and the target facial image is greater than a first preset threshold, obtaining a first target facial image corresponding to the first facial image; When the similarity between the second facial image and the target facial image is greater than a second preset threshold, a second target facial image corresponding to the second facial image is obtained.

4. The method according to claim 3, characterized in that The respectively calculating the similarities between the first face image and the second face image and the target face image includes: Determine facial skin color feature information of the first facial image and the second facial image; Dividing the first face image and the second face image respectively to obtain local face blocks of the first face image and the second face image, and determining contour feature information, texture feature information, and area feature information of the local face blocks; The similarities between the first face image and the second face image and the target face image are calculated based on the face skin color feature information, the contour feature information, the texture feature information and the area feature information of the first face image and the second face image.

5. The method according to claim 1, characterized in that The sending the target satisfaction level and the first video image to the second terminal, and sending the target service level and the second video image to the first terminal includes: Determine a loading position of the target satisfaction level in the first video image, load the target satisfaction level into the first video image according to the loading position, obtain a first target video image, and send the first target video image to the second terminal; Determine a loading position of the target service level in the second video image, load the target service level into the second video image according to the loading position, obtain a second target video image, and send the second target video image to the first terminal.

6. A video data processing device based on micro-expressions, characterized in that: The device comprises: A face detection module is used to receive a first video image sent by the first terminal and a second video image sent by the second terminal after a video call is established between the first terminal and the second terminal, perform face detection on the first video image and the second video image, and obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image; A micro-expression recognition module, used to input the first facial image and the second facial image into a trained micro-expression recognition model, respectively, to obtain a first micro-expression corresponding to the first facial image and a second micro-expression corresponding to the second facial image; The recognition result processing module is used to receive a first voice message and a second voice message, obtain the corresponding first text message according to the first voice message, and obtain the second text message according to the second voice message; extract keywords in the first text message and the second text message to obtain a first keyword corresponding to the first text message and a second keyword corresponding to the second text message; when the first keyword is consistent with a first preset keyword, determine the target satisfaction according to the first keyword and the first micro-expression, and when the second keyword is consistent with a second preset keyword, determine the target service level according to the second keyword and the second micro-expression, send the target satisfaction and the first video image to the second terminal, and send the target service level and the second video image to the first terminal; when the video call ends, count the time proportion of the target satisfaction and the target service level during the video call, use the target satisfaction and the target service level with the highest time proportion as the final satisfaction and final service level of the video call, and send the final satisfaction and the final service level to the corresponding terminal for display.

7. The device according to claim 6, characterized in that The face detection module comprises: A video capture module, configured to capture video images from the first video image and the second video image at a preset time interval; The face extraction module is used to extract the face part in the video image to obtain a first face image corresponding to the first video image and a second face image corresponding to the second video image.

8. The device according to claim 6, characterized in that The device further comprises: A similarity calculation module, used to calculate the similarity between the first face image and the second face image and the target face image respectively; A first target facial image determination module, configured to obtain a first target facial image corresponding to the first facial image when the similarity between the first facial image and the target facial image is greater than a first preset threshold; The second target facial image determination module is used to obtain a second target facial image corresponding to the second facial image when the similarity between the second facial image and the target facial image is greater than a second preset threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images