Method and device for training expression recognition model and electronic equipment

Abstract

The invention provides a method and device for training an expression recognition model and electronic equipment, and relates to the technical field of computers, in particular to the fields of large models, video processing and expression recognition. According to the implementation scheme, multiple pieces of video data are labeled to obtain expression labeling information of the video data, and the expression labeling information comprises at least one expression labeling text for at least one time period in the video data; classifying the expression labeling information so as to associate the expression labeling text with a preset expression category; and training the expression recognition model based on supervision fine tuning by using the video data, the timestamp of the video data and the expression labeling information, so that the expression recognition model predicts the expression category.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the fields of large models, video processing, and facial expression recognition. Specifically, it relates to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training facial expression recognition models. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] By using deep learning models to recognize and understand video semantics, it is possible to achieve facial expression recognition.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training facial expression recognition models.

[0006] According to one aspect of this disclosure, a data processing method for training an expression recognition model is provided, comprising: labeling multiple video data to obtain expression labeling information of the video data, wherein the expression labeling information includes at least one expression labeling text for at least one time period in the video data; classifying the expression labeling information to associate the expression labeling text with a predetermined expression category; and training the expression recognition model based on supervised fine-tuning using the video data, the timestamp of the video data, and the expression labeling information, so that the expression recognition model can predict the expression category.

[0007] According to another aspect of this disclosure, a data processing apparatus for training an expression recognition model is provided, comprising: an annotation module configured to annotate multiple video data to obtain expression annotation information of the video data, wherein the expression annotation information includes at least one expression annotation text for at least one time period in the video data; a classification module configured to classify the expression annotation information to associate the expression annotation text with a predetermined expression category; and a training module configured to perform supervised fine-tuning training on the expression recognition model using the video data, the timestamps of the video data, and the expression annotation information, so that the expression recognition model can predict the expression category.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements the method described according to embodiments of this disclosure.

[0011] According to one or more embodiments of this disclosure, an expression recognition model can be trained to learn semantic information about temporal transitions between video frames, thereby improving the model's ability to recognize expressions in videos.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1A schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure is shown.

[0015] Figure 2 An exemplary flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0016] Figure 3 An exemplary process for training an expression recognition model according to an embodiment of this application is shown.

[0017] Figure 4 An exemplary block diagram of a data processing apparatus for training an expression recognition model according to the present disclosure is shown.

[0018] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0021] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0024] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the training of an expression recognition model and the expression recognition method according to embodiments of this disclosure.

[0025] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0027] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to receive user input and provide corresponding output. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0029] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0033] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0034] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0036] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0037] While multimodal large models (VLMs) in related technologies can provide overall emotion category judgments (such as happiness, surprise, anger, etc.) when analyzing facial expressions in videos, they struggle to accurately characterize facial expression changes, local action units, and micro-expression events across different time periods. Furthermore, video analysis models in related technologies generally lack the ability to combine visual features with dynamic changes over time in their modeling.

[0038] It is evident that facial expression recognition models in related technologies suffer from insufficient understanding of the dynamic nature of facial expression changes and an inability to accurately capture fine-grained changes in facial expressions.

[0039] To address the aforementioned problems in related technologies, this disclosure provides a novel method for training facial expression recognition models.

[0040] Figure 2 An exemplary flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0041] In step S202, multiple video data are labeled to obtain facial expression labeling information for the video data. The facial expression labeling information includes at least one facial expression labeling text for at least one time period in the video data.

[0042] In step S204, the emoticon annotation information is classified to associate the emoticon annotation text with a predetermined emoticon category.

[0043] In step S206, the expression recognition model is trained using video data, video data timestamps, and expression annotation information, based on supervised fine-tuning, so that the expression recognition model can predict expression categories.

[0044] Using the data processing method provided in the embodiments of this disclosure, an expression recognition model can be trained to learn the semantic information of temporal transitions between video frames, thereby improving the model's ability to recognize expressions in videos.

[0045] The principles of this disclosure will now be described in detail.

[0046] In step S202, multiple video data can be labeled to obtain facial expression labeling information for the video data. The facial expression labeling information includes at least one facial expression labeling text for at least one time period in the video data.

[0047] In some embodiments, step S202 may include annotating multiple video data using a large model to obtain facial expression annotation information. Various multimodal large models, such as Gemini and GPT, can be used for annotation. Annotation using a large model can improve the efficiency and consistency of annotation. In other embodiments, facial expression annotation information can be achieved using manual annotation or a combination of manual annotation and automatic annotation using a large model. An exemplary facial expression annotation information could be "00:00–00:02 She raised her eyebrows and showed a focused state; 00:02–00:04 The corners of her mouth slightly turned up, showing a slight smile." By describing the facial expressions in the video in time segments, the facial expression annotation information can reflect the temporal changes of facial expressions with video frames. Furthermore, by using facial expression annotation text to provide a detailed natural language description of the facial expression information in the video, fine-grained facial expression information can be reflected in the facial expression annotation information.

[0048] In step S204, the emoticon annotation information can be categorized to associate the emoticon annotation text with a predetermined emoticon category.

[0049] In some embodiments, step S204 may include classifying the emoticon annotation information using a large model to obtain the emoticon category associated with the emoticon annotation text. The large model used for classification in step S204 and the large model used for annotation in step S202 may be the same or different. In other embodiments, step S204 may also include classifying the emoticon annotation information using a rule-based method to obtain the emoticon category associated with the emoticon annotation text.

[0050] Expression categories can be predefined, such as "happy," "angry," or "neutral." Those skilled in the art can define expression categories according to the actual situation.

[0051] In some cases, at least one of the annotation results obtained in step S202 and the classification results obtained in step S204 can be repaired to correct erroneous data.

[0052] In step S206, the expression recognition model can be trained using video data, video data timestamps, and expression annotation information, so that the expression recognition model can predict expression categories.

[0053] By training the facial expression recognition model with the facial expression annotation information of the video data obtained in steps S202-S204, the facial expression recognition model can learn the temporal information and facial expression description information in the facial expression annotation information.

[0054] The timestamp features of video data, image features of video data, and text features of facial expression annotation information can be fused together and provided to the facial expression recognition model for learning.

[0055] Step S206 may include: extracting frames from the video data at a predetermined frequency to obtain an image set including multiple video frames; determining the timestamp corresponding to each video frame in the image set; processing the video frame and the corresponding timestamp together using the current parameters of the expression recognition model to obtain a predicted expression description of the video frame; and training the parameters of the expression recognition model based on supervised fine-tuning based on the predicted expression description and the expression annotation information corresponding to the video frame to obtain the updated parameters of the expression recognition model.

[0056] To improve the efficiency of the facial expression prediction process, frame extraction from video data can reduce the amount of data the model needs to process, thus saving computational resources. Furthermore, by processing video frames and their corresponding timestamps together, temporal information associated with video frames (images) can be considered during facial expression recognition. This allows the model to understand the relationship between the time period information described in the facial expression annotation text and the timestamp information of the video frames during training, enabling the trained model to understand the temporal transitions between consecutive video frames.

[0057] In some embodiments, video frames and their corresponding timestamps can be encoded separately, and the encoded features of the video frames and timestamps can be concatenated to obtain a joint encoding result of the video frame and its timestamp. A visual encoder (such as ViT, CLIP model, etc.) can be used to encode the image information of the extracted video frames to obtain the encoded features of the video frames. For the timestamps, cyclic encoding can be used to obtain the encoded features of the timestamps. Any suitable method can be used to process the encoded features of the video frames and timestamps to achieve the concatenation of these encoded features.

[0058] By processing the image information and timestamps of the input video frames together using the current parameters of the facial expression recognition model, and further predicting facial expression descriptions from the joint encoded information of both, supervised fine-tuning of the model's parameters can be achieved. The predicted facial expression description can include time period information and predicted text annotated with the facial expression for that time period. The model parameters can be adjusted by comparing the differences between the predicted facial expression description and the time period information in the facial expression annotation information, and the differences between the predicted text annotated with the facial expression. Those skilled in the art can choose any suitable loss function to represent these differences and achieve supervised fine-tuning training based on the actual situation.

[0059] Exemplary facial expression recognition models may include Transformer-based models, Convolutional Neural Network (CNN)-based models, VGG models, MiniCPM models, etc. Those skilled in the art can select any suitable existing model or a modified model structure to implement the facial expression recognition model used in the embodiments of this disclosure, depending on the specific circumstances.

[0060] After training the parameters of the facial expression recognition model using supervised fine-tuning, the performance of the trained model can be evaluated, and the parameters of the facial expression recognition model can be further optimized based on the evaluation results.

[0061] In some embodiments, method 200 may further include: performing facial expression recognition on video data in the evaluation dataset using updated parameters of the facial expression recognition model to evaluate the updated parameters; and determining an updated dataset for optimizing the updated parameters in response to the evaluation results of the updated parameters indicating that the recognition accuracy of the facial expression category to be optimized is lower than a predetermined accuracy threshold or the number of recognition results of the facial expression category to be optimized is lower than a predetermined number threshold.

[0062] The evaluation dataset may include evaluation videos and facial expression descriptions used to describe the videos. For example, the facial expression descriptions in the evaluation dataset can be determined using expert network annotation, direct manual annotation, or by manually reviewing the model annotation results, to ensure the accuracy of the facial expression descriptions in the evaluation videos.

[0063] The accuracy of an expression recognition model with currently updated parameters can be evaluated by comparing the expression recognition results output by the model for video data in the evaluation dataset with the differences between the results and the pre-associated real expression descriptions of the evaluation videos. For example, the acceptable accuracy of the evaluation can be determined by comparing the expression recognition accuracy of the model with currently updated parameters with a predetermined accuracy threshold.

[0064] In some embodiments, the evaluation step described above can be categorized. In this case, the evaluation videos in the evaluation dataset can be labeled with corresponding expression categories, and the performance of the expression recognition model for different expression categories can be statistically analyzed separately to identify the current weaknesses of the expression recognition model, i.e., which expression categories it is not yet capable of recognizing.

[0065] In other embodiments, the facial expression recognition model can be used to statistically analyze the facial expression categories in the video data of the evaluation dataset, and the facial expression categories with a number of recognition results lower than a predetermined threshold can be identified as the facial expression categories to be optimized.

[0066] In some examples, expression categories with recognition accuracy below a predetermined accuracy threshold can be identified as expression categories to be optimized. In other examples, at least one expression category with a relatively low recognition accuracy can be identified as an expression category to be optimized. Without departing from the principles of this disclosure, those skilled in the art can use various methods to analyze the recognition accuracy of the current expression recognition model to determine the next optimization direction.

[0067] For each expression category to be optimized, training data containing video data and descriptive text corresponding to that category can be collected for further training. In some embodiments, training data corresponding to the expression category to be optimized can be collected by searching videos based on text.

[0068] In some examples, determining the updated dataset may include identifying multiple facial expression description texts associated with the facial expression category to be optimized. For example, a large model can be used to summarize and generalize the facial expression description texts related to optimizing the facial expression category, resulting in multiple typical text descriptions for that facial expression category. Then, the multiple facial expression description texts can be vectorized to determine the facial expression text vectors corresponding to each facial expression description text. Further, the multiple facial expression text vectors corresponding to the multiple facial expression description texts can be averaged to obtain a text vector representing the facial expression category to be optimized. Exemplarily, a weighted average of the multiple facial expression text vectors can be used to obtain the text vector for optimizing the facial expression category, and the text vector incorporates the content of multiple typical text descriptions for that facial expression category. Without departing from the principles of this disclosure, those skilled in the art can adjust the weights of each facial expression text vector according to the actual situation. In some other examples, various data processing techniques for text vectors, such as vector concatenation and vector addition, can also be used to fuse multiple facial expression text vectors. Further, video clip feature retrieval can be performed based on the text vectors to obtain video clips associated with the facial expression category to be optimized as the updated dataset. Exemplarily, the VideoClip model can be used to perform video clip feature retrieval based on text vectors. In text-based video retrieval using the VideoClip model, the facial expression description text can be converted into a suitable input language, such as English, before vectorization. Those skilled in the art can also choose any other suitable model capable of video retrieval based on text vector features, depending on the specific circumstances.

[0069] As mentioned earlier, the model has a low accuracy rate in facial expression recognition for the facial expression category to be optimized. This may be due to insufficient training data for the facial expression category to be optimized and / or inaccurate facial expression annotation information in the training data for the facial expression category to be optimized.

[0070] The above method can be used to quickly collect video clips for the expression categories to be optimized as an updated dataset for further model optimization, thereby increasing the amount of training data for the expression categories to be optimized.

[0071] Furthermore, the accuracy of the annotation information can be improved by updating the facial expression annotation information corresponding to the video segments in the updated dataset, and the more accurate annotation information can be used to train the model to optimize the model performance.

[0072] After retrieving video clips from the updated dataset used for optimizing the expression category, determining the updated dataset may further include: for each video clip in the updated dataset, determining whether the expression description text for the expression category to be optimized corresponds to the same expression category as the expression annotation text of the video clip; in response to determining that the expression description text for the expression category to be optimized and the expression annotation text of the video clip correspond to different expression categories, updating the expression annotation text of the video clip. Using this method, the original expression annotation information of the video clips in the updated dataset (such as the expression annotation information obtained in step S202) can be compared with the expression description information of the expression category to be optimized. If it is determined that the original expression annotation information of the video clip does not match the expression description information of the expression category to be optimized, the expression annotation text of the video clip can be updated so that the updated expression annotation text matches the expression category to be optimized. Using this method can improve the accuracy of the expression annotation information of video clips, thereby improving the model training effect.

[0073] In some embodiments, method 200 may further include supervising the training of the facial expression recognition model using video clips in the updated dataset, timestamps of the video clips, and updated facial expression annotation text for the video clips. By using video clips in the updated dataset as training data, the facial expression recognition model can be further trained using the same method as in step S204, thereby improving the optimization performance of the facial expression recognition model for the facial expression category to be optimized.

[0074] In some embodiments, during the process of updating the facial expression annotation information of a video clip, by identifying discrepancies between the original facial expression annotation information of the video clip and the facial expression description information of the facial expression category to be optimized, positive samples with correct annotations and negative samples with incorrect annotations can be determined for the video clip. In response to the determination that the facial expression description text of the facial expression category to be optimized corresponds to a different facial expression category than the facial expression annotation text of the video clip, the facial expression annotation text of the video clip is determined as a negative sample, and the updated facial expression annotation text of the video clip is determined as a positive sample. These positive and negative samples can be used to update the parameters of the facial expression recognition model based on reinforcement learning. During reinforcement learning training, the model receives a positive reward for making decisions similar to positive samples, making it more inclined to produce such behaviors; simultaneously, it is penalized or receives a lower reward for making decisions similar to negative samples, making it avoid such behaviors. Those skilled in the art can use any suitable reinforcement learning training method to update the facial expression recognition model according to the actual situation.

[0075] After training and optimizing the parameters of the facial expression recognition model using the methods described above, the trained model can be used to process the target video to obtain facial expression recognition results for the target video. These results include time-segmented natural language descriptions. Exemplary facial expression recognition results include: 00:00–00:04 Eyes filled with fear; 00:04–00:05 Eyes briefly widening in surprise. These facial expression recognition results can provide detailed descriptions of facial expressions appearing in the video across different time segments, thus reflecting the temporal changes and details of facial expressions in the video.

[0076] Figure 3 An exemplary process for training an expression recognition model according to an embodiment of this application is shown.

[0077] At position 301, obtain the pre-trained MiniCPM model. The MiniCPM model can be trained using a general training dataset, enabling the pre-trained MiniCPM model to possess general semantic recognition and understanding capabilities.

[0078] At position 302, a large model can be used to annotate the video data to obtain facial expression annotation information associated with the video data.

[0079] At point 303, manual or rule-based error correction can be performed on the facial expression annotation information output by point 302 to resolve obvious errors in the annotation data and improve the accuracy of the facial expression annotation information.

[0080] At point 304, supervised fine-tuning training can be performed on the MiniCPM pre-trained model. The MiniCPM pre-trained model can be fine-tuned using steps S202-S206 described in method 200, thereby enabling the fine-tuned MiniCPM pre-trained model to possess the corresponding facial expression recognition capabilities.

[0081] At position 305, the accuracy of the MiniCPM model output at position 304 can be analyzed.

[0082] At position 306, based on the accuracy analysis results obtained at position 305, expression categories with low accuracy and small proportions can be identified as expression categories to be optimized.

[0083] At point 307, a large model can be used to analyze the text associated with the expression category to be optimized, so as to obtain multiple typical texts related to the expression category to be optimized.

[0084] At position 308, the Videoclip model can be used to retrieve video clips related to the desired expression category based on typical text. After obtaining these clips, the process can return to position 302 and use a larger model to re-annotate the relevant video clips based on the desired expression category, thus improving the accuracy of the annotations. The optimized annotations can then be used to further train the MiniCPM model.

[0085] Figure 4 An exemplary block diagram of a data processing apparatus for training an expression recognition model according to the present disclosure is shown.

[0086] like Figure 4 As shown, the device 400 includes a labeling module 410, a classification module 420, and a training module 430.

[0087] The annotation module 410 can be configured to annotate multiple video data to obtain facial expression annotation information of the video data, wherein the facial expression annotation information includes at least one facial expression annotation text for at least one time period in the video data.

[0088] The classification module 420 can be configured to classify emoticon annotation information to associate emoticon annotation text with predefined emoticon categories.

[0089] The training module 430 can be configured to train the facial expression recognition model using video data, timestamps of the video data, and facial expression annotation information, so that the facial expression recognition model can predict facial expression categories.

[0090] In some embodiments, training an expression recognition model using video data, timestamps of the video data, and expression annotation information includes: extracting frames from the video data at a predetermined frequency to obtain an image set comprising multiple video frames; determining the timestamp corresponding to each video frame in the image set; processing the video frame and its corresponding timestamp together using the current parameters of the expression recognition model to obtain a predicted expression description for the video frame; and training the parameters of the expression recognition model using supervised fine-tuning based on the predicted expression description and the expression annotation information corresponding to the video frame to obtain updated parameters of the expression recognition model.

[0091] In some embodiments, annotating multiple video data includes: using a large model to annotate multiple video data to obtain facial expression annotation information.

[0092] In some embodiments, classifying emoticon annotation information includes: classifying the emoticon annotation information using a large model to obtain the emoticon category associated with the emoticon annotation text.

[0093] In some embodiments, the apparatus 400 may further include an optimization module configured to: perform facial expression recognition on video data in the evaluation dataset using the updated parameters of the facial expression recognition model to evaluate the updated parameters; and determine an updated dataset for optimizing the updated parameters in response to the evaluation results of the updated parameters indicating that the recognition accuracy of the facial expression category to be optimized is lower than a predetermined accuracy threshold or the number of recognition results of the facial expression category to be optimized is lower than a predetermined number threshold.

[0094] In some embodiments, determining the updated dataset includes: determining multiple expression description texts associated with the expression category to be optimized; vectorizing the multiple expression description texts to determine the expression text vectors corresponding to each expression description text; averaging the multiple expression text vectors corresponding to the multiple expression description texts to obtain a text vector representing the expression category to be optimized; and performing video segment feature retrieval based on the text vectors to obtain video segments associated with the expression category to be optimized as the updated dataset.

[0095] In some embodiments, the VideoClip model is used to retrieve video clip features based on text vectors.

[0096] In some embodiments, determining the updated dataset further includes: for video segments in the updated dataset, determining whether the expression description text of the expression category to be optimized is the same as the expression category corresponding to the expression annotation text of the video segment; and updating the expression annotation text of the video segment in response to determining that the expression description text of the expression category to be optimized is different from the expression category corresponding to the expression annotation text of the video segment.

[0097] In some embodiments, the training module is further configured to: train the facial expression recognition model using supervised fine-tuning with video clips in the updated dataset, timestamps of the video clips, and updated facial expression annotation text for the video clips.

[0098] In some embodiments, the training module is further configured to: in response to the fact that the expression description text for determining the expression category to be optimized is different from the expression annotation text for the video clip, determine the expression annotation text for the video clip as a negative sample and determine the updated expression annotation text for the video clip as a positive sample; The parameters of the facial expression recognition model are updated based on reinforcement learning using positive and negative samples.

[0099] In some embodiments, the training module is further configured to process the target video using a trained facial expression recognition model to obtain facial expression recognition results for the target video, wherein the facial expression recognition results include time-segmented natural language descriptions.

[0100] It should be understood that Figure 4 The various modules or units of the apparatus 400 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 400 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0101] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the various units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0102] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0103] According to embodiments of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described according to embodiments of the present disclosure.

[0104] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the methods described according to embodiments of the present disclosure.

[0105] According to embodiments of the present disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described according to embodiments of the present disclosure.

[0106] refer to Figure 5 The present invention describes a structural block diagram of an electronic device 500 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0108] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, output unit 507, storage unit 508, and communication unit 509. Input unit 506 can be any type of device capable of inputting information to electronic device 500. Input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 507 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 508 may include, but is not limited to, disk and optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0109] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, method 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform method 200 by any other suitable means (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0115] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0116] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0117] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data processing method for training an expression recognition model, comprising: annotating a plurality of video data to obtain expression annotation information of the video data, wherein the expression annotation information comprises at least one expression annotation text for at least one time period in the video data; classifying the expression annotation information to associate the expression annotation text with a predetermined expression category; performing supervised fine-tuning training on the expression recognition model using the video data, timestamps of the video data, and the expression annotation information, so that the expression recognition model predicts the expression category.

2. The data processing method of claim 1, wherein, performing supervised fine-tuning training on the expression recognition model using the video data, timestamps of the video data, and the expression annotation information comprises: frame-sampling the video data at a predetermined frequency to obtain an image set comprising a plurality of video frames; determining a timestamp corresponding to each video frame in the image set; processing the video frame and the corresponding timestamp together using current parameters of the expression recognition model to obtain a predicted expression description of the video frame; performing supervised fine-tuning training on the parameters of the expression recognition model based on the predicted expression description and the expression annotation information corresponding to the video frame to obtain updated parameters of the expression recognition model.

3. The data processing method of claim 1, wherein, annotating a plurality of video data comprises annotating the plurality of video data using a large model to obtain the expression annotation information.

4. The data processing method of claim 1, wherein, classifying the expression annotation information comprises classifying the expression annotation information using a large model to obtain the expression category associated with the expression annotation text.

5. The data processing method of claim 1, further comprising: performing expression recognition on video data in an evaluation data set using the updated parameters of the expression recognition model to evaluate the updated parameters; in response to determining that the recognition accuracy of a to-be-optimized expression category is lower than a predetermined accuracy threshold or the number of recognition results of the to-be-optimized expression category is lower than a predetermined number threshold based on the evaluation result of the updated parameters, determining an update data set for optimizing the updated parameters.

6. The data processing method of claim 5, wherein, determining an update data set comprises: determining a plurality of expression description texts associated with the to-be-optimized expression category; vectorizing the plurality of expression description texts to determine an expression text vector corresponding to each expression description text; averaging a plurality of expression text vectors corresponding to the plurality of expression description texts to obtain a text vector representing the to-be-optimized expression category; performing video clip feature retrieval based on the text vector to obtain a video clip associated with the to-be-optimized expression category as an update data set.

7. The data processing method of claim 6, wherein, performing video clip feature retrieval based on the text vector using a VideoClip model.

8. The data processing method of claim 6, wherein, determining an update data set further comprises: for a video clip in the update data set, determining whether the expression description text of the to-be-optimized expression category and the expression category corresponding to the expression annotation text of the video clip are the same; In response to determining that the facial description text of the facial category to be optimized is different from the facial category corresponding to the facial annotation text of the video segment, updating the facial annotation text of the video segment.

9. The data processing method of claim 8, further comprising: performing supervised fine-tuning based training of the facial recognition model using the video segment, the timestamp of the video segment, and the updated facial annotation text for the video segment.

10. The data processing method of claim 8, further comprising: In response to determining that the facial description text of the facial category to be optimized is different from the facial category corresponding to the facial annotation text of the video segment, determining the facial annotation text of the video segment as a negative sample and determining the updated facial annotation text of the video segment as a positive sample; performing reinforcement learning based parameter updating of the facial recognition model based on the positive sample and the negative sample.

11. The data processing method of claim 1, further comprising: processing a target video using the trained facial recognition model to obtain a facial recognition result for the target video, wherein the facial recognition result comprises time-periodic natural language descriptions.

12. A data processing apparatus for training a facial recognition model, comprising: an annotation module configured to annotate a plurality of video data to obtain facial annotation information of the video data, wherein the facial annotation information comprises at least one facial annotation text for at least one time period in the video data; a classification module configured to classify the facial annotation information to associate the facial annotation text with a predetermined facial category; a training module configured to perform supervised fine-tuning based training of the facial recognition model using the video data, timestamps of the video data, and the facial annotation information, so that the facial recognition model predicts the facial category.

13. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-11. The computer program, when executed by a processor, implements the method of any one of claims 1-11.

Images