Facial expression data set expansion method based on large language model

Abstract

The invention belongs to the technical field of emotion calculation, and particularly relates to a facial expression data set expansion method based on a large language model. The method comprises the specific steps that S1, an expression database is used as training data, and a large language model is finely adjusted; s2, taking the face database as input data, and utilizing a prompt project to guide a large language model to generate a face image with expressions; s3, performing prejudgement on the facial image with the expression generated in the S2, and verifying and optimizing a generation result; and S4, collecting the images which are judged and optimized in the step S3, and realizing expansion of the facial expression data set. According to the method, the facial expression image conforming to the scene is created by using the image generation capability of the large language model, so that the expansion of the facial expression data set is supported, meanwhile, the image quality evaluation algorithm is provided, the screening and verification of the generated facial expression image are realized, and the quality of the generated image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion computing, and in particular relates to a method for expanding a facial expression data set based on a large language model. Background Art

[0002] Facial expression databases have important applications in the fields of computer vision and biometrics, such as facial expression recognition, expression analysis, and identity authentication. However, the quality and scale of facial expression databases have a direct impact on algorithm performance. Traditional facial expression data collection methods mainly rely on manual annotation and collection, which is cumbersome and costly. Due to the insufficient amount of data in the expression database, the recognition of facial expressions has not achieved better recognition results, so there is an urgent need to improve the quantity and quality of facial expression databases for facial expression recognition.

[0003] At present, the existing facial expression dataset expansion methods are mainly divided into two categories, namely semi-automatic and automatic. Semi-automatic methods can improve generation efficiency while ensuring data quality, and are suitable for scenarios that require high-quality data. Automated methods are widely used in scenarios that require a large amount of data and do not require a single data sample. Nevertheless, the choice of dataset expansion method needs to be weighed according to specific application requirements. Blindly using automated methods may lead to a decline in data quality, thereby affecting the performance and stability of subsequent algorithms.

[0004] In recent years, artificial intelligence (AI) technology has made significant progress and innovations in supporting image generation. It can not only generate high-definition images, but also make very realistic performances in details, textures, light and shadow. For example, images generated by diffusion models, GANs (generative adversarial networks) and VQ-VAE have achieved extremely high fidelity. In particular, with the rise and development of large language models, the optimization of solutions for image generation tasks has been further promoted. For example, by taking text, speech or other modalities as input, large language models can convert them into images. Typical representatives in this regard are models such as OpenAI's DALL-E and ChatGPT-4o. Users can enter simple natural language descriptions, and large language models can generate corresponding images based on the descriptions. Large language models can also make image editing simpler and more efficient by completing damaged image areas, removing unnecessary elements, or enhancing low-resolution images to high resolution.

[0005] The image generation method of the large language model mainly generates images based on text descriptions. This type of method uses a large language model to generate descriptions, and then generates corresponding images through a text-to-image generation model. This method can quickly generate diverse images, but the quality and authenticity of the generated images may be unstable. Traditional machine learning methods for generating images include: First, data augmentation technology, which generates new data samples by rotating, cropping, flipping, and other operations on existing facial expression data. Although this method is simple and easy to implement, the generated samples are limited in diversity. The second is based on the adversarial generative network method, which makes the generated images realistic and diverse by training the generator and discriminator. This method can generate high-quality facial images, but the training process is complex and the computational cost is high. However, the hybrid large language model and traditional machine learning generation method, combined with the advantages of the above methods, can further improve the quality and diversity of the data set through the collaborative work of multiple technologies.

[0006] Therefore, it is of great significance to develop a facial expression dataset expansion method based on a large language model. Summary of the invention

[0007] The technical problem to be solved by the present invention is to address the deficiencies in the prior art and to provide a method for expanding a facial expression data set based on a large language model and a face database, so as to expand the existing expression database, provide more learning data for the training of methods for recognizing human emotions, and improve the recognition rate.

[0008] In order to solve the above technical problems, the present invention discloses a method for expanding a facial expression data set based on a large language model, and the specific steps are as follows:

[0009] S1. Use the expression database as training data to fine-tune the large language model;

[0010] S2. Take the face database as input data and use the prompt engineering to guide the large language model to generate facial images with expressions;

[0011] S3. Make bias judgment on the facial images with expressions generated by S2 and verify and optimize the generated results;

[0012] S4. Collect the images judged and optimized in S3 to expand the facial expression data set.

[0013] Among them, in S1, the specific steps of fine-tuning are to use Low-Rank Adaptation (LoRA), add a low-rank decomposition model to the weight matrix of the model, and only update the parameters of these modules to reduce the computational complexity.

[0014] Among them, in S2, based on the face database as input, combined with the prompt project to generate images, including zero-sample learning and few-sample learning methods to construct prompt words, so as to determine the interaction method with the large language model, generate facial expression images, and at the same time, generate expression images for the input ordinary face images combined with expression labels, wherein the labels of the generated images are seven categories, namely anger, anxiety, boredom, disgust, fear, happiness and uncertainty, and the generated images are matched one by one with the labels to generate a corresponding relationship.

[0015] Among them, in said S3, the specific steps are:

[0016] S31. Perform face detection using face recognition algorithm;

[0017] S32. Performing face recognition bias detection using a bias detection algorithm;

[0018] S33 is calculated by the expression recognition algorithm through S31 and S32 detected image and the expression label set whether it matches;

[0019] S34. Regenerate and optimize the output result of the unmatched image in S33.

[0020] In S31, the face recognition algorithm uses the DeepFace face recognition library, and the underlying model uses VGG-Face, where the convolution layer used to extract local features is shown in formula (1).

[0021]

[0022] Where f is the input image, g is the convolution kernel (filter), (i, j) is the location of the output feature map, and (m, n) is the spatial offset of the convolution kernel.

[0023] Among them, in S32, the bias detection algorithm uses the AI ​​Fairness 360 library to detect and understand the bias in the model, detect whether the generated image has bias in terms of race, gender, age, facial features, etc., and put the image data that does not meet the standards into a queue to be regenerated.

[0024] In S33, the expression recognition algorithm is used to calculate whether the generated image matches the set label and whether it reaches the set recognition threshold of 80%. Image data that does not reach the set recognition threshold enters the queue to be regenerated; the expression recognition algorithm is a facial expression recognition algorithm based on a convolutional neural network, wherein the calculation process of the spatial dimension of the feature map output by the convolution operation is shown in formulas (2) and (3).

[0025]

[0026] Among them, H out is the height of the output feature map, H in is the height of the input feature map, F H is the convolution kernel size, W out is the width of the output feature map, W in is the width of the input feature map, F W is the convolution kernel size, P is the padding size, and s is the stride.

[0027] Among them, in S34, the images below the recognition threshold in S33 are added to the loop queue and regenerated. After multiple cycles, when the recognition rate of the generated images tends to be stable, the image that finally meets the recognition threshold is saved, and the recognition threshold is updated to the average recognition rate of all saved images, so as to achieve dynamic adjustment of the threshold to improve the quality of the generated image; wherein, the stability of the recognition rate after step S33 is judged by the convergence of the mean and variance.

[0028] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method for expanding a facial expression data set based on a large language model of the present invention are implemented.

[0029] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for expanding a facial expression data set based on a large language model of the present invention are implemented.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. It utilizes the image generation capability of a large language model to create facial expression images that conform to the scene, thereby supporting the expansion of facial expression data sets; 2. It proposes an image quality assessment algorithm to achieve screening and verification of generated facial expression images, thereby improving the quality of generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0032] Figure 1 A flowchart of a method for expanding a facial expression data set according to the present invention;

[0033] Figure 2 This is a flow chart of fine-tuning a large language model in an embodiment of the present invention;

[0034] Figure 3 This is an example of generating a facial expression image in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments, so that the invention can be understood more clearly and intuitively.

[0036] Embodiment 1:

[0037] like Figure 1 As shown, the implementation of the facial expression data set expansion method based on the large language model of the present invention includes the following steps:

[0038] S1. Using the FER2013 expression database as training data, the large language model is fine-tuned using Low-Rank Adaptation (LoRA). By adding a low-rank decomposition model to the model's weight matrix, only the parameters of these modules are updated to reduce the computational complexity. In this process, the training parameters are continuously adjusted to obtain the best fine-tuning effect. The steps are as follows: Figure 2 As shown, the model can understand the domain knowledge and goals of the current task;

[0039] S2. Take the VGGFace2 face database as input data, and use the prompt engineering to guide the large language model to generate facial images with obvious expressions. Specifically, based on the face database as input, the prompt engineering is combined to generate images. The prompt engineering specifically uses zero-shot learning and few-shot learning methods to construct prompt words, so as to determine the interaction method with the large language model and generate facial expression images.

[0040] S3. In the evaluation phase, the rationality and ethics of the generated expression image results are judged, and then the facial expression recognition algorithm based on the convolutional neural network is used to cross-validate and optimize the available generated results. Images with recognition rates lower than the recognition threshold will be regenerated until the recognition rate of the image tends to be stable;

[0041] S4. The facial expression data is generated through the above steps to expand the facial expression data set.

[0042] Among them, in step S2, the prompt project is used to guide the large language model, and the expression image is generated for the input ordinary face image combined with the expression label, wherein the labels of the generated image are seven categories, namely anger, anxiety, boredom, disgust, fear, happiness and uncertainty. The generated image is matched one by one with the label, and the corresponding relationship is generated to facilitate step S3 to read the input.

[0043] The image evaluation algorithm based on face recognition detection, bias detection, and image review proposed by the present invention in step S3 is shown in Table 1, and the functions of each part are as follows:

[0044] S31. First, use the face recognition algorithm to detect the face. The algorithm is used to detect whether the corresponding face is generated in the image. The image that is not recognized normally will enter the queue to be regenerated. This method uses the DeepFace open source face recognition library, and the underlying model uses VGG-Face. The model is pre-trained on a large-scale data set based on a convolutional neural network. It can well capture the subtle differences of the face and has excellent performance. The convolution layer used to extract local features is shown in formula (1).

[0045]

[0046] Where f is the input image, g is the convolution kernel (filter), (i, j) is the location of the output feature map, and (m, n) is the spatial offset of the convolution kernel.

[0047] S32. Secondly, use the bias detection algorithm to detect bias in face recognition. Face recognition bias refers to the fact that in face detection and recognition systems, due to limitations of models, data sets or algorithms, face detection and recognition performance may be unfair to specific groups (such as different races, genders, ages, etc.). Use the algorithm to detect bias on images that pass through S31 to detect whether the generated images have bias in terms of race, gender, age and facial features, so as to ensure the diversity and fairness of the data set. Use the AI ​​Fairness 360 (AIF360) open source library to implement face detection bias, detect and understand bias in the model, adjust the results in the model output stage to make it more fair among different groups, and put the image data that does not meet the standards into the queue to be regenerated.

[0048] S33. Then use the facial expression recognition algorithm based on convolutional neural network to review the image. Using the expression recognition algorithm, calculate whether the generated image matches the set label and whether it reaches the set recognition threshold of 80%. Image data that does not reach the set recognition threshold enters the queue to be regenerated. The calculation process of the spatial dimension of the feature map output by the convolution operation in this algorithm is shown in formulas (2) and (3).

[0049]

[0050] Among them, H out is the height of the output feature map, H in is the height of the input feature map, F H is the convolution kernel size, W out is the width of the output feature map, W in is the width of the input feature map, F W is the convolution kernel size, P is the padding size, and s is the stride.

[0051] S34. Finally, for images that do not meet the above evaluation algorithm (recognition rate is lower than the recognition threshold), add them to the loop queue and regenerate them. In the process of multiple cycles, when the recognition rate of the generated image tends to be stable, save the last image that meets the recognition threshold, and update the recognition threshold to the mean recognition rate of all saved images, so as to dynamically adjust the threshold to improve the quality of the generated image. The convergence of the mean and variance is used to judge the stability of the recognition rate after the audit algorithm, and the calculation formulas of the mean and variance are shown in formulas (4) and (5).

[0052]

[0053] Where N is the number of total recognition rates, x i is the i-th recognition rate, μ is the mean, σ 2 is the variance.

[0054] Through the above steps, the images in the face database are combined with the set expression labels in turn according to the above evaluation and optimization process to generate facial expression images, thereby expanding the facial expression data set.

[0055] Table 1 Image evaluation algorithms based on face recognition detection, bias detection, and image review

[0056]

[0057]

[0058] Embodiment 2:

[0059] The computer-readable storage medium of this embodiment stores a computer program, which, when executed by a processor, implements the steps in the method for expanding a facial expression dataset based on a large language model in Example 1.

[0060] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.

[0061] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0062] Embodiment 3:

[0063] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for expanding a facial expression data set based on a large language model of Embodiment 1 are implemented.

[0064] In this embodiment, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, readily available programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0065] Those skilled in the art will appreciate that the disclosed content of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present solution may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present solution may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0066] The present solution is described with reference to the flowchart and / or block diagram of the method and computer program product according to the embodiment of the present solution. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0070] The present invention provides a method for expanding a facial expression data set based on a large language model. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for expanding a facial expression dataset based on a large language model, characterized in that: The specific steps are: S1. Use the expression database as training data to fine-tune the large language model; S2. Take the face database as input data and use the prompt engineering to guide the large language model to generate facial images with expressions; S3. Make bias judgment on the facial images with expressions generated by S2 and verify and optimize the generated results; S4. Collect the images judged and optimized in S3 to expand the facial expression data set.

2. The method according to claim 1, characterized in that In S1, the specific steps of fine-tuning are to use Low-Rank Adaptation to add a low-rank decomposition model to the weight matrix of the model, and only update the parameters of these modules to reduce the computational complexity.

3. The method according to claim 1, characterized in that In S2, based on the face database as input, the image is generated in combination with the prompt project, including constructing prompt words by zero-sample learning and few-sample learning methods, so as to determine the interaction method with the large language model and generate facial expression images. At the same time, the expression images are generated for the input ordinary face images in combination with the expression labels, wherein the labels of the generated images are seven categories, namely anger, anxiety, boredom, disgust, fear, happiness and uncertainty, and the generated images are matched one by one with the labels to generate a corresponding relationship.

4. The method according to claim 1, characterized in that: In S3, the specific steps are: S31. Perform face detection using face recognition algorithm; S32. Performing face recognition bias detection using a bias detection algorithm; S33 is calculated by the expression recognition algorithm through S31 and S32 detected image and the expression label set whether it matches; S34. Regenerate and optimize the output result of the unmatched image in S33.

5. The method according to claim 4, characterized in that In S31, the face recognition algorithm uses the DeepFace face recognition library, and the underlying model uses VGG-Face, where the convolution layer used to extract local features is shown in formula (1). Where f is the input image, g is the convolution kernel (filter), (i, j) is the location of the output feature map, and (m, n) is the spatial offset of the convolution kernel.

6. The method according to claim 4, characterized in that In S32, the bias detection algorithm uses the AI ​​Airness 360 library to detect and understand bias in the model, detect whether the generated images have bias in terms of race, gender, age, and facial features, and put image data that does not meet the standards into a queue to be regenerated.

7. The method according to claim 4, characterized in that In S33, an expression recognition algorithm is used to calculate whether the generated image matches the set label and whether it reaches a set recognition threshold of 80%. Image data that does not reach the set recognition threshold enters a queue to be regenerated. The expression recognition algorithm is a facial expression recognition algorithm based on a convolutional neural network, wherein the spatial dimension calculation process of the feature map output by the convolution operation is shown in formulas (2) and (3). Among them, H out is the height of the output feature map, H in is the height of the input feature map, F H is the convolution kernel size, W out is the width of the output feature map, W in is the width of the input feature map, F W is the convolution kernel size, P is the padding size, and s is the stride.

8. The method according to claim 4, characterized in that In S34, the images below the recognition threshold in S33 are added to the loop queue and regenerated. After multiple cycles, when the recognition rate of the generated images tends to be stable, the image that finally meets the recognition threshold is saved, and the recognition threshold is updated to the average recognition rate of all saved images, so as to dynamically adjust the threshold to improve the quality of the generated image; wherein, the stability of the recognition rate after step S33 is judged by the convergence of the mean and variance.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for expanding a facial expression data set based on a large language model as described in any one of claims 1 to 8 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for expanding a facial expression dataset based on a large language model according to any one of claims 1 to 8 is implemented.

Images