A Micro-Search-Based Method and System for Facial Expression Data Augmentation

Abstract

This invention discloses a micro-search-based method for facial expression data augmentation, comprising the following steps: performing face detection on the original image to obtain facial key points, filtering expression-related facial key points, and constructing an effective decision subset of the original image; using a differential evolution algorithm to search the effective decision subset of the original image to obtain facial key points of the augmented image, and constructing an effective decision subset of the augmented image; performing triangulation on the original image, and performing image transformation based on the effective decision subsets of the original image and the augmented image to obtain the augmented image; calculating the fitness value of the augmented image, adjusting the search parameters based on the fitness value, iteratively searching the effective decision subset of the original image, and outputting the optimal augmented image. This invention achieves simple, fast, and efficient facial expression data augmentation by incorporating the augmented dataset corresponding to the optimal augmented image into the original facial expression dataset corresponding to the original image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for augmenting facial expression data based on micro-search. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence, facial expression recognition technology has gradually become a research hotspot in the field of image processing technology. However, existing facial expression datasets generally suffer from problems such as small quantity and imbalanced categories, resulting in poor model generalization ability. Data augmentation algorithms can utilize existing datasets to generate similar data with more features, thereby making the features of the augmented dataset more diverse. Traditional data augmentation methods are mostly based on local image transformations, with less consideration for the content information of the image itself, making it difficult to achieve ideal results when processing small image datasets.

[0003] Deep learning-based data augmentation methods require a large amount of data to ensure learning effectiveness. Therefore, sufficient facial expression data is extremely important, both for training facial expression recognition models and for deep learning-based facial data augmentation.

[0004] Each pixel in the augmented sample is an individual entity, interconnected with its neighbors. The total number of pixels in the image collectively constitutes the whole of the sample image. Optimizing every single pixel in the sample image would be extremely time-consuming, leading to large-scale optimization problems. Micro-search assumes that the effective decision subset of a large-scale optimization problem is much smaller than the problem's decision set. By constructing a smaller effective decision subset and searching within that subset, search capabilities outside the effective decision subset are saved.

[0005] Therefore, in order to improve the efficiency of the sample augmentation method and reduce the number of pixels involved in the search optimization, the algorithm needs to construct an effective decision subset for the sample augmentation method, thus reducing the search optimization problem for all pixels of the sample image to a search optimization problem only within the effective decision subset. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing facial expression augmentation methods that lack attention to the feature content of the image itself, and to provide a facial expression data augmentation method and system based on micro-search, which can achieve simple, fast and efficient facial expression data augmentation.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A micro-search-based facial expression data augmentation method includes the following steps:

[0009] S1. Perform face detection on the original image, obtain facial key points in the original image, filter facial key points related to expressions, and construct an effective decision subset of the original image;

[0010] S2. Use the differential evolution algorithm to search for the effective decision subset of the original image, obtain the facial key points of the augmented image, and construct the effective decision subset of the augmented image;

[0011] S3. Triangulate the original image, and perform image transformation based on the effective decision subsets of the original image and the effective decision subsets of the augmented image to obtain the augmented image;

[0012] S4. Calculate the fitness value of the augmented image, adjust the search parameters according to the fitness value, iteratively search the effective decision subset of the original image, and output the optimal augmented image.

[0013] Further, in step S1, the facial landmark detection model is used to locate the facial landmarks of the original images in the original dataset to obtain the set of facial landmarks of the original images; facial landmarks related to facial expressions are filtered out, and facial landmarks that do not affect facial expressions are removed to construct an effective decision subset of the original images.

[0014] Furthermore, in step S2, a population is established for each original image. Each individual in the population consists of corresponding facial key points. The region to which the facial key points belong is determined by using the index of each facial key point in the individual. Different strategies are used to initialize the population according to the different regions.

[0015] Furthermore, in step S3, facial key points that do not affect facial expressions are added back to the effective decision subset of the original image, boundary points of the original image are added to the effective decision subset of the original image, and triangulation is performed on the points on the effective decision subset of the original image to obtain a triangular mesh corresponding to the effective decision subset of the original image.

[0016] The position coordinates of facial key points that do not affect facial expressions and the boundary points of the original image are added to the set of new position coordinates of facial key points. Using the affine transformation formula, the position coordinates used for affine transformation are calculated using the old and new position coordinates of facial key points.

[0017] Traverse the triangles in the triangular mesh, use the old position coordinates of each vertex of the triangle and the position coordinates used for affine transformation to generate an affine transformation matrix, and transform the original image according to the affine transformation matrix to generate an augmented image.

[0018] Furthermore, the formula for affine transformation is:

[0019] x m= (1-α)*x i +α*x j

[0020] y m =(1-α)*y i +α*y j

[0021] In the formula, (x i ,y i ) represents the old position coordinates of the facial landmarks, i.e., the position coordinates of the facial landmarks in the original image; (x j ,y j ) represents the new coordinates of the facial landmarks, i.e., the coordinates of the facial landmarks in the augmented image; α represents the affine transformation parameters; (x m ,y m ) represents the position coordinates used for affine transformation.

[0022] Furthermore, the boundary points of the original image include the four corner vertices of the original image and the center points of the four boundaries.

[0023] Furthermore, in step S4, the pre-trained facial expression recognition model is used to screen and evaluate the augmented image set, remove augmented images that do not meet the constraints, and select the best individuals to form a new parent population based on the fitness value of the augmented images.

[0024] If the search reaches the maximum number of iterations, the search is terminated, the optimal augmented image is output, and the augmented dataset corresponding to the optimal augmented image is merged into the original dataset. Otherwise, the optimal individual is set as the new parent population, and the search is repeated iteratively until the optimal augmented image is obtained.

[0025] A micro-search-based facial expression data augmentation system, employing any of the micro-search-based facial expression data augmentation methods described above, includes:

[0026] The facial landmark detection module is used to perform face detection on the original images corresponding to the original facial expression dataset, obtain the facial landmarks of the original images, and construct an effective decision subset of the original images;

[0027] The facial landmark search module is used to search for the effective decision subset of the original image using the differential evolution algorithm, obtain the facial landmarks of the augmented image, and construct the effective decision subset of the augmented image.

[0028] The image transformation module is used to perform image transformation on the original image based on the effective decision subset of the original image and the effective decision subset of the augmented image to obtain the augmented image;

[0029] The facial expression recognition module is used to screen and evaluate augmented images, select the best augmented image, and merge the augmented dataset corresponding to the best augmented image into the original dataset.

[0030] An electronic device includes a memory and a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the micro-search-based facial expression data augmentation method as described in any of the preceding claims.

[0031] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the micro-search-based facial expression data augmentation method as described in any of the preceding claims.

[0032] Compared with existing technologies, the micro-search-based facial expression data augmentation method of this invention reduces search parameters, narrows the search range of the algorithm, and improves the solution efficiency by constructing a decision-effective subset of the original images corresponding to the original facial expression dataset. This achieves efficient facial expression augmentation, and the augmentation process is simple and effective. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a micro-search-based method for augmenting facial expression data.

[0034] Figure 2 A diagram illustrating key facial features.

[0035] Figure 3 A diagram illustrating key facial features.

[0036] Figure 4 This is a schematic diagram of the Delaunay triangulation of the original image into a triangular mesh. Detailed Implementation

[0037] The following description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the micro-search-based facial expression data augmentation method and system of the present invention.

[0038] Facial landmarks, located in areas such as facial contours, eyebrows, and eyes, possess significant expressive power in capturing the complex and varied facial movements and expressions of humans. While the positions of facial landmarks are relatively stable, they exhibit a degree of variability due to factors such as changes in expression or viewing angle. Pixels associated with facial landmarks can serve as key feature pixels in sample images, and searching for facial landmarks can diversify the features of augmented datasets.

[0039] Please see Figure 1 This invention discloses a method for augmenting facial expression data based on micro-search, comprising the following steps:

[0040] S1. Perform face detection on the original image, obtain facial key points in the original image, filter facial key points related to expressions, and construct an effective decision subset of the original image.

[0041] S2. Use the differential evolution algorithm to search for the effective decision subset of the original image, obtain the facial key points of the augmented image, and construct the effective decision subset of the augmented image.

[0042] S3. Triangulate the original image, and perform image transformation based on the effective decision subsets of the original image and the effective decision subsets of the augmented image to obtain the augmented image.

[0043] S4. Calculate the fitness value of the augmented image, adjust the search parameters according to the fitness value, iteratively search the effective decision subset of the original image, and output the optimal augmented image.

[0044] Specifically, please refer to Figure 2 and Figure 3 In step S1, the facial landmark detection model is used to locate the facial landmarks in the original images of the original dataset, resulting in a set of facial landmarks for the original images. Facial landmarks related to facial expressions are selected, while those that do not affect facial expressions are removed, thus constructing an effective decision subset of the original images.

[0045] Specifically, in step S2, a population is established for each original image, and each individual in the population consists of corresponding facial key points. Using the index of each facial key point within an individual, the region to which the facial key point belongs is determined. Different strategies are used to initialize the population according to the different regions. The population is initialized based on a mutation factor, and new offspring populations are generated through crossover and mutation selection operations.

[0046] Specifically, in step S3, facial key points that do not affect facial expressions are re-added to the effective decision subset of the original image, and the boundary points of the original image are added to the effective decision subset of the original image. The boundary points of the original image include the four corner vertices of the original image and the center points of the four boundaries.

[0047] Please see Figure 4 The points on the effective decision subset of the original image are triangulated to obtain a triangular mesh corresponding to the effective decision subset of the original image.

[0048] The position coordinates of facial key points that do not affect facial expressions and the boundary points of the original image are added to the set of new position coordinates of facial key points. Using the affine transformation formula, the position coordinates used for affine transformation are calculated using the old and new position coordinates of the facial key points.

[0049] Traverse the triangles in the triangular mesh, use the old position coordinates of each vertex of the triangle and the position coordinates used for affine transformation to generate an affine transformation matrix, and transform the original image according to the affine transformation matrix to generate an augmented image.

[0050] The affine transformation formula is:

[0051] x m = (1-α)*x i +α*x j

[0052] y m =(1-α)*y i +α*y j

[0053] In the formula, (x i ,y i ) represents the old position coordinates of the facial landmarks, i.e., the position coordinates of the facial landmarks in the original image; (x j ,y j ) represents the new coordinates of the facial landmarks, i.e., the coordinates of the facial landmarks in the augmented image; α represents the affine transformation parameters; (x m ,y m ) represents the position coordinates used for affine transformation.

[0054] Specifically, in step S4, the pre-trained facial expression recognition model is used to screen and evaluate the augmented image set, remove augmented images that do not meet the constraints, and select the best individuals to form a new parent population based on the fitness value of the augmented images.

[0055] If the search reaches the maximum number of iterations, the search is terminated, the optimal augmented image is output, and the augmented dataset corresponding to the optimal augmented image is merged into the original dataset. Otherwise, the optimal individual is set as the new parent population, and the search is repeated iteratively until the optimal augmented image is obtained.

[0056] The augmented image set is constrained and the objective function is evaluated. The constraint condition is to use a pre-trained facial expression recognition model to evaluate the target category probability of the augmented image. If the evaluated probability is greater than a threshold, it means that the augmented image meets the constraint condition and is regarded as a feasible solution. The objective function is to use diversity to evaluate the augmented image.

[0057] In summary, the facial expression data augmentation method based on micro-search of this invention reduces search parameters, narrows the search range of the algorithm, and improves the solution efficiency by constructing a decision-effective subset of the original images corresponding to the original facial expression dataset. This achieves efficient facial expression augmentation, and the augmentation process is simple and effective.

[0058] This invention also discloses a micro-search-based facial expression data augmentation system, employing any of the micro-search-based facial expression data augmentation methods described above, comprising:

[0059] The facial landmark detection module is used to perform face detection on the original images corresponding to the original facial expression dataset, obtain the facial landmarks of the original images, and construct an effective decision subset of the original images;

[0060] The facial landmark search module is used to search for the effective decision subset of the original image using the differential evolution algorithm, obtain the facial landmarks of the augmented image, and construct the effective decision subset of the augmented image.

[0061] The image transformation module is used to perform image transformation on the original image based on the effective decision subset of the original image and the effective decision subset of the augmented image to obtain the augmented image;

[0062] The facial expression recognition module is used to screen and evaluate augmented images, select the best augmented image, and merge the augmented dataset corresponding to the best augmented image into the original dataset.

[0063] The micro-search-based facial expression data augmentation system of the present invention can execute the micro-search-based facial expression data augmentation method of the present invention, and can execute any combination of the implementation steps of the method embodiments, possessing the corresponding functions and beneficial effects of the method.

[0064] This invention constructs a facial landmark detection module, a facial landmark search module, an image transformation module, and a facial expression recognition module to form a micro-search-based facial expression data augmentation system. Users only need to input a facial expression dataset, and the system can automatically process the facial expression image data to obtain augmented facial expression image data.

[0065] This invention also discloses an electronic device, including a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the micro-search-based facial expression data augmentation method as described in any of the preceding claims. The electronic device of this invention can execute the micro-search-based facial expression data augmentation method of this invention, and can execute any combination of the steps of the method embodiments, possessing the corresponding functions and beneficial effects of the method.

[0066] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the micro-search-based facial expression data augmentation method as described in any of the preceding claims. The computer-readable storage medium of this invention can execute the micro-search-based facial expression data augmentation method of this invention, and can execute any combination of the steps of the method embodiments, possessing the corresponding functions and beneficial effects of the method.

[0067] Although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the system disclosed herein, the actual implementation of the module will be understood within the scope of conventional art for an engineer. Therefore, those skilled in the art can implement the invention set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0068] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0070] Various parts of this invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals; application-specific integrated circuits (ASICs) having suitable combinational logic gates; programmable gate arrays (PGAs); field-programmable gate arrays (FPGAs); etc.

[0071] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit disclosed in the present invention should fall within the patent scope covered by the present invention.

Claims

1. A method for augmenting facial expression data based on micro-search, characterized in that, Includes the following steps: S1. Perform face detection on the original image, obtain facial key points in the original image, filter facial key points related to expressions, and construct an effective decision subset of the original image; S2. Use the differential evolution algorithm to search for the effective decision subset of the original image, obtain the facial key points of the augmented image, and construct the effective decision subset of the augmented image; S3. Triangulate the original image, and perform image transformation based on the effective decision subsets of the original image and the effective decision subsets of the augmented image to obtain the augmented image; S4. Calculate the fitness value of the augmented image, adjust the search parameters according to the fitness value, iteratively search the effective decision subset of the original image, and output the optimal augmented image; In step S1, facial landmarks related to facial expressions are selected, and facial landmarks that do not affect facial expressions are removed to construct an effective decision subset of the original image. In step S3, facial key points that do not affect facial expressions are added back to the effective decision subset of the original image, boundary points of the original image are added to the effective decision subset of the original image, and triangulation is performed on the points on the effective decision subset of the original image to obtain a triangular mesh corresponding to the effective decision subset of the original image. The position coordinates of facial key points that do not affect facial expressions and the boundary points of the original image are added to the set of new position coordinates of facial key points. Using the affine transformation formula, the position coordinates used for affine transformation are calculated using the old and new position coordinates of facial key points. Traverse the triangles in the triangular mesh, use the old position coordinates of each vertex of the triangle and the position coordinates used for affine transformation to generate an affine transformation matrix, and transform the original image according to the affine transformation matrix to generate an augmented image; In step S4, the pre-trained facial expression recognition model is used to screen and evaluate the augmented image set, remove augmented images that do not meet the constraints, and select the best individuals to form a new parent population based on the fitness value of the augmented images. If the search reaches the maximum number of iterations, the search is terminated, the optimal augmented image is output, and the augmented dataset corresponding to the optimal augmented image is merged into the original dataset. Otherwise, the optimal individual is set as the new parent population, and the search is repeated iteratively until the optimal augmented image is obtained.

2. The facial expression data augmentation method based on micro-search according to claim 1, characterized in that, In step S1, the facial landmark detection model is used to locate the facial landmarks in the original images of the original dataset to obtain the set of facial landmarks in the original images.

3. The facial expression data augmentation method based on micro-search according to claim 2, characterized in that, In step S2, a population is established for each original image. Each individual in the population consists of corresponding facial key points. The region to which the facial key point belongs is determined by the index of each facial key point in the individual. Different strategies are used to initialize the population according to the different regions.

4. The facial expression data augmentation method based on micro-search according to claim 1, characterized in that, The formula for affine transformation is: x m =(1-a)*x i +a*x j and m =(1-α)*y i +α*y j In the formula, (x i ,y i ) represents the old position coordinates of the facial landmarks, i.e., the position coordinates of the facial landmarks in the original image; (x j ,y j ) represents the new coordinates of the facial landmarks, i.e., the coordinates of the facial landmarks in the augmented image; α represents the affine transformation parameters; (x m ,y m ) represents the position coordinates used for affine transformation.

5. The facial expression data augmentation method based on micro-search according to claim 1, characterized in that, The boundary points of the original image include the four corner vertices of the original image and the center points of the four boundaries.

6. A facial expression data augmentation system based on micro-search, employing the facial expression data augmentation method based on micro-search as described in any one of claims 1 to 5, characterized in that, include: The facial landmark detection module is used to perform face detection on the original images corresponding to the original facial expression dataset, obtain the facial landmarks of the original images, and construct an effective decision subset of the original images; The facial landmark search module is used to search for the effective decision subset of the original image using the differential evolution algorithm, obtain the facial landmarks of the augmented image, and construct the effective decision subset of the augmented image. The image transformation module is used to perform image transformation on the original image based on the effective decision subset of the original image and the effective decision subset of the augmented image to obtain the augmented image; The facial expression recognition module is used to screen and evaluate augmented images, select the best augmented image, and merge the augmented dataset corresponding to the best augmented image into the original dataset.

7. An electronic device comprising a memory and a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes a computer program to implement the micro-search-based facial expression data augmentation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the micro-search-based facial expression data augmentation method as described in any one of claims 1 to 5.

Images