Expression detection network training method and device and electronic equipment
By introducing the supervision signal of the expression renderer into the AU detection network, the problem of lack of supervision information in the existing AU detection network training is solved, the detection accuracy is improved and the technology is promoted.
Patent Information
- Application Number
- CN202310890812.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-19
- Publication Date
- 2025-01-21
AI Technical Summary
The training of existing AU detection networks lacks additional supervision information, resulting in low detection accuracy, limiting the application and promotion of technology.
Real face images are obtained and input into the AU detection network to be trained. After outputting AU labels, these labels are input into the pre-trained expression renderer, and the supervision signal of the renderer is used to assist in training the AU detection network.
It improves the accuracy of the AU detection network and promotes its promotion and development in fields such as human-computer interaction and autonomous driving.
Smart Images
Figure CN119339202A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a training method, device and electronic equipment for an expression detection network. Background Art
[0002] Facial expressions can be divided into dozens of basic action units according to the movement of facial muscle groups, such as raising the inner eyebrow, squinting, pulling down the chin, lifting the corners of the mouth, etc., forming a complete facial action unit coding system, also known as FACS (Facial Action Coding System). Any human expression can be represented by a group of action units and their combinations of different intensities. For example, a laughing expression can be composed of action units such as pulling down the chin, opening the mouth, lifting the corners of the mouth, lifting the cheeks, squinting, etc.
[0003] The AU (Action Unit) facial action unit detection task is based on the input face image, detecting the AU category or the basic expression category appearing in it. Therefore, AU detection helps to obtain detailed information related to facial expressions from the image. According to AU detection, the local muscle movements of facial expressions can also be obtained, and the expressions can also be inferred through the combination of AU detection.
[0004] Since AU detection has the advantages of objectivity, fine-grainedness, and quantifiability, and can characterize human expressions and emotions in detail, it has very broad application prospects in human-computer interaction, autonomous driving, film and television or animation production, medical services, and other fields.
[0005] In recent years, many domestic and foreign scholars have proposed many related technologies, but there are still many challenges that have not been resolved, and the current technology also has many shortcomings. For example, the training of the neural network used for AU detection lacks additional supervision information to improve the accuracy of the AU detection network, etc. This not only limits the accuracy of AU detection technology, but also limits the promotion and development of AU detection technology to a certain extent. Summary of the invention
[0006] In view of this, the object of the present invention is to provide a training method, device and electronic device for an expression detection network to alleviate the above technical problems.
[0007] In a first aspect, an embodiment of the present invention provides a training method for an expression detection network, the method comprising: obtaining a first facial image, wherein the first facial image is a real facial image; inputting the first facial image into the expression detection network to be trained, and outputting an AU label of the first facial image through the expression detection network to be trained; inputting the AU label into a pre-trained expression renderer, and supervising the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets preset conditions.
[0008] In a second aspect, an embodiment of the present invention further provides a training device for an expression detection network, the device comprising: a second acquisition module, used to acquire a first face image, wherein the first face image is a real face image; an input module, used to input the first face image into the expression detection network to be trained, and output the AU label of the first face image through the expression detection network to be trained; a supervision module, used to input the AU label into a pre-trained expression renderer, and supervise the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets preset conditions.
[0009] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.
[0010] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the first aspect.
[0011] The embodiments of the present invention bring the following beneficial effects:
[0012] The training method, device and electronic device of the expression detection network provided by the embodiment of the present invention can obtain a first face image containing a real face image, input the first face image into the expression detection network to be trained, output the AU label of the first face image through the expression detection network to be trained, and input the AU label into a pre-trained expression renderer, and supervise the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets the preset conditions. During the training process, the supervision process of the expression renderer can be used as a new supervision signal to assist the training of the expression detection network, which not only helps to improve the accuracy of the expression detection network, but also promotes the promotion and development of the expression detection network to a certain extent.
[0013] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.
[0014] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0016] Figure 1 A flowchart of a method for training an expression detection network provided by an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of a training process of an expression detection network to be trained provided in an embodiment of the present invention;
[0018] Figure 3 A flowchart of a training process of an expression renderer provided by an embodiment of the present invention;
[0019] Figure 4 A schematic diagram of a training process of an expression renderer provided by an embodiment of the present invention;
[0020] Figure 5 A schematic diagram of the structure of a training device for an expression detection network provided by an embodiment of the present invention;
[0021] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0023] AU detection technology can be used to detect facial movements. For example, based on the input face image, the AU categories appearing in it can be detected. Usually, multiple AU categories often appear in a face image, so this task is a multi-label classification problem. For example, if the category exists, the output of the category is 1, otherwise the output of the category is 0; for example, if the input is a laughing expression, the result of AU detection is usually 1 for open mouth, 1 for raised mouth corners, 0 for pouting, 1 for squinting, and so on.
[0024] Therefore, the current neural network models based on AU detection technology usually use neural networks to learn facial image features, and then predict the probability of each AU category from the image features. The most advanced method is based on deep learning. However, since the AU detection task itself is a multi-label binary classification task, during the training process of the neural network, the existing deep learning methods can only use conventional classification loss functions, such as binary cross entropy loss or alignment weighted loss, etc., and the supervision of the network training process is also relatively simple.
[0025] Based on this, the training method, device and electronic device of an expression detection network provided by the embodiments of the present invention can effectively alleviate the above technical problems.
[0026] To facilitate understanding of this embodiment, a training method for an expression detection network disclosed in an embodiment of the present invention is first introduced in detail.
[0027] In a possible implementation, an embodiment of the present invention provides a method for training an expression detection network.
[0028] Specifically, Figure 1 A flow chart of a method for training an expression detection network is shown, the method comprising the following steps:
[0029] Step S102, obtaining a first face image;
[0030] The first face image in the embodiment of the present invention is a real face image;
[0031] Step S104, inputting the first face image into the expression detection network to be trained, and outputting the AU label of the first face image through the expression detection network to be trained;
[0032] Step S106, inputting the AU label to a pre-trained expression renderer, and supervising the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets the preset conditions;
[0033] Among them, the expression detection network to be trained in the embodiment of the present invention is also called the AU detection network to be trained. Usually, the goal of the AU detection network is to identify whether certain AU categories appear in the input real face image. The input of the AU detection network is the real face image, and the output is the probability prediction value of each AU category. The traditional AU detection network training process can only impose classification loss on the output probability prediction value of the AU category. In the embodiment of the present invention, based on the supervised training process of the expression renderer, relevant losses can be imposed on the expression image to give the expression detection network to be trained more supervision signals.
[0034] Therefore, the training method of the expression detection network provided by the embodiment of the present invention can obtain a first face image containing a real face image, input the first face image into the expression detection network to be trained, output the AU label of the first face image through the expression detection network to be trained, and input the AU label into a pre-trained expression renderer, and supervise the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets the preset conditions. During the training process, the supervision process of the expression renderer can be used as a new supervision signal to assist the training of the expression detection network, which not only helps to improve the accuracy of the expression detection network, but also promotes the promotion and development of the expression detection network to a certain extent.
[0035] In actual use, the expression renderer in the embodiment of the present invention is a pre-constructed mapping network from AU labels to images. Therefore, in the above step S106, when the expression renderer supervises the training process of the expression detection network to be trained, the expression renderer can output the second face image corresponding to the AU label; then the loss of the first face image and the second face image is calculated until the loss is less than a preset loss threshold, indicating that the expression detection network to be trained meets the preset conditions, that is, the training process of the expression detection network to be trained is completed.
[0036] For ease of understanding, Figure 2 A schematic diagram of the training process of the expression detection network to be trained is also shown. Figure 2 As shown, it includes the following processes:
[0037] (1) Real face image input process;
[0038] In this process, a real face image is input into the expression detection network to be trained, and the expression detection network to be trained outputs the AU label of the real face image, wherein the AU label is usually a predicted label output by the expression detection network to be trained. Based on the AU label, a classification loss can be applied to the AU label through the traditional AU detection network training process to calculate the AU classification loss function.
[0039] In the embodiment of the present invention, the AU label output by the expression detection network to be trained can be further mapped to the expression image through the expression renderer. Figure 2 (2) process in .
[0040] (2) Expression renderer rendering process:
[0041] In this process, an expression renderer is used to map to an expression image, i.e., the second face image mentioned above, and then an image-related loss function is applied to the second face image, such as pixel loss, expression loss, perceptual loss, etc. of the true value image, so that the training process of the expression detection network to be trained can be supervised by more supervisory signals.
[0042] It should be noted that Figure 2 In the process (2), the expression renderer does not participate in training, but only makes predictions. Moreover, the network of the expression renderer is differentiable and can backpropagate the gradient.
[0043] In execution Figure 2 In the steps shown, an expression renderer that maps from AU labels to character images is pre-trained, and the expression renderer can be used in all AU-related detection and classification tasks to map an AU label to a character expression image. The expression detection network obtained through supervised training of the expression renderer, i.e., the AU detection network, can be used to detect the activation of certain AU categories in the input real face image, and can be used in scenarios such as expression analysis.
[0044] In actual use, in order to obtain the above expression renderer so as to supervise the training process of the expression detection network to be trained using the expression renderer, it is necessary to first construct the above expression renderer before training the expression detection network to be trained.
[0045] Specifically, the expression renderer in the embodiment of the present invention is a pre-built mapping network from AU labels to images, and the expression renderer is trained based on an expression image pairing data set, wherein each sample data in the expression image pairing data set includes an AU label and an expression sample image corresponding to the AU label, so as to train the expression renderer.
[0046] Therefore, the embodiment of the present invention also includes a training process of the expression renderer, specifically, Figure 3 The flowchart of a training process of an expression renderer shown includes the following steps:
[0047] Step S302, obtaining a pre-constructed expression image pairing dataset;
[0048] In the embodiment of the present invention, each sample data in the expression image pairing data set includes an AU label and an expression sample image corresponding to the AU label;
[0049] In the field of movies and games, the face of a virtual character is just a series of three-dimensional geometric vertices and patches. In order to make it present the same facial expressions as real-life people, animators will manually bind the facial expressions of the virtual character to obtain a series of basic expression channels for the virtual character, such as left / right eyebrow raising, left / right eye closing, left / right mouth corner pulling up, left / right pouting, open mouth, pursed lips, etc., and define an action range for each basic expression channel, for example, from 0 to 1, representing the minimum action amplitude (0) and the maximum action amplitude (1) respectively; in fact, when animators bind the basic expressions of the characters, they often refer to the Facial Action Coding System (FACS). System), that is to say, when an animator defines each base expression channel, it may have a physical correspondence with a specific facial action unit, and at the same time, the weight value of the base expression and the corresponding facial action unit also have a corresponding relationship. On the basis of the bound virtual character model, by assigning appropriate weight values to each base expression channel of the model, a variety of rich facial expressions can be combined. The expression image pairing data set in the embodiment of the present invention, wherein the AU label included in each sample data is equivalent to the weight value assigned to each base expression channel, and the expression sample image corresponding to the AU label is the facial image finally corresponding to the weight value assigned to each base expression channel.
[0050] Step S304, extracting AU labels from the sample data in sequence, inputting the AU labels into a pre-built initial expression renderer, and outputting a predicted expression image corresponding to the AU labels through the initial expression renderer;
[0051] Step S306, calculating the loss function of the predicted expression image and the expression sample image corresponding to the AU label until the initial expression renderer converges to obtain the expression renderer;
[0052] The expression renderer in the embodiment of the present invention is used to perform supervised training on the expression detection network.
[0053] In actual use, the initial expression renderer pre-constructed in the above step S304 is usually an initialized neural network model, and the expression image pairing data set in the above step S302 is the data set used to train the initial expression renderer. In addition, the input of the initial expression renderer is the AU label, and the output is the predicted expression image corresponding to the AU label. In addition, the initial expression renderer is constrained to convergence through the loss function of step S306, and a trained expression renderer can be obtained. The input of the expression renderer is any set of AU labels, and the output is the expression image corresponding to the AU label, that is, the process of mapping from AU to character expression image is realized.
[0054] And because the AU label of the expression renderer can be used for supervised training of the expression detection network, the expression renderer can be used as a new supervisory signal to assist the training process of the expression detection network, which not only helps to improve the accuracy of the expression detection network, but also promotes the promotion and development of the expression detection network to a certain extent.
[0055] In actual use, the above-mentioned expression image pairing dataset is actually a pairing dataset of AU labels to virtual character expression images, and the expression image pairing dataset is constructed based on the AU dataset. Usually, in order to realize the pairing of AU labels to virtual character expression images, it is necessary to first obtain a preset AU dataset, and then obtain the base expression data corresponding to the AU label of each sample in the AU dataset according to the correspondence between the AU label and the expression channel; and perform expression rendering on the base expression data to obtain the expression sample image corresponding to the AU label; and then construct the expression image pairing dataset according to the AU label and the expression sample image corresponding to the AU label.
[0056] For example, given an AU dataset, each sample has AU labels of multiple categories. For example, assuming there are 12 AU categories: (AU1\AU2\AU4\AU6\AU7\AU9\AU10\AU12\AU14\AU15\AU17\AU23\AU24) If the sample exists in a certain AU category, the AU category is marked as 1, otherwise it is marked as 0. Therefore, the AU label of each sample in the dataset is a set of 0 / 1 sequences of length 12, and the 0 / 1 at each position represents the presence or absence of a certain AU category. Then, the correspondence between the AU label and the base expression of the virtual character is obtained; the base expression data corresponding to the AU label can be obtained, such as the mapping of the AU label and the character expression rig, thereby obtaining the character rig data corresponding to the AU label, for example, the AU1 corresponding It is the rig channel for the inner eyebrow raising of the character, AU2 corresponds to the rig channel for the outer eyebrow raising of the character, AU17 corresponds to the rig channel for the chin raising of the character, and so on. The rig data corresponding to the AU label can be obtained in the above manner to form an AU label-rig data pair. The rig data therein is input into a pre-built three-dimensional software and the character's expression image is rendered to obtain the expression map corresponding to the rig data. The expression map is the expression sample image corresponding to the AU label, thereby constructing the paired data of AU label-rig data-expression sample image. By performing the above operation multiple times, an AU label-rig data-expression sample image data set can be obtained. The AU label-expression sample image therein can be extracted to obtain the above-mentioned expression image paired data set in the embodiment of the present invention.
[0057] Furthermore, in order to facilitate image processing during expression renderer training, in the embodiment of the present invention, when constructing an expression image pairing dataset, the expression sample image can also be scaled to a preset resolution to obtain a standard expression sample image; for example, each expression sample image is scaled to a standard expression sample image with a resolution of 256×256, and then the standard expression sample image and the above AU label form a data pair, which is saved as a sample data to the expression image pairing dataset. The specific resolution size can be set according to actual usage, and the embodiment of the present invention does not limit this.
[0058] Furthermore, in an embodiment of the present invention, the neural network model used to construct the above-mentioned initial expression renderer includes an input layer, a convolutional layer and an output layer connected in sequence; wherein the input layer is a linear layer, and the input dimension of the input layer is consistent with the number of categories of the AU label of the sample data; for example, if an AU label includes 12 categories, the input dimension of the input layer is also 12, and if an AU label includes 8 categories, the input dimension of the input layer is also 8. The specific input dimension is subject to actual usage, and the embodiment of the present invention is not limited to this.
[0059] Furthermore, the convolution layer includes a preset number of deconvolution network layers; and the output layer includes a preset activation function.
[0060] In actual use, since the deconvolution network layer is used when the convolution layer is used, the expression renderer in the embodiment of the present invention is actually a neural network model renderer based on the deconvolution network, and the input layer of the neural network model renderer is a linear layer. The input dimension of the linear layer is consistent with the number of categories of the AU label. For example, if an AU label includes 12 categories, 12 neurons can be set in the linear layer to correspond to the 12 categories of the AU label. In actual use, the number of the above-mentioned deconvolution network layers is generally 7 layers. The input AU label can output a feature map after passing through the 7 deconvolution network layers, and finally pass through the activation function mother of the output layer, such as the Sigmoid activation function, to obtain a predicted expression image.
[0061] Furthermore, based on the above constructed expression image pairing data set and the neural network model of the initial expression renderer, the neural network model of the initial expression renderer can be trained on the basis of the above expression image pairing data set.
[0062] During specific training, the AU label can be input into the input layer, and then the AU label can be input into the convolution layer through the input layer; then the feature map of the AU label is output through the convolution layer, and the predicted expression image corresponding to the AU label is obtained through the output layer, and then the loss function of the predicted expression image and the expression sample image corresponding to the AU label is further calculated until the initial expression renderer converges.
[0063] For ease of understanding, Figure 4 A schematic diagram of the training process of an expression renderer is also shown. Figure 4 As shown, it includes the following processes:
[0064] (1) randomly sampling a batch of sample data from the expression image pairing dataset as a training set, and training the initial expression renderer on the training set;
[0065] This process corresponds to Figure 4 (1) Random sampling in .
[0066] (2) extracting AU labels from sample data, inputting them into a pre-built initial expression renderer, and outputting predicted sample images through the initial expression renderer;
[0067] The process corresponds to Figure 4 In the rendering process (2), after 7 layers of deconvolution network, the final output is a 256*256*3 RGB image as the prediction sample image;
[0068] (3) The expression sample images corresponding to the AU labels in the sample data are used as the true values to supervise the training of the initial expression renderer, and the training loss function adopts the L1 loss function.
[0069] This process corresponds to Figure 4 In the actual training process, a suitable optimizer, such as the Adam optimizer, can be used for the loss function (3) in the above figure. In addition, the data batch size is set to 16. A larger learning rate (0.001) is first used to learn the initial expression renderer network. As the number of network iterations increases, the learning rate decreases exponentially until the network fully converges.
[0070] Furthermore, the L1 loss function used in training the initial expression renderer in the embodiment of the present invention includes three types, namely, mean square error loss (mse), mean absolute value loss (mae), and similarity loss (ssim).
[0071] based on Figure 4 The training process shown can obtain a neural network expression renderer that maps AU labels to character expression images. In practical applications, based on this expression renderer, as long as a set of AU labels are input, the corresponding character expression images can be obtained. This expression renderer can be used as a supervisory signal of the expression detection network to assist the training of the AU detection network.
[0072] therefore, Figure 4 The training process shown in the figure can obtain a differentiable expression renderer that maps AU to character expression images. First, a dataset of expression images with AU labels is required. The dataset is derived from the labels of the real AU dataset, so the AU labels contained are real and reasonable, and contain a variety of AU category combinations; secondly, through the process of constructing data from AU labels to character expression images, a huge amount of data can be constructed infinitely in theory, which is of great benefit to the subsequent neural network training; further, the expression renderer that maps AU labels to character expression images obtained above, in practical applications, can obtain the corresponding character expression images as long as a set of AU labels are input. The expression renderer can be used as a new supervisory signal to assist the training of the AU detection network. In short, the expression renderer is differentiable and can predict expression images under a set of AU labels, so it can be used as additional supervisory information for the AU detection network to assist the training of the expression detection network, which not only helps to improve the accuracy of the expression detection network, but also promotes the promotion and development of the expression detection network to a certain extent.
[0073] Furthermore, the embodiment of the present invention also provides a training device for an expression detection network, such as Figure 5 A structural schematic diagram of a training device for an expression detection network is shown, the device comprising:
[0074] A second acquisition module 50, configured to acquire a first face image, wherein the first face image is a real face image;
[0075] An input module 52, used for inputting the first face image into the expression detection network to be trained, and outputting the AU label of the first face image through the expression detection network to be trained;
[0076] The supervision module 54 is used to input the AU label to a pre-trained expression renderer, and supervise the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets a preset condition.
[0077] Furthermore, the expression renderer provided in the embodiment of the present invention is a pre-constructed mapping network from AU labels to images; in the above-mentioned supervision module 54, the step of supervising the training process of the expression detection network to be trained by the expression renderer includes:
[0078] Outputting a second facial image corresponding to the AU label through the expression renderer; calculating the loss of the first facial image and the second facial image until the loss is less than a preset loss threshold.
[0079] Further, in an embodiment of the present invention, the expression renderer is obtained by training based on an expression image pairing data set, and each sample data in the expression image pairing data set includes an AU label and an expression sample image corresponding to the AU label;
[0080] The above device is also used for:
[0081] Acquire the pre-constructed expression image pairing data set; extract AU labels from the sample data in sequence, input the AU labels into a pre-constructed initial expression renderer, and output the predicted expression image corresponding to the AU label through the initial expression renderer; calculate the loss function of the predicted expression image and the expression sample image corresponding to the AU label until the initial expression renderer converges to obtain the expression renderer;
[0082] Among them, the initial expression renderer includes an input layer, a convolutional layer and an output layer connected in sequence;
[0083] The input layer is a linear layer, and the input dimension of the input layer is consistent with the number of categories of the AU labels of the sample data;
[0084] The convolutional layer includes a preset number of deconvolutional network layers;
[0085] The output layer includes a preset activation function.
[0086] The loss functions used in the above training of the initial expression renderer include: mean square error loss, mean absolute value loss, and similarity loss.
[0087] Furthermore, the step of inputting the AU tag into a pre-built initial expression renderer and outputting a predicted expression image corresponding to the AU tag through the initial expression renderer includes: inputting the AU tag into the input layer, and inputting the AU tag into the convolution layer through the input layer; outputting a feature map of the AU tag through the convolution layer, and obtaining the predicted expression image corresponding to the AU tag through the output layer.
[0088] Furthermore, the above-mentioned device is also used to: obtain a pre-set AU data set, wherein the AU tag of each sample in the AU data set carries a sequence containing multiple base expression categories; obtain base expression data corresponding to the AU tag of each sample in the AU data set according to the correspondence between the AU tag and the expression channel; perform expression rendering on the base expression data to obtain an expression sample image corresponding to the AU tag; and construct an expression image pairing data set according to the AU tag and the expression sample image corresponding to the AU tag.
[0089] Among them, the step of constructing an expression image pairing data set based on the AU tag and the expression sample image corresponding to the AU tag includes: scaling the expression sample image to a preset resolution to obtain a standard expression sample image; forming a data pair with the standard expression sample image and the AU tag, and saving it as a sample data to the expression image pairing data set.
[0090] The training device for the expression detection network provided in the embodiment of the present invention has the same technical features as the training method for the expression detection network provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.
[0091] Furthermore, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0092] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.
[0093] Furthermore, an embodiment of the present invention also provides a schematic diagram of the structure of an electronic device, such as Figure 6As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer executable instructions that can be executed by the processor 61, and the processor 61 executes the computer executable instructions to implement the above method.
[0094] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63 , wherein the processor 61 , the communication interface 63 and the memory 60 are connected via the bus 62 .
[0095] Among them, the memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 62 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0096] The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 61 or the instructions in the form of software. The above processor 61 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0097] The computer program product of the expression detection network training method, device and electronic device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.
[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0099] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0101] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0102] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A training method for an expression detection network, characterized in that: The method comprises: Acquire a first face image, wherein the first face image is a real face image; Inputting the first face image into the expression detection network to be trained, and outputting the AU label of the first face image through the expression detection network to be trained; The AU label is input into a pre-trained expression renderer, and the expression renderer is used to supervise the training process of the expression detection network to be trained until the expression detection network to be trained meets a preset condition.
2. The method according to claim 1, characterized in that The expression renderer is a pre-built mapping network from AU labels to images; The step of supervising the training process of the expression detection network to be trained by the expression renderer comprises: Outputting a second face image corresponding to the AU label through the expression renderer; The loss of the first facial image and the second facial image is calculated until the loss is less than a preset loss threshold.
3. The method according to claim 2, characterized in that The expression renderer is trained based on an expression image pairing data set, and each sample data in the expression image pairing data set includes an AU label and an expression sample image corresponding to the AU label; The method comprises: Obtaining the pre-constructed expression image pairing dataset; Extracting AU tags from the sample data in sequence, inputting the AU tags into a pre-built initial expression renderer, and outputting a predicted expression image corresponding to the AU tags through the initial expression renderer; The loss function of the predicted expression image and the expression sample image corresponding to the AU label is calculated until the initial expression renderer converges to obtain an expression renderer.
4. The method according to claim 3, characterized in that The initial expression renderer comprises an input layer, a convolutional layer and an output layer connected in sequence; The input layer is a linear layer, and the input dimension of the input layer is consistent with the number of categories of the AU labels of the sample data; The convolutional layer includes a preset number of deconvolutional network layers; The output layer includes a preset activation function.
5. The method according to claim 3, characterized in that: The loss functions used in training the initial expression renderer include: mean square error loss, mean absolute value loss, and similarity loss.
6. The method according to claim 4, characterized in that The step of inputting the AU label into a pre-built initial expression renderer, and outputting a predicted expression image corresponding to the AU label through the initial expression renderer comprises: Inputting the AU label to the input layer, and inputting the AU label to the convolution layer through the input layer; The feature map of the AU label is output through the convolution layer, and the predicted expression image corresponding to the AU label is obtained through the output layer.
7. The method according to claim 3, characterized in that The method further comprises: Acquire a preset AU data set, wherein the AU label of each sample of the AU data set carries a sequence including multiple base expression categories; Obtaining base expression data corresponding to the AU label of each sample in the AU data set according to the correspondence between the AU label and the expression channel; Performing expression rendering on the base expression data to obtain an expression sample image corresponding to the AU label; An expression image pairing data set is constructed according to the AU label and the expression sample image corresponding to the AU label.
8. The method according to claim 7, characterized in that The step of constructing an expression image pairing data set according to the AU label and the expression sample image corresponding to the AU label comprises: Scaling the expression sample image to a preset resolution to obtain a standard expression sample image; The standard expression sample image and the AU label form a data pair, which is saved as a sample data in the expression image pairing data set.
9. A training device for an expression detection network, characterized in that: The device comprises: A second acquisition module, used to acquire a first face image, wherein the first face image is a real face image; An input module, used for inputting the first face image into the expression detection network to be trained, and outputting the AU label of the first face image through the expression detection network to be trained; The supervision module is used to input the AU label into a pre-trained expression renderer, and supervise the training process of the expression detection network to be trained through the expression renderer until the expression detection network to be trained meets the preset conditions.
10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are executed.