Lightweight facial beauty evaluation method and apparatus, device and medium

Abstract

Provided in the present application are a lightweight facial beauty evaluation method and apparatus, a device and a medium. The method comprises: using training image data to train an initial evaluation model to obtain a target evaluation model, and then inputting image data to be evaluated into the target evaluation model, so as to perform facial beauty evaluation. The process of training the initial evaluation model comprises: performing block-wise operation on the training image data; on the basis of a feature extraction operator, performing feature extraction on first image data; and performing multi-stage training on the basis of first feature data of different sizes and according to different regularization methods, wherein with the progression of the training stages, the size of the first feature data gradually increases, and the regularization strength of the regularization methods gradually increases. The present application enables networks to achieve quick and stable convergence at different stages, so as to help to prevent problems such as gradient explosion or vanishing gradient during training processes, and also enable models to better adapt to different data distributions and task requirements, thereby improving the generalization ability and performance stability of models.

Description

Lightweight facial beauty evaluation method, device, equipment and medium Technical Field

[0001] The embodiments of the present application relate to the field of image recognition, and in particular to lightweight facial beauty evaluation methods, devices, equipment, and media. Background Art

[0002] Facial beauty prediction, or facial beauty assessment, uses deep convolutional neural networks to evaluate facial images and generate scores. However, these deep convolutional neural networks typically have large network depth, a large number of parameters, and high computing power requirements, making them difficult to deploy efficiently on mobile devices. Furthermore, they are prone to exploding or vanishing gradients during training.

[0003] Summary of the Invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide a lightweight facial beauty evaluation method, device, equipment and medium, which enables the network to converge quickly and stably at different stages, thereby improving the generalization ability and performance stability of the model.

[0006] An embodiment of the first aspect of the present application is a lightweight method for evaluating facial beauty, comprising:

[0007] Obtain training image data;

[0008] Training an initial evaluation model using the training image data to obtain a target evaluation model;

[0009] Inputting the image data to be evaluated into the target evaluation model to evaluate the beauty of the face and obtain an evaluation result;

[0010] In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes, and feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

[0011] According to certain embodiments of the first aspect of the present application, there are M training stages, the size of the first feature data of the first stage is S0, the size of the first feature data of the Mth stage is Se , the size of the first feature data of the i-th stage is S i ,and i is an integer and 1≤i≤M.

[0012] According to certain embodiments of the first aspect of the present application, there are M training stages, and the regularization strength of the regularization method in the first stage is φ0, The regularization strength of the regularization method in the Mth stage is φ e , The regularization strength of the regularization method in the i-th stage is φ i , and i is an integer and 1≤i≤M.

[0013] According to certain embodiments of the first aspect of the present application, performing feature extraction on the first image data according to a feature extraction operator obtained by fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes includes:

[0014] calculating a mean value of the first image data;

[0015] calculating a variance of the first image data;

[0016] Obtaining a feature extraction operator according to the mean, the variance, and a two-dimensional convolution operator;

[0017] Feature extraction is performed on the first image data according to the feature extraction operator to obtain first feature data.

[0018] According to certain embodiments of the first aspect of the present application, performing feature extraction on the first image data according to a feature extraction operator obtained by fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes includes:

[0019] Performing feature extraction on the first image data by an encoder to obtain first feature data;

[0020] The encoder has a size corresponding to the input first image data, and the encoder is provided with a feature extraction operator.

[0021] According to certain embodiments of the first aspect of the present application, after performing a block operation on the training image data to obtain first image data of different sizes, the method includes:

[0022] The first image data is converted into a vector form by using an embedding layer of an initial evaluation model.

[0023] According to certain embodiments of the first aspect of the present application, before performing multi-stage training using first feature data of different sizes and according to different regularization methods, the method includes:

[0024] Average pooling is performed on the first feature data of each channel to obtain pooled first feature data.

[0025] An embodiment of the second aspect of the present application is a lightweight facial beauty evaluation device, comprising:

[0026] An input unit, used to obtain training image data;

[0027] A training unit, configured to train an initial evaluation model using the training image data to obtain a target evaluation model;

[0028] an application unit, configured to input the image data to be evaluated into the target evaluation model to evaluate the beauty of the face and obtain an evaluation result;

[0029] In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes, and feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

[0030] An embodiment of the third aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the lightweight facial beauty evaluation method as described above is implemented.

[0031] An embodiment of the fourth aspect of the present application is a computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the lightweight facial beauty evaluation method as described above.

[0032] The above scheme has at least the following beneficial effects: adopting a progressive training strategy, by gradually increasing the size and complexity of the training dataset and gradually improving the strength of the regularization method, the network can converge quickly and stably at different stages; it not only helps to avoid problems such as gradient explosion or gradient vanishing during training, but also enables the model to better adapt to different data distributions and task requirements, thereby improving the model's generalization ability and performance stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0034] FIG1 is a diagram showing the steps of the training phase of the lightweight face beauty evaluation method;

[0035] FIG2 is a diagram of sub-steps of performing feature extraction on first image data using a feature extraction operator obtained by fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes;

[0036] Figure 3 is a network structure diagram of the evaluation model;

[0037] Figure 4 is a schematic diagram of the principle of multi-stage training according to different regularization methods;

[0038] FIG5 is a structural diagram of a lightweight facial beauty evaluation device. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0041] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0042] The embodiments of the present application propose a lightweight facial beauty evaluation method.

[0043] A lightweight facial beauty evaluation method includes a training phase and an application phase.

[0044] The training phase includes: obtaining training image data, training the initial evaluation model using the training image data to obtain the target evaluation model;

[0045] The application stage includes: obtaining image data to be evaluated, inputting the image data to be evaluated into the target evaluation model to evaluate the beauty of the face, and obtaining the evaluation results.

[0046] In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes. Feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

[0047] In this embodiment, the network structure of the evaluation model is based on the MobileViT model; of course, in other embodiments, the network structure of the evaluation model can also be based on the network structure of other networks. Referring to Figure 3, the network structure of the evaluation model includes a block layer, a block embedding layer, a Transformer encoder, an average pooling layer, a regularization layer, a fully connected layer, and a softmax layer.

[0048] The training image data used to train the initial evaluation model can come from online image databases, such as the Large Scale Asia Facial Beauty Database (LSAFBD), or from local image databases collected in real time.

[0049] The training image data is input into the initial evaluation model to train the initial evaluation model.

[0050] 1 , the training phase includes the following steps.

[0051] Step S100 : performing a block operation on the training image data to obtain first image data of different sizes.

[0052] In step S100, the training image data is divided into blocks to obtain first image data of different sizes. According to the size of the first image data, the first image data is divided into multiple data sets, such as a small-size data set, a medium-size data set, a large-size data set, etc.

[0053] The first image data is converted into a vector form by using an embedding layer of the initial evaluation model.

[0054] Step S200 , performing feature extraction on the first image data according to a feature extraction operator obtained by fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes.

[0055] The first image data of different sizes is input into a plurality of Transformer encoders for feature extraction and representation learning in sequence. During the feature extraction and representation learning process, the Transformer encoder executes step S200 to output the first feature data or feature data further processed from the first feature data.

[0056] In step S200, the encoder performs feature extraction on the first image data based on a feature extraction operator derived from a fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. The first image data of different sizes are respectively input to encoders of corresponding sizes. The encoders have corresponding sizes to the input first image data and are provided with feature extraction operators.

[0057] 2 , feature extraction is performed on the first image data according to a feature extraction operator obtained by fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes, including the following steps:

[0058] Step S210, calculating the mean of the first image data;

[0059] Step S220, calculating the variance of the first image data;

[0060] Step S230, obtaining a feature extraction operator based on the mean, variance, and two-dimensional convolution operators;

[0061] Step S240 , performing feature extraction on the first image data according to a feature extraction operator to obtain first feature data.

[0062] Generally, the convolutional layer (Conv2d) is responsible for extracting image features, while the batch normalization (BN) layer is used to accelerate model training and improve performance. In traditional CNN structures, the Conv2d and BN layers are usually separated, which results in additional computational and storage overhead. In this embodiment, feature extraction is performed by combining Conv2d and batch normalization to form a feature extraction operator, which can effectively simplify the model structure and improve computational efficiency.

[0063] First, the convolution layer applies a filter (kernel) to the input data and produces an output feature map through the convolution operation. Specifically, for the input image X i , the calculation formula of the Conv2d layer is as follows: i =Conv2d(X i );where Y i is the output feature map, and Conv2d represents the convolution operation.

[0064] Next, consider the batch normalization layer, which is used to normalize the input data and speed up the training of the model. i , the calculation formula of the mean μ of the BN layer is as follows: and variance σ 2 The calculation formula is as follows: Where m is the number of input data. Then, for the normalized data The calculation formula of the BN layer is as follows: Here, ε is a small constant used to prevent division by zero.

[0065] Finally, the output Y of the convolutional layer is i Bringing it into the calculation of the BN layer, the feature extraction operator formed by combining Conv2d and batch normalization is expressed as:

[0066] The convolution operation and normalization operation are combined into one step through structural reparameterization, which greatly simplifies the model structure and improves computational efficiency.

[0067] Multiple Transformer encoders output first feature data, which is input to the average pooling layer. The average pooling layer performs global average pooling processing on the first feature data of each channel to obtain pooled first feature data.

[0068] Step S300 , performing multi-stage training using first feature data of different sizes and using different regularization methods to obtain regularized features.

[0069] As the training phase progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

[0070] In the early stage of training, a smaller training size and weak regularization method are used to enable the network to quickly learn simple expression capabilities; in the later stage of training, a larger training size and strong regularization method are used to make the network tend to the training target.

[0071] 4 , a weak regularization method is used for small-sized datasets, a medium regularization method is used for medium-sized datasets, and a strong regularization method is used for large-sized datasets.

[0072] There are M training stages. The size of the first feature data in the first stage is S0, and the size of the first feature data in the Mth stage is S e , the size of the first feature data of the i-th stage is S i ,and i is an integer and 1≤i≤M.

[0073] There are M training stages, and the regularization strength of the regularization method in the first stage is φ0. The regularization strength of the regularization method in the Mth stage is φ e , The regularization strength of the regularization method in the i-th stage is φ i , and i is an integer and 1≤i≤M.

[0074] Regularization methods include dropout rate, RandAugment magnitude, and mixup ratio.

[0075] Through regularization methods, features are randomly discarded to reduce overfitting and reduce the interdependence between neurons in the neural network.

[0076] Step S400: perform classification prediction based on the regularized features to obtain a prediction result.

[0077] In step S400, the retained regularized features are subjected to dimensionality reduction and linear transformation through the fully connected layer, and classification prediction is performed through the Softmax layer to generate a prediction result.

[0078] Step S500: Generate a loss function based on the prediction results, adjust the parameters of the initial evaluation model based on the loss function, and generate a target evaluation model.

[0079] In the application stage, the image data to be evaluated is obtained, and the image data to be evaluated is input into the target evaluation model to evaluate the beauty of the face and obtain the evaluation result.

[0080] In this example, a progressive training strategy is employed, gradually increasing the size and complexity of the training dataset and the strength of the regularization method, enabling the network to converge quickly and stably at different stages. This progressive training strategy not only helps avoid problems such as exploding or vanishing gradients during training, but also enables the model to better adapt to different data distributions and task requirements, thereby improving the model's generalization and performance stability.

[0081] Based on the lightweight model MobileViT, the computational load of the model on mobile devices is significantly reduced by optimizing the inference process. This method has higher computational efficiency and lower resource requirements, providing an efficient face beauty prediction solution for mobile devices.

[0082] The embodiment of the present application provides a lightweight face beauty evaluation device. The lightweight face beauty evaluation device applies the lightweight face beauty evaluation method described above.

[0083] 5 , the lightweight facial beauty evaluation apparatus includes an input unit 110 , a training unit 120 , and an application unit 130 .

[0084] The input unit 110 is used to obtain training image data; the training unit 120 is used to train the initial evaluation model through the training image data to obtain a target evaluation model; the application unit 130 is used to input the image data to be evaluated into the target evaluation model to evaluate the beauty of the face and obtain an evaluation result.

[0085] In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes, and feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

[0086] It can be understood that the lightweight face beauty evaluation device of this embodiment applies the object face beauty evaluation method as described above, and each unit of the lightweight face beauty evaluation device of this embodiment corresponds one-to-one to each step of the object face beauty evaluation method. The lightweight face beauty evaluation device and the object face beauty evaluation method have the same technical solution, solve the same technical problems, and bring the same beneficial effects.

[0087] An embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method for evaluating facial beauty when executing the computer program.

[0088] The electronic device may be any intelligent terminal including a computer.

[0089] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0090] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called by the processor to execute the methods of the embodiments of this application.

[0091] The input / output interface is used to realize information input and output.

[0092] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0093] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0094] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for executing the above-described method for evaluating facial beauty.

[0095] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. In the above description of this specification, the reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0096] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0097] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0101] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A lightweight facial beauty evaluation method, characterized by: include: Obtain training image data; Training an initial evaluation model using the training image data to obtain a target evaluation model; Inputting the image data to be evaluated into the target evaluation model to evaluate the beauty of the face and obtain an evaluation result; In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes, and feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

2. A lightweight facial beauty evaluation method according to claim 1, characterized in that: There are M training stages. The size of the first feature data in the first stage is S0, and the size of the first feature data in the Mth stage is S e , the size of the first feature data of the i-th stage is S i ,and i is an integer and 1≤i≤M.

3. A lightweight facial beauty evaluation method according to claim 1, characterized in that: There are M training stages, and the regularization strength of the regularization method in the first stage is φ0. The regularization strength of the regularization method in the Mth stage is φ e , The regularization strength of the regularization method in the i-th stage is φ i , and i is an integer and 1≤i≤M.

4. A lightweight facial beauty evaluation method according to claim 1, characterized in that: The step of performing feature extraction on the first image data according to a feature extraction operator obtained by fusing two-dimensional convolution and batch normalization to obtain first feature data of different sizes includes: calculating a mean value of the first image data; calculating a variance of the first image data; Obtaining a feature extraction operator according to the mean, the variance, and a two-dimensional convolution operator; Feature extraction is performed on the first image data according to the feature extraction operator to obtain first feature data.

5. A lightweight facial beauty evaluation method according to claim 4, characterized in that: The step of performing feature extraction on the first image data according to a feature extraction operator obtained by fusing two-dimensional convolution and batch normalization to obtain first feature data of different sizes includes: Performing feature extraction on the first image data by an encoder to obtain first feature data; The encoder has a size corresponding to the input first image data, and the encoder is provided with a feature extraction operator.

6. A lightweight facial beauty evaluation method according to claim 1, characterized in that: After performing a block operation on the training image data to obtain first image data of different sizes, the method includes: The first image data is converted into a vector form by using an embedding layer of an initial evaluation model.

7. A lightweight facial beauty evaluation method according to claim 1, characterized in that: Before performing multi-stage training using first feature data of different sizes and according to different regularization methods, the method includes: Average pooling is performed on the first feature data of each channel to obtain pooled first feature data.

8. A lightweight facial beauty evaluation device, characterized in that: include: An input unit, used to obtain training image data; A training unit, configured to train an initial evaluation model using the training image data to obtain a target evaluation model; an application unit, configured to input the image data to be evaluated into the target evaluation model to evaluate the beauty of the face and obtain an evaluation result; In the process of training the initial evaluation model, the training image data is divided into blocks to obtain first image data of different sizes, and feature extraction is performed on the first image data according to the feature extraction operator obtained by the fusion of two-dimensional convolution and batch normalization to obtain first feature data of different sizes. Multi-stage training is performed using the first feature data of different sizes and according to different regularization methods. As the training stage progresses, the size of the first feature data used for training gradually increases, and the regularization strength of the regularization method used for training gradually increases.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the lightweight facial beauty evaluation method according to any one of claims 1 to 7 is implemented.

10. A computer storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the lightweight face beauty evaluation method according to any one of claims 1 to 7.

Images