Model training method and device, nonvolatile storage medium and electronic equipment

By generating and using facial and hair feature images as training data, iterative training of the neural network model is solved, and the texture blur caused by the face sketch image model ignoring the facial structure in the comparison learning method is achieved, and clearer sketch image generation is achieved.

CN118918412APending Publication Date: 2024-11-08BEIJING INFORMATION SCI & TECH UNIV
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410931783.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The face sketch image model trained based on the contrast learning method in the prior art ignores the spatial distribution of the facial structure of the human face, resulting in the generated sketch image having problems such as blurred facial area texture and blurred hair area texture.

Method used

By obtaining optical images containing face images, mask processing to obtain the target image, generate facial feature images and hair feature images, combine these feature images as training data, and iteratively train the neural network model until the preset number is reached, ensuring that The model focuses on the facial area and hair area of ​​the face image during training, thereby generating a clear sketched image.

Benefits of technology

It effectively reduces the number of invalid samples, achieves the balance between positive and negative samples, improves the facial details and hair texture of the sketch image, and solves the problem of blurred local texture of the sketch image in traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918412A_ABST
    Figure CN118918412A_ABST
Patent Text Reader

Abstract

The invention discloses a model training method and device, a nonvolatile storage medium and electronic equipment. The method comprises the steps that an optical image containing a face image, a target image obtained by conducting mask processing on the optical image and a target sketch image corresponding to the optical image are obtained, and the target sketch image contains the face image; feature images are generated according to the optical image, and the feature images comprise a facial feature image and a hair feature image; and performing iterative training on the neural network model by taking the feature image, the optical image and the target image as training data and taking the target sketch image as test data until the number of iterations reaches a preset number of times, and determining that the training is completed. According to the method and the device, the technical problem that the face sketch image output by the model obtained by training has blurred face region texture and blurred hair region texture due to neglect of spatial distribution of the face structure when the model is trained by adopting a comparative learning method in related technologies is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a model training method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] In the related art, the model for generating face sketch images is usually trained by a shallow learning-based method, a deep learning-based method or a contrastive learning method. Among them, the shallow learning-based method trains the model through the idea of ​​reasoning. The model trained by the shallow learning method needs to perform reasoning processing on the input image before outputting the face sketch image, which has the problem of long data processing time. In addition, when the model trained by the shallow learning method is applied to different data sets, the quality of the generated image also has significant differences. The generalization ability of the deep learning-based method is enhanced, but the training cost of the model is significantly increased, and there is a problem of high training cost. The contrastive learning-based method training applies a random sampling strategy when training the model. When sampling the input image according to the random sampling strategy, there is a problem of ignoring the spatial distribution of the facial structure of the face, resulting in a large number of invalid samples in the training process of the model, which increases the complexity of the model training; in addition, because the random sampling strategy ignores the spatial distribution of the facial structure, there is also an imbalance between positive and negative samples. Therefore, the traditional contrastive learning method also has the problem of blurred local texture of the generated sketch face image caused by the imbalance of positive and negative samples.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a model training method and device, a non-volatile storage medium, and an electronic device, so as to at least solve the technical problem that when the contrast learning method is used in the related art to train the model, the spatial distribution of the facial structure of the human face is ignored, and the facial sketch image output by the trained model has blurred texture in the facial area and blurred texture in the hair area.

[0005] According to one aspect of an embodiment of the present application, a model training method is provided, including: acquiring an optical image including a face image, performing mask processing on the optical image to obtain a target image, and a target sketch image corresponding to the optical image, wherein the target sketch image includes a face image; generating a feature image according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image, and the pixel density of the facial region in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair region in the hair feature image is the highest; using the feature image, the optical image, and the target image as training data, and using the target sketch image as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and determining that the training is complete.

[0006] Optionally, a feature image is generated based on the optical image, wherein the hair feature image is generated according to the following method: using a preset pooling window to traverse a pixel matrix generated based on multiple first pixels of the optical image to obtain a global feature matrix of the optical image, wherein each first element in the global feature matrix is ​​an average value of multiple first pixels included in the preset pooling window at each traversal position; processing the global feature matrix to obtain a feature vector of the optical image; generating a first regional feature matrix based on the feature vector and the pixel matrix, wherein the first regional feature matrix includes multiple second elements, the multiple second elements are respectively mapped to multiple second pixels, and the multiple second pixels are used to form the hair feature image.

[0007] Optionally, a feature image is generated according to the optical image, wherein the facial feature image is generated according to the following method: a pixel matrix generated according to multiple first pixels of the optical image is convolved to obtain a convolution matrix, and the convolution matrix is ​​normalized; a second region feature matrix is ​​generated according to the normalized convolution matrix, the pixel matrix and the first region feature matrix, wherein the second region feature matrix includes multiple third elements, the multiple third elements are respectively mapped to multiple third pixels, and the multiple third pixels are used to form the facial feature image.

[0008] Optionally, each iterative training of the neural network model includes: inputting the optical image and the target image into the neural network model to obtain a sketch output by the neural network model, wherein the similarity between the sketch and the optical image is less than the similarity between the target sketch image and the optical image; generating training samples based on the feature image, the optical image and the sketch, wherein the training samples include: positive samples and negative samples, and the image used as the positive sample and the image used as the negative sample are images of different areas in the optical image; determining multiple loss function values ​​based on the training samples, the sketch and the target sketch image; and using the multiple loss function values ​​to perform the next iterative training of the neural network model.

[0009] Optionally, generating training samples based on feature images, optical images and sketches includes: taking the feature images as constraints, performing feature extraction on the optical images to obtain a first type of feature stack, and taking the feature images as constraints, performing feature extraction on the sketches to obtain a second type of feature stack, wherein the first type of feature stack indicates an image of a facial region and an image of a hair region in the optical image, and the second type of feature stack indicates an image of a facial region and an image of a hair region in the sketch; determining any image block in the image indicated by the second type of feature stack as a query image block; and generating training samples based on the query image block and the first type of feature stack.

[0010] Optionally, generating training samples based on the query image block and the first type of feature stack includes: determining first position information of the query image block in the sketch, and determining second position information of each target image block in the optical image among a plurality of target image blocks corresponding to the first type of feature stack, wherein the plurality of target image blocks constitute an image of a facial region and an image of a hair region in the optical image; determining a target image block corresponding to second position information identical to the first position information as a positive sample, and determining the remaining target image blocks as negative samples, wherein the number of positive samples is positively proportional to the pixel density of the query image block.

[0011] Optionally, multiple loss function values ​​are determined based on training samples, sketch sketches and target sketch images, including: determining a patch contrast loss function value among multiple loss function values ​​based on positive samples, negative samples and query samples in the sketch sketch as query image blocks, wherein the patch contrast loss function value is used to indicate the image similarity between the sketch sketch and the optical image, and the patch contrast loss function value and the similarity are inversely proportional; determining an adversarial loss function value among multiple loss function values ​​based on the sketch sketch and the target sketch image, wherein the adversarial loss function value is used to indicate the degree of image similarity between the sketch sketch and the target sketch image, and the adversarial loss function value and the similarity are inversely proportional; determining a pixel-by-pixel loss function value among multiple loss function values ​​based on the sketch sketch and the target sketch image, wherein the pixel-by-pixel loss function value is used to indicate the clarity of the sketch sketch, and the pixel-by-pixel loss function value is inversely proportional to the clarity; determining a perceptual loss function value among multiple loss function values ​​based on the sketch sketch and the target sketch image, wherein the perceptual loss function value is used to evaluate the visual similarity between the sketch sketch and the target sketch image, and the perceptual loss function value and the visual similarity are inversely proportional.

[0012] Optionally, the neural network model is trained for the next iterative training using multiple loss function values, including: obtaining a coefficient corresponding to each loss function value, wherein each coefficient is used to indicate the degree of influence of the corresponding loss function value on the neural network model when training the neural network model, and the coefficient is directly proportional to the degree of influence; determining the product of each loss function value and the corresponding coefficient to obtain multiple products, and determining the accumulation result of the multiple products as the total loss function value; and using the total loss function value to train the neural network model for the next iterative training, wherein each time the iterative training is performed, the old hyperparameters of the neural network model before the iteration are replaced with new hyperparameters of the neural network model corresponding to the total loss function value with the smallest numerical value.

[0013] According to another aspect of an embodiment of the present application, a model training device is also provided, including: an acquisition module, used to acquire an optical image including a face image, a target image obtained by masking the optical image, and a target sketch image corresponding to the optical image, wherein the sketch image includes a face image; a generation module, used to generate a feature image based on the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image, and the pixel density of the facial area in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair area in the hair feature image is the highest; a training module, used to use the feature image, the optical image and the target image as training data, and use the target sketch image as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and it is determined that the training is completed.

[0014] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which implement the steps of the above-mentioned model training method when executed by a processor.

[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, in which a computer program is stored, wherein the above-mentioned model training method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0016] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned model training method through the computer program.

[0017] In an embodiment of the present application, an optical image including a face image is acquired, a target image is obtained by masking the optical image, and a target sketch image corresponding to the optical image is obtained, wherein the target sketch image includes a face image; a feature image is generated according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image, and the pixel density of the face region in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair region in the hair feature image is the highest; the feature image, the optical image, and the target image are used as training data, and the target sketch image is used as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and a method for determining the completion of the training is determined, and a feature image with the highest pixel density in the face region and the highest pixel density in the hair region are obtained by processing the optical image as input. Feature images are generated as key samples of training data according to the feature images during the training process, thereby achieving the purpose of reducing the number of invalid samples. Key samples are generated according to the feature images during iterative training. This dynamic sampling strategy achieves the purpose of balancing positive samples and negative samples; and, the model is trained with the key samples as training data, so that the model pays more attention to the facial area and hair area of ​​the face image when generating the sketch image, thereby achieving the purpose of making the texture features of the facial area and the texture features of the hair area of ​​the face sketch image output by the trained model, thereby achieving the technical effect of training a model that can output sketch images with rich facial details, and further solving the technical problem that the face sketch image output by the trained model has blurred texture in the facial area and blurred texture in the hair area due to the fact that the spatial distribution of the facial structure of the face is ignored when the contrast learning method is used to train the model in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a model training method according to an embodiment of the present application;

[0020] Figure 2 It is a flowchart of the steps of a model training method provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of generating a hair feature image;

[0022] Figure 4 is a schematic diagram of generating facial feature images;

[0023] Figure 5 is a structural diagram of a training device of a model according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a training process of a model according to an embodiment of the present application;

[0025] Figure 7 It is an experimental result record table according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0029] Neural network model: A mathematical model that simulates the human nervous system. A neural network model learns the patterns and features of input data by adjusting weights so that it can make accurate predictions or classifications.

[0030] Pixel density: Measured in pixels per inch (PPI), the more pixels per inch, the higher the pixel density.

[0031] Feature stack: extracted from the image and used to represent the key information in the image; the key information represented by the feature stack can be local features such as edges, corners, textures in the image, or global features such as the overall color histogram and gradient histogram; the feature stack can be obtained by processing the pixels of the image through an extraction algorithm.

[0032] In the related art, a model for generating sketch images is trained by contrastive learning algorithm, and samples are selected by random sampling strategy during model training, resulting in a large number of invalid samples. Therefore, there is a problem of high difficulty in model training; at the same time, the positive samples and negative samples in the samples selected by the random sampling strategy are unevenly distributed, resulting in the local details of the sketch images generated by the model trained by the above method being blurred when applied, and there is a problem of poor model training effect. In order to solve this problem, a relevant solution is provided in the embodiment of the present application, which is described in detail below.

[0033] According to an embodiment of the present application, a method embodiment of a model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a model training method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0035] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the training method of the model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the training method of the above-mentioned model. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0039] The present application embodiment provides a model training method that can be implemented in the above operating environment. Figure 2 is a flowchart of the steps of the training method of the model provided in the embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0040] Step S202, obtaining an optical image including a face image, a target image obtained by performing mask processing on the optical image, and a target sketch image corresponding to the optical image, wherein the target sketch image includes a face image.

[0041] The neural network model trained by the method provided in the embodiment of the present application is a neural network model for generating a sketch image, or it can also be called a sketch image generator; the sketch image generator has an image translation function, which can translate the optical image as input into the corresponding sketch image, so the optical image and the sketch image are used as training data in the training process. In step S202, an image for training the model is obtained, including: an optical image containing a face image, a face contour image containing the contour of the face image (i.e., a target image), and a sketch image containing the same face image as the optical image (i.e., a target sketch image); wherein the face contour image (i.e., the target image) is a black and white image with high contrast, and the pixel value of the background part in the optical image in the face contour image (i.e., the target image) is 0, which is displayed as black; the pixel value of the face part (complete face image) in the optical image in the face contour image (i.e., the target image) is 255, which is displayed as white; the face contour image (i.e., the target image) can be obtained by masking the optical image.

[0042] The method provided in the embodiment of the present application can be implemented by any computer device. For example, when the neural network model is trained by the method provided in the embodiment, the neural network model and its training data and test data (in the embodiment, the optical image containing the face image, the face contour image and the target sketch image constitute the training data and the test data) are loaded into the memory. For example, the raw data of the neural network model can be loaded from the non-volatile memory to the volatile memory so that the processor can run the neural network model. The raw data of the first neural network model refers to the unprocessed data, which usually includes the parameters and structural data of the neural network model before training. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between the intermediate layers and the neurons. Specifically, the structural data can include the code related to the structure of the neural network model, such as the code for performing the related calculation between the intermediate layers and the neurons. In one embodiment, a data storage area and a parameter storage area for loading the neural network model and its training data and test data can be divided in the memory. The structural data storage area is used to store the relevant code of the structural data, and the parameters referenced by it can point to the address of the specific parameters in the parameter storage area through a pointer. In the training process of the neural network model, it may be necessary to update the parameters frequently, and the parameter values ​​in the parameter storage area can be updated.

[0043] Step S204, generating a feature image according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, the facial feature image is a complete face image, and the pixel density of the facial area in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair area in the hair feature image is the highest.

[0044] After the optical image including the face image is acquired in step S202, in step S204, a facial structure spatial distribution map (i.e., feature image) of the face image is generated according to the optical image, wherein the facial structure spatial distribution map (i.e., feature image) includes two types of images, one is a facial feature map, and the other is a hair feature map; wherein the facial feature map includes a complete face image, and at the same time, in the facial feature image, the pixel density of the face area is the highest, therefore, the neural network model pays the highest attention to the face area when processing the facial feature image according to the attention mechanism; wherein the hair feature map includes a complete face image, and at the same time, in the hair feature image, the pixel density of the hair area is the highest, therefore, the neural network model pays the highest attention to the hair area when processing the hair feature image according to the attention mechanism.

[0045] Optionally, a feature image is generated based on the optical image, wherein the hair feature image is generated according to the following method: using a preset pooling window to traverse a pixel matrix generated based on multiple first pixels of the optical image to obtain a global feature matrix of the optical image, wherein each first element in the global feature matrix is ​​an average value of multiple first pixels included in the preset pooling window at each traversal position; processing the global feature matrix to obtain a feature vector of the optical image; generating a first regional feature matrix based on the feature vector and the pixel matrix, wherein the first regional feature matrix includes multiple second elements, the multiple second elements are respectively mapped to multiple second pixels, and the multiple second pixels are used to form the hair feature image.

[0046] The training method provided in the embodiment of the present application can be executed by a complete network architecture, wherein generating a feature image based on an optical image can be executed by a regional sampling module (RCS) in the network architecture for implementing regional constrained sampling. The RCS calls a convolutional neural network and adopts a dual-branch attention mechanism to obtain a spatial distribution map of the facial structure of the optical image (hair feature image and facial feature image). Figure 3 is a schematic diagram of generating hair feature images, such as Figure 3 As shown in the figure, the model used to implement regional constrained sampling performs pooling on the input optical image in the average pooling layer (Avgpool1) to obtain the global feature matrix (Z c), in this embodiment, the optical image exists in the form of a pixel matrix in the convolutional neural network, and the pixel matrix of the optical image is also recorded as P i , the pooling process of the optical image is actually the pixel matrix P i Pooling processing; next, in the fully connected layer (F c ) uses activation functions (ReLU and Sigmoid) to transform the global feature matrix (Z c ) is processed to obtain the feature vector S of the optical image; according to the feature vector S and the optical image (P i ) can generate an enhanced channel attention matrix (P i ) c (i.e., the first regional feature matrix); there is a mapping relationship between the elements in the enhanced channel attention matrix (i.e., the second elements) and the pixels of the hair feature image (i.e., the second pixels); according to the enhanced channel attention matrix P c The (second) pixel constituting the hair feature image may be determined; for example, the second element is directly used as the second pixel, or the second element is processed to obtain a value within a preset range, and the value is used as the second pixel. The above process of generating the hair feature image may be expressed by the following formula: c =Avgpool(P i )、S=Sigmoid(F c (ReLU(F c (Z c )))))、(P i ) c =Conv(SP i ). Among them, Z c =Avgpool(P i ) represents the optical image / pixel matrix (P i ) is used for pooling processing. Specifically, a preset pooling window is used to move on the optical image. Since the optical image exists in the form of a pixel matrix in the convolutional neural network, the movement of the preset pooling window on the optical image is equivalent to using the preset pooling window to traverse the pixel matrix of the optical image. Each element in the pixel matrix is ​​a pixel of the optical image (i.e., the first pixel); the preset pooling window is a square. When the preset pooling window stays at each traversal position, the average value of the (first) pixel contained in the preset pooling window is determined. After the traversal is completed, multiple average values ​​will be obtained, and these multiple average values ​​will be used as the global feature matrix (Z c ) in the global feature matrix (Z c ) is the same as the number of traversed positions. c (ReLU(F c (Z c)))) indicates that in the fully connected layer F c Use activation functions (ReLU and Sigmoid) to transform the global feature matrix (Z c ) is processed to obtain the feature vector S of the optical image. i ) c =Conv(SP i ) represents the combination of the feature vector S and the pixel matrix (P i ) and convolve the result to get the enhanced channel attention matrix (P i ) c (i.e., the first region feature matrix). During verification, the enhanced channel attention matrix (P i ) c Visualization processing is performed to obtain Figure 3 The visual results of the image a show that the pixel density of the hair area in image a is the highest. Therefore, it can be determined that Figure 3 The processing flow in can generate hair feature images.

[0047] According to an optional embodiment of the present application, a feature image is generated according to an optical image, wherein the facial feature image is generated according to the following method: a pixel matrix generated according to a plurality of first pixels of the optical image is convolved to obtain a convolution matrix, and the convolution matrix is ​​normalized; a second region feature matrix is ​​generated according to the normalized convolution matrix, the pixel matrix and the first region feature matrix, wherein the second region feature matrix includes a plurality of third elements, the plurality of third elements are respectively mapped to a plurality of third pixels, and the plurality of third pixels are used to form the facial feature image.

[0048] In order to improve the sketch image generator's attention to the facial area of ​​the face image during the process of processing the optical image, in this embodiment, a fused spatial and channel attention mechanism is used to generate a facial feature image for limiting the sampling area to the facial area of ​​the face image. Figure 4 It is a schematic diagram of generating facial feature images. The compression matrix (i.e., convolution matrix) (Z) is obtained by performing 1×1 convolution on the input optical image (Input optical image) s ), the optical image exists in the form of a pixel matrix in the convolutional neural network. Therefore, in this embodiment, the convolution processing of the optical image is actually the pixel matrix (P i ) is convolved; next, the activation function (Sigmoid) is used to compress the matrix (Z s ) is normalized; according to the compressed matrix (Z s ), pixel matrix (P i ) and the first regional feature matrix (Pi ) c The second regional feature matrix (f f ) Among them, the second regional feature matrix (f f There is a mapping relationship between the elements in ) (i.e., the third elements) and the pixels in the facial feature image (i.e., the third pixels); according to the second regional feature matrix (f f ) can determine the (third) pixel constituting the facial feature image; for example, the third element is directly used as the third pixel, or the third element is processed to obtain a value within a preset range, and the value is used as the third pixel value. The above process of generating the facial feature image can be expressed by the following formula: s =Conv(P i )、(P i ) s =Sigmoid(Z s ) i 、f f =Conv((P i ) c +(P i ) s ). Among them, Z s =Avgpool(P i ) represents the optical image / pixel matrix (P i ) convolution processing, and obtain the convolution matrix (Z s ). Sigmoid(Z s ) represents the convolution matrix (Z s ) is normalized, (P i ) s =Sigmoid(Z s ) i and f f =Conv((P i ) c +(P i ) s ) represents the convolution matrix, pixel matrix and first regional feature matrix after normalization (P i ) c Generate the second regional feature matrix f f , where (P i ) s =Sigmoid(Z s ) i Represents the normalized convolution matrix (Z s ) and the pixel matrix P i Multiply them together to get the matrix (P i ) s .f f =Conv((P i )c +(P i ) s ) represents the matrix (P i ) s and the first regional feature matrix (P i ) c The sum of is convolved to obtain the second regional feature matrix (f f In addition, since the generation of the facial feature image requires the use of the first region feature matrix (P i ) c , so, if Figure 4 As shown in FIG. 1 , the process of generating a facial feature image includes part of the process of generating a hair feature image. During verification, the second region feature matrix (f f ) is visualized to obtain Figure 4 The visual results of the image b show that the pixel density of the face area in image b is the highest, so it can be determined based on Figure 4 The processing flow in can generate facial feature images.

[0049] It should be noted that the convolution processing mentioned in the embodiment of the present application may be a 1×1 convolution with a convolution kernel.

[0050] Step S206, using the feature image, the optical image and the target image as training data, and using the target sketch image as test data, iteratively trains the neural network model until the number of iterations reaches a preset number, and determines that the training is completed.

[0051] In step S206, the neural network model is iteratively trained using the feature image generated in step S204, the optical image and the face contour image (i.e., the target image) obtained in step S202, and the target sketch image, until the number of iterations reaches a preset number of training times (i.e., the preset number of times), and the training is determined to be completed. The trained neural network model (sketch image generator) is used to output a sketch image containing the same face image as the input image; the clarity of the texture features of the hair area and the clarity of the texture features of the face area in the sketch image output by the sketch image generator are greater than the clarity of the texture features of the hair area and the clarity of the texture features of the face area in the sketch image output by the model trained by other methods.

[0052] According to another optional embodiment of the present application, each iterative training of the neural network model includes: inputting the optical image and the target image into the neural network model to obtain a sketch output by the neural network model, wherein the similarity between the sketch and the optical image is less than the similarity between the target sketch image and the optical image; generating training samples according to the feature image, the optical image and the sketch, wherein the training samples include: positive samples and negative samples, and the images used as positive samples and the images used as negative samples are images of different areas in the optical image; determining multiple loss function values ​​according to the training samples, the sketch and the target sketch image; and performing the next iterative training of the neural network model using the multiple loss function values.

[0053] As mentioned in the above embodiments, the training method provided in the embodiments of the present application can be executed by a complete network architecture, wherein the iterative training of the neural network model can be jointly participated by the region sampling module (RCS), feature extraction module / feature extractor (H) and generator (G / GAN) and discriminator (D) in the network architecture; wherein, the region sampling module (RCS) and feature extractor (H) participate in the generation process of the feature image mentioned in the above embodiments, the encoder module (E) and feature extractor (H) contained in the generator (G / GAN) participate in the process of generating training samples in the iterative training, and the generator (G) is the trained neural network model; the discriminator (D) participates in the process of determining the loss function value in the iterative training. In this embodiment, when the neural network model is iteratively trained, the optical image and the face contour image (i.e., the target image) obtained in step S202 are input as input data to the neural network model to obtain the sketch image (i.e., the sketch rough) output by the neural network model. Since the neural network model has not yet been trained, the similarity between the sketch image output by the neural network model and the optical image must be lower than the similarity between the target sketch image as the test data and the optical image. Therefore, in this embodiment, the sketch images output by the neural network model during the iterative training process are all referred to as sketch roughs. The encoder module (E) extracts features from the optical image based on the sketch rough to generate training samples; the feature extractor (H) and the discriminator (D) determine multiple loss function values ​​for evaluating the image quality of the sketch rough based on the training samples, the sketch rough and the target sketch image; wherein the loss function value obtained in each iterative training will be used for the next iterative training of the neural network model. The training samples mentioned above include two categories: positive samples and negative samples. Since this embodiment hopes that the neural network model will pay more attention to the facial area and hair area of ​​the image during the model training process, in this embodiment, the (first) image / image blocks constituting the first image used as positive samples can be queried in the facial area / hair area in the optical image; while the (second) image / image blocks constituting the second image used as negative samples cannot be queried in the facial area / hair area in the optical image.

[0054] Optionally, generating training samples based on feature images, optical images and sketches includes: taking the feature images as constraints, performing feature extraction on the optical images to obtain a first type of feature stack, and taking the feature images as constraints, performing feature extraction on the sketches to obtain a second type of feature stack, wherein the first type of feature stack indicates an image of a facial region and an image of a hair region in the optical image, and the second type of feature stack indicates an image of a facial region and an image of a hair region in the sketch; determining any image block in the image indicated by the second type of feature stack as a query image block; and generating training samples based on the query image block and the first type of feature stack.

[0055] In this embodiment, a training sample is generated by constructing a feature stack, wherein the feature stack includes a first type of feature stack (Z l ) L and the second type of feature stack extracted from the sketch Will form the second type of feature stack Any one of the multiple image blocks of the indicated image is used as a query image block, and the first type of feature stack (Z l ) L The multiple image blocks contained in the indicated image (i.e., the target image blocks) are classified into positive samples and negative samples. Generate the first type of feature stack (Z l ) L When the multi-level feature map is extracted from the optical image, the extracted multi-level feature map is input into the feature extractor (H). The feature extractor (H) generates the first type of feature stack (Z) with position constraints under the guidance of the RCS module. l ) L Specifically, the feature extractor (H) uses the feature image (facial feature map and hair feature map) output by the RCS module as a constraint. Since the facial feature map and hair feature map are used as constraints, when the feature extractor (H) extracts features from the optical image, the feature extraction area is limited to the facial area and the hair area. Therefore, the generated first type of feature stack (Z l ) L The indicated images are an image of the face region and an image of the hair region in the optical image. Generating the second type of feature stack When , a multi-level feature map is first extracted from the sketch, and then the extracted multi-level feature map is input into the feature extractor (H). The feature extractor (H) generates a second type of feature stack with position constraints under the guidance of the RCS module. In generating the second type of feature stack The feature images (facial feature maps and hair feature maps) output by the RCS module are also used as constraints. Therefore, the generated second type of feature stack The indicated image is an image of the face area and hair area in the sketch. The process of constructing the feature stack can be expressed by the following formula: (Z l ) L =H(RCS(E l )(P))、 Among them, (Z l ) L =H(RCS(E l )(P)) indicates that the feature extractor (H) processes the optical image (P) through the encoder (E) at the lth layer of the encoder under the constraint of the feature image output by the RCS, and extracts the first type of feature stack (Z l ) L ; It represents that the feature extractor (H) is constrained by the feature image output by the RCS through the encoder (E) in the encoder's l-th layer to sketch the sketch Processing, extraction into the second type of feature stack The lth layer of the encoder can be the 0th layer, the 4th layer, the 8th layer, etc. of the encoder. For example, l∈{0, 4, 8, 12, 16} can be set. The multi-level feature map of the optical image is a plurality of feature maps including facial texture features and hair texture features obtained after processing the optical image. The facial texture features and hair texture features in the plurality of feature maps are slightly different.

[0056] According to an optional embodiment of the present application, a training sample is generated based on a query image block and a first type of feature stack, including: determining first position information of the query image block in a sketch, and determining second position information of each target image block in a plurality of target image blocks corresponding to the first type of feature stack in an optical image, wherein the plurality of target image blocks constitute an image of a facial region and an image of a hair region in the optical image; determining a target image block corresponding to second position information identical to the first position information as a positive sample, and determining the remaining target image blocks as negative samples, wherein the number of positive samples is positively proportional to the pixel density of the query image block.

[0057] The above embodiment involves the following scheme: according to the query image block, the first type of feature stack (Z l)L, the multiple image blocks (i.e., target image blocks) contained in the indicated image are classified into positive samples and negative samples. This is because the training method provided in the embodiment of the present application is to enable the trained model / sketch image generator to pay more attention to the facial area and hair area of ​​the optical image when generating a sketch image for the input optical image. Therefore, in the iterative training process, it is necessary to focus on the facial area and hair area of ​​the sketch. In this embodiment, when a sample is generated in each iterative training, one of the multiple image blocks that constitute the facial area and hair area in the sketch is used as a query image block, and the query image block is used to extract training samples from the optical image, and the number of samples is determined according to the pixel density of the selected query sample (query image block). For example, it can be set that the higher the pixel density of the query sample, the greater the number of positive samples in the selected training sample. Due to the second type of feature stack The first type of feature stack (Zl) L is constructed based on the sketch. The corresponding image is constructed based on the optical image, and the sketch and the optical image correspond to the same face image. Therefore, the layout of pixels in the two images remains unchanged, that is, the image block representing the same image (such as the image of the eye part of the face) in the optical image (first) position information and the (second) position information in the sketch are exactly the same. The above (first and second) position information can be in the form of coordinates. For example, the optical image coordinate system is established with the lower left corner of the optical image, and the sketch coordinate system is established with the lower left corner of the sketch. The optical image coordinate system and the sketch coordinate system are two completely identical systems. Coordinate system, the first position information is the coordinate of the query image block in the sketch coordinate system, and the second position information is the coordinate of the image block of the optical image in the optical image coordinate system. Therefore, positive samples and negative samples can be screened based on the position information of the image block; for example, after determining the (first) position information of the query image block in the sketch, determine the (second) position information of the image block (i.e., the target image block) contained in the facial area and the hair area of ​​the optical image in the optical image, and use the target image block on the second position information corresponding to the same coordinate as the first position information as the positive sample, and use the image blocks other than the positive samples in the multiple (target) image blocks corresponding to the first type of feature stack as negative samples. Each iteration collects training samples from the optical image in the above manner. This dynamic sampling strategy dynamically adjusts the number of samples according to the feature density of the query image block, which can effectively solve the problem of imbalance between positive and negative samples. In addition, the cross entropy loss function is introduced in this embodiment: To evaluate the association between the positive sample and the query sample (i.e., the query image block), where v represents the query sample (query image block), v + represents a positive sample, v - represents negative samples, and τ is the temperature coefficient, which is set to 0.07.

[0058] According to some optional embodiments of the present application, multiple loss function values ​​are determined according to training samples, sketch sketches and target sketch images, including: determining a patch contrast loss function value among multiple loss function values ​​according to positive samples, negative samples and query samples in the sketch sketch as query image blocks, wherein the patch contrast loss function value is used to indicate the image similarity between the sketch sketch and the optical image, and the patch contrast loss function value and the similarity are inversely proportional; determining an adversarial loss function value among multiple loss function values ​​according to the sketch sketch and the target sketch image, wherein the adversarial loss function value is used to indicate the degree of image similarity between the sketch sketch and the target sketch image, and the adversarial loss function value and the similarity are inversely proportional; determining a pixel-by-pixel loss function value among multiple loss function values ​​according to the sketch sketch and the target sketch image, wherein the pixel-by-pixel loss function value is used to indicate the clarity of the sketch sketch, and the pixel-by-pixel loss function value is inversely proportional to the clarity; determining a perceptual loss function value among multiple loss function values ​​according to the sketch sketch and the target sketch image, wherein the perceptual loss function value is used to evaluate the visual similarity between the sketch sketch and the target sketch image, and the perceptual loss function value and the visual similarity are inversely proportional.

[0059] In order to ensure the stability of the network training process and accelerate convergence, the method provided in the embodiment of the present application adopts multiple loss functions to evaluate the image quality of the sketch (evaluated from multiple aspects such as the texture feature clarity of the face area and the hair area, and the similarity with the target sketch image), wherein the above-mentioned multiple loss functions include: a patch contrast loss function (L) for evaluating the image similarity between the sketch and the optical image; patch ), an adversarial loss function (L GAN ), a pixel-by-pixel loss function for evaluating the clarity of the sketch (L1), and a perceptual loss function for evaluating the visual similarity between the sketch and the real sketch image (L per ). The function value of the patch contrast loss function (patch contrast loss function value) is calculated according to the formula Calculated, where L patch (E l , H, p) represents the contrast loss function value, H represents the feature extractor, p represents the optical image, (p i ~p p ) represents the sample distribution of the optical image, represents the query sample (i.e., query image patch) contained in the sketch, represents the positive sample, represents negative samples, E l represents the lth layer of the encoder (E), s = {1, 2, 3, 4..., S ι},S lRepresents the total number of samples (positive and negative samples) in each layer; the smaller the value of the patch contrast loss function, the higher the image similarity between the sketch and the optical image. The function value of the adversarial loss function (adversarial loss function value) is calculated by the formula Calculated, where L GAN (G, D, P, S) represents the adversarial loss function value, P represents the optical image, G represents the generator, D represents the discriminator, E represents the encoder, (s i ~p s ) represents the sample distribution of sketches, s i represents the target sketch image, p i represents the optical image as input, G(p i ) is the trained neural network model / generator for the optical image p as input i The output sketch. The smaller the adversarial loss function value, the higher the image similarity between the sketch and the real sketch image. The pixel-by-pixel loss function value is calculated by the formula Calculated, where L1(G, P, s) represents the pixel-by-pixel loss function value, ||G(p i )-s i || l Denotes the L1 norm (l) used to measure the generator G for the input optical image p i The output sketch G(p i ) and the target sketch image s i The difference between the two, the smaller the pixel loss function value, the higher the clarity of the sketch. The function value of the perceptual loss function (perceptual loss function value) is calculated by the formula Calculated, where L per (G, P, S) represents the perceptual loss function value, It represents the feature extractor H at the jth layer for the target pixel map s i The extracted perceptual features, It represents the feature extractor H at layer j for the optical image p i The output sketch G(p i ) extracted perceptual features, ω j represents the weight parameter, j = {1, 2, 3, 4, 5},

[0060] According to some optional embodiments of the present application, the neural network model is trained for the next iterative training using multiple loss function values, including: obtaining a coefficient corresponding to each loss function value, wherein each coefficient is used to indicate the degree of influence of the corresponding loss function value on the neural network model when training the neural network model, and the coefficient is directly proportional to the degree of influence; determining the product of each loss function value and the corresponding coefficient to obtain multiple products, and determining the accumulation result of the multiple products as the total loss function value; performing the next iterative training on the neural network model using the training samples and the total loss function value, wherein each time the iterative training is performed, the old hyperparameters of the neural network model before the iteration are replaced with new hyperparameters of the neural network model corresponding to the total loss function value with the smallest numerical value.

[0061] In some embodiments, a total loss function value is determined based on the function values ​​of the multiple loss functions determined in the above embodiments, and new samples are continuously generated and the total loss function value is updated during the iterative training process. Specifically, a coefficient pre-set for each type of loss function is obtained, wherein the value of the coefficient is related to the degree of influence of this type of loss function on the training results of the neural network model. The larger the value of the coefficient, the greater the influence of this type of loss function on the training results of the neural network model, and the more emphasis is placed on the evaluation criteria corresponding to this type of loss function during training. For example, the coefficient corresponding to the pixel loss function value is the largest, indicating that the training process focuses on the clarity of the sketch image. Next, the coefficient is multiplied by the value of the loss function of the corresponding type, and the products obtained by multiplying the values ​​of multiple types of loss functions by their corresponding coefficients are accumulated, and the accumulated result is used as the total loss function value. For example, a contrast loss function (L patch ) corresponds to the coefficient β, the adversarial loss function (L GAN ) corresponds to 1, the coefficient corresponding to the pixel loss function (L1) is α, and the perceptual loss function (L per )The corresponding coefficient is Y, then the total loss function F full =L GAN +αL1+βL patch +γL per . Finally, the total loss function value and training samples are used to perform the next iteration training of the neural network model. In each iteration training, the total loss function value is used to determine whether to update the hyperparameters of the neural network model. If the total loss function value obtained in this iteration training is less than the total loss function value obtained in the previous iteration training, the hyperparameters of the neural network model at the beginning of this iteration (i.e., the old hyperparameters) are replaced with the hyperparameters of the neural network model at the end of this iteration (i.e., the new hyperparameters).

[0062] Through the above steps, it is possible to generate feature images for constraining the sampling area. When a random sampling strategy is applied to generate samples during training, the sampling area is limited to a specific area (such as the hair area and the face area) to avoid generating a large number of invalid samples. New samples are generated by resampling each iteration. The sampling frequency is adjusted through this dynamic sampling strategy to avoid the imbalance between positive and negative samples and improve the expressiveness of the sketch face image in the local area. In the training process, the face contour image is added as training data, so that the sketch image generator focuses on the extraction of the features of the face image, thereby making the facial texture and hair texture of the generated sketch image clearer, and at the same time enhancing the character contour information of the sketch image.

[0063] Figure 5 is a structural diagram of a model training device provided according to an embodiment of the present application, such as Figure 5 As shown, the training device of the type includes: an acquisition module 50, which is used to acquire an optical image including a face image, a target image obtained by masking the optical image, and a target sketch image corresponding to the optical image, wherein the sketch image includes a face image; a generation module 52, which is used to generate a feature image according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image, and the pixel density of the facial area in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair area in the hair feature image is the highest; a training module 54, which is used to iteratively train the neural network model using the feature image, the optical image and the target image as training data, and using the target sketch image as test data, until the number of iterations reaches a preset number, and the training is determined to be completed.

[0064] Figure 6 This is a schematic diagram of the model training process, such as Figure 6 As shown, the training process of using the training device of the above model to train the neural network model, i.e., the generator (Global Generator), is as follows Figure 6 As shown, the training device can run a network architecture including a sampler (RCS), a feature extractor (H), a discriminator (D), a deep convolutional neural network model (VGG-19) and a deep learning model (Global Discriminator); when training the neural network model (Global Generator), the acquisition module 50 acquires an optical image (Optical image), a target image obtained by masking the optical image, namely, a mask (Mask image), and a target sketch image (Input real sketch) corresponding to the optical image, and the generation module 52 calls RCS to process the optical image to generate the following: Figure 3 The hair feature image shown in Figure 4 The facial feature image shown in the figure; the feature image generated by RCS is used to limit the sampling area when the neural network model samples the optical image, so that when the trained neural network model generates a sketch image, more facial areas and hair areas of the face image contained in the optical image are sampled; the training module 54 trains the neural network model according to the feature image, optical image and target image obtained above. Figure 6 As shown in the figure, the components of the neural network model include an encoder (E) component, a generator (G) and an output. During the training process, the feature extractor (H) extracts features based on the optical image and the pseudo-sketch processed by the encoder under the restriction of the feature image output by the RCS, and generates training samples (positive samples and negative samples). In addition, in order to maximize the mutual information between the optical image and the pseudo-sketch during the training process, a multi-level patch contrast loss (Multilayer Patchwise contrastive loss) is introduced, which is calculated based on the training samples. Figure 6 As shown in Figure 2, during the training process, the discriminator (D) is used to calculate the adversarial loss function (L) based on the sketch (Pseudo-sketch) and the target sketch image (Input real sketch). GAN ), through L GAN To evaluate the similarity between the sketch and the real sketch image (target sketch image), the discriminator identifies the sketch as a fake sketch and the real sketch image as Real. When the number of training times reaches the preset number, a fake sketch as close to the real sketch image as possible can be generated. In addition, in order to ensure that the model considers the global structure of the image and makes the generated sketch image more natural, the perceptual loss function (L per ) to evaluate the visual similarity between the sketch and the target pixel map, such as Figure 6 As shown in the figure, the training process consists of a deep convolutional neural network model (VGG-19) calculating the perceptual loss function (L per ).

[0065] It should be noted that Figure 5 The preferred implementation of the illustrated embodiment can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0066] An embodiment of the present application also provides a non-volatile storage medium, in which a computer program is stored, wherein the above model training method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0067] The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: obtaining an optical image containing a face image, a target image obtained by masking the optical image, and a target sketch image corresponding to the optical image, wherein the target sketch image contains a face image; generating a feature image based on the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image, and the pixel density of the facial area in the facial feature image is the highest, and the hair feature image is a complete face image, and the pixel density of the hair area in the hair feature image is the highest; using the feature image, the optical image and the target image as training data, and using the target sketch image as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and determining that the training is completed.

[0068] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above model training method through the computer program.

[0069] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: obtaining an optical image containing a facial image, a target image obtained by masking the optical image, and a target sketch image corresponding to the optical image, wherein the target sketch image contains a facial image; generating a feature image based on the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete facial image, and the pixel density of the facial area in the facial feature image is the highest, and the hair feature image is a complete facial image, and the pixel density of the hair area in the hair feature image is the highest; using the feature image, the optical image, and the target image as training data, and using the target sketch image as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and determining that the training is completed.

[0070] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the above-mentioned model training method when executed by a processor.

[0071] In addition, different optical images are used to verify the sketch image generator trained by the training method provided in the embodiment of the present application, and the three indicators of peak signal-to-noise ratio (PSNR), structural similarity index measurement (SSIM) and Fréchet initial distance (FID) are used to evaluate the sketch image output by the sketch image generator. The higher the values ​​of SNR and SSIM and the lower the value of FID, the better the quality of the generated sketch image (the texture features of the facial area and the texture features of the hair area are clearer). Figure 7 is the experimental result record table, such as Figure 7The table shown records the PSNR, SSIM and FID of the sketch image obtained in the image segmentation (CUT) step, the PSNR, SSIM and FID of the sketch image obtained after the RCS is introduced in the process, the PSNR, SSIM and FID of the sketch image obtained after the discriminator is further introduced in the process, and the PSNR, SSIM and FID of the sketch image obtained after the mask is further introduced in the process. By gradually introducing the method provided by the embodiment of the present application in the experiment and then comparing it with the feature image (FSS), it is observed that the FID is significantly reduced. In particular, the FID is reduced by about 25 on the first and second test data sets, and by about 45 on the third test data set. In addition, the improvements in SSIM and PSNR are obvious in all data sets, although to different degrees. The results confirm the key role played by the method provided by the embodiment of the present application in enhancing the overall performance of FSS.

[0072] It should be noted that the various modules in the training device of the above-mentioned model can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0073] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0074] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0076] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0077] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0079] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training method, characterized in that: include: Acquire an optical image containing a face image, a target image obtained by performing mask processing on the optical image, and a target sketch image corresponding to the optical image, wherein the target sketch image contains the face image; Generate a feature image according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image of the person, and the face region of the facial feature image has the highest pixel density, and the hair feature image is a complete face image of the person, and the hair region of the hair feature image has the highest pixel density; The feature image, the optical image and the target image are used as training data, and the target sketch image is used as test data to iteratively train the neural network model until the number of iterations reaches a preset number, and the training is determined to be completed.

2. The method according to claim 1, characterized in that Generate a characteristic image according to the optical image, wherein the hair characteristic image is generated according to the following method: Using a preset pooling window to traverse a pixel matrix generated according to a plurality of first pixels of the optical image, to obtain a global feature matrix of the optical image, wherein each first element in the global feature matrix is ​​an average value of a plurality of first pixels contained in the preset pooling window at each traversal position; Processing the global feature matrix to obtain a feature vector of the optical image; A first regional feature matrix is ​​generated according to the feature vector and the pixel matrix, wherein the first regional feature matrix includes a plurality of second elements, the plurality of second elements are respectively mapped to a plurality of second pixels, and the plurality of second pixels are used to form the hair feature image.

3. The method according to claim 2, characterized in that Generate a feature image according to the optical image, wherein the facial feature image is generated according to the following method: Performing convolution processing on a pixel matrix generated according to a plurality of first pixels of the optical image to obtain a convolution matrix, and performing normalization processing on the convolution matrix; A second region feature matrix is ​​generated according to the normalized convolution matrix, the pixel matrix and the first region feature matrix, wherein the second region feature matrix includes a plurality of third elements, the plurality of third elements are respectively mapped to a plurality of third pixels, and the plurality of third pixels are used to form the facial feature image.

4. The method according to claim 1, characterized in that: The neural network model is iteratively trained each time, including: Inputting the optical image and the target image into the neural network model to obtain a sketch output by the neural network model, wherein the similarity between the sketch and the optical image is less than the similarity between the target sketch image and the optical image; Generate training samples according to the feature image, the optical image and the sketch, wherein the training samples include: positive samples and negative samples, and the image used as the positive sample and the image used as the negative sample are images of different regions in the optical image; Determining a plurality of loss function values ​​based on the training samples, the sketch image, and the target sketch image; The neural network model is trained for the next iteration using the multiple loss function values.

5. The method according to claim 4, characterized in that Generating a training sample according to the feature image, the optical image and the sketch includes: Using the feature image as a constraint, performing feature extraction on the optical image to obtain a first type of feature stack, and using the feature image as a constraint, performing feature extraction on the sketch to obtain a second type of feature stack, wherein the first type of feature stack indicates an image of a facial region and an image of a hair region in the optical image, and the second type of feature stack indicates an image of a facial region and an image of a hair region in the sketch; Determine any image block in the image indicated by the second type of feature stack as a query image block; The training sample is generated according to the query image block and the first type of feature stack.

6. The method according to claim 5, characterized in that Generating the training sample according to the query image block and the first type of feature stack includes: Determine first position information of the query image block in the sketch, and determine second position information of each of the target image blocks in the optical image among a plurality of target image blocks corresponding to the first type of feature stack, wherein the plurality of target image blocks constitute an image of a facial region and an image of a hair region in the optical image; The target image block corresponding to the second position information which is the same as the first position information is determined as the positive sample, and the remaining target image blocks are determined as the negative samples, wherein the number of the positive samples is in direct proportion to the pixel density of the query image block.

7. The method according to claim 4, characterized in that Determining a plurality of loss function values ​​according to the training sample, the sketch image, and the target sketch image includes: Determine a patch contrast loss function value among the multiple loss function values ​​according to the positive sample, the negative sample, and a query sample in the sketch as a query image block, wherein the patch contrast loss function value is used to indicate the image similarity between the sketch and the optical image, and the patch contrast loss function value is in inverse proportion to the similarity; Determining an adversarial loss function value among the multiple loss function values ​​according to the sketch image and the target sketch image, wherein the adversarial loss function value is used to indicate the degree of image similarity between the sketch image and the target sketch image, and the adversarial loss function value is inversely proportional to the similarity; Determining a pixel-by-pixel loss function value among the plurality of loss function values ​​according to the sketch image and the target sketch image, wherein the pixel-by-pixel loss function value is used to indicate the clarity of the sketch image, and the pixel-by-pixel loss function value is inversely proportional to the clarity; A perceptual loss function value among the multiple loss function values ​​is determined according to the sketch sketch and the target sketch image, wherein the perceptual loss function value is used to evaluate the visual similarity between the sketch sketch and the target sketch image, and the perceptual loss function value is inversely proportional to the visual similarity.

8. The method according to claim 4, characterized in that The neural network model is trained for the next iteration using the multiple loss function values, including: Obtaining a coefficient corresponding to each of the loss function values, wherein each of the coefficients is used to indicate the degree of influence of the corresponding loss function value on the neural network model when training the neural network model, and the coefficient is in direct proportion to the degree of influence; Determine the product of each loss function value and the corresponding coefficient to obtain a plurality of the products, and determine the cumulative result of the plurality of the products as the total loss function value; The neural network model is subjected to the next iterative training using the total loss function value, wherein each time the iterative training is performed, the old hyperparameters of the neural network model before the iteration are replaced with new hyperparameters of the neural network model corresponding to the total loss function value having the smallest numerical value.

9. A model training device, characterized in that: include: An acquisition module, used for acquiring an optical image including a face image, a target image obtained by performing mask processing on the optical image, and a target sketch image corresponding to the optical image, wherein the target sketch image includes the face image; A generating module, configured to generate a feature image according to the optical image, wherein the feature image includes: a facial feature image and a hair feature image, wherein the facial feature image is a complete face image of the human being, and the pixel density of the face region in the facial feature image is the highest, and the hair feature image is a complete face image of the human being, and the pixel density of the hair region in the hair feature image is the highest; The training module is used to iteratively train the neural network model using the feature image, the optical image and the target image as training data and the target sketch image as test data until the number of iterations reaches a preset number, thereby determining that the training is completed.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the model training method described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Sketch face recognition method and device based on domain alignment sampling network

    CN115457616A