Sketch image generation method and device and electronic equipment

Abstract

The invention discloses a sketch image generation method and device and electronic equipment. The method comprises the steps that an optical face image to be processed is acquired, and the optical face image is an optical domain image; the optical face image is processed through a sketch generation model, a sketch image corresponding to the optical face image is obtained, the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is obtained through inter-domain distance optimization loss function training; and through the inter-domain distance optimization loss function, learning the mapping of the image from the optical domain to the sketch domain, so that the similarity between the sketch image and a real sketch image corresponding to the optical face image is greater than a preset threshold. The technical problems of low feature distribution matching accuracy and large modal difference between the generated sketch image and the real sketch image in the sketch synthesis method in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a method, device and electronic device for generating a sketch image. Background Art

[0002] In recent years, sketch face synthesis technology has attracted more and more attention from researchers, which aims to convert optical face images into sketch-style face images. Existing sketch face synthesis methods are mainly divided into two categories: data-driven sketch face synthesis methods and model-driven sketch face synthesis methods.

[0003] In data-driven synthesis methods, images are divided into small blocks, and the style transfer of the entire image is achieved by analyzing and splicing small image blocks. Since the image blocks are processed, the generated sketch image will appear clumping, structurally incoherent, and have poor global consistency. In addition, when processing images with complex textures or details, the image blocks cannot effectively retain local detail features. Model-driven synthesis methods have the problem of local detail distortion in the generated image. Existing algorithms usually focus on the consistency of global style and content, and suppress the retention of high-frequency information in operations such as upsampling and regularization. At the same time, they lack attention to local details (such as hair, wrinkles, etc.), resulting in the generated image being too smooth and lacking details. At the same time, the generated sketch image retains the modal features of the optical face image. Since face images and sketches belong to different image domains, the commonly used loss design focuses on making the generated image modality close to the sketch domain, but lacks attention to the modality of the optical domain face image, resulting in the model's lack of ability to distinguish modal differences between domains, causing the generated sketch image to retain the modal information of the optical domain image, which is quite different from the style of the real sketch.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for generating a sketch image, so as to at least solve the technical problems in the sketch synthesis method in the related art, that is, the feature distribution matching accuracy is low and the generated sketch image has a large modality difference with the real sketch image.

[0006] According to one aspect of an embodiment of the present application, a method for generating a sketch image is provided, comprising: obtaining an optical face image to be processed, wherein the optical face image is an optical domain image; processing the optical face image using a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and a real sketch image corresponding to the optical face image is greater than a preset threshold.

[0007] In some embodiments of the present application, the sketch generation model is trained in the following manner: obtaining a historical optical face image, and obtaining a historical real sketch image corresponding to the historical optical face image; using a first generator of the initial sketch generation model to process the historical optical face image to obtain an initial sketch generation image; using a second generator of the initial sketch generation model to optimize the details of the initial sketch generation image to obtain a historical sketch generation image; determining an adversarial loss based on the historical real sketch image, the initial sketch generation image, and the historical sketch generation image; determining a first total perceptual loss based on the historical real sketch image and the initial sketch generation image, and determining a second total perceptual loss based on the historical real sketch image and the historical sketch generation image. perceptual loss; fusing the first feature of the historical optical face image and the second feature of the historical real sketch image to obtain a fused feature; determining the first inter-domain distance optimization loss based on the third feature, the first feature and the fused feature of the initial sketch generated image, and determining the second inter-domain distance optimization loss based on the fourth feature, the first feature and the fused feature of the historical sketch generated image, wherein the first inter-domain distance optimization loss and the second inter-domain distance optimization loss are determined by the inter-domain distance optimization loss function; determining the total loss based on the adversarial loss, the first total perceptual loss, the second total perceptual loss, the first inter-domain distance optimization loss and the second inter-domain distance optimization loss, and training the initial sketch generation model based on the total loss to obtain the sketch generation model.

[0008] In some embodiments of the present application, the initial sketch generated image is optimized in detail to obtain a historical sketch generated image, including: extracting high-frequency texture information of the historical optical face image to obtain a historical optical residual texture image; splicing the initial sketch generated image and the historical optical residual texture image to obtain a sketch residual texture image; and obtaining a historical sketch generated image based on the sketch residual texture image and the initial sketch generated image.

[0009] In some embodiments of the present application, an adversarial loss is determined based on a historical real sketch image, an initial sketch generated image, and a historical sketch generated image, including: inputting the historical real sketch image into the discriminator of the initial sketch generation model to obtain a first score that the historical real sketch image belongs to a real sketch image; determining a first expectation corresponding to the historical real sketch image based on the pixel value of the historical real sketch image and the first score; inputting the historical sketch generated image into the discriminator to obtain a second score that the historical sketch generated image belongs to the real sketch image; determining a second expectation corresponding to the historical sketch generated image based on the pixel value and the second score of the historical sketch generated image; inputting a smoothed image obtained after smoothing the historical real sketch image into the discriminator to obtain a third score that the smoothed image belongs to the real sketch image; determining a third expectation corresponding to the smoothed image based on the pixel value and the third score of the smoothed image; determining the adversarial loss based on the first expectation, the second expectation, and the third expectation.

[0010] In some embodiments of the present application, a first total perceptual loss is determined based on the historical real sketch image and the initial sketch generated image, and a second total perceptual loss is determined based on the historical real sketch image and the historical sketch generated image, including: determining a target convolutional layer from a feature extraction network in an initial sketch generation model, wherein the target convolutional layer includes multiple convolutional layers, and the feature map sizes output by each convolutional layer are different; determining a second feature corresponding to the historical real sketch image, a third feature corresponding to the initial sketch generated image, and a fourth feature corresponding to the historical sketch generated image through the target convolutional layer; determining a first total perceptual loss based on the second feature, the third feature, and the length and width of the second feature or the third feature; determining a second total perceptual loss based on the second feature, the fourth feature, and the length and width of the second feature or the fourth feature.

[0011] In some embodiments of the present application, a first feature of a historical optical facial image and a second feature of a historical real sketch image are fused to obtain a fused feature, including: converting the first feature into a one-dimensional vector and sorting the resultant feature in a preset order to obtain a first sequence, wherein the sequence number of the first sequence is in ascending order; sequentially reading the eigenvalues ​​of the first feature after it is converted into a one-dimensional vector to obtain a first eigenvalue set, and determining the sequence number of the first eigenvalue set according to the correspondence between the first eigenvalue set and the eigenvalues ​​of the first sequence; converting the second feature into a one-dimensional vector and sorting the resultant feature in a preset order to obtain a second sequence, wherein the sequence number of the second sequence is in ascending order; sequentially reading the eigenvalues ​​of the second feature after it is converted into a one-dimensional vector to obtain a second eigenvalue set; concatenating the sequence number of the first eigenvalue set and the second eigenvalue set to obtain an initial fused feature, wherein the initial fused feature is a one-dimensional vector; sorting the initial fused feature in ascending order according to the sequence number, converting the sorted initial fused feature into the length and width of the first feature or the second feature to obtain a fused feature.

[0012] In some embodiments of the present application, a first inter-domain distance optimization loss is determined based on the third feature, the first feature and the fused feature of the image generated by the initial sketch, and a second inter-domain distance optimization loss is determined based on the fourth feature, the first feature and the fused feature of the image generated by the historical sketch, including: determining the first distance between the third feature and the fused feature, determining the second distance between the third feature and the first feature; determining the first inter-domain distance optimization loss based on the first distance, the second distance and the distance threshold; determining the third distance between the fourth feature and the fused feature, determining the fourth distance between the fourth feature and the first feature; determining the second inter-domain distance optimization loss based on the third distance, the fourth distance and the distance threshold.

[0013] According to another aspect of an embodiment of the present application, a device for generating a sketch image is also provided, including: an acquisition module, used to acquire an optical face image to be processed, wherein the optical face image is an optical domain image; a generation module, used to process the optical face image using a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, comprising: a memory for storing program instructions; a processor, connected to the memory, for executing program instructions to implement the following functions: obtaining an optical face image to be processed, wherein the optical face image is an optical domain image; processing the optical face image using a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

[0015] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned method for generating a sketch image when executed by a processor.

[0016] In an embodiment of the present application, an optical face image to be processed is obtained, wherein the optical face image is an optical domain image; a sketch generation model is used to process the optical face image to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function. The mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold, thereby achieving the purpose of improving the accuracy of image conversion from the optical domain to the sketch domain, thereby realizing the technical effect of efficient and accurate sketch synthesis, and further solving the technical problems in the sketch synthesis method in the related art that the feature distribution matching accuracy is low and the modal difference between the generated sketch image and the real sketch image is large. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a method for generating a sketch image according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for generating a sketch image according to an embodiment of the present application;

[0020] Figure 3 is a network framework diagram of a sketch generation model according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of artifacts and blur caused by low-order statistics matching according to an embodiment of the present application;

[0022] Figure 5 This is the experimental result of comparing various methods on the CUHK dataset;

[0023] Figure 6 This is the experimental result of comparing various methods on the AR dataset;

[0024] Figure 7 This is the experimental result of comparing various methods on the XM2VTS dataset;

[0025] Figure 8 It is a structural diagram of a sketch image generation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0029] Existing sketch face synthesis methods are mainly divided into two categories: data-driven sketch face synthesis methods and model-driven sketch face synthesis methods.

[0030] The data-driven synthesis method divides the sketch image in the data set into image blocks, then selects the sketch image blocks that best match the optical image content, and finally obtains the sketch image by splicing the selected image blocks. Earlier, Roweis et al. proposed an unsupervised Locally Linear Embedding (LLE) algorithm, which can retain local linear features in the process of input dimensionality reduction, so that training samples can be screened and combined according to the retained features in sketch face synthesis. Tang et al. constructed facial feature vectors using covariance matrices and synthesized sketch face images by projecting test images into feature vector space. Later, Tang et al. proposed a sketch face synthesis method combined with a multi-scale Markov Random Field (MRF) model. This method divides the facial area into overlapping image blocks, and then uses the MRF model to learn the optical face image and sketch image pairs at multiple scales, thereby synthesizing higher quality sketch face images. Gao et al. proposed a sketch face synthesis algorithm based on sparse representation. The algorithm automatically finds the adjacent image block closest to the target image block, and then learns the mapping between the sketch image block and the optical image block by building a coupled sparse representation model, thereby improving the quality of the sketch face synthesis image. Wang et al. proposed an algorithm model that effectively utilizes test data. The model innovatively incorporates the test image into the model learning process. The model is used to alternately optimize the reconstruction fidelity of the input optical face image and the synthesis fidelity of the target output sketch image, thereby improving the synthesis performance and quality of the model. However, the data-driven synthesis method cannot well maintain the global structure of the face image, and consumes a high computational cost in the process of searching and matching local patches. Zhang et al. proposed a sketch face synthesis method using prior knowledge. The method calculates the similarity between different image blocks, learns the sparse coefficient matrix from the training image block, and then obtains the sketch synthesis image corresponding to the final test image through the sparse coefficient matrix and prior knowledge. Wang et al. proposed a simple and efficient offline random sampling and local constraints (RSLCR) synthesis method, which uses offline random sampling instead of the common K-NN algorithm, and combines local constraints to simulate the obvious correlation between the test patch and the randomly sampled patch, and finally synthesizes the sketch result of the test image. Jiang et al. proposed a sketch face synthesis method using residual learning (RL), which predicts the residual image by learning the mapping relationship between the optical face image and the residual, and improves the quality of the sketch face synthesis image by optimizing the residual mapping.

[0031] The model-driven synthesis method learns the mapping relationship between optical images and sketch images by training the neural network model. In the inference stage, the input optical face image can quickly output the corresponding sketch style image, which makes up for the time-consuming problem of the data-driven method. Among them, the method based on the Generative Adversarial Network (GAN) uses the adversarial training of the generator and the discriminator to enable the discriminator to distinguish between real sketch images and generated sketch images by learning, prompting the generator to generate more and more realistic sketch images to "deceive the discriminator". Adaptive Instance Normalization (AdaIN) is based on the Gaussian prior assumption and performs sketch stylization by matching the mean and standard deviation of features. The Whitening and Coloring Transform (WCT) proposed by Li et al. achieves style transfer by matching the covariance matrix of the content image and the style image. However, the matching of the low-order statistics of the features does not conform to the distribution of the real data, resulting in a significant difference between the modality of the generated image and the real sketch. Some studies use the GAN, introduce the discriminator, and train against the generator to achieve Nash equilibrium, thereby generating images with more realistic visual effects. Isola et al. proposed a conditional generative adversarial network Pix2Pix, which uses the U-Net model to effectively retain feature information of different scales. However, the L1 loss used in this model will lead to overfitting and poor generalization ability when the training data is limited. Zhu et al. proposed a cycle-consistent generative adversarial network (CycleGAN), which uses two generative adversarial networks to achieve cycle consistency from the source domain to the target domain and from the target domain back to the source domain. In this way, CycleGAN can learn the mapping between the two image domains and maintain content consistency without one-to-one paired training samples. Zhang et al. proposed a sketch face synthesis framework consisting of two processes: inter-domain transfer and intra-domain transfer. The intra-domain transfer process is used to learn the regressor of the optical face image in the test data relative to the optical face image in the training data, and transfer it to the sketch image domain to ensure that the facial structure of the face is reconstructed in the process of synthesizing the sketch, while the inter-domain transfer process is used to learn the mapping relationship between the optical face image and the sketch image, thereby suppressing the problem of losing specific structural information in the synthesis process. Based on the idea of ​​disentanglement, Lee et al. embedded images into a cross-domain content information space that captures shared information between domains and a domain-specific attribute space. They took disentangled features as input, solved the problem of mode collapse, and constrained content features not to carry domain attributes through content adversarial loss.Sheng et al. regard the sketch face synthesis task as an optimization task based on neural network feature maps, use enhanced 3D block matching and cross-layer cost aggregation methods to obtain the feature maps of the test image, and make full use of the overall sketch style information hidden in the training data set to finally synthesize the sketch result image with relatively excellent texture and sketch style performance. Hong et al. proposed a domain-aware style transfer network (DSTN) to transmit the attributes of the sketch domain while transmitting style information, and proposed a new indicator to capture domain attributes from the texture and structure of the sketch image. In order to improve the visual fidelity of the generated image, Huang et al. proposed a new style transfer framework (QuanArt), which models artistic images of different styles through vector quantization, and pushes the potential representation of the generated image to the centroid of the distribution of real artworks to synthesize sketch images with consistent structure, similar style, and visual fidelity. However, the above methods have problems such as low feature distribution matching accuracy and large modal differences between the generated image and the real sketch image.

[0032] In order to solve the problems of missing local details and residual optical domain modal features in the sketch image generated in the related art, the present application provides a method for generating a sketch image, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0033] The sketch image generation method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a method for generating a sketch image is shown. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for generating a sketch image in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned method for generating a sketch image. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0038] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0039] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for generating a sketch image. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 is a flow chart of a method for generating a sketch image according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0041] Step S202, obtaining an optical face image to be processed, wherein the optical face image is an optical domain image.

[0042] In the above step S202, the optical face image refers to a face image captured by an optical imaging device (such as a digital camera, a webcam, etc.). These optical face images are usually stored in the form of a two-dimensional pixel matrix, and each pixel contains color (which may be grayscale or color) and brightness information. Optical face images are projections of real-world faces on optical sensors, so they reflect the physical characteristics of the face in terms of lighting, shadows, texture, color, etc. The optical domain refers to the collection or space of image data captured by an optical imaging device.

[0043] Step S204, using a sketch generation model to process the optical face image to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

[0044] In the above step S204, the sketch generation model is a network structure based on deep learning, which is specially designed to convert one image style (here is the optical domain image) into another style (sketch domain image). In order to train the sketch generation model, it is necessary to define a loss function to measure the difference between the generated image and the real sketch image. The inter-domain distance optimization loss function is one of the special loss functions for training the sketch generation model. It aims to reduce the "distance" or difference between the generated image and the target domain (here is the sketch domain) image. This difference may include, for example, pixel-level differences, structural differences, texture differences, etc. By optimizing the loss function, the sketch generation model can better learn to extract key features from the optical domain image and re-express these features in a sketch style, so that the sketch generation model can distinguish modal differences.

[0045] Through the above steps S202 to S204, the purpose of improving the accuracy of image conversion from the optical domain to the sketch domain is achieved, thereby achieving the technical effect of efficient and accurate sketch synthesis, and further solving the technical problems of low feature distribution matching accuracy and large modality difference between the generated sketch image and the real sketch image in the sketch synthesis method in the related art. The following is an explanation.

[0046] In the above-mentioned sketch image generation method, the sketch generation model is trained in the following manner: obtaining a historical optical face image, and obtaining a historical real sketch image corresponding to the historical optical face image; using the first generator of the initial sketch generation model to process the historical optical face image to obtain an initial sketch generation image; using the second generator of the initial sketch generation model to optimize the details of the initial sketch generation image to obtain a historical sketch generation image; determining an adversarial loss based on the historical real sketch image, the initial sketch generation image, and the historical sketch generation image; determining a first total perceptual loss based on the historical real sketch image and the initial sketch generation image, and determining a second total perceptual loss based on the historical real sketch image and the historical sketch generation image. total perceptual loss; fusing the first feature of the historical optical face image and the second feature of the historical real sketch image to obtain a fused feature; determining the first inter-domain distance optimization loss based on the third feature, the first feature and the fused feature of the initial sketch generated image, and determining the second inter-domain distance optimization loss based on the fourth feature, the first feature and the fused feature of the historical sketch generated image, wherein the first inter-domain distance optimization loss and the second inter-domain distance optimization loss are determined by the inter-domain distance optimization loss function; determining the total loss based on the adversarial loss, the first total perceptual loss, the second total perceptual loss, the first inter-domain distance optimization loss and the second inter-domain distance optimization loss, and training the initial sketch generation model based on the total loss to obtain the sketch generation model.

[0047] The sketch image generation method provided in the embodiment of the present application is a generative adversarial sketch face synthesis method based on inter-domain distance optimization. In the training process of the sketch generation model, the high-frequency texture information of the historical optical face image is extracted by Laplace transform, and the mapping relationship of the high-frequency texture information from the optical domain to the sketch domain is learned by the detail optimization module. The generated historical optical residual texture image (or referred to as the sketch domain high-frequency texture image) is combined with the initial sketch generation image to optimize the local details and obtain the historical sketch generation image; then, the historical real sketch image after the historical optical face image is smoothed is input into the discriminator as a false sample, so that the discriminator can more easily distinguish the blurred historical face image. Sketch-generated images are generated, thereby improving the generalization ability of the model; finally, an inter-domain distance optimization loss is proposed, which filters out the residual optical domain modal features in the historical sketch-generated images by expanding the Euclidean distance between the features of the historical sketch-generated images (anchor points) and the features of the historical optical face images (negative samples), and reduces the Euclidean distance between the features of the historical sketch-generated images and the features of the historical real sketch images (positive samples), so that the modality of the historical sketch-generated images is close to the real sketch images. Among them, a feature fusion module is proposed, which fuses the features of the historical optical face images and the historical real sketch images as positive samples through a quick sorting algorithm, so that the historical sketch-generated images are closer to the real data distribution and the visual quality is improved.

[0048] The following combination Figure 3 Explain the training process of the sketch generation model. Figure 3 This is a network framework diagram of a sketch generation model according to an embodiment of the present application, including: (A) Framework Overview, (B) Adversarial Loss, and (C) Domain's Distance Optimization Loss. Figure 3 In part (A), we first transform the historical optical face image P i The generator of the input encoder-decoder structure (i.e. the first generator mentioned above) generates the corresponding sketch image S' i (i.e. the image generated by the initial sketch above), and then the historical optical face image P i Perform Laplace filtering to generate the historical optical residual texture image R i , the historical optical residual texture image and the initial sketch generated image S' i Stitching, input detail optimization module (Detail Optimization Module) to generate sketch residual texture image R' i , and the initial sketch generates an image S' i Stack to get the historical sketch generated image S" i In the feature extraction stage, the corresponding Figure 3In part (C), the historical optical face image P i , historical real sketch image S i And the initial sketch generates the image S' i Or historical sketch generates image S" i The pre-trained VGG-16 network is input to extract multi-level features, wherein the mean square error is used to calculate the norms between the historical real sketch image features and the initial sketch generated image features, and the historical real sketch image features and the historical sketch generated image features, respectively, to obtain the first total perceptual loss and the second total perceptual loss, and the inter-domain distance optimization loss required for training the first generator or the second generator is obtained through feature fusion, wherein the first inter-domain distance optimization loss is required for training the first generator, and the second inter-domain distance optimization loss is required for training the second generator. i , historical real sketch image S i And the initial sketch generated image S' i As an example, we get F Pij 、F Sij 、F S'ij The feature fusion module fuses the historical optical face image features F Pij and historical real sketch image features F Sij Output fusion feature F Fusion , the fused features are used as positive samples of the inter-domain distance optimization module, and the initial sketch generates image features F S'ij As an anchor, the historical optical face image feature F Pij As negative samples (Negative). The inter-domain distance optimization module supervises the generator to distinguish the modal differences between the optical face image domain and the sketch domain by pushing the Euclidean distance between the anchor point and the negative sample farther and shortening the Euclidean distance between the anchor point and the positive sample closer. In order to further improve the performance of the detail optimization module, the discriminator D (Discriminator) is introduced to conduct adversarial training with the detail optimization module to obtain the adversarial loss L' adv ,correspond Figure 3 In part (B), the historical real sketch image S i Input the edge smoothing module to obtain the smoothed sketch E i , S i With E i , S” i Input the discriminator to improve its ability to distinguish blurred generated images.

[0049] The following is a detailed description of the training process of the above sketch generation model with specific steps.

[0050] In the above steps, the initial sketch generated image is optimized in detail to obtain a historical sketch generated image, including: extracting high-frequency texture information of the historical optical face image to obtain a historical optical residual texture image; splicing the initial sketch generated image and the historical optical residual texture image to obtain a sketch residual texture image; and obtaining a historical sketch generated image based on the sketch residual texture image and the initial sketch generated image.

[0051] In some embodiments of the present application, the face image has more detailed information, such as complex lines and textures such as wrinkles, hair and glasses. It is difficult to effectively process these detailed information using the traditional sketch image generation method. In order to enhance the model's ability to process detailed information, the embodiments of the present application optimize the initial sketch generated image through a detail optimization module, such as Figure 3 First, the historical optical face image P is extracted by Laplace filtering. i The high-frequency texture information of the historical optical residual texture image R is generated i , spliced ​​with the initial sketch image S'i, input into the detail optimization module, the Laplace filter expression is as follows:

[0052]

[0053] in, is the historical optical face image P i 2 times down-sampled version, L is Laplace filtering, U p is a 2x upsampling operation; then, the detail optimization module generates a sketch residual texture image R' by learning the mapping relationship between high-frequency texture information from the optical domain to the sketch domain. i , and the smoothed initial sketch generates the image S' i Stacking, optimizing high-frequency textures and local details. This process ensures that the global structure of the generated sketch image is correctly preserved while the details are richer and the contours and textures are clearer. The specific formula is as follows:

[0054] R′ i =DetOpt(R i ,S′ i )

[0055] S″ i =S′ i +R′ i

[0056] Wherein, DetOpt represents the above-mentioned detail optimization module, and the detail optimization network (corresponding to the above-mentioned second generator) is designed as an encoder-decoder framework, such as Figure 3As shown in part (A) of , it includes the following parts: convolution layer (Conv_Block), downsampling layer (Downsample_Block), 9 residual blocks (Res_Block), and upsampling layer (Upsample_Block). It should be noted that the architecture of the second generator is similar to that of the first generator, specifically the number of upsampling layers and downsampling layers is different. In order to pay more attention to local detail information, the detail optimization network contains only one downsampling layer and the corresponding upsampling layer to reduce the receptive field. The convolution kernel of the convolution layer can be 3×3, with a step size of 1. Before the convolution operation, the outer side of the feature map is filled with a reflection of scale 1. Because the edge content of the image is repeated instead of simply filling with zero or other constant values, this method can expand the image edge more naturally than traditional zero filling. The residual block can effectively retain the original structural information while deepening the network depth. Therefore, 9 residual blocks are added after the downsampling block to learn the mapping relationship of texture information.

[0057] In the above steps, the adversarial loss is determined based on the historical real sketch image, the initial sketch generated image and the historical sketch generated image, including: inputting the historical real sketch image into the discriminator of the initial sketch generation model to obtain a first score that the historical real sketch image belongs to the real sketch image; determining the first expectation corresponding to the historical real sketch image based on the pixel value and the first score of the historical real sketch image; inputting the historical sketch generated image into the discriminator to obtain a second score that the historical sketch generated image belongs to the real sketch image; determining the second expectation corresponding to the historical sketch generated image based on the pixel value and the second score of the historical sketch generated image; inputting the smoothed image obtained after smoothing the historical real sketch image into the discriminator to obtain a third score that the smoothed image belongs to the real sketch image; determining the third expectation corresponding to the smoothed image based on the pixel value and the third score of the smoothed image; determining the adversarial loss based on the first expectation, the second expectation and the third expectation.

[0058] In the embodiment of the present application, since adversarial training can promote the model to synthesize more realistic images through the feedback of the discriminator, and the discriminator of PatchGAN has a smaller receptive field and is more sensitive to local structures, the discriminator of PatchGAN is added to the embodiment of the present application to help the detail optimization network capture finer local details, and an edge smoothing processing module is proposed, which is reflected in Figure 3 Part (B) of the

[0059] The embodiment of the present application selects WGAN-GP as the adversarial loss function in generative adversarial training, which uses Wasserstein distance to calculate the loss, avoiding the mode collapse and gradient vanishing problems during the training process, and improving the robustness of GAN during the training process. The adversarial loss function can be expressed as:

[0060]

[0061] Among them, P r (s) is the sample distribution of historical real sketch images, P g (s) is the sample distribution of images generated by historical sketches. represents the expected fraction of historical real sketch images after inputting them into the discriminator, It represents the expectation of the fraction of the historical sketch generated image that belongs to the real sketch image after it is input into the discriminator.

[0062] During the training process, some generated result images without clear edge contours can successfully deceive the discriminator and affect the training of the discriminator, resulting in the fact that simply using the standard adversarial loss function to train the discriminator D is not enough to generate excellent sketch images. Therefore, before the start of training, the application introduces an edge smoothing processing module to smooth the images in the real sketch training set r(s) (i.e., the above-mentioned historical real sketch images) to obtain the processed smoothed sketch data set e(s). The specific steps are: 1) Use the standard Canny edge detector to detect the edge pixels of the face; 2) Expand the edge area; 3) Use the standard Gaussian smoothing operation on the expanded edge area. The specific formula is as follows:

[0063] E=Canny(S i )

[0064] E d =Dilate(E,K)

[0065] S smooth =Gauss(E d ,σ)

[0066] Among them, E represents the face edge pixels corresponding to the historical real sketch image, K is the 5x5 expansion kernel, and E d It represents the area obtained by expanding the edge area of ​​the face edge pixels, S smooth represents the smoothing operation, and σ is the standard deviation of the Gaussian kernel. The improved adversarial loss function can be expressed as:

[0067]

[0068] in, where D(S) represents the first score mentioned above, Expressing the first expectation mentioned above, D(S) in represents the second fraction mentioned above, represents the second expectation mentioned above, P s (s) is the sample distribution of the sketch image after edge smoothing of the historical real sketch image, D(S) in represents the third fraction mentioned above, It represents the expectation of the fraction of historical real sketch images after the sketch image with edge smoothing is input into the discriminator, that is, the third expectation mentioned above, L adv ′(D) is the above-mentioned confrontation loss, such as Figure 3 As shown in part (B) of FIG.

[0069] It should be noted that the loss function used in the embodiment of the present application is Wasserstein Loss, which is also a loss function of the generative adversarial network. It no longer outputs a probability, but outputs a real-valued score with an interval of negative infinity to positive infinity. When calculating the loss of the generator, the opposite number of the discriminator score of the generated false sample (at the beginning, the discriminator is very easy to distinguish false samples, and the output score is a negative number with a large absolute value, and then the opposite number is taken, which is the loss (positive number), and iterative optimization makes this loss smaller). When calculating the loss of the discriminator, the false sample score is subtracted from the average value of the real sample score (in order to more easily distinguish true and false samples, the discriminator should try to make the real sample score larger and the false sample score lower). The above losses all take the expectation of the score of each training batch, that is, E in the formula.

[0070] In the above steps, a first total perceptual loss is determined based on the historical real sketch image and the initial sketch generated image, and a second total perceptual loss is determined based on the historical real sketch image and the historical sketch generated image, including: determining a target convolutional layer from a feature extraction network in an initial sketch generation model, wherein the target convolutional layer includes multiple convolutional layers, and the feature map sizes output by each convolutional layer are different; determining a second feature corresponding to the historical real sketch image, a third feature corresponding to the initial sketch generated image, and a fourth feature corresponding to the historical sketch generated image through the target convolutional layer; determining a first total perceptual loss based on the second feature, the third feature, and the length and width of the second feature or the third feature; determining a second total perceptual loss based on the second feature, the fourth feature, and the length and width of the second feature or the fourth feature.

[0071] In some embodiments of the present application, sketch face synthesis aims to make the generated sketch image retain the identity information of the optical face image while integrating the modal information of the sketch, and the corresponding sketch image can be generated using perceptual loss and style loss constraint generators respectively.

[0072] First, in the training sketch generation model, the first generator ( Figure 3 Part (A) from P i to S′ i The process is realized by the first generator) when the historical real sketch image S i and the initial sketch to generate the image S′ iInput to the pre-trained VGG-16 network, assuming that the input image is X, the embodiment of the present application selects different convolutional layers V in multiple VGG-16 networks j (X) (i.e. the target convolution layer mentioned above) extracts the feature F of the input image X X , where j is the number of layers in the network, for example, 3 convolutional layers are selected. After extracting the features, the norm between the generated sketch features and the corresponding real sketch features is calculated using the mean square error as the perceptual loss function L′ perceptual , as shown in the formula:

[0073]

[0074]

[0075]

[0076]

[0077] Among them, F Sij represents the second feature mentioned above, F S'ij represents the third feature, w and h represent the width and height of the second or third feature, n and k represent the pixel value, ||F Sij –F S'ij || is defined as:

[0078]

[0079] If F Sij –F S'ij Let it be M, then M can be expressed as:

[0080]

[0081] Among them, tr{(F Sij –F S'ij ) T ·(F Sij –F S'ij )} is defined as:

[0082]

[0083] Among them, tr means to find the sum of the main diagonal.

[0084] In an optional embodiment, the 2nd, 7th and 14th layers in the feature extraction network V can be selected as the output layers of feature extraction, namely V2(X), V7(X), V 14 (X). Finally, the outputs of the above three convolutional layers are added after perceptual loss calculation to obtain the total perceptual loss L perceptual, through this loss, the generator G can be guided to better preserve the detailed features of the facial content and provide accurate semantic content for the generated image. The total perceptual loss expression is as follows:

[0085] L perceptual (S i , S′ i ) = L′ perceptual (S i2 , S′ i2 , V2)+L′ perceptual (S i7 , S′ i7 , V7)+L′ perceptual (S i14 , S i14 , V 14 )

[0086] Among them, L perceptual (S i , S i ) is the first total perceptual loss mentioned above, which can also be expressed as L perceptual (G).

[0087] In training the second generator of the sketch generative model ( Figure 3 Part (A) from R i to R′ i The process is realized by the second generator) when the historical real sketch image S i and historical sketch to generate image S″ i Input to the pre-trained VGG-16 network, select different convolutional layers V in multiple VGG-16 networks j (X) (i.e. the target convolution layer mentioned above) extracts the feature F of the input image X X The second total perceptual loss is determined in the same way as the first total perceptual loss, replacing S′ in the first total perceptual loss i Replace with S″ i , other parameters remain unchanged, and the second total perceptual loss L can be obtained perceptual (S i , S″ i ), which can also be expressed as L perceptual (Rev), specifically, the second feature and the fourth feature F corresponding to the image generated by the historical sketch are needed to determine the second total perceptual loss. S″ij and the length and width of the second or fourth feature.

[0088] In the above steps, the first feature of the historical optical face image and the second feature of the historical real sketch image are fused to obtain a fused feature, including: converting the first feature into a one-dimensional vector and sorting them in a preset order to obtain a first sequence, wherein the sequence number of the first sequence is in ascending order; sequentially reading the eigenvalues ​​of the first feature after it is converted into a one-dimensional vector to obtain a first eigenvalue set, and determining the sequence number of the first eigenvalue set according to the correspondence between the first eigenvalue set and the eigenvalues ​​of the first sequence; converting the second feature into a one-dimensional vector and sorting them in a preset order to obtain a second sequence, wherein the sequence number of the second sequence is in ascending order; sequentially reading the eigenvalues ​​of the second feature after it is converted into a one-dimensional vector to obtain a second eigenvalue set; splicing the sequence number of the first eigenvalue set and the second eigenvalue set to obtain an initial fused feature, wherein the initial fused feature is a one-dimensional vector; sorting the initial fused feature in ascending order according to the sequence number, converting the sorted initial fused feature into the length and width of the first feature or the second feature to obtain a fused feature.

[0089] In the above steps, the first inter-domain distance optimization loss is determined based on the third feature, the first feature and the fused feature of the image generated by the initial sketch, and the second inter-domain distance optimization loss is determined based on the fourth feature, the first feature and the fused feature of the image generated by the historical sketch, including: determining the first distance between the third feature and the fused feature, determining the second distance between the third feature and the first feature; determining the first inter-domain distance optimization loss based on the first distance, the second distance and the distance threshold; determining the third distance between the fourth feature and the fused feature, determining the fourth distance between the fourth feature and the first feature; determining the second inter-domain distance optimization loss based on the third distance, the fourth distance and the distance threshold.

[0090] In some embodiments of the present application, for style loss, simply reducing the distance between the generated sketch image and the historical real sketch image in the feature space will cause the model to lack the ability to distinguish the modal differences between the optical domain and the sketch domain images, making the sketch style of the generated image lack realism. In the work of Schroff et al., triplet loss was originally used for face recognition and clustering, aiming to distinguish non-matching faces from faces of the same identity by increasing the distance between negative pairs and positive pairs in the embedding space. Inspired by Schroff et al., different from distinguishing identity information, the embodiments of the present application introduce inter-domain distance optimization loss as style loss, and supervise the model to filter optical domain modal information that is not related to the sketch domain, such as Figure 3 As shown in part (C) of FIG.

[0091] First, the historical optical face image P i , historical real sketch image S i and the initial sketch to generate the image S′ i Input to the pre-trained VGG-16 network to extract the historical optical face image feature FPij , Historical real sketch image features F Sij and the initial sketch to generate image features F S′ij , as shown in the formula:

[0092]

[0093] Then, by reducing the Euclidean distance between the anchor point and the positive sample and increasing the Euclidean distance between the anchor point and the negative sample, the model is promoted to learn the modal differences between the optical domain and the sketch domain. Among them, selecting appropriate positive and negative samples is particularly critical. In the selection process of positive samples, due to the subjectivity of manual hand-drawing and other factors (such as cropping and tilt), the correspondence between sketches and faces may be disturbed. Therefore, the embodiment of the present application does not simply select a real sketch image as a positive sample, but selects a fusion of optical images and sketch image features F. Fij As positive samples. For the selection of negative samples, since other optical facial images do not belong to the same identity as the currently trained face, and there are modal differences and content differences between them and the current sketch face, selecting optical facial images of other identities in the data set will increase the spatial distance between the two. In order to filter out the modal information of the optical domain image, the embodiment of the present application selects the input historical optical facial images as negative samples. The input historical optical facial images and the input historical real sketch facial images only have inter-modal differences, which can enable the style loss to model the details of the modal differences, prompting the generator to synthesize facial images with more realistic sketch styles.

[0094] For the acquisition of positive samples, the historical optical face image features Fpij extracted by VGG-16 are compared with the historical real sketch image features F Sij Input to the feature fusion module as a positive sample, the fused feature expression is as follows:

[0095]

[0096] Fusion means feature fusion, and feature fusion can be regarded as aligning the features of the optical face image with the features of the sketch face image, so that the statistics of the generated image feature distribution match the sketch domain image, thereby achieving sketch stylization. The traditional AdaIN fusion method fuses features by matching the mean and variance of the channel level of the optical image and the sketch image, and its expression is:

[0097]

[0098] Among them, x represents the first feature of the input and y represents the second feature of the input.

[0099] like Figure 4 As shown, Figure 4This is a schematic diagram of artifacts and blur caused by low-order statistical matching in an embodiment of the present application. During the training process of the experiment, artifacts appeared, such as "double chin", "thick lips" and other unsatisfactory effects. This is because the matching of low-order statistics makes the model unable to correctly model the complex distribution of real data, resulting in the structure of the generated image tending to be an intermediate effect between optical images and sketch images, and the correct mapping relationship cannot be learned. In order to improve the matching accuracy of feature distribution, the embodiment of the present application adopts a sorting matching algorithm to achieve matching of higher-level statistics. The embodiment of the present application adopts a quick sorting algorithm for sorting matching, which is achieved by matching after sorting two vectors. Its index is represented by a single-row vector as follows:

[0100] x: t=(t1, t2, t3,..., t n ),

[0101] y:k=(k1,k2,k3,...,k n ),

[0102] Among them, t1, t2, t3, ..., t n represents the first sequence, k1, k2, k3, ..., k n represents the second sequence, and Sort in ascending order, that is, if i < j, then x i ≤x j , similarly, y ki The definition is similar. Sorting matching output o, then τ i Element o τi for:

[0103]

[0104] Among them, τi represents the sequence number of the first eigenvalue set mentioned above, represents the second eigenvalue set.

[0105] For ease of understanding, the above sorting and matching process is explained below with reference to specific examples.

[0106] Assume the first feature Second feature After the first feature is converted into a one-dimensional vector, it is sorted in a preset order (i.e., the order of eigenvalues ​​from small to large) to obtain a first sequence. The first sequence is [0 0 11 2 3]. The serial numbers of the first sequence are t1, t2, t3, t4, t4, t6, and the first eigenvalue set after the first feature is converted into a one-dimensional vector is {0, 1, 0, 2, 1, 3}. According to the corresponding relationship between the first eigenvalue set and the eigenvalue of the first sequence, the serial number of the first eigenvalue set is determined, that is, the serial numbers corresponding to {0, 1, 0, 2, 1, 3} are t1, t3, t2, t5, t4, t6, respectively.

[0107] After the second feature is converted into a one-dimensional vector and sorted in a preset order, a second sequence is obtained, and the second sequence is [013456]. The serial numbers of the second sequence are k1, k2, k3, k4, k5, k6 respectively. The second eigenvalue set after the second feature is converted into a one-dimensional vector is {1, 0, 4, 3, 5, 6}. According to the corresponding relationship between the second eigenvalue set and the eigenvalue of the second sequence, the serial number of the second eigenvalue set is determined, that is, the serial numbers corresponding to {1, 0, 4, 3, 5, 6} are k2, k1, k4, k3, k5, k6 respectively.

[0108] Concatenate the sequence number of the first eigenvalue set and the second eigenvalue set to obtain the initial fusion feature, that is, to obtain o τ1 =y k2 =1,o τ3 =y k1 =0,o τ2 =y k4 =4,o τ5 =y k3 =3,o τ4 =y k5 =5,o τ6 =y k6 =6, the sequence number of the initial fusion feature is o τi After sorting in ascending order, we get {1, 4, 0, 5, 3, 6}, whose corresponding serial numbers are o τ1 -o τ6 The sorted initial fusion features are converted into the length and width of the first feature or the second feature to obtain the fusion feature. The vector representation of the fusion feature is

[0109] In order to implement gradient back propagation in the deep model, the embodiment of the present application modified the fusion formula in the experiment as follows:

[0110]

[0111] Among them, <> means stopping the gradient operation.

[0112] Regarding the selection of negative samples, using other optical facial images as negative samples will cause the discriminator to easily distinguish between positive and negative samples, which will not provide sufficient guidance for the training of the discriminator, and thus affect the generation effect of the generator. The embodiment of the present application selects the currently input historical optical facial image as a negative sample. Since it only has a modal difference with the input historical real sketch image, it can force the inter-domain distance optimization loss to model the details of the modal difference, train the discriminator's sketch style judgment ability, and prompt the generator to synthesize sketch facial images with a more realistic sketch style. The inter-domain distance optimization loss function expression is as follows:

[0113]

[0114] Where, L ddo (G) is the first inter-domain distance optimization loss, For the third feature, To fusion features, As the first feature, is the first distance, is the second distance, α is the distance threshold, which can be set to 2 according to experience, and d is the Euclidean distance, which is expressed as follows:

[0115]

[0116]

[0117] Among them, a is the anchor image, p is the positive sample image, and n is the negative sample image.

[0118]

[0119] Where, L ddo (Rev) is the second domain distance optimization loss, The fourth characteristic is is the third distance, The fourth distance.

[0120] In summary, the total loss function L of the training (initial) sketch generation model in the embodiment of the present application is total It can be expressed as:

[0121] L total (G, Rev, D) = α1L perceptual (G)+β1L ddo (G)+α2L perceptual (Rev)+β2L ddo (Rev)+μL′ adv , (D)

[0122] Among them, α1, α2, β1, β2, and μ are the weights of the loss function. After many experiments, they are finally set to: α1=1, β1=10, α2=1, β2=16, and μ=1 respectively.

[0123] The initial sketch generation model is trained through the above total loss function to obtain a sketch generation model.

[0124] The sketch image generation method provided in the embodiment of the present application utilizes the Laplace algorithm to extract the local texture of the optical face image, combines it with the initial sketch generated image, and inputs it into a neural network to learn the mapping relationship of the detail texture, thereby improving the generation effect; by introducing the blurred sketch image as a supplement to the discriminator training data, the robustness of the training is improved; a sketch face synthesis method with inter-domain distance optimization is used, by pushing away the Euclidean distance between the generated sketch image and the optical face image in the feature space, and shortening the Euclidean distance between the generated image and the sketch in the feature space, the supervised model learns the mapping from the optical face image to the sketch image, thereby distinguishing the modal differences between the optical face image and the sketch image.

[0125] In addition, the sketch image generation method provided in the embodiment of the present application benefits from the inter-domain distance optimization loss, which constrains the generated sketch image to be close to the sketch domain in the feature space, making the sketch style more realistic, and away from the optical domain in the feature space, filtering out modal information irrelevant to the sketch style, and accurately matching the feature distribution of the optical image and the sketch image through a quick sorting algorithm; and using a detail optimization module and an edge smoothing processing module to overcome the problems of blurred and deformed contours of the generated image, generate richer texture details, and perform more stably in different data sets.

[0126] In order to verify the effectiveness of the sketch image generation method provided in the embodiment of the present application in different sketch style data sets, the embodiment of the present application selects the CUHK data set, the AR data set and the XM2VTS data set, each data set contains a pair of optical face images and corresponding manual hand-drawn sketch images.

[0127] In the data preprocessing stage, the optical-sketch face images are aligned and cropped to a size of 256×256. The overall model training is divided into two stages. In the first stage, the sketch generator is trained with a batch size of 1, a learning rate of 0.0002 for the first 100 iterations, and a linear decay of the learning rate to 0 for the next 200 iterations. The Adam optimizer is used with momentum parameters of 0.5 and 0.999, respectively. In the second stage, the detail optimization network is trained with a batch size of 5, a learning rate of 0.0001 for the first 50 iterations, and a linear decay of the learning rate to 0 for the next 100 iterations. The Adam optimizer is used with momentum parameters of 0.9 and 0.99, respectively.

[0128] In order to verify the effectiveness of the sketch image generation method of the embodiment of the present application, this section will conduct a comparative experiment between the method proposed in the embodiment of the present application and the existing mainstream algorithms CycleGAN, MDAL, DRIT, pix2pix and AdaIN. Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS) are selected as measurement indicators.

[0129] The comparative experimental results of various methods on CUHK, AR, and XM2VTS datasets are shown in Figure 5 , 6 , 7. Among them, Figure 5 The experimental results of the comparison of various methods on the CUHK dataset are shown in Figure 2. Figure 6 The experimental results of the AR dataset are compared with each method. Figure 7 The results of the comparative experiments of various methods on the XM2VTS dataset are shown in Tables 1, 2, and 3.

[0130] Table 1 Comparison of quantitative indicators of different methods on CUHK dataset

[0131]

[0132] Table 2 Comparison of quantitative indicators of different methods on AR dataset

[0133]

[0134] Table 3 Comparison of quantitative indicators of different methods on the XM2VTS dataset

[0135]

[0136] As can be seen from the table above, the method proposed in the embodiment of the present application has higher scores on the three data sets than other comparison methods, which is consistent with the subjective feeling of the synthetic sketch image results in the figure. This is due to the inter-domain distance optimization loss, which ensures that the network generates rich texture feature details without generating redundant pixels. At the same time, due to the addition of detail optimization module and edge smoothing processing module, the method for generating sketch images in the embodiment of the present application also performs better in dealing with problems such as image edge blur and deformation. Therefore, considering the subjective visual experience and objective test results, the method provided in the embodiment of the present application is more competitive.

[0137] Figure 8 is a structural diagram of a sketch image generation device according to an embodiment of the present application, such as Figure 8 As shown, the device comprises:

[0138] An acquisition module 40 is used to acquire an optical face image to be processed, wherein the optical face image is an optical domain image;

[0139] A generation module 42 is used to process the optical face image using a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function. The mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

[0140] Through the acquisition module 42 and the generation module 42 in the above-mentioned sketch image generation device, the purpose of improving the accuracy of image conversion from the optical domain to the sketch domain is achieved, thereby realizing the technical effect of efficient and accurate sketch synthesis, and further solving the technical problems in the sketch synthesis method in the related technology that the feature distribution matching accuracy is low and the generated sketch image has a large modal difference with the real sketch image.

[0141] The sketch image generation device further includes a training module 44, which is used to train the sketch generation model. Specifically, the sketch generation model is trained in the following manner: obtaining a historical optical face image and a historical real sketch image corresponding to the historical optical face image; using a first generator of the initial sketch generation model to process the historical optical face image to obtain an initial sketch generation image; using a second generator of the initial sketch generation model to optimize the details of the initial sketch generation image to obtain a historical sketch generation image; determining an adversarial loss based on the historical real sketch image, the initial sketch generation image and the historical sketch generation image; determining a first total perceptual loss based on the historical real sketch image and the initial sketch generation image, and determining a first total perceptual loss based on the historical real sketch image and the initial sketch generation image. The method comprises the following steps: generating an image by combining the optical face image and the historical sketch to generate an image, determining a second total perceptual loss; fusing the first feature of the historical optical face image and the second feature of the historical real sketch image to obtain a fused feature; determining a first inter-domain distance optimization loss based on the third feature, the first feature and the fused feature of the image generated by the initial sketch, and determining a second inter-domain distance optimization loss based on the fourth feature, the first feature and the fused feature of the image generated by the historical sketch, wherein the first inter-domain distance optimization loss and the second inter-domain distance optimization loss are determined by an inter-domain distance optimization loss function; determining a total loss based on the adversarial loss, the first total perceptual loss, the second total perceptual loss, the first inter-domain distance optimization loss and the second inter-domain distance optimization loss, and training the initial sketch generation model based on the total loss to obtain a sketch generation model.

[0142] In the training module in the above-mentioned sketch image generation device, the training module is also used to extract high-frequency texture information of the historical optical face image to obtain a historical optical residual texture image; splice the initial sketch generated image and the historical optical residual texture image to obtain a sketch residual texture image; and obtain a historical sketch generated image based on the sketch residual texture image and the initial sketch generated image.

[0143] In the training module in the above-mentioned sketch image generation device, the training module is also used to input the historical real sketch image into the discriminator of the initial sketch generation model to obtain a first score that the historical real sketch image belongs to the real sketch image; determine the first expectation corresponding to the historical real sketch image based on the pixel value and the first score of the historical real sketch image; input the historical sketch generated image into the discriminator to obtain a second score that the historical sketch generated image belongs to the real sketch image; determine the second expectation corresponding to the historical sketch generated image based on the pixel value and the second score of the historical sketch generated image; input the smoothed image obtained after smoothing the historical real sketch image into the discriminator to obtain a third score that the smoothed image belongs to the real sketch image; determine the third expectation corresponding to the smoothed image based on the pixel value and the third score of the smoothed image; determine the adversarial loss based on the first expectation, the second expectation and the third expectation.

[0144] In the training module in the above-mentioned sketch image generation device, the training module is also used to determine the target convolution layer from the feature extraction network in the initial sketch generation model, wherein the target convolution layer includes multiple convolution layers, and the feature map output by each convolution layer has a different size; the second feature corresponding to the historical real sketch image, the third feature corresponding to the initial sketch generated image, and the fourth feature corresponding to the historical sketch generated image are determined through the target convolution layer; the first total perceptual loss is determined based on the second feature, the third feature, and the length and width of the second feature or the third feature; the second total perceptual loss is determined based on the second feature, the fourth feature, and the length and width of the second feature or the fourth feature.

[0145] In the training module in the above-mentioned sketch image generation device, the training module is also used to convert the first feature into a one-dimensional vector and then sort it in a preset order to obtain a first sequence, wherein the sequence number of the first sequence is in ascending order; read the eigenvalues ​​of the first feature after it is converted into a one-dimensional vector in sequence to obtain a first eigenvalue set, and determine the sequence number of the first eigenvalue set according to the correspondence between the first eigenvalue set and the eigenvalue of the first sequence; convert the second feature into a one-dimensional vector and then sort it in a preset order to obtain a second sequence, wherein the sequence number of the second sequence is in ascending order; read the eigenvalues ​​of the second feature after it is converted into a one-dimensional vector in sequence to obtain a second eigenvalue set; splice the sequence number of the first eigenvalue set and the second eigenvalue set to obtain an initial fused feature, wherein the initial fused feature is a one-dimensional vector; sort the initial fused feature in ascending order according to the sequence number, convert the sorted initial fused feature into the length and width of the first feature or the second feature, and obtain a fused feature.

[0146] In the training module in the above-mentioned sketch image generation device, the training module is also used to determine the first distance between the third feature and the fused feature, determine the second distance between the third feature and the first feature; determine the first inter-domain distance optimization loss based on the first distance, the second distance and the distance threshold; determine the third distance between the fourth feature and the fused feature, determine the fourth distance between the fourth feature and the first feature; determine the second inter-domain distance optimization loss based on the third distance, the fourth distance and the distance threshold.

[0147] It should be noted that Figure 8 The sketch image generation device shown is used to perform Figure 2 The method for generating a sketch image shown in the figure, therefore the relevant explanations in the above-mentioned method for generating a sketch image are also applicable to the device for generating a sketch image, and will not be repeated here.

[0148] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned method for generating a sketch image.

[0149] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above sketch image generation method by running the computer program.

[0150] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method for generating a sketch image in each embodiment of the present application.

[0151] An embodiment of the present application also provides a computer program, which, when executed by a processor, implements the steps of the method for generating a sketch image in each embodiment of the present application.

[0152] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0153] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0158] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a sketch image, characterized in that: include: Acquire an optical face image to be processed, wherein the optical face image is an optical domain image; The optical face image is processed by a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

2. The method according to claim 1, characterized in that The sketch generation model is trained in the following way: Acquire a historical optical face image, and acquire a historical real sketch image corresponding to the historical optical face image; Using a first generator of an initial sketch generation model to process the historical optical face image to obtain an initial sketch generation image; Using the second generator of the initial sketch generation model to optimize the details of the initial sketch generation image to obtain a historical sketch generation image; Determining adversarial loss according to the historical real sketch image and the historical sketch generated image; Determining a first total perceptual loss based on the historical real sketch image and the initial sketch generated image, and determining a second total perceptual loss based on the historical real sketch image and the historical sketch generated image; Fusing the first feature of the historical optical face image and the second feature of the historical real sketch image to obtain a fused feature; Determine a first inter-domain distance optimization loss according to a third feature of the image generated by the initial sketch, the first feature, and the fused feature, and determine a second inter-domain distance optimization loss according to a fourth feature of the image generated by the historical sketch, the first feature, and the fused feature, wherein the first inter-domain distance optimization loss and the second inter-domain distance optimization loss are determined by the inter-domain distance optimization loss function; The total loss is determined according to the adversarial loss, the first total perceptual loss, the second total perceptual loss, the first inter-domain distance optimization loss, and the second inter-domain distance optimization loss, and the initial sketch generation model is trained according to the total loss to obtain the sketch generation model.

3. The method according to claim 2, characterized in that Optimizing the details of the initial sketch generated image to obtain a historical sketch generated image, including: Extracting high-frequency texture information of the historical optical face image to obtain a historical optical residual texture image; splicing the initial sketch generated image and the historical optical residual texture image to obtain a sketch residual texture image; The historical sketch generation image is obtained based on the sketch residual texture image and the initial sketch generation image.

4. The method according to claim 2, characterized in that: Determining the adversarial loss according to the historical real sketch image and the historical sketch generated image includes: Inputting the historical real sketch image into the discriminator of the initial sketch generation model to obtain a first score that the historical real sketch image belongs to a real sketch image; Determining a first expectation corresponding to the historically authentic sketch image according to the pixel value of the historically authentic sketch image and the first score; Inputting the historical sketch generated image into the discriminator to obtain a second score of whether the historical sketch generated image belongs to a real sketch image; Determining a second expectation corresponding to the image generated by the historical sketch according to the pixel value of the image generated by the historical sketch and the second score; Inputting a smoothed image obtained after smoothing the historical real sketch image into the discriminator to obtain a third score indicating whether the smoothed image belongs to a real sketch image; Determining a third expectation corresponding to the smoothed image according to the pixel value of the smoothed image and the third score; The countermeasure loss is determined according to the first expectation, the second expectation and the third expectation.

5. The method according to claim 2, characterized in that: Determining a first total perceptual loss based on the historical real sketch image and the initial sketch generated image, and determining a second total perceptual loss based on the historical real sketch image and the historical sketch generated image, including: Determining a target convolutional layer from a feature extraction network in the initial sketch generation model, wherein the target convolutional layer includes a plurality of convolutional layers, and feature maps output by each convolutional layer have different sizes; Determine, through the target convolution layer, a second feature corresponding to the historical real sketch image, a third feature corresponding to the initial sketch generated image, and a fourth feature corresponding to the historical sketch generated image; determining the first total perceived loss according to the second feature, the third feature, and the length and width of the second feature or the third feature; The second total perceptual loss is determined according to the second feature, the fourth feature, and the length and width of the second feature or the fourth feature.

6. The method according to claim 5, characterized in that The first feature of the historical optical face image and the second feature of the historical real sketch image are fused to obtain a fused feature, including: The first features are converted into one-dimensional vectors and then sorted in a preset order to obtain a first sequence, wherein the sequence numbers of the first sequence are in ascending order; Sequentially read the eigenvalues ​​of the first feature after being converted into a one-dimensional vector to obtain a first eigenvalue set, and determine the sequence number of the first eigenvalue set according to the correspondence between the first eigenvalue set and the eigenvalues ​​of the first sequence; The second feature is converted into a one-dimensional vector and then sorted according to the preset order to obtain a second sequence, wherein the sequence number of the second sequence is in ascending order; Sequentially read the eigenvalues ​​of the second feature converted into a one-dimensional vector to obtain a second eigenvalue set; Concatenate the sequence number of the first eigenvalue set and the second eigenvalue set to obtain an initial fusion feature, wherein the initial fusion feature is a one-dimensional vector; The initial fusion features are sorted in ascending order of serial numbers, and the sorted initial fusion features are converted into the length and width of the first feature or the second feature to obtain the fusion feature.

7. The method according to claim 2, characterized in that Determining a first inter-domain distance optimization loss according to a third feature of the image generated by the initial sketch, the first feature, and the fused feature, and determining a second inter-domain distance optimization loss according to a fourth feature of the image generated by the historical sketch, the first feature, and the fused feature, including: Determine a first distance between the third feature and the fused feature, and determine a second distance between the third feature and the first feature; Determine the first inter-domain distance optimization loss according to the first distance, the second distance and a distance threshold; Determine a third distance between the fourth feature and the fused feature, and determine a fourth distance between the fourth feature and the first feature; The second inter-domain distance optimization loss is determined according to the third distance, the fourth distance and the distance threshold.

8. A device for generating a sketch image, characterized in that: include: An acquisition module, used for acquiring an optical face image to be processed, wherein the optical face image is an optical domain image; A generation module is used to process the optical face image using a sketch generation model to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor, connected to the memory, is used to execute program instructions that implement the following functions: obtaining an optical face image to be processed, wherein the optical face image is an optical domain image; using a sketch generation model to process the optical face image to obtain a sketch image corresponding to the optical face image, wherein the sketch image generated by the sketch generation model is a sketch domain image, and the sketch generation model is trained by an inter-domain distance optimization loss function, and the mapping of the image from the optical domain to the sketch domain is learned by the inter-domain distance optimization loss function, so that the similarity between the sketch image and the real sketch image corresponding to the optical face image is greater than a preset threshold.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for generating a sketch image described in any one of claims 1 to 7 is implemented.

Images