Sketch image translation method and system

Abstract

The invention provides a sketch image translation method and system, and the method comprises the steps: constructing a multi-information encoder based on a DINOV2 model and one-hot coding, carrying out the feature extraction of a sketch image through the multi-information encoder, and obtaining a sketch multi-information code; pairing the sketch multi-information code and the sketch real image to construct a corresponding data set, and training a conditional diffusion model through the data set; after a sketch image input by a user is received, sketch multi-information codes are output through the multi-information encoder to serve as guiding information, and a real image corresponding to the input sketch image is generated through the trained conditional diffusion model. Through the scheme, the generalization ability of image translation can be improved, and the quality of the generated image is guaranteed.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to a sketch image translation method and system. Background Art

[0002] Sketch image translation is to map the input image of the sketch field into a color image of the real field. In recent years, sketch image translation has become a research topic that has attracted much attention in the fields of computer vision, artificial intelligence, and image processing. It has broad application prospects in many fields. For example, in the field of industrial design, designers can quickly generate rendering images of products based on sketches. In the fields of auxiliary design and virtual reality, it can help the system restore the real scene more accurately.

[0003] The work of image translation can be traced back to the method proposed by Resales et al., but with the passage of time, the rapid development of deep learning technology has brought revolutionary changes to image translation. At present, the more mainstream image translation methods are mainly implemented by Generative Adversarial Networks (GANs), which can be divided into: matching image translation, one-to-one mapping between input image and output image, such as Pix2Pix, BicycleGAN, etc.; non-matching image translation, there is no corresponding relationship between input image and output image, and image translation tasks are performed without real image correspondence, such as CycleGAN, UGATIT, CUT, etc.; and multimodal unsupervised image translation, which generates multiple translations from a single image in one domain to another domain, such as MUNIT, UNIT, FUNIT, etc.

[0004] GANs is a high-quality image generation model recognized by most people, but the adversarial goal of GANs makes model training difficult and may encounter the problem of mode collapse. Obviously, the current image translation methods based on generative adversarial networks have poor generalization and insufficient generation quality. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a sketch image translation method and system for solving the problems of poor generalization and insufficient generation quality of current image translation methods.

[0006] In a first aspect of an embodiment of the present invention, a sketch image translation method is provided, comprising:

[0007] A multi-information encoder is constructed based on the DINOV2 model and one-hot encoding, and features of the sketch image are extracted by the multi-information encoder to obtain a sketch multi-information encoding;

[0008] Pair the sketch multi-information encoding with the sketch real image to build a corresponding dataset, and use the dataset to train the conditional diffusion model;

[0009] After receiving the sketch image input by the user, the sketch multi-information encoding is output by the multi-information encoder as the guiding information, and the real image corresponding to the input sketch image is generated through the trained conditional diffusion model.

[0010] In a second aspect of an embodiment of the present invention, a system for sketch image translation is provided, comprising:

[0011] A sketch encoding module is used to construct a multi-information encoder based on the DINOV2 model and one-hot encoding, and extract features of the sketch image through the multi-information encoder to obtain a sketch multi-information encoding;

[0012] The model training module is used to pair the sketch multi-information encoding with the sketch real image to construct a corresponding data set, and train the conditional diffusion model through the data set;

[0013] The image generation module is used to receive the sketch image input by the user, output the sketch multi-information encoding as the guiding information through the multi-information encoder, and generate a real image corresponding to the input sketch image through the trained conditional diffusion model.

[0014] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.

[0015] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.

[0016] In an embodiment of the present invention, a sketch multi-information encoding is extracted through a multi-information encoder as guiding information to guide the training of a conditional diffusion model and image generation, thereby realizing sketch image translation based on the conditional diffusion model, which not only improves the generalization ability of the model but also ensures the quality of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic flow chart of a sketch image translation method provided by one embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the architecture of a sketch image translation method provided by one embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the structure of a sketch image translation system provided by one embodiment of the present invention;

[0021] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that the term "including" and other similar expressions in the specification or claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, such as a process, method, system, or device including a series of steps or units is not limited to the listed steps or units. In addition, "first" and "second" are used to distinguish different objects, not to describe a specific order.

[0024] It can be understood that the conditional diffusion model requires appropriate conditions to complete image generation under the premise of limited data. This condition cannot be learned with a small number of samples relying solely on a single type of information. The images generated by the model will have problems such as lack of image content and inability to denoise. Therefore, an encoder-guided conditional diffusion model based on a limited data set is needed to achieve image generation.

[0025] See also Figure 1 , a schematic diagram of a process of a sketch image translation method provided by an embodiment of the present invention includes:

[0026] S101, constructing a multi-information encoder based on the DINOV2 model and one-hot encoding, and extracting features of the sketch image through the multi-information encoder to obtain a sketch multi-information encoding;

[0027] The DINOV2 model is a computer vision self-supervised model that can be used for image recognition and visual feature extraction. The one-hot encoding is also a single-bit effective encoding, which can convert categorical variables into binary vector representations as a feature representation method. By combining the DINOV2 model and one-hot encoding as an encoder, feature extraction and encoding can be performed on sketch images.

[0028] Specifically, the visual information of the sketch image is encoded through the pre-trained DINOV2 model; the category information of the sketch image is encoded through one-hot encoding; and the visual information encoding and the category information encoding of the sketch image are added together to obtain the sketch multi-information encoding.

[0029] The visual information and category information encoding of the sketch image are both 768 dimensions.

[0030] S102, pairing the sketch multi-information encoding with the sketch real image to construct a corresponding data set, and training the conditional diffusion model through the data set;

[0031] Based on a dataset constructed by pairing sketches with real images, the conditional diffusion model is trained through the multi-information encoding output by the multi-information encoder. The trained conditional diffusion model can generate similar images based on the input image and the corresponding multi-information encoding.

[0032] Among them, the backbone of the conditional diffusion model is the Transformer architecture, and the conditional diffusion model realizes image generation through a back diffusion process according to a conditional vector, and the conditional vector is a sketch multi-information encoding.

[0033] Specifically, the sketch multi-information encoding and time splicing are converted into conditional embedding; the conditional embedding is input into the conditional diffusion model, in which the noise of each diffusion step is predicted and the corresponding output image is calculated based on the original noise image, the number of diffusion steps and the conditional embedding, and the category information in the multi-information encoding is randomly blanked during the training process.

[0034] Before entering the Transformer, the sketch multi-information encoding f is obtained from the multi-information encoder as a conditional vector and concatenated with time t to obtain a conditional embedding and sent to the Transformer, so that the diffusion model can measure the noise ∈ t .

[0035] In the process of training the conditional diffusion model, it is necessary to provide the conditional diffusion model with an image x with added noise. t , time step t and multiple information encoding f, the objective function of the conditional diffusion model is:

[0036]

[0037] In the formula, represents the training objective of the conditional diffusion model, ∈ t is random noise.

[0038] In order to enhance generalization and prevent overfitting during training, the conditional diffusion model has a certain probability (p) to empty the sketch category information. At this time, the objective function is expressed as:

[0039]

[0040] After the conditional diffusion model is trained, the image generation is realized through the back diffusion process according to the condition vector. In order to control the use of conditions and guide the back diffusion process to generate images with the required features, the noise of the model in the back diffusion process can be described as:

[0041]

[0042] Where w represents the conditional scale used by the control diffusion model in image generation.

[0043] Conditional diffusion model∈ θ Prediction is added to image x t The noise in t , from the image x t Subtract the noise ∈ t , we can get the image x of the previous step t-1 , and we get the following expression:

[0044]

[0045] In the last step, according to x 1 and noise∈ 1 , the output image x can be calculated 0 , and the noise z in the last step is 0.

[0046] S103. After receiving the sketch image input by the user, the sketch multi-information encoding is output as guiding information through the multi-information encoder, and a real image corresponding to the input sketch image is generated through the trained conditional diffusion model.

[0047] Based on the multi-information encoder output sketch multi-information encoding as a guide, the conditional diffusion model can be guided to generate a real image corresponding to the input image.

[0048] The input sketch image will be converted into a multi-information encoding f by the multi-information encoder i ={f 1 ,f 2 ,f 3 ,…,f n}, input the diffusion step number t into the conditional diffusion model ∈ θ In the prediction, the noise ∈t , repeat until the output image x is calculated 0 ,x 0 It can be used as the real image corresponding to the sketch.

[0049] like Figure 2 As shown in the figure, for the sketch image, the visual feature v is extracted by the DINOV2 model in the multi-information encoder, and the category information c is encoded by one-hot encoding. The two are combined to obtain the multi-information encoding f. t The conditional diffusion model performs reverse reasoning based on the multi-information encoding and the number of diffusion steps to predict the noise and obtain the real image x corresponding to the sketch by denoising. 0 .

[0050] Preferably, the user controls the content of the real image generated by the conditional diffusion model by editing the sketch, and controls the style of the real image generated by the conditional diffusion model by modifying the random seed. Editing the sketch can change the content of the generated image, and modifying the random seed can change the style of the generated image.

[0051] In this embodiment, by using the multi-information encoding of the sketch output by the multi-information encoder as guiding information to guide the training and image generation of the conditional diffusion model, not only the generalization ability of the model can be improved, but also the quality of image growth can be guaranteed. At the same time, the model has strong reliability and high accuracy, which is convenient for practical application.

[0052] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0053] Figure 3 A schematic diagram of a system for translating sketch images provided by an embodiment of the present invention, the system comprising:

[0054] A sketch encoding module 310 is used to construct a multi-information encoder based on the DINOV2 model and one-hot encoding, and extract features of the sketch image through the multi-information encoder to obtain a sketch multi-information encoding;

[0055] The step of extracting features from the sketch image by using the multi-information encoder to obtain the sketch multi-information encoding comprises:

[0056] The visual information of the sketch image is encoded through the pre-trained DINOV2 model; the category information of the sketch image is encoded through one-hot encoding; the visual information encoding and the category information encoding of the sketch image are added together to obtain the sketch multi-information encoding.

[0057] A model training module 320 is used to pair the sketch multi-information encoding with the sketch real image to construct a corresponding data set, and train the conditional diffusion model through the data set;

[0058] Among them, the backbone of the conditional diffusion model is the Transformer architecture, and the conditional diffusion model realizes image generation through a back diffusion process according to a conditional vector, and the conditional vector is a sketch multi-information encoding.

[0059] The step of training the conditional diffusion model through the data set includes:

[0060] Transform sketch multi-information encoding and temporal concatenation into conditional embedding;

[0061] The conditional embedding is input into the conditional diffusion model. In the conditional diffusion model, the noise of each diffusion step is predicted according to the original noise image, the number of diffusion steps and the conditional embedding, and the corresponding output image is calculated. The category information in the multi-information coding is randomly emptied during the training process.

[0062] The image generation module 330 is used to, after receiving the sketch image input by the user, output the sketch multi-information encoding as guidance information through the multi-information encoder, and generate a real image corresponding to the input sketch image through the trained conditional diffusion model.

[0063] Preferably, the image generation module 330 includes:

[0064] The editing and modification unit is used for users to control the content of real images generated by the conditional diffusion model by editing sketches, and to control the style of real images generated by the conditional diffusion model by modifying random seeds.

[0065] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0066] Figure 4 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is used for sketch image translation. Figure 4 As shown, the electronic device 4 of this embodiment includes: a memory 410, a processor 420 and a system bus 430, wherein the memory 410 includes an executable program 4101 stored thereon, and those skilled in the art can understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0067] Combine the following Figure 4 A detailed introduction to the various components of electronic equipment:

[0068] The memory 410 can be used to store software programs and modules, and the processor 420 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 410. The memory 410 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cache data), etc. In addition, the memory 410 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0069] The memory 410 includes an executable program 4101 of a network request method, and the executable program 4101 can be divided into one or more modules / units, which are stored in the memory 410 and executed by the processor 420 to achieve image generation, etc. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 4101 in the electronic device 4. For example, the computer program 4101 can be divided into functional modules such as a sketch encoding module, a model training module, and an image generation module.

[0070] The processor 420 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 410, and calling data stored in the memory 410, so as to monitor the overall status of the electronic device. Optionally, the processor 420 may include one or more processing units; preferably, the processor 420 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 420.

[0071] The system bus 430 is used to connect the various functional components inside the computer, and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, CAN bus, etc. The instructions of the processor 420 are transmitted to the memory 410 through the bus, and the memory 410 feeds back data to the processor 420. The system bus 430 is responsible for the data and instruction exchange between the processor 420 and the memory 410. Of course, the system bus 430 can also be connected to other devices, such as network interfaces, display devices, etc.

[0072] In the embodiment of the present invention, the executable program executed by the processor 420 included in the electronic device includes:

[0073] A multi-information encoder is constructed based on the DINOV2 model and one-hot encoding, and features of the sketch image are extracted by the multi-information encoder to obtain a sketch multi-information encoding;

[0074] Pair the sketch multi-information encoding with the sketch real image to build a corresponding dataset, and use the dataset to train the conditional diffusion model;

[0075] After receiving the sketch image input by the user, the sketch multi-information encoding is output by the multi-information encoder as the guiding information, and the real image corresponding to the input sketch image is generated through the trained conditional diffusion model.

[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0077] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0078] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sketch image translation method, characterized in that: include: A multi-information encoder is constructed based on the DINOV2 model and one-hot encoding, and features of the sketch image are extracted by the multi-information encoder to obtain a sketch multi-information encoding; Pair the sketch multi-information encoding with the sketch real image to build a corresponding dataset, and use the dataset to train the conditional diffusion model; After receiving the sketch image input by the user, the sketch multi-information encoding is output by the multi-information encoder as the guiding information, and the real image corresponding to the input sketch image is generated through the trained conditional diffusion model.

2. The method according to claim 1, characterized in that The extracting features of the sketch image by the multi-information encoder to obtain the sketch multi-information encoding comprises: Encode the visual information of the sketch image through the pre-trained DINOV2 model; Encode the category information of the sketch image through one-hot encoding; The visual information encoding and category information encoding of the sketch image are added together to obtain the sketch multi-information encoding.

3. The method according to claim 1, characterized in that The conditional diffusion model is trained by the data set including: Transform sketch multi-information encoding and temporal concatenation into conditional embedding; The conditional embedding is input into the conditional diffusion model. In the conditional diffusion model, the noise of each diffusion step is predicted according to the original noise image, the number of diffusion steps and the conditional embedding, and the corresponding output image is calculated. The category information in the multi-information coding is randomly emptied during the training process.

4. The method according to claim 1, characterized in that: The backbone of the conditional diffusion model is the Transformer architecture. The conditional diffusion model realizes image generation through a back-diffusion process according to a conditional vector, and the conditional vector is a sketch multi-information encoding.

5. The method according to claim 1, characterized in that After receiving the sketch image input by the user, outputting the sketch multi-information encoding as the guide information through the multi-information encoder, and generating the real image corresponding to the input sketch image through the trained conditional diffusion model also includes: Users control the content of real images generated by the conditional diffusion model by editing sketches, and control the style of real images generated by the conditional diffusion model by modifying random seeds.

6. A system for translating sketch images, characterized in that include: A sketch encoding module is used to construct a multi-information encoder based on the DINOV2 model and one-hot encoding, and extract features of the sketch image through the multi-information encoder to obtain a sketch multi-information encoding; The model training module is used to pair the sketch multi-information encoding with the sketch real image to construct a corresponding data set, and train the conditional diffusion model through the data set; The image generation module is used to receive the sketch image input by the user, output the sketch multi-information encoding as the guiding information through the multi-information encoder, and generate a real image corresponding to the input sketch image through the trained conditional diffusion model.

7. The system according to claim 6, characterized in that The extracting features of the sketch image by the multi-information encoder to obtain the sketch multi-information encoding comprises: Encode the visual information of the sketch image through the pre-trained DINOV2 model; Encode the category information of the sketch image through one-hot encoding; The visual information encoding and category information encoding of the sketch image are added together to obtain the sketch multi-information encoding.

8. The system according to claim 6, characterized in that The conditional diffusion model is trained by the data set including: Transform sketch multi-information encoding and temporal concatenation into conditional embedding; The conditional embedding is input into the conditional diffusion model. In the conditional diffusion model, the noise of each diffusion step is predicted according to the original noise image, the number of diffusion steps and the conditional embedding, and the corresponding output image is calculated. The category information in the multi-information coding is randomly emptied during the training process.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the sketch image translation method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of a sketch image translation method as claimed in any one of claims 1 to 5 are implemented.

Images