An illusion hiding image creation method based on a text-to-image large model

Abstract

The application discloses an optical illusion hidden image creation method based on a text-to-image large model. The method is as follows: 1) an initial feature z0 of a reference image x is extracted by using a pre-trained text-to-image large model; 2) a denoising diffusion implicit model (DDIM) is used to invert z0 based on a constructed inversion trajectory to obtain Gaussian noise; 3) the DDIM is used to sample to obtain a reconstruction feature based on a constructed reconstruction trajectory; 4) a generated trajectory is constructed, and the DDIM is used to sample a noise signal randomly sampled according to a standard Gaussian distribution based on the generated trajectory to obtain a latent space feature, wherein the generated trajectory is equal in length to the reconstruction trajectory, and each time step of the generated trajectory is guided by a target prompt text v; and 5) the text-to-image large model is used to decode to obtain a generated image that conforms to the semantics of v and has the visual structure of x.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence image creation, focusing on digital art and media creation empowered by artificial intelligence, and particularly relates to a visual illusion hidden image creation method and system based on a text-to-image large model. BACKGROUND

[0002] With the continuous progress of multi-modal representation learning and visual Transformer, text-to-image large models have achieved great success. Under its impetus, the fields of AI painting, digital art, intelligent media, etc. are developing at a high speed, and the intelligent content creation community is flourishing unprecedentedly. At present, generative visual large models represented by text-to-image large models have become the focus of industry research, and they have achieved remarkable results in the generation and creation of digital media such as images, videos, and 3D scenes. In addition to direct applications in content generation, a large number of research works have explored the derivative applications of visual large models, such as image and video editing, virtual digital people, virtual fitting, layout design, artistic character design, image and video stylization, etc.

[0003] Although visual large models have greatly improved the efficiency of content creation, their potential in the field of digital art creative design still needs to be tapped. Visual illusion hidden images are a form of digital art that integrates artistic, creative, interesting, and imaginative elements. They cleverly dissolve or hide one image in the scene details of another image, creating a visual illusion effect that different images are observed from different angles. The creation of such images often relies on artificial creative inspiration and aesthetic wisdom, and has a high degree of difficulty and cost. Existing computational methods for creating image visual illusions can be roughly divided into two categories: (1) camouflage image generation based on texture migration; (2) visual illusion generation based on modeling of human visual perception. The first method aims to study the texture migration method of natural scene images, migrating the texture style of the background image to the specific target of the foreground image block, achieving the "camouflage" of the foreground content in the environment. Since it places too much emphasis on visual concealment, the hidden content in the generated camouflage image is less perceptible, making it difficult to apply to digital art creative design. The second method starts from the perspective of human visual perception mechanism, and simulates simple visual illusion effects such as geometric visual illusion, color visual illusion, and motion visual illusion by explicitly modeling the human visual stimulus response process. This method can usually only produce simple images composed of simple geometric elements such as straight lines, curves, and color blocks, and cannot build visual illusion effects based on real and complex images. SUMMARY

[0004] In view of the technical problems existing in the prior art, the purpose of the present application is to provide a visual illusion hidden image creation method based on a text-to-image large model. The present application realizes intelligent creation of creative visual illusion hidden images based on a text-to-image large model, and expands the application of visual large models in the field of digital art and digital creativity. In terms of technical implementation, the present application relies on a text-to-image large model based on a diffusion model paradigm. As the current mainstream generation model paradigm, the denoising diffusion probability model (DDPM) maps an image to a Gaussian noise space through a forward diffusion process with step-by-step noise addition, and learns an inverse denoising process (sampling process) from Gaussian noise to image to generate an image. The denoising diffusion implicit model (DDIM) improves the inverse sampling process of the diffusion model, greatly shortening the sampling time. On this basis, the text-to-image large model realizes open domain image creation guided by text by increasing the denoising network parameter quantity, introducing cross-modal text guidance, and constructing a large amount of paired image-text training data. Further, the latent diffusion model (LDM) greatly reduces the training and inference calculation overhead of the text-to-image large model by migrating the training of the diffusion model from a high-dimensional pixel space to a low-dimensional feature space. The present application takes LDM as the technical basis and proposes a plug-and-play latent diffusion model latent space phase migration mechanism, which realizes the harmonious integration of the visual structural clues of any reference image into any scene described by the target text, and generates high-quality visual illusion hidden art.

[0005] The present application relies on a large-scale text-to-image diffusion model to realize the conversion of any input image into a visual illusion hidden image. This task can be described as inputting an arbitrary reference image x and an arbitrary target text y to the model to obtain the converted visual illusion hidden image So that it observes the detailed scene content described by the text y at a close distance, and observes the structured visual clues of the input reference image x at a long distance, that is, realizes the harmonious integration of the structured information of x and the content semantic information of y.

[0006] For the above tasks, the present application first realizes the generation and creation of an optical illusion hidden image from the perspective of a text-driven image translation method, which promotes the powerful content creation capability of a visual large model to a new application field. The present application is also the first method to use the phase spectrum of the intermediate feature of the diffusion model sampling trajectory to model and control the spatial structure of the large model generated image, which realizes an application breakthrough with the aid of traditional signal processing technology assisting cutting-edge AIGC technology. Based on a pre-trained large-scale text-to-image diffusion model, the present application ingeniously realizes the fusion and penetration of the visual structure clues of the reference image into the scene image generated by the target text without additional model training and model fine-tuning. In addition, the present application proposes an asynchronous phase migration method to control the intensity of the penetration of the reference image structure, thereby flexibly controlling the visual saliency of the hidden content of the generated optical illusion hidden image.

[0007] The technical solutions adopted by the present application are as follows:

[0008] An optical illusion hidden image creation method based on a text-to-image large model, comprising the following steps:

[0009] 1) Construct a pre-trained large-scale text-to-image latent diffusion model LDM, load the pre-trained model parameters, input a reference image x, and extract the feature z after encoding by the VAE encoder of the LDM;

[0010] 2) Construct an inversion trajectory, apply DDIM inversion to the encoded feature z to gradually project it to the Gaussian noise space, obtain the noise feature after inversion, and the inversion trajectory uses an empty text as the guide condition of the LDM;

[0011] 3) Construct a reconstruction trajectory, apply DDIM sampling to the inverted Gaussian noise to gradually obtain the reconstruction feature of the initial encoded feature z The reconstruction trajectory also uses an empty text guide condition to ensure the reconstructability of the sampling result, that is,

[0012] 4) Construct a generation trajectory with the same length as the reconstruction trajectory, which gradually denoises the randomly initialized Gaussian noise feature by using the target prompt text y as the LDM guide condition through a same number of DDIM sampling processes, to obtain the final generated latent space feature Then, the VAE decoder of the LDM is used to decode the latent space feature generated by sampling to obtain the final generated image which is faithful to the content semantics of the target prompt text y;

[0013] 5) embedding the phase migration module proposed in the present application between the reconstructed trajectory and the generated trajectory, which migrates the phase spectrum of the intermediate denoising feature in the reconstructed trajectory to the denoising feature at the corresponding time step in the generated trajectory in a plug-and-play manner without training, thereby embedding the structural visual clues of the input reference image x into the sampling trajectory, realizing the deep fusion of the structural information of the reference image x and the semantic information of the target text y in the diffusion model feature hidden space, and making the decoded generated image both consistent with the content semantics of y and reflecting the visual structure of x, so as to obtain the visual illusion artistic effect of harmoniously hiding and melting the reference image x into the scene described by the arbitrary prompt text y.

[0014] As the technical core of the present application, the specific technical solution of the dynamic phase migration mechanism is described as follows:

[0015] Firstly, for a pair of features at the same time step of the reconstructed trajectory and the generated trajectory (each sampling step is a time step), which are called the reconstructed trajectory feature and the generated trajectory feature, a two-dimensional discrete Fourier transform 2D-FFT is used to transform them from the spatial domain to the frequency domain, and the real part and the imaginary part of them in the frequency domain are obtained, and the amplitude spectrum and the phase spectrum of the reconstructed trajectory feature and the generated trajectory feature are calculated from the real part and the imaginary part, respectively, wherein the amplitude spectrum is the square root of the square sum of the real part and the imaginary part, and the phase spectrum is the inverse tangent of the ratio of the imaginary part to the real part. Further, the phase spectrum of the generated trajectory feature is replaced by the phase spectrum of the reconstructed trajectory feature. Specifically, the phase spectrum of the reconstructed trajectory feature and the amplitude spectrum of the generated trajectory feature are used to synthesize a new frequency domain feature, and the two-dimensional inverse discrete Fourier transform 2D-IFFT is used to transform the new synthesized frequency domain feature back to the spatial domain, and the obtained spatial domain feature is used as the new generated trajectory feature at this time step. In this way, the phase migration process is repeated at each time step of the generated trajectory, realizing the deep fusion of the structural information of the reference image and the semantic information of the target text in the generated trajectory.

[0016] In order to guarantee the high visual quality of the generated image, the text-guided generated trajectory is divided into a migration phase and a non-migration phase. The migration phase is the early part of the denoising process of the generated trajectory, which has a decisive influence on the structural composition of the final generated image, and the non-migration phase is the later part of the generated trajectory, which mainly functions to improve the detail quality of the generated image. Therefore, the present application only applies gradual phase migration in the early migration phase to inject the structural information of the reference image, and removes the phase migration module in the later non-migration phase, so as to fully utilize the original denoising process of the text generation model to ensure the high visual quality of the generated image.

[0017] In view of the fact that the structural information of the intermediate features of the reconstruction trajectory becomes more and more significant as the denoising process proceeds, the present application proposes an attenuation strategy for the phase migration in the latent space, so as to avoid the over-strong structural embedding caused by directly migrating the feature phase spectrum in the later stage of the denoising process, thereby affecting the natural degree of the generated image. Specifically, in the early part of the migration stage, direct phase replacement is adopted, while in the later part, linear fusion of the phase spectrum of the reconstruction trajectory feature and the generated trajectory feature is adopted to replace the phase spectrum of the generated trajectory feature, and the fusion weight of the phase spectrum of the reconstruction trajectory feature is gradually reduced, so as to avoid over-strong structural penetration.

[0018] In order to better fuse the structural information of the reference image and the semantic information of the prompt text in the diffusion model latent space, the present application proposes to append a self-correction module after each phase migration module. The module takes the generated trajectory feature after phase migration as input, and predicts the generated trajectory feature itself at the current time step again under the guidance of the prompt text, which to some extent promotes the fusion of the input reference image structural information and the target prompt text semantic information in the feature space, thereby improving the naturalness and visual quality of the generated visual illusion image.

[0019] In order to realize flexible regulation of the visual saliency of the generated visual illusion image In order to realize flexible regulation of the visual saliency of the generated visual illusion image

[0020] The present application has the following advantages:

[0021] 1. The application first realizes the creation of an optical illusion hidden image from the perspective of a text-driven image translation method, expanding the application of a text-to-image large model in the field of digital art and digital creativity. Compared with existing advanced text-driven image translation methods, the application is the only technical method that realizes the deep fusion of reference image structure information and prompt text content semantics, making the generated image faithful to the description of the prompt text in content scene details, and presenting the hidden content of the reference image at a long distance.

[0022] 2. The application relies on a large-scale text-to-image diffusion model, without model training, model fine-tuning, and any online optimization process, with high inference efficiency, providing a plug-and-play method for the creation of optical illusion hidden art, greatly reducing the cost and expense of artificial design and creation.

[0023] 3. The application has high flexibility, allowing users to create optical illusion hidden images with arbitrary scenes and arbitrary hidden content, and allowing users to flexibly control the visual saliency of the hidden content in the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the overall architecture diagram of the embodiment.

[0025] Figure 2 is a detailed implementation diagram of the phase migration module which is the core of the embodiment.

[0026] Figure 3 is a schematic diagram of the asynchronous phase migration method proposed in the embodiment. DETAILED DESCRIPTION

[0027] To make the technical solutions and implementation modes of the application more obvious and easy to understand, the model architecture and technical details of the application are described in detail below with the aid of the accompanying drawings.

[0028] The embodiment discloses an optical illusion hidden image creation method based on a text-to-image large model, and the overall architecture is as shown in Figure 1 , and the specific steps are described as follows:

[0029] Step 1: Build and load a pre-trained text-to-image large model, and the text-to-image large model of the embodiment uses a latent diffusion model (LDM) of StableDiffusion v1.5 version.

[0030] Step 2: Input the reference image x, and extract the features of x using the encoder E of LDM to obtain the initial features z0=E(x). Based on z0, build the DDIM inversion trajectory of , and project z0 to the Gaussian noise space step by step to obtain the corresponding noise representation , wherein T inv represents the length of the inversion trajectory, and Tinv = 1000. Each time step of the inversion trajectory utilizes the empty text As the guiding condition, its expression is:

[0031]

[0032] where is the predefined noise diffusion coefficient of the DDPM, ∈ θ is the noise estimation network in the LDM model, f θ (z t , t, v φ ) is the approximated z0estimated by the current step's z t .

[0033] Step 3: Based on the noise representation obtained by inversion constructs the T-step reconstruction trajectory of z , where The reconstruction trajectory utilizes the same empty text as in the inversion trajectory as the guiding condition to ensure that the final reconstructed feature is approximately consistent with the initial feature z0of the reference image, i.e. This embodiment sets T = 100. Specifically, the expression of each time step in the reconstruction trajectory is:

[0034]

[0035] Step 4: Construct a T-step generation trajectory of z parallel to the reconstruction trajectory by DDIM sampling, where is a random noise signal sampled from a standard Gaussian distribution, i.e. is the final denoising result of the generation trajectory, i.e., the final generated latent space feature. To make the scene content of the generated image determined by the prompt text, the target prompt text v is used as the guiding condition in the generation trajectory. Further, to enhance the influence of the prompt text on the semantic content of the generated image, the classic classifier-free guidance technique of diffusion models is adopted, and the linear combination of the noise estimation guided by the target prompt text and the noise estimation guided by the empty text is used as the final noise prediction of each time step in the generation trajectory:

[0036]

[0037] where ω is the guidance scale in the classifier-free guidance technique, and this embodiment sets it to ω = 7.5.

[0038] Step 5: Based on the constructed reconstructed trajectory and generated trajectory, a phase transfer module is embedded between them to achieve the gradual fusion of the reference image structural information and the target text semantic information in the latent space of the generated trajectory. The implementation details of the phase transfer module are as follows: Figure 2 As shown. Taking time step t as an example, the phase transfer module (PTM) receives the reconstructed trajectory features at time step t. and generated trajectory features Will Phase spectrum shift to In the middle, the updated generated trajectory features are output. Then in step 4, based on the updated generated trajectory features Using the Denoising Diffusion Implicit Model (DDIM) to analyze randomly sampled noise signals that follow a standard Gaussian distribution Stepwise sampling is performed to obtain latent space features. Specifically, two-dimensional discrete Fourier transforms are applied to both the reconstructed features and the generated features to transform them from the spatial domain to the frequency domain:

[0039]

[0040] in and These are the reconstructed trajectory features The real and imaginary parts, and These are the generated trajectory features The real and imaginary parts of the feature are given by FFT, which is a two-dimensional Fast Fourier Transform. Based on the results of the Discrete Fourier Transform, the amplitude and phase spectra of the feature can be further calculated:

[0041]

[0042] in and These are the amplitude spectrum and phase spectrum of the reconstructed trajectory features, respectively. and These are the amplitude spectrum and phase spectrum of the generated trajectory features, respectively. The phase spectra of the reconstructed features and the generated features are fused using the fusion coefficient b at time step t. t The phase spectrum and the amplitude spectrum of the generated features are combined to obtain the phase-transferred frequency domain features. Then, a two-dimensional inverse Fourier transform is used to transform them back to the spatial domain to obtain the updated generated trajectory features. The above process can be formalized as follows:

[0043]

[0044] IFFT stands for Inverse Fast Fourier Transform (IFFT). Finally, a self-correction module is added to the end of the phase transfer module, which is based on the generated features after phase transfer. Re-predicting with target prompt text as a guiding condition It itself promotes the fusion of the embedded reference image structure information and the semantic information of the target text. The formula expression of the self-correction module is as follows:

[0045]

[0046] The setting of the fusion coefficient {b t} of phase migration is described below. To ensure the text fidelity and visual quality of the generated image, the generated trajectory is divided into an early migration stage and a later non-migration stage by using a time step λT, and the phase migration is only applied to the early migration stage. In addition, as the denoising process proceeds, the structural information of the reconstructed trajectory features is continuously enhanced, and to avoid the reduction of the naturalness of the generated image caused by excessive structural penetration, the migration stage of the generated trajectory is further divided into a direct migration and a decay migration by using a time step τT. In the direct migration sub-stage, direct phase replacement is adopted, that is, b t = 1; in the decay migration sub-stage, the phase fusion coefficient b t is gradually decayed to zero, continuously weakening the strength of phase migration. The above process can be expressed by the following formula:

[0047]

[0048] Wherein, λ = 0.4 and τ = 0.6 are set in the embodiment.

[0049] To achieve flexible control of the saliency of the hidden content in the generated visual illusion image, the proposed phase migration module (PTM) is further extended from migration at the same time step to migration at different time steps to obtain an asynchronous phase migration module (APTM), and the method schematic diagram is shown in Figure 3 Based on the PTM, the APTM first predicts the corresponding feature of the reconstructed trajectory feature after d-step denoising according to each time step, and then migrates the phase spectrum of to the generated trajectory feature of the current time step. When d is positive, the saliency of the hidden content in the generated visual illusion image (i.e. the generated image ) is enhanced as d increases; when d is negative, the saliency of the hidden content is weakened as d decreases. The process can be expressed by the following formula:

[0050]

[0051] The embodiment allows the saliency of the hidden visual content to be flexibly regulated by adjusting the time step interval d, and d ∈ [-10, 10] is a recommended value range.

[0052] Model inference:

[0053] The present application is based on a text-to-image large model, and proposes a plug-and-play phase migration technology, which realizes the creation of anamorphic hidden images without any model training and model fine-tuning. The above steps 1 to 5 are both model construction processes and model inference processes.

[0054] The above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and the protection scope of the present application should be subject to the description of the claims.

Claims

1. An optical illusion hidden image creation method based on a text-to-image large model, comprising the steps of: 1) extracting initial features z0 of a reference image x using a pre-trained text-to-image large model; 2) based on the constructed inversion trajectory, using a denoising diffusion implicit model DDIM to perform inversion on the initial feature z0 to obtain an inverted Gaussian noise Each time step of the inversion trajectory uses empty text as a guide condition; 3) using a denoising diffusion implicit model DDIM on the inverted Gaussian noise based on the constructed reconstruction trajectory sampling to obtain a reconstruction feature corresponding to the initial feature z0 each time step of the reconstruction trajectory uses empty text as a guide condition; 4) constructing a generation trajectory, sampling a noise signal subject to a standard Gaussian distribution based on the generation trajectory using a denoising diffusion implicit model (DDIM) to obtain a latent space feature wherein the generation trajectory is equal in length to the reconstruction trajectory, and each time step of the generation trajectory is conditioned on a target prompt text v; a phase shift module is embedded between the reconstruction trajectory and the generation trajectory, configured to gradually fuse the structural information of the reference image x and the semantic information of the target prompt text v in the generation trajectory latent space. 5) using the text-to-image large model to process the latent space feature to obtain a generated image that conforms to the target prompt text v semantics and has the reference image x visual structure 2. The method of claim 1, wherein, In step 4), at the t-th time step, the phase shift module receives the reconstructed trajectory feature at the t-th time step and generates a trajectory feature The phase spectrum of the reconstructed trajectory feature is shifted to the generated trajectory feature , and the generated trajectory feature is updated Then, based on the updated generated trajectory feature , a noise signal sampled randomly from a standard Gaussian distribution is sampled step by step using a denoising diffusion implicit model DDIM to obtain a latent space feature 3. The method of claim 2, wherein, updating the generated trajectory feature The method is: respectively performing two-dimensional discrete Fourier transform on the reconstructed trajectory feature and the generated trajectory feature ​ wherein, and are respectively the real part and the imaginary part of the reconstructed trajectory feature , and are respectively the real part and the imaginary part of the generated trajectory feature , FFT is a two-dimensional fast Fourier transform; then the amplitude spectrum and the phase spectrum of the feature are calculated: wherein, and are respectively the amplitude spectrum and the phase spectrum of the reconstructed trajectory feature , and are respectively the amplitude spectrum and the phase spectrum of the generated trajectory feature ; then the phase spectrum of the reconstructed trajectory feature and the phase spectrum of the generated trajectory feature are fused by using the fusion coefficient b t of the t-th time step, to obtain the fused phase spectrum The phase spectrum and the amplitude spectrum of the generated trajectory feature are combined to obtain the phase-shifted frequency domain feature, and the phase-shifted frequency domain feature is transformed back to the spatial domain through a two-dimensional inverse Fourier transform, to obtain the updated generated trajectory feature 4. The method of claim 3, wherein, The generated trajectory is divided into a migration phase and a non-migration phase, and the migration phase is divided into a direct migration sub-phase and a decay migration sub-phase; that is Wherein, T is the total number of time steps of the generated trajectory, and lambda, tau are proportional coefficients, lambda+tau=1.

5. The method according to claim 2 or 3 or 4, characterized in that, An auto-correction module is added at the tail of the phase migration module; the auto-correction module is based on the updated generated trajectory feature The generated trajectory feature is re-predicted based on the target prompt text v as a guide condition The image structure information of the reference image x is promoted to be fused with the semantic information of the target prompt text v.

6. The method of claim 1, wherein, An asynchronous phase migration module is embedded between the reconstructed trajectory and the generated trajectory, for gradually fusing the structural information of the reference image x and the semantic information of the target prompt text v in the generated trajectory latent space; at the t-th time step, the asynchronous phase migration module first migrates the phase spectrum of the reconstructed trajectory feature at the t-th time step to the generated trajectory feature at the t-th time step according to the reconstructed trajectory feature at the t-th time step to obtain the corresponding feature after d time steps of denoising Then the phase spectrum of is migrated to the generated trajectory feature at the current t-th time step When d is positive, the hidden content saliency of the generated image is enhanced with the increase of d; when d is negative, the hidden content saliency of the generated image is weakened with the decrease of d.

7. The method of claim 1, wherein, In step 4), a linear combination of the noise estimates guided with the target prompt text v and the empty text the linear combination of the noise estimates guided with the target prompt text v and the empty text as the final noise prediction for each time step in the generated trajectory.

8. The method of claim 1, wherein, The reconstructed trajectory is based on the inverted Gaussian noise constructed T-step reconstructed trajectory; the inversion trajectory is constructed based on the initial feature z0 DDIM inversion trajectory.

9. The method of claim 1, wherein, the text-to-image large model is a latent diffusion model (LDM).

Images