Image-text correlation multi-distribution sampling method and device based on diffusion model

Abstract

The application provides a kind of method and device for sampling multiple distributions based on diffusion model of image-text correlation, comprising: obtaining data to be processed, the data to be processed is one of image data, text data and image-text data;According to the pre-set sampling type, determine the noise type and / or noise parameter value to be input;The data to be processed, the noise type and / or noise parameter value to be input are input into the pre-trained image-text correlation multiple distribution sampling model to obtain the sampling result;Wherein, the image-text correlation multiple distribution sampling model is obtained by training the noise added in sample data pair based on the pre-constructed neural network, and the sample data pair is a data pair formed by image data sample and text data sample.The application simultaneously considers multiple image-text distributions, adds noise according to the pre-set sampling type, uses the image-text correlation multiple distribution sampling model, realizes multifunction, high universality, and is suitable for multiple distribution sampling of multi-modal data.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a multi-distribution sampling method and apparatus for image-text correlation based on a diffusion model. Background Technology

[0002] A diffusion model is a deep generative model used to generate data similar to the data used to train it. The diffusion model works by continuously adding Gaussian noise to corrupt the training data, and then learning to recover the data by reversing this noise process. After training, we can use the diffusion model to generate data by simply passing in randomly sampled noise through the learned denoising process. In other words, the model is determined by the denoising model on noisy data.

[0003] The image-text related multi-distribution modeling problem refers to the problem of modeling a family of potential distributions of image and text data. This family of distributions includes the marginal distribution of image data, the marginal distribution of text data, the joint distribution of image and text data, the conditional distribution of text to image, and the conditional distribution of image to text.

[0004] For the problem of modeling multi-distribution correlations between text and image data, existing diffusion models are designed to model a specific distribution, such as the conditional distribution from text to image. During training, the diffusion model adds noise to the images in the text-image data, and then uses the noisy image, the original text, and the magnitude of the image noise as input to predict the noise in the image. This training method only considers a single distribution, meaning that existing diffusion models cannot support multi-distribution modeling of arbitrary multimodal (text-image) data. For example, a model cannot simultaneously support modeling the edge distributions of image data, text data, the joint distribution of text-image data, the conditional distribution from text to image, or the conditional distribution from image to text. This results in existing diffusion models having only a single function when used for sampling.

[0005] In summary, existing sampling methods suffer from limited functionality and low universality. Summary of the Invention

[0006] This invention provides a multi-distribution sampling method and apparatus for image-text correlation based on a diffusion model, which solves the defects of existing technologies such as single function and low universality, and achieves the effect of multi-distribution sampling with multiple functions and high universality, applicable to any multi-modal data.

[0007] This invention provides a multi-distribution sampling method for image-text correlation based on a diffusion model, comprising:

[0008] Acquire data to be processed, wherein the data to be processed is one of image data, text data, and graphic-text data;

[0009] Based on the preset sampling type, determine the noise type and / or noise parameter value to be input;

[0010] The data to be processed, the noise type to be input, and / or the noise parameter value are input into a pre-trained image-text related multi-distribution sampling model to obtain the sampling results;

[0011] The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples.

[0012] According to the present invention, a multi-distribution sampling method for image-text relevance based on a diffusion model is provided. This method trains a pre-built neural network using sample data pairs and noise added to the sample data pairs to obtain an image-text relevance multi-distribution sampling model. The method specifically includes the following steps:

[0013] S1: Obtain sample data pairs, which include image data samples and text data samples;

[0014] S2: Obtain target image noise and target text noise based on standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values;

[0015] S3: Based on the image noise parameter value, calculate the linear combination of the image data sample and the image noise parameter value to obtain a noisy image sample; based on the text noise parameter value, calculate the linear combination of the text data sample and the text noise parameter value to obtain a noisy text sample.

[0016] S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result;

[0017] S5: Calculate the squared L2 norm of the difference between the image noise prediction result and the target image noise, and the squared L2 norm of the difference between the text noise prediction result and the target text noise, and train the parameters of the image-text related multi-distribution sampling model with the goal of minimizing the sum of the two squared L2 norms.

[0018] S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model.

[0019] According to the diffusion model-based image-text correlation multi-distribution sampling method provided by the present invention, the noise type and / or noise parameter value to be input are determined according to a pre-set sampling type, specifically including:

[0020] If the preset sampling type is image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as standard Gaussian noise. The parameter value of the text noise to be input is determined to be the maximum value within the preset range.

[0021] According to the diffusion model-based image-text correlation multi-distribution sampling method provided by the present invention, the noise type and / or noise parameter value to be input are determined according to a pre-set sampling type, specifically including:

[0022] If the preset sampling type is text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as standard Gaussian noise. The image noise parameter value to be input is determined to be the maximum value within the preset range.

[0023] According to the diffusion model-based image-text correlation multi-distribution sampling method provided by the present invention, the noise type and / or noise parameter value to be input are determined according to a pre-set sampling type, specifically including:

[0024] If the preset sampling type is image-text joint sampling, the image noise parameter value to be input and the text noise parameter value to be input are determined to be the same preset value.

[0025] According to the diffusion model-based image-text correlation multi-distribution sampling method provided by the present invention, the noise type and / or noise parameter value to be input are determined according to a pre-set sampling type, specifically including:

[0026] If the preset sampling type is image-to-text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as a preset image, and the image noise parameter value to be input is determined to be 0.

[0027] According to the diffusion model-based image-text correlation multi-distribution sampling method provided by the present invention, the noise type and / or noise parameter value to be input are determined according to a pre-set sampling type, specifically including:

[0028] If the preset sampling type is text-to-image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as preset text. The image noise parameter value to be input is determined to be 0.

[0029] The present invention also provides a multi-distribution sampling device for image-text correlation based on a diffusion model, comprising:

[0030] A data acquisition unit is used to acquire data to be processed, wherein the data to be processed is one of image data, text data, and graphic data;

[0031] The noise determination unit is used to determine the noise type and / or noise parameter value to be input based on a preset sampling type.

[0032] The sampling unit is used to input the data to be processed, the noise type to be input, and / or the noise parameter value into a pre-trained image-text related multi-distribution sampling model to obtain the sampling result;

[0033] The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image-text correlation multi-distribution sampling method based on the diffusion model as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image-text correlation multi-distribution sampling method based on the diffusion model as described above.

[0036] This invention provides a multi-distribution sampling method and apparatus for image-text correlation based on a diffusion model. The method involves acquiring data to be processed, which can be image data, text data, or image-text data; determining the noise type and / or noise parameter value to be input according to a pre-defined sampling type; and inputting the data to be processed, the noise type, and / or noise parameter value into a pre-trained image-text correlation multi-distribution sampling model to obtain sampling results. The image-text correlation multi-distribution sampling model is trained based on a pre-built neural network using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples. This invention simultaneously considers multiple image-text distributions, adds noise according to a pre-defined sampling type, and inputs the noise type and noise parameter value into the pre-trained image-text correlation multi-distribution sampling model to obtain sampling results. This achieves a multi-functional, highly universal, and applicable multi-distribution sampling effect suitable for any multimodal data. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1This is one of the flowcharts of the multi-distribution sampling method for image-text correlation based on the diffusion model provided by the present invention;

[0039] Figure 2 This is the second flowchart of the multi-distribution sampling method for image-text correlation based on the diffusion model provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of the image-text correlation multi-distribution sampling device based on the diffusion model provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0042] Figure label:

[0043] 310: Data acquisition unit; 320: Noise determination unit; 330: Sampling unit;

[0044] 410: Processor; 420: Communication interface; 430: Memory; 440: Communication bus. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] The following is combined Figures 1-2 This invention describes a diffusion-based multi-distribution sampling method for image-text correlation. Figure 1 This is one of the flowcharts illustrating the multi-distribution sampling method for image-text correlation based on a diffusion model provided by this invention, such as... Figure 1 As shown, this invention provides a multi-distribution sampling method for image-text correlation based on a diffusion model, comprising:

[0047] Step 110: Obtain the data to be processed, which is one of image data, text data, and graphic data.

[0048] The data to be processed may be image data, text data, or a combination of image and text data.

[0049] Step 120: Determine the noise type and / or noise parameter value to be input based on the preset sampling type.

[0050] The preset sampling types include: image sampling, text sampling, combined image and text sampling, image-to-text sampling, and text-to-image sampling.

[0051] The noise types to be input include two modalities: image and text. The noise parameter values ​​are set according to the sampling type.

[0052] Step 130: Input the data to be processed, the noise type to be input, and / or the noise parameter value into the pre-trained image-text related multi-distribution sampling model to obtain the sampling results;

[0053] The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples.

[0054] The data to be processed, the noise type to be input, and the noise parameter values ​​are input into a pre-trained image-text related multi-distribution sampling model to obtain the sampling results.

[0055] This invention aims to propose a modeling method for image-text related multi-distribution. During training, noise is simultaneously added to both the image and the text, with the noise levels for each being independent. Then, based on the noisy image, the noisy text, and the individual noise levels of both the image and text as input, the noise in both the image and text is predicted simultaneously. During inference, by specifying the parameter value of the input noise, sampling of images, text, image-text combinations, image-to-text, and text-to-image can be performed.

[0056] This invention improves the universality of the diffusion model, transforming it from a single-function model (e.g., text-to-image sampling) into a multi-function model (simultaneously possessing sampling functions for images, text, combined text and images, image-to-text, and text-to-image).

[0057] Based on the above embodiments, in this method, a multi-distribution sampling model for image-text relevance is obtained by training a pre-built neural network using sample data pairs and noise added to the sample data pairs. Specifically, the method includes the following steps:

[0058] S1: Obtain sample data pairs, which include image data samples and text data samples;

[0059] S2: Obtain target image noise and target text noise based on standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values;

[0060] S3: Based on the image noise parameter value, calculate the linear combination of the image data sample and the image noise parameter value to obtain a noisy image sample; based on the text noise parameter value, calculate the linear combination of the text data sample and the text noise parameter value to obtain a noisy text sample.

[0061] S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result;

[0062] S5: Calculate the squared L2 norm of the difference between the image noise prediction result and the target image noise, and the squared L2 norm of the difference between the text noise prediction result and the target text noise, and train the parameters of the image-text related multi-distribution sampling model with the goal of minimizing the sum of the two squared L2 norms.

[0063] S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model.

[0064] Specifically, such as Figure 2 For a given image and text dataset, the model training steps are as follows:

[0065] S1: Obtain sample data pairs, which include image data samples and text data samples. In some embodiments, the sample data pairs are derived from a given image-text dataset, by sampling image-text data pairs from the image-text dataset to obtain the sample data pairs.

[0066] S2: Obtain target image noise and target text noise based on a standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values. In the specific operation, noise is sampled from the standard Gaussian distribution, and an image noise parameter value is sampled. Similarly, noise is sampled from the standard Gaussian distribution, and a text noise parameter value is sampled.

[0067] S3: Based on the image noise parameter values, calculate a linear combination of the image data sample and the image noise parameter values ​​to obtain a noisy image sample. Similarly, based on the text noise parameter values, calculate a linear combination of the text data sample and the text noise parameter values ​​to obtain a noisy text sample. In other words, based on the image noise parameter values, calculate a linear combination of the image data sample and the image noise parameter values ​​to obtain an image with added Gaussian noise (i.e., a noisy image), called a noisy image sample. Likewise, calculate a linear combination of the text data sample and the text noise parameter values ​​to obtain text with added Gaussian noise (i.e., noisy text), called a noisy text sample. The noisy image sample and the image noise parameter values ​​together constitute the target text noise; the noisy text sample and the text noise parameter values ​​together constitute the target text noise.

[0068] S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result. The image-text correlation multi-distribution sampling model here is a pre-constructed noise prediction neural network model. Inputting the noisy image sample, image noise parameter value, noisy text sample, and text noise parameter value into the image-text correlation multi-distribution sampling model yields the prediction results for image noise and text noise.

[0069] S5: Calculate the L2 squared difference between the image noise prediction result and the target image noise, and the L2 squared difference between the text noise prediction result and the target text noise, with the goal of minimizing the sum of the two L2 squared differences, and train the parameters of the pre-constructed image-text related multi-distribution sampling model.

[0070] S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model. The preset number of training iterations can be 300,000.

[0071] For the problem of multi-distribution modeling of image-text correlation, existing technologies only consider the modeling of a single distribution. Therefore, they only add noise to a single modality (e.g., image), input the noise magnitude of the single modality into a noise prediction network, and predict the noise of the single modality. This invention considers the modeling of multiple distributions simultaneously. Therefore, it adds noise to all modalities (image and text), inputs the noise magnitude of each modality into a noise prediction network, and predicts the noise of each modality.

[0072] Based on the above embodiments, in this method, determining the noise type and / or noise parameter value to be input according to a pre-set sampling type specifically includes:

[0073] If the preset sampling type is image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as standard Gaussian noise. The parameter value of the text noise to be input is determined to be the maximum value within the preset range.

[0074] Specifically, if the preset sampling type is image sampling, then the noise type to be input is text noise. The text noise to be input to the model is fixed as standard Gaussian noise, and the text noise parameter value is fixed as the maximum value within the preset range. The sampler of the diffusion model contained in the image-text related multi-distribution sampling model is used to sample the noise prediction network with the fixed text noise input and text noise size.

[0075] Based on the above embodiments, in this method, determining the noise type and / or noise parameter value to be input according to a pre-set sampling type specifically includes:

[0076] If the preset sampling type is text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as standard Gaussian noise. The image noise parameter value to be input is determined to be the maximum value within the preset range.

[0077] Specifically, if the preset sampling type is text sampling, then the noise type to be input is image noise. The image noise to be input to the model is fixed as standard Gaussian noise, and the image noise parameter value is fixed as the maximum value within the preset range. The sampler of the diffusion model contained in the image-text related multi-distribution sampling model is used to sample the noise prediction network with the fixed image noise input and image noise size.

[0078] Based on the above embodiments, in this method, determining the noise type and / or noise parameter value to be input according to a pre-set sampling type specifically includes:

[0079] If the preset sampling type is image-text joint sampling, the image noise parameter value to be input and the text noise parameter value to be input are determined to be the same preset value.

[0080] Specifically, if the preset sampling type is image-text joint sampling, the input image noise parameter value and text noise parameter value of the model are set to the same preset value. It should be emphasized that the input image noise parameter value and text noise parameter value are the same, and the preset value can be any value that meets the actual operational requirements; this invention does not impose any limitations on this. The sampler of the diffusion model included in the image-text related multi-distribution sampling model is used to sample the noise prediction network after the above-mentioned preset image noise magnitude value and text noise magnitude value are the same.

[0081] Based on the above embodiments, in this method, determining the noise type and / or noise parameter value to be input according to a pre-set sampling type specifically includes:

[0082] If the preset sampling type is image-to-text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as a preset image, and the image noise parameter value to be input is determined to be 0.

[0083] Specifically, if the pre-defined sampling type is image-to-text sampling, and the noise type to be input is determined to be image noise, the image noise input to the model is fixed to a given image, and the image noise parameter value is fixed to 0. The sampler of the diffusion model contained in the image-text correlated multi-distribution sampling model is used to sample the noise prediction network with the fixed image noise input and image noise magnitude.

[0084] Based on the above embodiments, in this method, determining the noise type and / or noise parameter value to be input according to a pre-set sampling type specifically includes:

[0085] If the preset sampling type is text-to-image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as preset text. The image noise parameter value to be input is determined to be 0.

[0086] Specifically, if the pre-defined sampling type is text-to-image sampling, and the noise type to be input is determined to be text noise, the text noise input to the model is fixed to the given text, and the text noise parameter value is fixed to 0. The sampler of the diffusion model contained in the text-image related multi-distribution sampling model is used to sample the noise prediction network with the fixed text noise input and text noise magnitude.

[0087] In some specific embodiments, the image-text relevance multi-distribution sampling method based on the diffusion model provided by this invention can run on a computer equipped with eight Nvidia GeForce 2080Ti graphics cards. After configuring the corresponding dataset on the computer, the method of this invention can be run to train the model. Specifically, the program requires a system capable of running the PyTorch deep learning framework, equipped with graphics cards with a total of at least 96GB of video memory (e.g., eight 12GB graphics cards). The actual program execution flow is as follows:

[0088] 1. Configure the training batch size according to the actual GPU memory used for the operation.

[0089] 2. Train the noise prediction neural network model.

[0090] In the above specific embodiments, the image-text related multi-distribution sampling method based on a diffusion model provided by the present invention acquires data to be processed, which is one of image data, text data, and image-text data; determines the noise type and / or noise parameter value to be input according to a pre-set sampling type; and inputs the data to be processed, the noise type and / or noise parameter value to be input into a pre-trained image-text related multi-distribution sampling model to obtain sampling results. The image-text related multi-distribution sampling model is trained based on a pre-constructed neural network using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples. The present invention simultaneously considers multiple image-text distributions, adds noise according to a pre-set sampling type, and inputs the noise type and noise parameter value into the pre-trained image-text related multi-distribution sampling model to obtain sampling results, achieving a multi-functional, highly universal, and applicable multi-distribution sampling effect suitable for any multimodal data.

[0091] The following describes the image-text correlation multi-distribution sampling device based on the diffusion model provided by the present invention. The image-text correlation multi-distribution sampling device based on the diffusion model described below and the image-text correlation multi-distribution sampling method based on the diffusion model described above can be referred to in correspondence with each other.

[0092] Figure 3 This is a schematic diagram of the structure of a multi-distribution sampling device for image-text correlation based on a diffusion model provided in an embodiment of the present invention, as shown below. Figure 3 As shown, this embodiment of the invention provides a multi-distribution sampling device for image-text correlation based on a diffusion model, including: a data acquisition unit 310; a noise determination unit 320; and a sampling unit 330.

[0093] in,

[0094] The data acquisition unit 310 is used to acquire data to be processed, wherein the data to be processed is one of image data, text data, and graphic data;

[0095] The noise determination unit 320 is used to determine the noise type and / or noise parameter value to be input based on a preset sampling type.

[0096] The sampling unit 330 is used to input the data to be processed, the noise type to be input, and / or the noise parameter value into a pre-trained image-text related multi-distribution sampling model to obtain the sampling result;

[0097] The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples.

[0098] Based on the above embodiments, in this device, a multi-distribution sampling model for image-text correlation is obtained by training a pre-built neural network using sample data pairs and noise added to the sample data pairs, specifically including the following steps:

[0099] S1: Obtain sample data pairs, which include image data samples and text data samples;

[0100] S2: Obtain target image noise and target text noise based on standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values;

[0101] S3: Based on the image noise parameter value, calculate the linear combination of the image data sample and the image noise parameter value to obtain a noisy image sample; based on the text noise parameter value, calculate the linear combination of the text data sample and the text noise parameter value to obtain a noisy text sample.

[0102] S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result;

[0103] S5: Calculate the squared L2 norm of the difference between the image noise prediction result and the target image noise, and the squared L2 norm of the difference between the text noise prediction result and the target text noise, and train the parameters of the image-text related multi-distribution sampling model with the goal of minimizing the sum of the two squared L2 norms.

[0104] S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model.

[0105] Based on the above embodiments, in this device, determining the noise type and / or noise parameter value to be input according to a preset sampling type specifically includes:

[0106] If the preset sampling type is image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as standard Gaussian noise. The parameter value of the text noise to be input is determined to be the maximum value within the preset range.

[0107] Based on the above embodiments, in this device, determining the noise type and / or noise parameter value to be input according to a preset sampling type specifically includes:

[0108] If the preset sampling type is text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as standard Gaussian noise. The image noise parameter value to be input is determined to be the maximum value within the preset range.

[0109] Based on the above embodiments, in this device, determining the noise type and / or noise parameter value to be input according to a preset sampling type specifically includes:

[0110] If the preset sampling type is image-text joint sampling, the image noise parameter value to be input and the text noise parameter value to be input are determined to be the same preset value.

[0111] Based on the above embodiments, in this device, determining the noise type and / or noise parameter value to be input according to a preset sampling type specifically includes:

[0112] If the preset sampling type is image-to-text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as a preset image, and the image noise parameter value to be input is determined to be 0.

[0113] Based on the above embodiments, in this device, determining the noise type and / or noise parameter value to be input according to a preset sampling type specifically includes:

[0114] If the preset sampling type is text-to-image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as preset text. The image noise parameter value to be input is determined to be 0.

[0115] In the above specific embodiments, the image-text related multi-distribution sampling device based on a diffusion model provided by the present invention acquires data to be processed, which is one of image data, text data, and image-text data; determines the noise type and / or noise parameter value to be input according to a pre-set sampling type; and inputs the data to be processed, the noise type and / or noise parameter value to be input into a pre-trained image-text related multi-distribution sampling model to obtain sampling results. The image-text related multi-distribution sampling model is trained based on a pre-constructed neural network using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples. The present invention simultaneously considers multiple image-text distributions, adds noise according to a pre-set sampling type, and inputs the noise type and noise parameter value into the pre-trained image-text related multi-distribution sampling model to obtain sampling results, achieving a multi-functional, highly universal, and applicable multi-distribution sampling effect suitable for any multimodal data.

[0116] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a diffusion-based image-text correlation multi-distribution sampling method. This method includes: acquiring data to be processed, wherein the data to be processed is one of image data, text data, and image-text data; determining the noise type and / or noise parameter value to be input according to a pre-set sampling type; and inputting the data to be processed, the noise type and / or noise parameter value to be input into a pre-trained image-text correlation multi-distribution sampling model to obtain a sampling result; wherein the image-text correlation multi-distribution sampling model is trained based on a pre-built neural network using sample data pairs and noise added to the sample data pairs, wherein the sample data pairs are data pairs composed of image data samples and text data samples.

[0117] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described diffusion-based image-text related multi-distribution sampling method, the method comprising: acquiring data to be processed, wherein the data to be processed is one of image data, text data, and image-text data; determining a noise type and / or noise parameter value to be input according to a pre-set sampling type; inputting the data to be processed, the noise type and / or noise parameter value to be input into a pre-trained image-text related multi-distribution sampling model to obtain a sampling result; wherein the image-text related multi-distribution sampling model is trained based on a pre-constructed neural network using sample data pairs and noise added to the sample data pairs, wherein the sample data pairs are data pairs composed of image data samples and text data samples.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-distribution sampling method for image-text correlation based on a diffusion model, characterized in that, include: Acquire data to be processed, wherein the data to be processed is one of image data, text data, and graphic-text data; Based on the preset sampling type, determine the noise type and / or noise parameter value to be input; The data to be processed, the noise type to be input, and / or the noise parameter value are input into a pre-trained image-text related multi-distribution sampling model to obtain the sampling results; The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples. The image-text related multi-distribution sampling model is trained through the following steps: S1: Obtain the sample data pairs; S2: Obtain target image noise and target text noise based on standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values; S3: Based on the image noise parameter value, calculate the linear combination of the image data sample and the image noise parameter value to obtain a noisy image sample; based on the text noise parameter value, calculate the linear combination of the text data sample and the text noise parameter value to obtain a noisy text sample. S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result; S5: Calculate the squared L2 norm of the difference between the image noise prediction result and the target image noise, and the squared L2 norm of the difference between the text noise prediction result and the target text noise, and train the parameters of the image-text related multi-distribution sampling model with the goal of minimizing the sum of the two squared L2 norms. S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model.

2. The image-text correlation multi-distribution sampling method based on diffusion model according to claim 1, characterized in that, Based on the pre-defined sampling type, determine the noise type and / or noise parameter value to be input, specifically including: If the preset sampling type is image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as standard Gaussian noise. The parameter value of the text noise to be input is determined to be the maximum value within the preset range.

3. The multi-distribution sampling method for image-text correlation based on the diffusion model according to claim 1, characterized in that, Based on the pre-defined sampling type, determine the noise type and / or noise parameter value to be input, specifically including: If the preset sampling type is text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as standard Gaussian noise. The image noise parameter value to be input is determined to be the maximum value within the preset range.

4. The image-text correlation multi-distribution sampling method based on the diffusion model according to claim 1, characterized in that, Based on the pre-defined sampling type, determine the noise type and / or noise parameter value to be input, specifically including: If the preset sampling type is image-text joint sampling, the image noise parameter value to be input and the text noise parameter value to be input are determined to be the same preset value.

5. The image-text correlation multi-distribution sampling method based on the diffusion model according to claim 1, characterized in that, Based on the pre-defined sampling type, determine the noise type and / or noise parameter value to be input, specifically including: If the preset sampling type is image-to-text sampling, the noise type to be input is determined to be image noise, and the image noise to be input is specified as a preset image, and the image noise parameter value to be input is determined to be 0.

6. The multi-distribution sampling method for image-text correlation based on diffusion model according to claim 1, characterized in that, Based on the pre-defined sampling type, determine the noise type and / or noise parameter value to be input, specifically including: If the preset sampling type is text-to-image sampling, the noise type to be input is determined to be text noise, and the text noise to be input is specified as preset text. The image noise parameter value to be input is determined to be 0.

7. A multi-distribution sampling device for image-text correlation based on a diffusion model, characterized in that, include: A data acquisition unit is used to acquire data to be processed, wherein the data to be processed is one of image data, text data, and graphic data; The noise determination unit is used to determine the noise type and / or noise parameter value to be input based on a preset sampling type. The sampling unit is used to input the data to be processed, the noise type to be input, and / or the noise parameter value into a pre-trained image-text related multi-distribution sampling model to obtain the sampling result; The image-text related multi-distribution sampling model is based on a pre-built neural network trained using sample data pairs and noise added to the sample data pairs. The sample data pairs are data pairs composed of image data samples and text data samples. The following steps were used to train the following: S1: Obtain sample data pairs; S2: Obtain target image noise and target text noise based on standard Gaussian distribution, and obtain image noise parameter values ​​and text noise parameter values; S3: Based on the image noise parameter value, calculate the linear combination of the image data sample and the image noise parameter value to obtain a noisy image sample; based on the text noise parameter value, calculate the linear combination of the text data sample and the text noise parameter value to obtain a noisy text sample. S4: Input the image noise parameter value, the text noise parameter value, the noisy image sample, and the noisy text sample into the image-text correlation multi-distribution sampling model to obtain the image noise prediction result and the text noise prediction result; S5: Calculate the squared L2 norm of the difference between the image noise prediction result and the target image noise, and the squared L2 norm of the difference between the text noise prediction result and the target text noise, and train the parameters of the image-text related multi-distribution sampling model with the goal of minimizing the sum of the two squared L2 norms. S6: Repeat steps S1-S5 until the preset number of training iterations is reached. Use the parameters of the last trained model as the parameters of the final model to obtain the trained image-text related multi-distribution sampling model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image-text correlation multi-distribution sampling method based on the diffusion model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image-text correlation multi-distribution sampling method based on the diffusion model as described in any one of claims 1 to 6.

Images