Fusion method and system of low-resolution data and small sample data

Abstract

The invention discloses a low-resolution data and small sample data fusion method and system, and the method comprises the steps: inputting a low-resolution image into a trained diffusion model, carrying out the noise reduction of the low-resolution image data, carrying out the denoising through a reverse process, restoring a noisy image into a noiseless image, and carrying out the fusion of the low-resolution data and the small sample data. Converting the low-resolution image data into high-resolution image data; obtaining a high-resolution image, and obtaining mask labeling information, a foreground image and a background image; setting a background region position of the foreground reply, calculating a divergence map of the foreground image and the background image by applying a Laplace operator, and performing boundary filling; s4, performing image fusion on the foreground image, the mask labeling information and the background image through Poisson fusion; according to the method, low-resolution data is converted into high-resolution data, it is ensured that richer information can be obtained through image fusion, the data subjected to noise reduction are fused in combination with multi-source information, and the uniformity of sample distribution is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and more particularly to a method and system for fusing low-resolution data and small sample data. Background Art

[0002] At present, image fusion algorithms mainly include spatial domain fusion methods and frequency domain fusion methods. Spatial domain fusion includes alpha fusion, pyramid fusion, Poisson fusion and other fusion algorithms; spatial domain fusion includes DCT fusion, wavelet fusion and other fusion algorithms.

[0003] However, the shortcomings of existing image data fusion mainly include:

[0004] (1) When extracting target features, due to the uneven distribution of samples of various types in some data sets, some small sample data are often difficult to extract effective features when fused with other data because the clarity and background of the images are not similar;

[0005] (2) Some fusion algorithms require large amounts of computation, resulting in slow data processing speed. Especially when faced with large-scale data sets, their real-time performance is even more unsatisfactory. This is mainly because many high-precision fusion algorithms require complex mathematical operations, such as matrix operations and feature extraction. As the amount of data increases, the processing time increases significantly.

[0006] (3) Information redundancy is common during the fusion process. This is mainly because multiple data sources provide similar or repeated information. Such redundancy not only increases the computational complexity, but may also cause obvious seams in the new fused data, affecting the overall quality and visual effect of the image. Especially in image fusion or video processing applications, when data from different sources are merged, due to differences between the source data, such as lighting, viewing angle, and resolution, unnatural boundaries and seams are easily formed, making the newly generated image lack coherence and consistency.

[0007] Therefore, how to achieve the fusion of low-resolution data and small sample data in order to address the current problems of poor effectiveness of feature extraction, slow processing speed, and information redundancy in data fusion is an urgent problem that technicians in this field need to solve. Summary of the invention

[0008] In view of this, the present invention provides a method and system for fusing low-resolution data and small sample data to solve some of the technical problems mentioned in the background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A method for fusing low-resolution data and small sample data comprises the following steps:

[0011] S1. Input the low-resolution image into the trained diffusion model, restore the noise of the low-resolution image data, and denoise it through the inverse process to restore the noisy image to a noise-free image, thereby converting the low-resolution image data into high-resolution image data;

[0012] S2. Obtain a high-resolution image, and obtain mask annotation information, a foreground image, and a background image;

[0013] S3. Set the background area position of the foreground that needs to be replied, apply the Laplacian operator to calculate the scatter diagram of the foreground image and the background image and fill the border;

[0014] S4. The foreground image, the mask annotation information and the background image are fused through Poisson fusion.

[0015] Preferably, in step S1, the specific content of training the diffusion model is:

[0016] Forward process: add noise to the low-resolution image at each moment to obtain the image distribution after adding noise at each moment;

[0017] Noise estimation: The noise estimation model is trained based on the image distribution and step size after adding noise to obtain the predicted noise;

[0018] Reverse process: based on the low-resolution image, the image distribution after adding noise and the predicted noise, denoising is performed to obtain the denoised image;

[0019] Training and learning: The denoised image is input into the noise estimation model again, and the noise estimation and reverse process operations are repeated until the data is close to the original image, and finally the trained diffusion model is obtained.

[0020] Preferably, the image distribution after adding noise at each moment is:

[0021]

[0022] Among them, x0 is the input low-resolution image, and Represents the corresponding weight, To add noise, follow Gaussian distribution;

[0023] The loss function of the noise estimation model is:

[0024]

[0025] Among them, μ θ (x t, x0) is the real noise z θ Next x t-1 The mean of the predicted noise z is obtained after training t ;

[0026] The image distribution after denoising by the reverse process is:

[0027]

[0028]

[0029] Preferably, the specific content of step S2 is:

[0030] S21. Obtain high-resolution images from diffusion model results;

[0031] S22. extracting corresponding mask annotation information to clearly identify a specific area in the image;

[0032] S23. Crop the foreground image from the high-resolution image according to the mask annotation information, and integrate the three parts of data into one data structure.

[0033] Preferably, the specific contents of step S3 are: determining the position and range of the background area of ​​the foreground image reply in the background image; using a graphic tool to draw the boundary of the selected background area on the high-resolution image, and marking the background area.

[0034] Preferably, the Laplace operator is applied to perform convolution operation on the foreground image and the background image to obtain the divergence information of each pixel in the image, and the Neumann boundary condition is used to fill the boundary by calculating the boundary value of the background image to obtain the divergence map of the foreground image and the background image.

[0035] Preferably, the specific content of the image fusion in step S4 is:

[0036] Assume that the boundary value of the background image calculated using the Neumann boundary condition is f k * , the background image at the background area of ​​the background image is f i , the foreground image is g i , then the fusion formula is:

[0037] L(f i )=L(g i )

[0038] L(g i ) is the pixel value of the foreground image after Laplace convolution, and the calculation method is:

[0039] L(g i )=Δ*g i|g i ∈(gg k * )

[0040] Among them, g i ∈(gg k * ) represents the foreground image excluding the boundary value;

[0041] In the background image, the boundary value f k * If is a known value, the matrix formula can be used for calculation:

[0042] A i =b

[0043] Where A is the corresponding pixel f i The Laplace calculation matrix of L(g i )-f k * , the calculation result f i That is the pixel value of the corresponding area, and then the entire image after Poisson fusion is obtained.

[0044] A fusion system of low-resolution data and small sample data, based on the fusion method of low-resolution data and small sample data, comprises: an image resolution conversion module, a multivariate image data acquisition module, a multivariate image data processing module and an image fusion module;

[0045] An image resolution conversion module is used to input a low-resolution image into a trained diffusion model, and to convert the low-resolution image data into high-resolution image data by performing noise reduction on the low-resolution image data and denoising through an inverse process to restore the noisy image to a noise-free image;

[0046] A multi-image data acquisition module is used to acquire high-resolution images, and obtain mask annotation information, foreground images, and background images;

[0047] The multivariate image data processing module is used to set the background area position, apply the Laplace operator to calculate the foreground scatter map and perform boundary filling;

[0048] The image fusion module is used to fuse the foreground image, mask annotation information and background image through Poisson fusion.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for fusing low-resolution data and small sample data.

[0050] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the method for fusing low-resolution data and small sample data is implemented.

[0051] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for fusing low-resolution data and small sample data. By utilizing a diffusion model and performing noise restoration on low-resolution data, the clarity and resolution of the data are significantly improved, ensuring that richer information can be obtained in the subsequent fusion process. The data after noise restoration is then combined with multi-source information for fusion, which can further improve the distribution of samples and make them more uniform. The present invention not only improves the accuracy of small target detection, but also reduces the deviation caused by small sample data. At the same time, the results after data fusion show stronger robustness in the feature extraction process, which helps to better adapt to subsequent analysis and application, and promotes research and application development in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a method for fusing low-resolution data and small sample data provided by the present invention;

[0054] Figure 2 A schematic diagram of the diffusion model principle provided by the present invention;

[0055] Figure 3 This is a diffusion model effect diagram provided by the present invention;

[0056] Figure 4 A schematic diagram of image fusion provided by the present invention;

[0057] Figure 5 This is a rendering of a method for fusing low-resolution data and small sample data provided by the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The embodiment of the present invention discloses a method for fusing low-resolution data and small sample data, such as Figure 1 , including the following steps:

[0060] S1. Input the low-resolution image into the trained diffusion model, restore the noise of the low-resolution image data, and denoise it through the inverse process to restore the noisy image to a noise-free image, thereby converting the low-resolution image data into high-resolution image data;

[0061] S2. Obtain a high-resolution image, and obtain mask annotation information, a foreground image, and a background image;

[0062] S3. Set the background area position of the foreground that needs to be replied, apply the Laplacian operator to calculate the scatter diagram of the foreground image and the background image and fill the border;

[0063] S4. Perform image fusion of the foreground image, mask annotation information and background image through Poisson fusion.

[0064] In order to further implement the above technical solution, in step S1, Figure 2 , the specific content of training diffusion model is:

[0065] Forward process: add noise to the low-resolution image at each moment to obtain the image distribution after adding noise at each moment;

[0066] Noise estimation: The noise estimation model is trained based on the image distribution and step size after adding noise to obtain the predicted noise;

[0067] Reverse process: based on the low-resolution image, the image distribution after adding noise and the predicted noise, denoising is performed to obtain the denoised image;

[0068] Training and learning: The denoised image is input into the noise estimation model again, and the noise estimation and reverse process operations are repeated until the data is close to the original image, and finally the trained diffusion model is obtained.

[0069] In order to further implement the above technical solution, the image distribution after adding noise at each moment is:

[0070]

[0071] Among them, x0 is the input low-resolution image, and Represents the corresponding weight, To add noise, follow Gaussian distribution;

[0072] The loss function of the noise estimation model is:

[0073]

[0074] Among them, μ θ (x t , x0) is the real noise z θ Next x t-1 The mean of the predicted noise z is obtained after training t ;

[0075] The image distribution after denoising by the reverse process is:

[0076]

[0077]

[0078] In this embodiment, the forward process of the diffusion model is a process of continuously adding noise to the model. Noise is added at each moment, and the image at the next moment is obtained by adding noise to the image at the previous moment. The noise obeys a Gaussian distribution:

[0079]

[0080] Among them, z represents the gray value, v represents the average value or expected value of z, and σ represents the standard deviation of z;

[0081] The method for obtaining the image distribution at each moment is as follows:

[0082] α t =1-β t

[0083] Among them, β t Represents the weight of the noise, in the forward process β t The changes will become greater and greater over time;

[0084] When the image distribution after adding noise at time t is:

[0085]

[0086] Among them, x t represents the distribution of the image at time t, z1 represents the noise at time 1, and represents the corresponding weight;

[0087] Similarly:

[0088]

[0089] After iteration, we can get:

[0090]

[0091] The reverse process is to restore the noisy image to the noise-free image x t-1 , the known condition is x t and x0, using the Bayesian formula:

[0092]

[0093] in:

[0094]

[0095]

[0096]

[0097] thereby:

[0098]

[0099] After simplification, we can get x t-1 The mean of :

[0100]

[0101]

[0102] z t For the noise at each moment, a deep learning algorithm is used for simulation. In this embodiment, Unet is used for simulation. The input parameters of the image are the distribution at the current moment and the step length t. The learning process is: given a noisy image and a step length t, input it into the Unet network, and the loss function is the mean square error. After training, the predicted noise is obtained, and the predicted noise is brought into the obtained x t-1 Formula, get the denoised image, then input the denoised image into Unet again, repeat the operation until it is close to the data of the original image, and finally apply the entire trained model to the low-resolution image to infer the high-resolution image, such as Figure 3 .

[0103] In order to further implement the above technical solution, Figure 4 , the specific content of step S2 is:

[0104] S21. Obtain high-resolution images from diffusion model results;

[0105] S22. extracting corresponding mask annotation information to clearly identify a specific area in the image;

[0106] S23. Crop the foreground image from the high-resolution image according to the mask annotation information, and integrate the three parts of data into one data structure.

[0107] In practical applications, the integration results can be selectively displayed through visualization tools to ensure the clarity and correspondence of each part, and the integrated data can be saved in a suitable format for subsequent use.

[0108] In order to further implement the above technical solution, the specific content of step S3 is: determine the position and range of the background area of ​​the foreground image reply in the background image; use graphic tools to draw the boundary of the selected background area on the high-resolution image and mark the background area.

[0109] In this embodiment, the position and range of the background area are usually represented by coordinates; when marking the background area, different colors or patterns can be used to distinguish it from the foreground part to help the model or user understand the background that needs attention.

[0110] In order to further implement the above technical solution, the Laplace operator is applied to perform convolution operation on the foreground image and the background image to obtain the divergence information of each pixel in the image, and the Neumann boundary condition is used to fill the boundary by calculating the boundary value of the background image to obtain the divergence map of the foreground image and the background image.

[0111] In this embodiment, in a discrete case, the Laplacian operator can be expressed by a Laplacian convolution kernel as:

[0112]

[0113] Apply the Laplace operator to perform convolution operation on the foreground image to obtain the divergence information of each pixel in the image. This process involves calculating the second-order derivative of the gradient in the image, so that the high-frequency information of the image (such as edges and noise) is enhanced;

[0114] When an image is convolved or filtered, the way the edge is processed will affect the generation of the divergence map. The Neumann boundary condition is that when processing the edge of an image, the pixel values ​​outside the edge are regarded as the same as the edge pixels. This processing method can effectively reduce the errors caused by the edge effect. When the Laplace operator is applied to calculate the foreground divergence map in this embodiment, the use of the Neumann boundary condition can ensure that the derivative calculation at the edge will not be affected by external noise, thereby maintaining the smoothness and consistency of the overall image.

[0115] In order to further implement the above technical solution, the specific content of the image fusion in step S4 is:

[0116] Assume that the boundary value of the background image calculated using the Neumann boundary condition is f k * , the background image at the background area of ​​the background image is f i , the foreground image is g i, then the fusion formula is:

[0117] L(f i )=L(g i )

[0118] L(g i ) is the pixel value of the foreground image after Laplace convolution, and the calculation method is:

[0119] L(g i )=Δ*g i |g i ∈(gg k * )

[0120] Among them, g i ∈(gg k * ) represents the foreground image excluding the boundary value;

[0121] In the background image, the boundary value f k * If is a known value, the matrix formula can be used for calculation:

[0122] A i =b

[0123] Where A is the corresponding pixel f i The Laplace calculation matrix of L(g i )-f k * , the calculation result f i That is the pixel value of the corresponding area, and then the entire image after Poisson fusion is obtained.

[0124] The present invention combines the diffusion model and Poisson fusion to solve the problem of data scarcity in small sample learning. The diffusion model is used to generate diverse high-quality images, and Poisson fusion is used to seamlessly synthesize these generated images with the original images, thereby expanding the data set. The use of the diffusion model for data enhancement can significantly improve the availability of large sample fuzzy data. The diffusion model generates high-quality images by gradually denoising, which can effectively extract useful features from fuzzy data, and then generate clearer and more diverse samples. This process makes fuzzy data not only usable for training, but also improves the model's generalization ability for different types of data. Experimental results show that the generated images are superior to traditional data enhancement methods in visual quality and diversity, and after Poisson fusion processing, details and edge information are effectively retained, such as Figure 5 .

[0125] A fusion system of low-resolution data and small sample data, based on a fusion method of low-resolution data and small sample data, comprising: an image resolution conversion module, a multivariate image data acquisition module, a multivariate image data processing module and an image fusion module;

[0126] An image resolution conversion module is used to input a low-resolution image into a trained diffusion model, and to convert the low-resolution image data into high-resolution image data by performing noise reduction on the low-resolution image data and denoising through an inverse process to restore the noisy image to a noise-free image;

[0127] A multi-image data acquisition module is used to acquire high-resolution images, and obtain mask annotation information, foreground images, and background images;

[0128] The multivariate image data processing module is used to set the background area position, apply the Laplace operator to calculate the foreground scatter map and perform boundary filling;

[0129] The image fusion module is used to fuse the foreground image, mask annotation information and background image through Poisson fusion.

[0130] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for fusing low-resolution data and small sample data.

[0131] A processing terminal comprises a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a method for fusing low-resolution data and small sample data is implemented.

[0132] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for fusing low-resolution data and small sample data, characterized in that: The following steps are involved: S1. Input the low-resolution image into the trained diffusion model, perform noise reduction on the low-resolution image data, perform denoising through the inverse process, and restore the noisy image to a noise-free image, thereby converting the low-resolution image data into high-resolution image data; S2. Obtain a high-resolution image, and obtain mask annotation information, a foreground image, and a background image; S3. Set the background area position of the foreground that needs to be replied, apply the Laplacian operator to calculate the scatter diagram of the foreground image and the background image and fill the border; S4. The foreground image, the mask annotation information and the background image are fused through Poisson fusion.

2. The method for fusing low-resolution data and small sample data according to claim 1, characterized in that: In step S1, the specific content of training the diffusion model is: Forward process: add noise to the low-resolution image at each moment to obtain the image distribution after adding noise at each moment; Noise estimation: The noise estimation model is trained based on the image distribution and step size after adding noise to obtain the predicted noise; Reverse process: based on the low-resolution image, the image distribution after adding noise and the predicted noise, denoising is performed to obtain the denoised image; Training and learning: The denoised image is input into the noise estimation model again, and the noise estimation and reverse process operations are repeated until the data is close to the original image, and finally the trained diffusion model is obtained.

3. The method for fusing low-resolution data and small sample data according to claim 2, characterized in that: The image distribution after adding noise at each moment is: Among them, x0 is the input low-resolution image, and Represents the corresponding weight, To add noise, follow Gaussian distribution; The loss function of the noise estimation model is: Among them, μ θ (x t , x0) is the real noise z θ Next x t-1 The mean of the predicted noise z is obtained after training t ; The image distribution after denoising by the reverse process is:

4. The method for fusing low-resolution data and small sample data according to claim 1, characterized in that: The specific content of step S2 is: S21. Obtain high-resolution images from diffusion model results; S22. extracting corresponding mask annotation information to clearly identify a specific area in the image; S23. Crop the foreground image from the high-resolution image according to the mask annotation information, and integrate the three parts of data into one data structure.

5. The method for fusing low-resolution data and small sample data according to claim 1, characterized in that: The specific contents of step S3 are: determining the position and range of the background area of ​​the foreground image reply in the background image; using graphic tools to draw the boundary of the selected background area on the high-resolution image, and marking the background area.

6. The method for fusing low-resolution data and small sample data according to claim 1, characterized in that: The Laplace operator is applied to perform convolution operation on the foreground image and the background image to obtain the divergence information of each pixel in the image, and the Neumann boundary condition is used to fill the boundary by calculating the boundary value of the background image to obtain the foreground scatter map and the background scatter map.

7. The method for fusing low-resolution data and small sample data according to claim 6, characterized in that: The specific content of the image fusion in step S4 is: Assume that the boundary value of the background image calculated by the Neumann boundary condition is fk*, the background image at the background area of ​​the background image is fi, and the foreground image is gi, then the fusion formula is: L(fi)=L(g i ) L(g i ) is the pixel value of the foreground image after Laplace convolution, and the calculation method is: L(g i )=Δ*g i |g i ∈(g-g k * ) Among them, g i ∈(gg k * ) represents the foreground image excluding the boundary value; In the background image, the boundary value f k * If is a known value, the matrix formula can be used for calculation: Of i =b Among them, A is the corresponding pixel f i The Laplace calculation matrix of L(g i )-f k * , the calculation result f i That is the pixel value of the corresponding area, and then the entire image after Poisson fusion is obtained.

8. A fusion system for low-resolution data and small sample data, characterized in that: A method for fusing low-resolution data and small sample data based on any one of claims 1 to 7, comprising: an image resolution conversion module, a multivariate image data acquisition module, a multivariate image data processing module and an image fusion module; An image resolution conversion module is used to input a low-resolution image into a trained diffusion model, and to convert the low-resolution image data into high-resolution image data by performing noise reduction on the low-resolution image data and denoising the image through an inverse process to restore the noisy image to a noise-free image; A multi-image data acquisition module is used to acquire high-resolution images, and obtain mask annotation information, foreground images, and background images; The multivariate image data processing module is used to set the background area position, apply the Laplace operator to calculate the foreground scatter map and perform boundary filling; The image fusion module is used to fuse the foreground image, mask annotation information and background image through Poisson fusion.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for fusing low-resolution data and small sample data described in any one of claims 1 to 7 is implemented.

10. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method for fusing low-resolution data and small sample data as described in any one of claims 1 to 7 is implemented.

Images