Laser image translation method and system based on dual-condition improved diffusion model

Abstract

The invention relates to the technical field of laser image translation, and particularly discloses a laser image translation method and system based on a dual-condition improved diffusion model, and the method comprises the steps: collecting a laser image and a visible light image, and carrying out the data preprocessing of the laser image and the visible light image, so as to construct a laser-visible light image data set; constructing an improved diffusion model based on double conditions to learn mapping between a laser image domain and a visible light image domain in the laser-visible light image data set; and translating a to-be-processed laser image into a visible light image by using the improved diffusion model based on double conditions.

Description

Technical Field

[0001] The invention relates to the technical field of laser image translation, and in particular to a laser image translation method and system based on a dual-condition improved diffusion model. Background Art

[0002] It is difficult for visible light imaging systems to collect images with clear contours, clear targets and backgrounds in low light conditions, rainy and foggy weather, and long distances. However, laser active imaging systems can make up for the shortcomings of visible light imaging systems in poor imaging quality under harsh environmental conditions and obtain clear images. The combination of the two can achieve all-day and all-weather security and monitoring tasks. However, the modal difference between laser images and visible light images is large, which is not conducive to human observation and computer processing. It is necessary to translate laser images into visible light images across modalities to improve human interpretability and apply existing visual algorithms for processing visible light images. Existing cross-modal image translation algorithms have problems such as training difficulties and color penetration.

[0003] The laser active imaging system can make up for the imaging quality of the visible light imaging system in harsh environments and obtain clear images. The combination of the two can achieve all-day and all-weather security and monitoring tasks. However, the modal difference between laser images and visible light images is large, which is not conducive to human observation and computer processing. It is necessary to translate laser images into visible light images across modalities to improve human interpretability and apply existing visual algorithms for processing visible light images. At present, there is little research on laser image translation. The current technologies related to visible light image translation can be mainly divided into two categories. One is the technology based on generative adversarial networks, which generally has problems of unstable training and mode collapse; the other is the technology based on Transformer, which has problems of cumulative errors and color penetration.

[0004] Therefore, how to solve the technical problems of unstable training and low quality of generated images in the prior art is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0005] To achieve the purpose of the present invention, the present application provides a laser image translation method based on a dual-condition improved diffusion model, comprising: Step S1: collecting visible light images and laser images, and performing data preprocessing on the laser images and visible light images to construct a laser-visible light image data set; Step S2: constructing an improved diffusion model based on dual conditions, and using the laser-visible light image dataset for training, so as to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset; The improved diffusion model based on dual conditions includes a forward diffusion process and a backward diffusion process; the forward diffusion process gradually adds random Gaussian noise to the visible light image, so that the visible light image is completely transformed into a standard Gaussian distribution, and introduces a Brownian bridge strategy in the noise diffusion process, so that the noise addition end point is changed from pure Gaussian noise to a laser image; Step S3: using the improved diffusion model based on dual conditions, translating the laser image to be processed into a visible light image.

[0006] In one specific embodiment, step S1 includes: The laser image and the visible light image are sheared, and the sheared laser image and the visible light image are horizontally flipped and vertically flipped.

[0007] In one specific embodiment, in step S2, the backward diffusion process uses a neural network model to fit the conditional probability distribution, wherein, in the denoising process, the laser image is interpolated and transformed using an interpolation conditional injection method and then combined with the noisy image and uploaded to the neural network model, so that the neural network model predicts the random Gaussian noise added in each step and restores it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

[0008] In one specific embodiment, the loss function of the improved diffusion model based on dual conditions is determined according to the following formula: ; In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding feature after interpolation change, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, m t Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

[0009] In one specific embodiment, the interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with the noisy image and inputting them into the neural network model to weaken the uncertainty of the diffusion model.

[0010] To achieve the same invention purpose, the present application also provides a laser image translation system based on a dual-condition improved diffusion model, comprising: Dataset construction module: used for collecting visible light images and laser images, and performing data preprocessing on the laser images and visible light images to construct a laser-visible light image dataset; Model building module: used to build an improved diffusion model based on dual conditions to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset; The improved diffusion model based on dual conditions includes a forward diffusion process and a backward diffusion process; the forward diffusion process gradually adds random Gaussian noise to the visible light image, so that the visible light image is completely transformed into a standard Gaussian distribution, and introduces a Brownian bridge strategy in the noise diffusion process, so that the noise addition end point is changed from pure Gaussian noise to a laser image; Image translation module: used for translating the laser image to be processed into a visible light image by using the improved diffusion model based on dual conditions.

[0011] In one specific embodiment, the data set construction module is used to: The laser image and the visible light image are sheared, and the sheared laser image and the visible light image are horizontally flipped and vertically flipped.

[0012] In one specific embodiment, the model building module includes: The back diffusion process uses a neural network model to fit the conditional probability distribution. In the denoising process, the laser image is interpolated and transformed using an interpolation conditional injection method and then combined with the noisy image and uploaded to the neural network model, so that the neural network model predicts the random Gaussian noise added in each step and restores it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

[0013] In one specific embodiment, the loss function of the improved Brownian bridge diffusion model based on dual conditions is determined according to the following formula: ; In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding feature after interpolation change, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, m t Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

[0014] In one specific embodiment, the interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with the noisy image and inputting them into the neural network model to weaken the uncertainty of the diffusion model.

[0015] Beneficial effects of the above technical solution: This application provides a laser image translation method and system based on a dual-condition improved diffusion model, which introduces a diffusion model to achieve translation from laser images to visible light images. Based on the Brownian bridge strategy, the conditional image is directly integrated with the diffusion process, so that the diffusion model can directly learn the mapping between the two domains. In order to further limit the randomness of the generation of the diffusion model, an interpolation condition injection method is proposed, and a self-attention mechanism is used to enhance the model's ability to learn the scene, color and edge features of the target in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic flow chart of a laser image translation method based on a dual-condition improved diffusion model provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a laser image translation system based on a dual-condition improved diffusion model provided by an embodiment of the present invention; Figure 3 A schematic diagram of a laser-visible light image acquisition system for a laser image translation method based on a dual-condition improved diffusion model provided by an embodiment of the present invention; Figure 4 A schematic flow chart of a laser image translation method based on a dual-condition improved diffusion model provided by an embodiment of the present invention; Figure 5 A schematic flow chart of a diffusion process based on a dual-condition improved diffusion model of a laser image translation method based on a dual-condition improved diffusion model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0019] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0020] Embodiment 1 An embodiment of the present invention provides a laser image translation method based on a dual-condition improved diffusion model, referring to Figure 1 As shown, including: Step S1: Acquire visible light images and laser images acquired using laser active imaging under the same field of view and shooting conditions, and perform data preprocessing on the laser images and visible light images to construct a laser-visible light image dataset.

[0021] In a specific embodiment of the present invention, step S1 includes: The laser image and the visible light image are cut using image processing software, and the cut laser image and the visible light image are flipped horizontally and vertically.

[0022] Specifically, in order for the model to better learn the color and edge features of various scenes and objects in the real world, the constructed dataset should contain as many categories of scenes and targets as possible.

[0023] For the same object, the images taken from different angles may look completely different, so the scene and target need to be photographed from different angles.

[0024] In order to meet the requirements of supervised and unsupervised model training and subsequent model evaluation, the data collected by the dataset should be one-to-one corresponding.

[0025] The collection of visible light images needs to be carried out under lighting conditions, and the collection of laser images requires the use of laser active illumination. In order to meet the application scenarios of this topic, the shooting environment is chosen to be outdoors with sunlight.

[0026] Since the wavelength of sunlight covers both the invisible light band (ultraviolet and infrared) and the visible light band, when performing laser active imaging, light outside the laser light source band will affect the collected image. Therefore, in order to capture effective laser images, the influence of light in other bands should be weakened as much as possible.

[0027] In order to reduce the impact of outdoor sunlight on laser image acquisition, the shooting time is selected in the evening, when the sunlight is very weak, but the outdoor brightness can also meet the requirements of shooting visible light images. At the same time, lasers outside the visible light band are used for irradiation to further reduce the impact of sunlight on the collection of laser images. When shooting, try to choose as many types of scenes and targets as possible, mainly including flowers, trees, vehicles, buildings, house numbers, street scenes, etc., and the shooting distance is selected between 50m-100m.

[0028] In order to construct the data set as required, this application designed a laser-visible light image acquisition system. Figure 3 The overall structure of the acquisition system is shown. The system mainly consists of five parts, namely, camera, electric zoom lens, laser, electric beam expander and narrow-band filter. The camera is a color CCD industrial camera that can collect laser and visible light images. The camera's photosensitivity size is 1 / 3″, the frame rate is 30FPS, the collected image resolution is 1024×768, and the image bit depth is 24 bits; the electric zoom lens is a 400mm electric zoom lens with a focal length of 10mm-400mm, an aperture size of F2.8, and a field of view of 1 / 2″; the electric beam expander is used to expand the diameter and divergence angle of the laser beam so that the laser covers the entire field of view as much as possible; the laser is an 850nm near-infrared laser, which is used for active laser irradiation when shooting laser images; the central wavelength of the narrow-band filter is 850nm, in order to avoid the influence of other bands of light as much as possible when shooting laser images.

[0029] Specifically, the original resolution of the image obtained by the image acquisition system is 1024 × 768. Since the image resolution of the dataset used by the current commonly used image translation network model is 256 × 256 or 512 × 512, and the amount of data in the dataset samples is increased, the original image is cut using image processing software. Based on the principle of not changing the image content quality and ensuring that there are relatively complete objects and scenes in the image, each image is cut at a resolution of 512 × 512.

[0030] In order to further expand the dataset, a typical data augmentation method, mirror flipping, was used to flip the cropped images horizontally and vertically respectively.

[0031] Specifically, 100 laser images and 100 visible light images were captured using an image acquisition system, a total of 100 pairs. After the shearing operation, 250 pairs of laser-visible light images were obtained. For subsequent model training, the image data was divided into training sets and test sets in a ratio of 9:1, and finally 215 pairs of training sets and 35 pairs of test sets were obtained. In order to ensure the robustness of the network model after training, this application only performs mirror flipping data enhancement on the training set. The laser-visible light image dataset includes 630 pairs of training set data and 35 pairs of test set data.

[0032] Step S2: constructing an improved diffusion model based on dual conditions to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset.

[0033] Specifically, the Brownian bridge is a common research object in probability theory. It is a random process in continuous time and also conforms to the properties of Markov chains. It is a special case of Brownian motion. The special thing about the Brownian bridge is that it is a Brownian motion with fixed starting and ending states. For example, suppose there is a set of random variables that conform to the normal distribution , time step t starts from 0 and ends at T, then the Brownian bridge process can be defined as: (1) Where N represents normal distribution, t represents the current time step, T represents the total time step, x 0 represents the random variable at the 0th time step, I represents the identity matrix, x t represents the random variable at time step t, x T represents the random variable under T time steps. When t=0 and t=T, the whole process is and Fixed, the middle process is also connected by these two ends to form a bridge.

[0034] In a specific embodiment of the present invention, referring to Figure 4 , Figure 5 As shown, the improved diffusion model based on dual conditions includes: forward diffusion process and backward diffusion process; The forward diffusion process gradually adds random Gaussian noise to the visible light image, so that the visible light image is completely transformed into a standard Gaussian distribution, and introduces the Brownian bridge principle in the noise diffusion process to change the noise addition end point from pure Gaussian noise to a laser image.

[0035] Specifically, based on Markov chain theory, DDPM defines a source image data Start by adding noise gradually and end with standard Gaussian noise The forward diffusion process ends, and the whole process can be expressed as follows: (2) (3) in, t represents the time step, represents a normal distribution, Represents a value between 0 and 1. .

[0036] This noise addition method allows the model to sample images from pure noise, so the images generated by the model have significant diversity. However, this diversity is a disadvantage for learning the mapping between two certain domains. This application implements a more deterministic diffusion model for end-to-end image generation by weakening the randomness in the sampling process.

[0037] Furthermore, the present application adopts the Brownian bridge strategy to directly introduce conditional control in the process of adding noise and denoising to weaken the randomness of the diffusion model. Different from the DDPM (denoising diffusion probability model) in the prior art, the denoising process of the present application starts with the visible light image and ends with the laser image. For the sake of consistency, the present application will be described in detail in the following sections. represents the target image (visible light image), and y represents the source image (laser image). The forward diffusion process can be expressed as follows: (4) in, represents the variance, Normally, according to formula (1), the variance here is Should be However, when When , the variance is , when the total time step T increases, the variance of the process points will also increase. If the standard normal distribution is satisfied, then the variance of the steps in the noise addition process should be kept consistent, which is contrary to the principle and will affect the training of the model. To this end, this application adopts a new variance design: (5) When t=0, =0, the mean is the target image ; When t=T, =1, at this time the mean is the source image y. According to formula (5), the variance gradually increases from 0 to the midpoint It reaches its maximum value when , and then gradually decreases to 0. In the Brownian bridge process, this application adjusts the diversity of sampling by controlling the maximum variance of the intermediate time step. For this purpose, the coefficient s can be used to expand the variance. : (6) According to formula (4), the marginal distribution of each time step t can be obtained. However, during training and inference, it is also necessary to deduce arrive The forward transition probability By recursive calculation, we can first obtain The following expression: (7) in represents noise. According to formula (7), Indicates that the forward transition probability can be obtained The specific representation is as follows: (8) in, , which establishes the to the source image y.

[0038] The backward diffusion process uses a neural network to fit the conditional probability distribution. When predicting noise during the denoising process, the laser image is interpolated and transformed using the interpolation conditional injection method and then combined with the noisy image and uploaded to the neural network model. The neural network model can predict the random Gaussian noise added at each step and restore it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

[0039] Specifically, unlike most diffusion models, this application abandons the practice of starting denoising from pure Gaussian noise, and instead uses the Brownian bridge strategy to change the denoising starting point to the source image y, thereby directly injecting conditions into the diffusion process and reducing the generation uncertainty of the diffusion model to better adapt to the end-to-end task of laser-visible light image translation. According to the core idea of ​​the diffusion model, the posterior probability required in the reverse process can be obtained as follows: (9) in, is the mean of the noise distribution predicted by the neural network at each time step, is the variance of the noise distribution at each time step.

[0040] In the experiment, it was found that the ability of laser-visible light image translation to enhance the diffusion model only through the Brownian bridge strategy is still insufficient, and the generated images have problems such as color cross-talk and color loss. In order to solve the above problems, this application further restricts the diffusion model to generate a more certain image. Unlike the Brownian bridge diffusion model, which directly injects conditions into the diffusion process to establish a connection between the two domains, most diffusion models impose conditional restrictions by inputting additional conditional information such as the feature encoding of the reference image into the neural network model during the reverse diffusion process, thereby guiding the denoised image to move closer to the target domain. Based on this, this application combines the advantages of both and proposes an improved Brownian bridge diffusion model with dual conditional control.

[0041] In a specific embodiment of the present invention, the interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with the noisy image and inputting it into the neural network to weaken the uncertainty of the diffusion model.

[0042] This application inputs the encoded image of the laser image after interpolation into the neural network for conditional guidance. This operation is usually cited in super-resolution tasks. In order to make the resolution of the source image reach the target image resolution, the input source image size is converted to the size of the target image through interpolation. The interpolation operation in super-resolution is essentially to predict the distribution of pixels around each pixel of the source image, providing prior information for the neural network to better learn the connection between each part of the image. Based on this, this application encodes the laser image through a bicubic interpolation operation, and then combines the encoded image with the laser image and inputs it into the neural network, providing additional conditional control and further weakening the uncertainty of the diffusion model.

[0043] by It represents the encoding features after interpolation change, and the backward diffusion process after adding interpolation condition injection is as follows: (10) In a specific embodiment of the present invention, the loss function of the improved Brownian bridge diffusion model based on dual conditions is determined according to the following formula: ; In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding feature after interpolation change, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, m t Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

[0044] Specifically, like most diffusion models, the model of the present application completes the training of the model by optimizing the variational lower bound, which can be expressed as: (11) This application fixes the end point of the forward diffusion process so that , then the first term in formula (10) can be regarded as a constant and ignored. Through formulas (4) and (8) and using the Bayesian formula and the properties of the Markov chain, the posterior probability of the second term in formula (10) is expressed as follows: (12) By reparameterization and simplification It is expressed as: (13) in is the noise distribution that the neural network needs to predict, , , The specific form is as follows: ; ; (14) Finally, combining the above formulas, formula (10) is simplified to: (15) in, is the random noise added to enhance the robustness of the neural network model. Then, the second conditional control after the interpolation operation is introduced to Represents the encoding features after interpolation, and the final loss function is obtained as follows: (16) Specifically, the self-attention mechanism reduces the original attention mechanism's reliance on external information, and implements a modeling method that effectively captures global context information through a triple (key, query, value), which can better capture the internal correlation of data or features. It calculates the attention score of the input itself through the self-attention mechanism to obtain the importance and connection of each part in the image data. In order to enable the model in this application to better identify the background and each target in the image for more accurate restoration, the self-attention mechanism is used in the middle layer of the neural network.

[0045] Step S3: translating the laser image to be processed into a visible light image using the improved diffusion model based on dual conditions.

[0046] In a specific embodiment of the present invention, the present application uses the Adam optimizer for training and verification of the constructed laser-visible light image dataset. In order to accelerate the reasoning process of the model, the non-Markov process is used to reduce the original reasoning process that requires T=1000 to T=200, which can ensure both the speed of reasoning and the quality of generation.

[0047] For the laser-visible light image translation task, in order to compare the quality, clarity and fidelity of the generated images, this application selected two indicators commonly used to evaluate the effect of the generated model, FID and LPIPS.

[0048] FID: It is an indicator for evaluating the performance of the generative model. It evaluates the image quality by comparing the distribution difference between the generated image and the real image in the feature space. Based on the pre-trained Inception network, feature extraction is performed on a set of real images and a set of generated images to obtain their respective expressions in the feature space. The mean and covariance matrix of the real image and the generated image in the feature space are calculated, and then the Fréchet distance formula is used to calculate the difference between the two sets of mean and covariance matrices. Fréchet distance is a measure of the similarity between two probability distributions. A smaller FID value indicates that the distribution of the generated image in the feature space is closer to that of the real image, and the generated image quality is higher.

[0049] LPIPS: is an indicator used to evaluate image quality and similarity. It uses deep learning methods to simulate human sensitivity to image differences. The key idea of ​​LPIPS is to use a pre-trained convolutional neural network (CNN) to extract features from input images (generated images and target images) to obtain feature expressions at different levels, compare these feature expressions and calculate the differences between them. This usually includes weighting and normalizing the feature maps, and then calculating the Euclidean distance between them. The weighted sum of the feature differences at each level is used as the LPIPS value to measure the similarity between images, which can more accurately capture the differences in human visual perception. A smaller LPIPS value indicates a smaller visual difference between images.

[0050] In order to obtain more accurate and reliable FID and LPIPS, this application generates 5 samples for each laser image input in the test set, and then calculates the average error of the corresponding generated sample quality in the test set data.

[0051] This application selects several representative benchmark models from the field of cross-modal translation of visible light images, including Pix2pix, BigColor, CT2, ColorFormers, and DDColor. Pix2pix and BigColor are GAN-based models. CT2, ColorFormers, and DDColor are Transformer-based models.

[0052] Table 1 Quantitative evaluation scores of the proposed method and the baseline model

[0053] In the quantitative analysis, the present application calculated the FID and LPIPS values ​​of the selected comparison model and the present application model for comparison, and the comparison results are shown in Table 1. It can be seen from the table that the present application model has achieved the best performance in both FID and LPIPS scores, and has obvious advantages over other models.

[0054] This application conducted ablation experiments on the two conditional controls and self-attention mechanism in the model to verify the effectiveness of each design. This application conducted experiments on three situations: single conditional control, dual conditional control, and dual conditional control with self-attention mechanism. The experimental results are shown in Table 2.

[0055] Table 2 Quantitative results of ablation experiments

[0056] From the quantitative results of the ablation experiment, it can be seen in this application that when the Brownian bridge strategy is used to control the first condition, thanks to the integration of the condition into the diffusion process, the model can directly learn the mapping between domains, and the obtained diffusion model has been improved compared with other benchmark models in Table 2; on the basis of the first condition, the interpolation of the laser image is injected to enhance the model's ability to learn the edge features of the scene and the target, and the two indicators are also improved; finally, the self-attention mechanism is added to enable the model to better capture the intrinsic relationship between the scene and the target, and improve the color and edge recovery ability. The model effect achieves the best of the three.

[0057] This application explores the field of laser-visible light image translation and introduces a diffusion model to complete this new task. Inspired by the Brownian Bridge strategy, this application directly integrates the conditional image with the diffusion process, allowing the diffusion model to directly learn the mapping between the two domains. In order to further limit the randomness of the generation of the diffusion model, this application proposes a method of interpolation condition injection and adds a self-attention module to enhance the model's ability to learn the scene, color and edge features of the target in the image. Compared with the five baseline models, the method proposed in this application has achieved excellent performance.

[0058] Embodiment 2 An embodiment of the present invention provides a laser image translation system based on a dual-condition improved diffusion model, referring to Figure 2 As shown, including: Dataset construction module 10: used to collect visible light images and laser images collected using laser active imaging under the same field of view and shooting conditions, and perform data preprocessing on the laser images and visible light images to construct a laser-visible light image dataset; Model building module 20: used to build an improved diffusion model based on dual conditions to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset; The improved diffusion model based on dual conditions includes a forward diffusion process and a backward diffusion process; the forward diffusion process gradually adds random Gaussian noise to the visible light image, so that the visible light image is completely transformed into a standard Gaussian distribution, and introduces a Brownian bridge strategy in the noise diffusion process, so that the noise addition end point is changed from pure Gaussian noise to a laser image; The image translation module 30 is used to translate the laser image to be processed into a visible light image by using the improved diffusion model based on dual conditions.

[0059] In a specific embodiment of the present invention, the data set construction module 10 is used to: The laser image and the visible light image are sheared, and the sheared laser image and the visible light image are horizontally flipped and vertically flipped.

[0060] In a specific embodiment of the present invention, the model building module 20 includes: The back diffusion process uses a neural network model to fit the conditional probability distribution. In the denoising process, the laser image is interpolated and transformed using an interpolation conditional injection method and then uploaded to the neural network model together with the noisy image, so that the neural network model predicts the random Gaussian noise added in each step and restores it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

[0061] In a specific embodiment of the present invention, the loss function of the improved diffusion model based on dual conditions is determined according to the following formula: ; In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding feature after interpolation change, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, mt Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

[0062] In a specific embodiment of the present invention, the interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with the noisy image and inputting them into the neural network model to weaken the uncertainty of the diffusion model.

[0063] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0064] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although the preferred embodiments of the embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention. Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0065] The method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0066] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "one specific embodiment" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A laser image translation method based on a dual-condition improved diffusion model, characterized in that: include: Step S1: collecting visible light images and laser images, and performing data preprocessing on the laser images and visible light images to construct a laser-visible light image data set; Step S2: constructing an improved diffusion model based on dual conditions, and using the laser-visible light image dataset for training, so as to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset; The improved diffusion model based on dual conditions includes a forward diffusion process and a backward diffusion process; the forward diffusion process gradually adds random Gaussian noise to the visible light image and introduces a Brownian bridge strategy in the noise diffusion process to change the noise addition end point from pure Gaussian noise to a laser image; Step S3: using the improved diffusion model based on dual conditions, translating the laser image to be processed into a visible light image.

2. The laser image translation method based on the dual-condition improved diffusion model according to claim 1 is characterized in that: Step S1 includes: The laser image and the visible light image are sheared, and the sheared laser image and the visible light image are horizontally flipped and vertically flipped.

3. The laser image translation method based on the dual-condition improved diffusion model according to claim 1 is characterized in that: In step S2, the backward diffusion process uses a neural network model to fit the conditional probability distribution. In the denoising process, the laser image is interpolated and transformed using an interpolation conditional injection method and then combined with the noisy image and uploaded to the neural network model, so that the neural network model predicts the random Gaussian noise added in each step and restores it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

4. The laser image translation method based on the dual-condition improved diffusion model according to claim 3 is characterized in that: The loss function of the improved diffusion model based on dual conditions is determined according to the following formula: ； In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding feature after interpolation change, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, m t Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

5. The laser image translation method based on the dual-condition improved diffusion model according to claim 3 is characterized in that: The interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with a noisy image and inputting them into a neural network model to weaken the uncertainty of the diffusion model.

6. A laser image translation system based on a dual-condition improved diffusion model, characterized in that: include: Dataset construction module: used for collecting visible light images and laser images, and performing data preprocessing on the laser images and visible light images to construct a laser-visible light image dataset; Model building module: used to build an improved diffusion model based on dual conditions to learn the mapping between the laser image domain and the visible light image domain in the laser-visible light image dataset; The improved diffusion model based on dual conditions includes a forward diffusion process and a backward diffusion process; the forward diffusion process gradually adds random Gaussian noise to the visible light image and introduces a Brownian bridge strategy in the noise diffusion process to change the noise addition end point from pure Gaussian noise to a laser image; Image translation module: used for translating the laser image to be processed into a visible light image by using the improved diffusion model based on dual conditions.

7. The laser image translation system based on the dual-condition improved diffusion model according to claim 6 is characterized in that: The dataset building blocks are used to: The laser image and the visible light image are sheared, and the sheared laser image and the visible light image are horizontally flipped and vertically flipped.

8. The laser image translation system based on the dual-condition improved diffusion model according to claim 6 is characterized in that: The model building module includes: The backward diffusion process uses a neural network model to fit the conditional probability distribution. In the denoising process, the laser image is interpolated and transformed using an interpolation conditional injection method, and then combined with the noisy image and uploaded to the neural network model, so that the neural network model predicts the random Gaussian noise added in each step and restores it to a visible light image, thereby learning the mapping between the laser image domain and the visible light image domain.

9. The laser image translation system based on the dual-condition improved diffusion model according to claim 8 is characterized in that: The loss function of the improved Brownian bridge diffusion model based on dual conditions is determined according to the following formula: ； In the formula, represents the added random noise, and y represents the mean value of the laser image; represents the encoding features after interpolation changes, x 0 represents the mean of the visible light image, t represents the time step, x t represents the noisy image at the tth time step, m t Represents the ratio of the current time step t to the total time step T, represents the noise distribution that the neural network needs to predict, It represents the coefficients remaining after simplification of the posterior probability distribution in the denoising process. Represents the variational lower bound that needs to be optimized during training.

10. The laser image translation system based on the dual-condition improved diffusion model according to claim 8, characterized in that: The interpolation condition injection method includes: encoding the laser image through a bicubic interpolation operation, and then combining the encoded image with a noisy image and inputting them into a neural network model to weaken the uncertainty of the diffusion model.

Images