Space image generation method and system based on visible light and thermal imaging

Abstract

The present invention discloses a method and system for generating spatial images based on visible light and thermal imaging, which relates to the field of image generation technology. The method comprises the following steps: obtaining thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, performing unified coordinate system and spatial alignment; performing patch segmentation on the collected data, and adding position encoding. The method of the present invention uses ViT to calculate the global attention distribution, so that each patch can pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but also combine the information of the entire image. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fusion feature contains the global information calculated by ViT. The spatial attention weights obtained by calculation directly act on the channel enhancement feature matrix, further optimizing the expression of spatial features.

Description

Technical Field

[0001] The present invention relates to the field of image generation technology, and in particular to a method and system for generating space images based on visible light and thermal imaging. Background Art

[0002] With the development of computer vision, remote sensing, autonomous driving, and medical imaging, multimodal data fusion technology has received widespread attention. Among them, visible light and thermal imaging data fusion technology is a key means to improve environmental perception capabilities, target recognition accuracy, and adaptability to harsh environments. Existing multimodal imaging technologies usually rely on a single or a small number of sensors and use methods based on traditional image registration and feature fusion for information integration.

[0003] However, there are still many limitations in processing high-dynamic scenes, temporal synchronization of different modal data, spatial alignment, and feature fusion. For example, traditional image registration methods mainly rely on feature point matching or spatial mapping based on geometric transformation, which is difficult to cope with perspective differences and sensor distortion in complex scenes. At the same time, due to the different physical imaging principles of visible light images and infrared images, direct pixel-level fusion is prone to information redundancy or modality mismatch. In addition, existing multimodal feature extraction methods are mostly based on convolutional neural networks (CNNs). Their local receptive field limits the ability to model global information, making it difficult to fully utilize the complementary characteristics of different modal data. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a spatial image generation method based on visible light and thermal imaging to solve the many limitations in processing high-dynamic scenes, time synchronization, spatial alignment and feature fusion of different modal data. It is difficult to cope with the perspective differences and sensor distortion in complex scenes. At the same time, due to the different physical imaging principles of visible light images and infrared images, direct pixel-level fusion is likely to lead to information redundancy or modality mismatch. In addition, most of the existing multimodal feature extraction methods are based on convolutional neural networks (CNNs), and their local receptive fields limit the ability to model global information, making it difficult to fully utilize the complementary characteristics of different modal data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for generating a spatial image based on visible light and thermal imaging, comprising:

[0008] Acquire thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and perform unified coordinate system and spatial alignment;

[0009] The collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear transformation is performed to obtain multimodal features.

[0010] The convolutional block attention mechanism (CBAM) is used to calculate the channel attention weights and spatial attention weights of multimodal features to obtain channel-enhanced features. The 3D position is output based on the MLP neural network to predict the color. The light color value is calculated using NeRF and a 3D feature matrix is ​​generated. The depthwise separable convolution is used to perform feature dimensionality reduction and extract a multi-scale feature matrix.

[0011] A generative adversarial network is used for the final mapping output of the fused image, which is losslessly compressed and encrypted for secure transmission.

[0012] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the collected data is segmented into patches and position codes are added, the attention score matrix is ​​calculated using the ViT network, a multi-head attention calculation is performed to form a comprehensive feature matrix, and a linear transformation is performed to obtain multimodal features, including:

[0013] The ViT network is used to segment the spatially aligned collected data into patches and linearly transform each patch into a feature vector of fixed length.

[0014] According to the feature vector sequence composed of all patches, the number of feature channels is determined;

[0015] Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding patch to form a patch sequence with position information;

[0016] Calculate the query, key, and value according to the patch sequence, and calculate the dot product of the query and key to obtain the attention score matrix, which represents the attention weight of each patch to all patches;

[0017] Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted value calculation on the normalized attention weight matrix to calculate the global features;

[0018] Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix;

[0019] Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, batch size, and number of channels.

[0020] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, wherein: the channel attention weight and spatial attention weight of the multimodal feature are calculated using the convolutional block attention mechanism CBAM to obtain the channel enhancement feature, including:

[0021] Using the convolutional block attention mechanism (CBAM), global average pooling and global maximum pooling are calculated based on the multimodal features of each patch to obtain the average pooling feature and maximum pooling feature respectively. The channel attention weight is calculated through a network consisting of two fully connected layers.

[0022] The calculated channel attention weights will be applied to the multimodal features calculated by ViT to obtain an enhanced feature matrix;

[0023] Calculate channel average pooling and channel maximum pooling according to the enhanced feature matrix corresponding to the number of feature channels, and calculate the spatial attention weight through the convolution kernel;

[0024] The calculated channel attention weights will be applied to the enhanced feature matrix to obtain channel enhanced features.

[0025] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, wherein: the 3D position prediction color is output based on the MLP neural network, the light color value is calculated using NeRF, and a 3D feature matrix is ​​generated. The feature dimension reduction is performed using depthwise separable convolution to extract the multi-scale feature matrix, including:

[0026] Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix;

[0027] Build a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer;

[0028] According to the channel enhancement features, a multi-layer perceptron (MLP) is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color.

[0029] Based on the 3D projection data for each light direction r, the channel enhancement feature calculates the volume density and predict the color to determine the color values ​​of different lights, and generate a 3D feature matrix through NeRF training;

[0030] Using depthwise separable convolution (DSConv), the 3D feature matrix generated by NeRF training is subjected to channel-independent convolution in the spatial dimension. Feature dimensionality reduction is performed by performing point convolution to extract a multi-scale feature matrix.

[0031] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the use of a generative adversarial network for final mapping to output a fused image refers to using a generative adversarial network for final mapping based on a multi-scale feature matrix. The generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation for upsampling and reconstruction to generate a fused image. The discriminator D uses a pre-trained convolutional neural network for authenticity prediction, and the generative adversarial network is trained with the goal of minimizing the error of the discriminator D. After the training is completed, the final fused image is output through the input multi-scale feature matrix.

[0032] As a preferred solution of the method for generating spatial images based on visible light and thermal imaging of the present invention, the method of acquiring thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data and performing unified coordinate system and spatial alignment includes:

[0033] Visible light image data is collected based on a visible light camera, and the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients to obtain the grayscale value of a single channel;

[0034] The infrared thermal imaging image data is collected based on the thermal imaging sensor to remove noise, and the event camera data is collected based on the event camera, and time-aligned with the infrared thermal imaging image data and visible light grayscale data;

[0035] Perform noise filtering on the depth map data collected by the LiDAR sensor and perform normalization on the collected data including thermal imaging image data, visible light grayscale data, depth map data, and event camera data;

[0036] Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

[0037] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the lossless compression of the fused image and encrypted secure transmission refers to using the lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, securely transmitting the data to cloud storage via the SFTP protocol, generating a fused image storage log based on the storage location, and performing local terminal storage.

[0038] In a second aspect, the present invention provides a system for generating a spatial image based on visible light and thermal imaging, comprising:

[0039] The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, and event camera data, and performing data processing and unified coordinate system alignment;

[0040] The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features;

[0041] The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM);

[0042] The multi-scale feature module uses NeRF to calculate the light color value, generate a 3D volume feature matrix, and use depthwise separable convolution (DSConv) for dimensionality reduction to extract multi-scale features.

[0043] The fusion image generation module uses GAN to perform final image synthesis and map the multi-scale feature matrix into a high-quality fusion image;

[0044] The secure transmission module performs storage, lossless compression, secure encryption and transmission of fused images.

[0045] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for generating a spatial image based on visible light and thermal imaging as described in the first aspect of the present invention is implemented.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for generating a spatial image based on visible light and thermal imaging as described in the first aspect of the present invention is implemented.

[0047] The beneficial effects of the present invention are: by using ViT to calculate the global attention distribution, each patch can pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but are combined with the information of the entire image. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fusion feature contains both the global information calculated by ViT, and the spatial attention weights obtained by calculation directly act on the channel enhancement feature matrix, further optimizing the expression of spatial features, making the key areas more recognizable in the multimodal features, and by using the multi-layer perceptron MLP for nonlinear transformation, the channel enhancement features can learn color information through deep feature mapping, ensuring that the predicted color can accurately reflect the light color characteristics of the 3D volume, and by using GAN to ensure that the deep network can still retain key local information and texture details, so that the final generated fusion image is closer to the real visible light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 Schematic diagram of the flow of the method for generating a spatial image based on visible light and thermal imaging in Example 1.

[0050] Figure 2 Schematic diagram of the structure of the space image generation system based on visible light and thermal imaging in Example 2. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0054] Example 1, with reference to Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for generating a space image based on visible light and thermal imaging, comprising the following steps:

[0055] S1, obtain thermal imaging image data, visible light grayscale data, depth map data and event camera data as collected data, and perform unified coordinate system and spatial alignment;

[0056] Preferably, obtaining thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and performing unified coordinate system and spatial alignment includes:

[0057] Based on the visible light camera to collect visible light image data, the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients respectively. The red channel can be multiplied by 0.299, the green channel by 0.587, and the blue channel by 0.114 to obtain the grayscale value of a single channel;

[0058] The thermal imaging sensor collects infrared thermal imaging image data for noise removal, and the event camera (DVS sensor) collects event camera data and performs time alignment with the infrared thermal imaging image data and visible light grayscale data.

[0059] Noise filtering is performed on the depth map data collected by the LiDAR sensor. The mean and standard deviation of the point cloud data are calculated using a statistical filtering method. Outliers exceeding a set threshold are removed. The collected data, including thermal imaging image data, visible light grayscale data, depth map data, and event camera data, are normalized.

[0060] Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

[0061] By converting RGB data into grayscale images, color redundant information can be effectively removed, making the information structure of visible light data more consistent with that of infrared and depth images. Noise reduction of thermal imaging data during the data acquisition stage can reduce interference in low-signal areas and improve the contrast of the target area. By temporally aligning event camera data with other modal data, different modal data can be analyzed within the same time frame. Statistical filtering (calculating the mean and standard deviation of point cloud data and removing outliers) can reduce noise points caused by sensor errors or environmental interference, ensuring that the depth information collected by LiDAR is more accurate. Obtaining the homography transformation matrix through camera calibration and projecting all collected data into a unified global reference coordinate system can ensure that the geometric structures of different modal data are aligned, avoiding fusion errors caused by coordinate mismatch. By calculating the field of view difference and adjusting the data resolution using bilinear interpolation, different modal data can be calculated at the same spatial resolution, avoiding information loss caused by scale mismatch.

[0062] S2, performs patch segmentation on the collected data, adds position encoding, uses the ViT network to calculate the attention score matrix, performs multi-head attention calculation to form a comprehensive feature matrix, and performs linear transformation to obtain multimodal features;

[0063] Preferably, the collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features, including:

[0064] The ViT network is used to perform patch segmentation on the spatially aligned collected data, and each patch is linearly transformed into a feature vector of fixed length, which is expressed as:

[0065] ;

[0066] in represents the feature vector of the i-th Patch, Represents the i-th Patch, (shape is ), represents the Patch bias term, Represents the Patch weight matrix, (shape is ), weights and biases can be determined through training;

[0067] According to the feature vector sequence composed of all patches, the number of feature channels is determined and expressed as:

[0068] ;

[0069] Where d represents the number of feature channels after Patch transformation, which is a hyperparameter in ViT network design and is usually 384, 512, 768 or 1024. N represents the number of patches.

[0070] Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding patch to form a patch sequence with position information;

[0071] Calculate the query, key, and value according to the patch sequence, and calculate the dot product of the query and key to obtain the attention score matrix, which represents the attention weight of each patch to all patches;

[0072] Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted value calculation on the normalized attention weight matrix to calculate the global features;

[0073] Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix;

[0074] Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, batch size, and number of channels.

[0075] Patch segmentation and position encoding ensure consistent alignment of data from different modalities at the feature level, solving the modality misalignment problem caused by traditional CNNs' reliance on fixed receptive fields. Because ViT can calculate attention distribution based on global patches, even if the input data comes from different sensors (visible light, infrared, LiDAR, etc.), it can still be aligned in the same feature space after linear transformation, ensuring that key information is not lost when fusing data from different modalities.

[0076] Using ViT to calculate the global attention distribution allows each patch to pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but also combined with the information of the entire image. The multi-head attention mechanism (MHA) allows ViT to simultaneously calculate multiple feature representations (such as texture, shape, edge information, etc.) and perform Softmax normalization to ensure that effective detail information can be retained when different modal data are fused. Linear transformation and bias terms are used to directly convert each patch into a fixed-length vector. Compared with CNN, which needs to gradually extract features through multiple layers of convolution, this method can map data to a high-dimensional space at one time, reduce computational complexity, improve feature conversion efficiency, and enhance the flexibility of the model to adapt to different computing environments. By calculating the comprehensive feature matrix, the use of linear transformation can ensure that the final fused features contain both the global information calculated by ViT and the scale of the original feature space, ensuring the stability of the fused features in terms of information integrity, feature scale and computational efficiency.

[0077] S3 uses the convolutional block attention mechanism (CBAM) to calculate the channel attention weights and spatial attention weights of the multimodal features to obtain channel-enhanced features. Based on the MLP neural network, the 3D position is output to predict the color. NeRF is used to calculate the light color value and generate a 3D feature matrix. Depthwise separable convolution is used to perform feature dimensionality reduction and extract a multi-scale feature matrix.

[0078] Preferably, the convolutional block attention mechanism (CBAM) is used to calculate the channel attention weight and spatial attention weight of the multimodal feature to obtain the channel enhanced feature, including:

[0079] Using the convolutional block attention mechanism (CBAM), global average pooling and global maximum pooling are calculated based on the multimodal features of each patch. The average feature distribution and the strongest response feature are extracted from the perspective of global features, respectively, and the average pooling feature and the maximum pooling feature are obtained. The channel attention weight is calculated through a network consisting of two fully connected layers, which is expressed as:

[0080] ;

[0081] in represents the channel attention weight of multimodal features, represents the Sigmoid function, and Represent parameter weights (calculated through training), represents the average pooling value, represents the maximum pooling value;

[0082] The calculated channel attention weights will be applied to the multimodal features calculated by ViT to obtain an enhanced feature matrix. This will achieve the effect that important feature channels will be enhanced and the influence of irrelevant or redundant channels will be weakened before the image is fused. It can be expressed as:

[0083] ;

[0084] in represents the enhanced feature matrix, represents the channel attention weight, Represents multimodal features;

[0085] The channel average pooling and channel maximum pooling are calculated according to the enhanced feature matrix corresponding to the number of feature channels, and the spatial attention weight is calculated through the convolution kernel, which is expressed as:

[0086] ;

[0087] in Represents the spatial attention weights of multimodal features, express The convolution kernel size, represents the channel average pooling value, Indicates the maximum pooling value of the channel;

[0088] The calculated channel attention weight will act on the enhanced feature matrix to obtain the channel enhanced feature, which is expressed as:

[0089] ;

[0090] in Represents channel enhancement features.

[0091] The overall feature distribution is extracted through global average pooling to obtain the mean information of global features. The strongest response features are selected through global maximum pooling, and the areas with high activation are strengthened. Important channels are enhanced, making key features more prominent in subsequent calculations, improving the representation ability of channel features, and reducing the interference of noise features. The channel attention weights are calculated using a two-layer fully connected network, which can dynamically adjust the importance of each channel based on the training data, so that the model can adapt to the feature distribution of different data modalities. The calculated channel attention weights are directly applied to the multimodal features calculated by ViT, enhancing the features of important channels while reducing the influence of irrelevant or redundant channels, making the final fused features more accurate in information expression. Compared with directly using ViT output features, this method performs channel optimization before fusion to ensure that the model pays more attention to key information, thereby improving the quality of the final fused image. The spatial attention is calculated using channel average pooling and channel maximum pooling, which allows the spatial attention calculation to fully consider the comprehensive information of all channels, rather than relying solely on a single modality.

[0092] The calculated spatial attention weights are directly applied to the channel-enhanced feature matrix, further optimizing the expression of spatial features and making key areas more recognizable within the multimodal features, thereby improving the quality of the subsequent fused image. By jointly calculating channel and spatial attention, the resulting channel-enhanced features possess stronger semantic expression and spatial consistency, ensuring that the fused image accurately retains the valid portion of multimodal information during the final generation process, while reducing inconsistencies between modalities and improving the visual clarity and effectiveness of the fused image. Compared to fusion using only ViT output features, this method ensures full optimization before information interaction, resulting in a more accurate and robust final fusion effect.

[0093] Furthermore, based on the output of the MLP neural network, the 3D position is predicted, the light color value is calculated using NeRF, and a 3D feature matrix is ​​generated. The feature dimension is reduced using depthwise separable convolution, and a multi-scale feature matrix is ​​extracted, including:

[0094] Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix, including the camera's rotation R and translation T;

[0095] Build a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer;

[0096] According to the channel enhancement features, a multi-layer perceptron (MLP) is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color.

[0097] Based on the 3D projection data for each light direction r, the channel enhancement feature calculates the volume density The color values ​​of different lights are determined by the predicted colors, and a 3D feature matrix is ​​generated through NeRF training, which is expressed as:

[0098] ;

[0099] in Represents the final color value of the light direction r, and The farthest depth that the light leaves the 3D volume and the closest depth that the light enters the 3D volume, respectively. represents the volume density of ray r at depth t, represents the predicted color of ray r at depth t, represents the transmittance from the starting point of the ray r to the depth t;

[0100] Using depth-wise separable convolution DSConv, the 3D feature matrix generated by NeRF training is subjected to channel-independent convolution in the spatial dimension, which can be expressed as:

[0101] ;

[0102] in Represents the feature map after depth convolution calculation, Indicates the total number of channels, represents the depth convolution kernel for processing channel i, Represents the data of channel i of the 3D feature matrix;

[0103] Feature dimensionality reduction is performed by performing point convolution to extract a multi-scale feature matrix.

[0104] Through 3D projection calculation based on the depth map, the camera's intrinsic parameter matrix K and extrinsic parameter matrix are used to convert the 2D acquired data into a point cloud in the 3D coordinate system, ensuring that the spatial positions of all modal data can be aligned. The multi-layer perceptron (MLP) is used for nonlinear transformation, which enables the channel enhancement features to learn color information through deep feature mapping, ensuring that the predicted color can accurately reflect the light color characteristics of the 3D volume. Using NeRF to calculate the light color value, the light transmittance and color accumulation in the 3D volume can be calculated based on the volume density and predicted color, so that the final 3D feature matrix not only contains global spatial information, but also can accurately represent the light color changes at different depths. DSConv is used for feature dimensionality reduction. Through deep convolution and point convolution, the computational overhead of the 3D feature matrix can be effectively reduced. It can retain cross-channel information interaction while reducing the amount of calculation, ensuring that the final feature expression can still retain multi-scale information. The final multi-scale feature matrix allows the model to focus on global features and local details at the same time, improve the information expression ability of the fused data at different levels, and provide more accurate feature input for the final fused image generation.

[0105] S4, uses the generative adversarial network to perform the final mapping and output the fused image, performs lossless compression on the fused image, and performs encrypted and secure transmission;

[0106] Preferably, using a generative adversarial network for final mapping to output a fused image means using a generative adversarial network for final mapping based on a multi-scale feature matrix, the generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation for upsampling and reconstruction to generate a fused image (ensuring that the resolution of the generated image is the same as the original input image), and uses a pre-trained convolutional neural network through the discriminator D to perform authenticity prediction, and the generative adversarial network is trained with the goal of minimizing the error of the discriminator D. After training is completed, the final fused image is output through the input multi-scale feature matrix.

[0107] By using ResNet residual blocks to transform the multi-scale feature matrix, the gradient vanishing problem can be effectively avoided, ensuring that the deep network can still retain key local information and texture details. Using bilinear interpolation for upsampling reconstruction can ensure that the resolution of the generated image is the same as the original input image, ensuring that spatial structure information is not lost during the fusion process. The GAN training objective, that is, minimizing the error of the discriminator D, can ensure that the generator G is gradually optimized, making the final generated fusion image closer to the real visible light image while still retaining the key areas of the infrared data.

[0108] Furthermore, lossless compression of the fused image and encrypted secure transmission refer to using lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, and securely transmitting data through the SFTP protocol to the cloud for storage, generating a fused image storage log according to the storage location, and performing local storage on the terminal.

[0109] By adopting TIFF LZW lossless compression, the file size of the fused image can be effectively reduced while ensuring that the data does not lose its original quality due to compression. Using AES-256 to encrypt the fused image file can prevent unauthorized access to the data during storage and transmission, ensuring that confidential information is not leaked. Using a key management system (KMS) for key management can dynamically control the access rights of encryption keys to ensure that only authorized users or system processes can decrypt the fused image data. Using SFTP for end-to-end encrypted data transmission can prevent data from being intercepted, tampered with, or forged during transmission. Storing fused image data in the cloud can improve data accessibility and durability. Combined with storage log records, the storage location, access time, download, and modification history of the data can be tracked.

[0110] This embodiment also provides a system for generating a spatial image based on visible light and thermal imaging, comprising:

[0111] The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, and event camera data, and performing data processing and unified coordinate system alignment;

[0112] The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features;

[0113] The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM);

[0114] The multi-scale feature module uses NeRF to calculate the light color value, generate a 3D volume feature matrix, and use depthwise separable convolution (DSConv) for dimensionality reduction to extract multi-scale features.

[0115] The fusion image generation module uses GAN to perform final image synthesis and map the multi-scale feature matrix into a high-quality fusion image;

[0116] The secure transmission module performs storage, lossless compression, secure encryption and transmission of fused images.

[0117] This embodiment also provides a computer device, which is applicable to the case of a space image generation method based on visible light and thermal imaging, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the space image generation method based on visible light and thermal imaging proposed in the above embodiment.

[0118] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0119] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the spatial image generation method based on visible light and thermal imaging as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0120] In summary, the present invention uses ViT to calculate the global attention distribution, so that each patch can pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but are combined with the information of the entire image. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fusion feature contains both the global information calculated by ViT, and the calculated spatial attention weights are directly applied to the channel enhancement feature matrix, further optimizing the expression of spatial features, making the key areas more recognizable in the multimodal features. By using the multi-layer perceptron MLP for nonlinear transformation, the channel enhancement features can learn color information through deep feature mapping, ensuring that the predicted color can accurately reflect the light color characteristics of the 3D volume. By using GAN, it is ensured that the deep network can still retain key local information and texture details, so that the final generated fusion image is closer to the real visible light image.

[0121] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for generating a spatial image based on visible light and thermal imaging, characterized in that: include: Acquire thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and perform unified coordinate system and spatial alignment; The collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear transformation is performed to obtain multimodal features. The convolutional block attention mechanism (CBAM) is used to calculate the channel attention weights and spatial attention weights of multimodal features to obtain channel-enhanced features. The 3D position is output based on the MLP neural network to predict the color. The light color value is calculated using NeRF and a 3D feature matrix is ​​generated. The depthwise separable convolution is used to perform feature dimensionality reduction and extract a multi-scale feature matrix. Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix, including the camera's rotation R and translation T; Build a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer; According to the channel enhancement features, a multi-layer perceptron (MLP) is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color. Based on the 3D projection data for each light direction r, the channel enhancement feature calculates the volume density The color values ​​of different lights are determined by the predicted colors, and a 3D feature matrix is ​​generated through NeRF training, which is expressed as: ； in Represents the final color value of the light direction r, and The farthest depth that the light leaves the 3D volume and the closest depth that the light enters the 3D volume, respectively. represents the volume density of ray r at depth t, represents the predicted color of ray r at depth t, represents the transmittance from the starting point of the ray r to the depth t; Using depth-wise separable convolution DSConv, the 3D feature matrix generated by NeRF training is subjected to channel-independent convolution in the spatial dimension, which can be expressed as: ； in Represents the feature map after depth convolution calculation, Indicates the total number of channels, represents the depth convolution kernel for processing channel i, Represents the data of channel i of the 3D feature matrix; Perform feature dimensionality reduction by performing point convolution and extract multi-scale feature matrix; A generative adversarial network is used for the final mapping output of the fused image, which is losslessly compressed and encrypted for secure transmission.

2. The method for generating a spatial image based on visible light and thermal imaging according to claim 1, wherein: The collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features, including: The ViT network is used to segment the spatially aligned collected data into patches and linearly transform each patch into a feature vector of fixed length. According to the feature vector sequence composed of all patches, the number of feature channels is determined; Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding patch to form a patch sequence with position information; Calculate the query, key, and value according to the patch sequence, and calculate the dot product of the query and key to obtain the attention score matrix, which represents the attention weight of each patch to all patches; Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted value calculation on the normalized attention weight matrix to calculate the global features; Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix; Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, batch size, and number of channels.

3. The method for generating a spatial image based on visible light and thermal imaging according to claim 2, wherein: The convolutional block attention mechanism CBAM is used to calculate the channel attention weight and spatial attention weight of the multimodal feature to obtain the channel enhanced feature, including: Using the convolutional block attention mechanism (CBAM), global average pooling and global maximum pooling are calculated based on the multimodal features of each patch to obtain the average pooling feature and maximum pooling feature respectively. The channel attention weight is calculated through a network consisting of two fully connected layers. The calculated channel attention weights will be applied to the multimodal features calculated by ViT to obtain an enhanced feature matrix; Calculate channel average pooling and channel maximum pooling according to the enhanced feature matrix corresponding to the number of feature channels, and calculate the spatial attention weight through the convolution kernel; The calculated channel attention weights will be applied to the enhanced feature matrix to obtain channel enhanced features.

4. The method for generating a spatial image based on visible light and thermal imaging according to claim 3, wherein: The use of a generative adversarial network for final mapping and outputting a fused image refers to using a generative adversarial network for final mapping based on a multi-scale feature matrix. The generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation for upsampling and reconstruction to generate a fused image. The discriminator D uses a pre-trained convolutional neural network for authenticity prediction, and the generative adversarial network is trained with the goal of minimizing the error of the discriminator D. After training, the final fused image is output through the input multi-scale feature matrix.

5. The method for generating a spatial image based on visible light and thermal imaging according to claim 4, wherein: The acquisition of thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and performing unified coordinate system and spatial alignment includes: Visible light image data is collected based on a visible light camera, and the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients to obtain the grayscale value of a single channel; The infrared thermal imaging image data is collected based on the thermal imaging sensor to remove noise, and the event camera data is collected based on the event camera, and time-aligned with the infrared thermal imaging image data and visible light grayscale data; Perform noise filtering on the depth map data collected by the LiDAR sensor and perform normalization on the collected data including thermal imaging image data, visible light grayscale data, depth map data, and event camera data; Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

6. The method for generating a spatial image based on visible light and thermal imaging according to claim 5, wherein: The lossless compression of the fused image and encrypted secure transmission refer to using the lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, securely transmitting the data to the cloud storage through the SFTP protocol, generating a fused image storage log according to the storage location, and performing local storage on the terminal.

7. A system for generating a spatial image based on visible light and thermal imaging, based on the method for generating a spatial image based on visible light and thermal imaging according to any one of claims 1 to 6, characterized in that: include, The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, and event camera data, and performing data processing and unified coordinate system alignment; The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features; The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM); The multi-scale feature module uses NeRF to calculate the light color value, generate a 3D volume feature matrix, and use depthwise separable convolution (DSConv) for dimensionality reduction to extract multi-scale features. The fusion image generation module uses GAN to perform final image synthesis and map the multi-scale feature matrix into a high-quality fusion image; The secure transmission module performs storage, lossless compression, secure encryption and transmission of fused images.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating a spatial image based on visible light and thermal imaging are implemented as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a spatial image based on visible light and thermal imaging according to any one of claims 1 to 6 are implemented.

Images