Space image generation method and system based on visible light and thermal imaging

Abstract

The invention discloses a space image generation method and system based on visible light and thermal imaging, and relates to the technical field of image generation, and the method comprises the steps: obtaining thermal imaging image data, visible light gray data, depth map data and event camera data as collection data, and carrying out the coordinate system unification and space alignment; patch segmentation is carried out on the collected data, and position codes are added; according to the method, the global attention distribution is calculated by adopting ViT, so that each Patch can pay attention to all other Patches, it is ensured that the characteristics of cross-modal data are not only influenced by local neighborhoods, but the characteristics of the cross-modal data are combined with the information of the whole image, and a comprehensive characteristic matrix is calculated, so that the cross-modal data are obtained. The use of linear transformation can ensure that the final fusion feature comprises global information of ViT calculation, and the spatial attention weight obtained through calculation directly acts on the channel enhancement feature matrix, so that the expression of the spatial feature is further optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method and system for generating a space image based on visible light and thermal imaging. Background Art

[0002] With the development of computer vision, remote sensing detection, autonomous driving, and medical imaging, multimodal data fusion technology has received widespread attention. Among them, visible light and thermal imaging data fusion technology is a key means to improve environmental perception, target recognition accuracy, and adaptability to harsh environments. Existing multimodal imaging technologies usually rely on a single or a small number of sensors and use methods based on traditional image registration and feature fusion for information integration.

[0003] However, there are still many limitations in processing high-dynamic scenes, time synchronization of different modal data, spatial alignment and feature fusion. For example, traditional image registration methods mainly rely on feature point matching or spatial mapping based on geometric transformation, which is difficult to cope with perspective differences and sensor distortion in complex scenes. At the same time, due to the different physical imaging principles of visible light images and infrared images, direct pixel-level fusion is prone to information redundancy or modality mismatch. In addition, most of the existing multimodal feature extraction methods are based on convolutional neural networks (CNNs), whose local receptive fields limit the ability to model global information and make it difficult to fully utilize the complementary characteristics of different modal data. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a spatial image generation method based on visible light and thermal imaging to solve the many limitations in processing high-dynamic scenes, time synchronization of different modal data, spatial alignment and feature fusion. It is difficult to cope with the perspective difference and sensor distortion in complex scenes. At the same time, due to the different physical imaging principles of visible light images and infrared images, direct pixel-level fusion is prone to information redundancy or modality mismatch. In addition, most of the existing multimodal feature extraction methods are based on convolutional neural networks (CNNs), and their local receptive fields limit the ability to model global information, making it difficult to fully utilize the complementary characteristics of different modal data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for generating a space image based on visible light and thermal imaging, which comprises:

[0008] Acquire thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and perform unified coordinate system and spatial alignment;

[0009] The collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features.

[0010] The convolutional block attention mechanism (CBAM) is used to calculate the channel attention weight and spatial attention weight of the multimodal features to obtain channel enhancement features. The 3D position is output based on the MLP neural network to predict the color. NeRF is used to calculate the light color value and generate a 3D feature matrix. The deep separable convolution is used to reduce the feature dimension and extract the multi-scale feature matrix.

[0011] A generative adversarial network is used for the final mapping output of the fused image, which is losslessly compressed and encrypted for secure transmission.

[0012] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the collected data is segmented into patches, position encoding is added, the attention score matrix is ​​calculated using the ViT network, multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features, including:

[0013] The ViT network is used to segment the spatially aligned collected data into patches, and each patch is linearly transformed into a feature vector of fixed length.

[0014] According to the feature vector sequence composed of all patches, the number of feature channels is determined;

[0015] Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding Patch to form a Patch sequence with position information;

[0016] Calculate the query, key, and value according to the Patch sequence, and calculate the dot product of the query and the key to obtain the attention score matrix, which represents the attention weight of each patch to all patches;

[0017] Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted Value calculation on the normalized attention weight matrix to calculate the global features;

[0018] Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix;

[0019] Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, the batch size, and the number of channels.

[0020] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, wherein: the channel attention weight and spatial attention weight of the multimodal feature are calculated using the convolutional block attention mechanism CBAM to obtain the channel enhancement feature, including:

[0021] Using the convolutional block attention mechanism (CBAM), global average pooling and global maximum pooling are calculated based on the multimodal features of each patch to obtain the average pooling feature and the maximum pooling feature, respectively. The channel attention weight is calculated through a network consisting of two fully connected layers.

[0022] The calculated channel attention weights will act on the multimodal features calculated by ViT to obtain an enhanced feature matrix;

[0023] The channel average pooling and channel maximum pooling are calculated according to the enhanced feature matrix corresponding to the number of feature channels, and the spatial attention weight is calculated through the convolution kernel;

[0024] The calculated channel attention weights will be applied to the enhanced feature matrix to obtain channel enhanced features.

[0025] As a preferred solution of the spatial image generation method based on visible light and thermal imaging of the present invention, wherein: the 3D position prediction color is output based on the MLP neural network, the light color value is calculated using NeRF, and a 3D feature matrix is ​​generated, and a deep separable convolution is used to perform feature dimensionality reduction, and a multi-scale feature matrix is ​​extracted, including:

[0026] Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix;

[0027] Construct a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer;

[0028] According to the channel enhancement features, a multi-layer perceptron MLP is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color;

[0029] Based on the 3D projection data, for each light direction r, the channel enhancement feature calculates the volume density and predict the color to determine the color values ​​of different lights, and generate a 3D feature matrix through NeRF training;

[0030] The deep separable convolution (DSConv) is used to perform channel-independent convolution on the 3D feature matrix generated by NeRF training in the spatial dimension, and feature dimensionality reduction is performed by performing point convolution to extract a multi-scale feature matrix.

[0031] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the use of a generative adversarial network for final mapping to output a fused image refers to using a generative adversarial network for final mapping based on a multi-scale feature matrix, the generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation to upsample and reconstruct to generate a fused image, and the discriminator D uses a pre-trained convolutional neural network to perform authenticity prediction, and the generative adversarial network is trained based on minimizing the error of the discriminator D. After the training is completed, the final fused image is output through the input multi-scale feature matrix.

[0032] As a preferred solution of the method for generating a spatial image based on visible light and thermal imaging of the present invention, the method of acquiring thermal imaging image data, visible light grayscale data, depth map data and event camera data as collected data and performing unified coordinate system and spatial alignment includes:

[0033] Based on the visible light camera to collect visible light image data, the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients to obtain the grayscale value of a single channel;

[0034] The infrared thermal imaging image data is collected based on the thermal imaging sensor to remove noise, and the event camera data is collected based on the event camera, and time-aligned with the infrared thermal imaging image data and visible light grayscale data;

[0035] Perform noise filtering based on the depth map data collected by the LiDAR sensor, and perform normalization processing on the collected data including thermal imaging image data, visible light grayscale data, depth map data, and event camera data;

[0036] Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

[0037] As a preferred solution of the spatial image generation method based on visible light and thermal imaging described in the present invention, the lossless compression of the fused image and the encrypted secure transmission refer to using the lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, securely transmitting the data to the cloud storage through the SFTP protocol, generating a fused image storage log according to the storage location, and performing terminal local storage.

[0038] In a second aspect, the present invention provides a system for a space image generation method based on visible light and thermal imaging, comprising:

[0039] The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, event camera data, and performing data processing and unified coordinate system alignment;

[0040] The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features;

[0041] The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM);

[0042] The multi-scale feature module uses NeRF to calculate the color value of light, generate a 3D volume feature matrix, and uses deep separable convolution DSConv to reduce the dimension and extract multi-scale features;

[0043] The fusion image generation module uses GAN to synthesize the final image and maps the multi-scale feature matrix into a high-quality fusion image;

[0044] The secure transmission module is used to store, losslessly compress, securely encrypt and transmit the fused image.

[0045] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for generating a spatial image based on visible light and thermal imaging as described in the first aspect of the present invention is implemented.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for generating a spatial image based on visible light and thermal imaging as described in the first aspect of the present invention is implemented.

[0047] The beneficial effects of the present invention are as follows: by using ViT to calculate the global attention distribution, each patch can pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but are combined with the information of the entire image. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fusion feature contains both the global information calculated by ViT, and the calculated spatial attention weights directly act on the channel enhancement feature matrix to further optimize the expression of spatial features, making the key areas more recognizable in the multimodal features. By using a multi-layer perceptron MLP for nonlinear transformation, the channel enhancement features can learn color information through deep feature mapping to ensure that the predicted color can accurately reflect the light color characteristics of the 3D volume. By using GAN, it is ensured that the deep network can still retain key local information and texture details, so that the final generated fusion image is closer to the real visible light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0049] Figure 1 Schematic diagram of the flow of the method for generating a spatial image based on visible light and thermal imaging in Example 1.

[0050] Figure 2 This is a schematic diagram of the structure of the space image generation system based on visible light and thermal imaging in Example 2. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0054] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for generating a space image based on visible light and thermal imaging, comprising the following steps:

[0055] S1, obtaining thermal imaging image data, visible light grayscale data, depth map data and event camera data as collected data, and performing unified coordinate system and spatial alignment;

[0056] Preferably, thermal imaging image data, visible light grayscale data, depth map data, and event camera data are obtained as collected data, and unified coordinate system and space alignment are performed, including:

[0057] Based on the visible light camera to collect visible light image data, the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients, that is, the red channel is multiplied by 0.299, the green channel is multiplied by 0.587, and the blue channel is multiplied by 0.114 to obtain the grayscale value of a single channel;

[0058] The infrared thermal imaging image data is collected based on the thermal imaging sensor to remove noise, and the event camera data is collected based on the event camera (DVS sensor), and time-aligned with the infrared thermal imaging image data and visible light grayscale data;

[0059] Based on the depth map data collected by the LiDAR sensor, noise filtering is performed, the mean and standard deviation of the point cloud data are calculated using a statistical filtering method, and outliers exceeding the set threshold are removed. The collected data including thermal imaging image data, visible light grayscale data, depth map data, and event camera data are normalized;

[0060] Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

[0061] By converting RGB data into grayscale images, color redundant information can be effectively removed, making the information structure of visible light data more consistent with infrared and depth images. Noise reduction of thermal imaging data during the data acquisition stage can reduce interference in low-signal areas and improve the contrast of target areas. By time-aligning event camera data with other modal data, it is ensured that different modal data are analyzed within the same time frame. Statistical filtering (calculating the mean and standard deviation of point cloud data and removing abnormal points) can reduce noise points caused by sensor errors or environmental interference, ensuring that the depth information collected by LiDAR is more accurate. Obtaining the homography transformation matrix through camera calibration and projecting all collected data into a unified global reference coordinate system can ensure that the geometric structures of different modal data are aligned and avoid fusion errors caused by coordinate mismatch. By calculating the field of view difference and adjusting the data resolution using bilinear interpolation, it is ensured that different modal data are calculated at the same spatial resolution and avoid information loss caused by scale mismatch.

[0062] S2, perform patch segmentation on the collected data, add position encoding, use ViT network to calculate the attention score matrix, perform multi-head attention calculation to form a comprehensive feature matrix, and perform linear transformation to obtain multimodal features;

[0063] Preferably, the collected data is segmented into patches and position codes are added, the ViT network is used to calculate the attention score matrix, multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features, including:

[0064] The ViT network is used to perform patch segmentation on the spatially aligned collected data, and each patch is linearly transformed into a feature vector of fixed length, expressed as:

[0065] ;

[0066] in represents the feature vector of the ith Patch, Represents the i-th Patch, (shape is ), represents the Patch bias term, Represents the Patch weight matrix (shape is ), weights and biases can be determined through training;

[0067] According to the feature vector sequence composed of all patches, the number of feature channels is determined, which is expressed as:

[0068] ;

[0069] Where d represents the number of feature channels after Patch transformation, which is a hyperparameter in ViT network design and is usually 384, 512, 768 or 1024. N represents the number of patches.

[0070] Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding Patch to form a Patch sequence with position information;

[0071] Calculate the query, key, and value according to the Patch sequence, and calculate the dot product of the query and the key to obtain the attention score matrix, which represents the attention weight of each patch to all patches;

[0072] Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted Value calculation on the normalized attention weight matrix to calculate the global features;

[0073] Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix;

[0074] Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, the batch size, and the number of channels.

[0075] Through patch segmentation and position encoding, data from different modalities are aligned consistently at the feature level, solving the problem of modal misalignment caused by the traditional CNN's reliance on fixed receptive fields. Since ViT can calculate attention distribution based on the global patch, even if the input data comes from different sensors (visible light, infrared, LiDAR, etc.), they can still be aligned in the same feature space after linear transformation, ensuring that key information is not lost when data from different modalities are fused;

[0076] Using ViT to calculate the global attention distribution can make each patch pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but also combined with the information of the entire image. The multi-head attention mechanism (MHA) allows ViT to simultaneously calculate multiple feature representations (such as texture, shape, edge information, etc.), and perform Softmax normalization to ensure that effective detail information can be retained when different modal data are fused. Linear transformation and bias terms are used to directly convert each patch into a vector of fixed length. Compared with CNN, which needs to gradually extract features through multiple layers of convolution, this method can map data to a high-dimensional space at one time, reduce computational complexity, improve feature conversion efficiency, and improve the flexibility of the model to adapt to different computing environments. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fused features contain both the global information calculated by ViT and the scale of the original feature space, ensuring the stability of the fused features in terms of information integrity, feature scale and computational efficiency.

[0077] S3, uses the convolutional block attention mechanism CBAM to calculate the channel attention weight and spatial attention weight of the multimodal features, obtains the channel enhanced features, outputs the 3D position prediction color based on the MLP neural network, uses NeRF to calculate the light color value, and generates a 3D feature matrix. It uses deep separable convolution to perform feature dimensionality reduction and extract a multi-scale feature matrix.

[0078] Preferably, the convolutional block attention mechanism CBAM is used to calculate the channel attention weight and spatial attention weight of the multimodal feature to obtain the channel enhancement feature, including:

[0079] Using the convolutional block attention mechanism CBAM, global average pooling and global maximum pooling calculations are performed based on the multimodal features of each patch. The average feature distribution and the strongest response feature are extracted from the perspective of global features, and the average pooling feature and the maximum pooling feature are obtained respectively. The channel attention weight is calculated through a network composed of two fully connected layers, which is expressed as:

[0080] ;

[0081] in represents the channel attention weights of multimodal features, represents the Sigmoid function, and Respectively represent parameter weights (calculated through training), represents the average pooling value, Represents the maximum pooling value;

[0082] The calculated channel attention weights will act on the multimodal features calculated by ViT to obtain an enhanced feature matrix, so that before the image is fused, the important feature channels will be enhanced, while the influence of irrelevant or redundant channels will be weakened, which is expressed as:

[0083] ;

[0084] in represents the enhanced feature matrix, represents the channel attention weight, Represents multimodal features;

[0085] The channel average pooling and channel maximum pooling are calculated according to the enhanced feature matrix corresponding to the number of feature channels, and the spatial attention weight is calculated through the convolution kernel, which is expressed as:

[0086] ;

[0087] in Represents the spatial attention weights of multimodal features, express The convolution kernel size is represents the channel average pooling value, Indicates the maximum pooling value of the channel;

[0088] The calculated channel attention weights will act on the enhanced feature matrix to obtain the channel enhanced features, expressed as:

[0089] ;

[0090] in Represents channel enhancement characteristics.

[0091] The overall feature distribution is extracted through global average pooling to obtain the mean information of global features. The strongest response feature is selected through global maximum pooling to strengthen the area with high activation. The important channels are enhanced, making the key features more prominent in subsequent calculations, improving the representation ability of channel features, and reducing the interference of noise features. The channel attention weight is calculated using a two-layer fully connected network, which can dynamically adjust the importance of each channel based on the training data, so that the model can adapt to the feature distribution of different data modalities. The calculated channel attention weight directly acts on the multimodal features calculated by ViT to enhance the features of important channels and reduce the influence of irrelevant or redundant channels, so that the final fused features are more accurate in information expression. Compared with directly using ViT output features, this method performs channel optimization before fusion to ensure that the model pays more attention to key information, thereby improving the quality of the final fused image. The spatial attention is calculated using channel average pooling and channel maximum pooling, which can make the spatial attention calculation fully consider the comprehensive information of all channels, rather than relying solely on a single modality.

[0092] The calculated spatial attention weights are directly applied to the channel enhancement feature matrix to further optimize the expression of spatial features, making the key areas more recognizable in the multimodal features, thereby improving the quality of subsequent fused images. Through the joint calculation of channel attention and spatial attention, the final channel enhancement features have stronger semantic expression ability and spatial consistency, ensuring that the fused image can accurately retain the effective part of the multimodal information in the final generation process, while reducing the inconsistency between modalities and improving the visual clarity and effectiveness of the fused image. Compared with using only ViT output features for fusion, this method ensures full optimization before information interaction, making the final fusion effect more accurate and robust.

[0093] Furthermore, based on the output of the MLP neural network, the 3D position is predicted, the light color value is calculated using NeRF, and a 3D feature matrix is ​​generated. The feature dimension is reduced using deep separable convolution, and a multi-scale feature matrix is ​​extracted, including:

[0094] Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix, including the camera's rotation R and the camera's translation T;

[0095] Construct a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer;

[0096] According to the channel enhancement features, a multi-layer perceptron MLP is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color;

[0097] Based on the 3D projection data, for each light direction r, the channel enhancement feature calculates the volume density The color values ​​of different lights are determined by the predicted colors, and a 3D feature matrix is ​​generated through NeRF training, which is expressed as:

[0098] ;

[0099] in Represents the final color value of the light direction r, and They represent the farthest depth that the light leaves the 3D volume and the closest depth that the light enters the 3D volume. represents the volume density of ray r at depth t, represents the predicted color of ray r at depth t, represents the transmittance from the starting point of the ray r to the depth t;

[0100] Using deep separable convolution DSConv, the 3D feature matrix generated by NeRF training is convolved channel-independently in the spatial dimension, expressed as:

[0101] ;

[0102] in Represents the feature map after deep convolution calculation, Indicates the total number of channels, represents the depth convolution kernel for processing channel i, Represents the data of channel i of the 3D feature matrix;

[0103] Feature dimension reduction is performed by performing point convolution to extract a multi-scale feature matrix.

[0104] Through 3D projection calculation based on depth map, the camera's intrinsic parameter matrix K and extrinsic parameter matrix are used to convert 2D acquisition data into point clouds in the 3D coordinate system to ensure that the spatial positions of all modal data can be aligned. Multi-layer perceptron MLP is used for nonlinear transformation, which allows channel enhancement features to learn color information through deep feature mapping to ensure that the predicted color can accurately reflect the light color characteristics of the 3D volume. NeRF is used to calculate the light color value, and the light transmittance and color accumulation in the 3D volume can be calculated based on the volume density and predicted color, so that the final 3D feature matrix not only contains global spatial information, but also can accurately represent the light color changes at different depths. DSConv is used for feature dimensionality reduction. Through deep convolution and point convolution, the computational overhead of the 3D feature matrix can be effectively reduced. It can reduce the amount of calculation while retaining cross-channel information interaction, ensuring that the final feature expression can still retain multi-scale information. The final multi-scale feature matrix allows the model to focus on global features and local details at the same time, improve the information expression ability of fused data at different levels, and provide more accurate feature input for the final fused image generation.

[0105] S4, using the generative adversarial network to perform the final mapping output of the fused image, performing lossless compression of the fused image, and performing encrypted and secure transmission;

[0106] Preferably, using a generative adversarial network for final mapping to output a fused image means using a generative adversarial network for final mapping based on a multi-scale feature matrix, wherein the generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation to perform upsampling reconstruction to generate a fused image (ensuring that the resolution of the generated image is the same as the original input image), and uses a pre-trained convolutional neural network through the discriminator D to perform authenticity prediction, and the generative adversarial network is trained with the goal of minimizing the error of the discriminator D. After the training is completed, the final fused image is output through the input multi-scale feature matrix.

[0107] By using the ResNet residual block to transform the multi-scale feature matrix, the gradient vanishing problem can be effectively avoided, ensuring that the deep network can still retain key local information and texture details. Bilinear interpolation is used for upsampling reconstruction to ensure that the resolution of the generated image is the same as the original input image, ensuring that spatial structure information is not lost during the fusion process. The GAN training objective, that is, minimizing the error of the discriminator D, can ensure that the generator G is gradually optimized, so that the final generated fusion image is closer to the real visible light image, while still retaining the key areas of the infrared data.

[0108] Furthermore, lossless compression of the fused image and encrypted secure transmission refer to using lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, securely transmitting data through the SFTP protocol to the cloud for storage, generating a fused image storage log according to the storage location, and performing terminal local storage.

[0109] By adopting TIFF LZW lossless compression, the file size of the fused image can be effectively reduced while ensuring that the data will not lose its original quality due to compression. Using AES-256 to encrypt the fused image file can prevent unauthorized access to the data during storage and transmission, ensuring that confidential information will not be leaked. Using the key management system (KMS) for key management can dynamically control the access rights of the encryption key to ensure that only authorized users or system processes can decrypt the fused image data. Using SFTP for end-to-end encrypted data transmission can prevent the data from being intercepted, tampered with or forged during transmission. Storing fused image data in the cloud can improve the accessibility and durability of the data. At the same time, combined with storage log records, the storage location, access time, download and modification history of the data can be tracked.

[0110] This embodiment also provides a system for a method for generating a space image based on visible light and thermal imaging, comprising:

[0111] The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, event camera data, and performing data processing and unified coordinate system alignment;

[0112] The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features;

[0113] The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM);

[0114] The multi-scale feature module uses NeRF to calculate the color value of light, generate a 3D volume feature matrix, and uses deep separable convolution DSConv to reduce the dimension and extract multi-scale features;

[0115] The fusion image generation module uses GAN to synthesize the final image and maps the multi-scale feature matrix into a high-quality fusion image;

[0116] The secure transmission module is used to store, losslessly compress, securely encrypt and transmit the fused image.

[0117] This embodiment also provides a computer device, which is applicable to the case of a space image generation method based on visible light and thermal imaging, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the space image generation method based on visible light and thermal imaging proposed in the above embodiment.

[0118] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0119] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating a space image based on visible light and thermal imaging as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a disk or an optical disk.

[0120] In summary, the present invention uses ViT to calculate the global attention distribution, so that each patch can pay attention to all other patches, ensuring that the features of cross-modal data are not only affected by the local neighborhood, but combined with the information of the entire image. By calculating the comprehensive feature matrix and using linear transformation, it can be ensured that the final fusion feature contains both the global information calculated by ViT, and the calculated spatial attention weights are directly applied to the channel enhancement feature matrix to further optimize the expression of spatial features, so that the key areas are more recognizable in the multimodal features. By using a multi-layer perceptron MLP for nonlinear transformation, the channel enhancement features can learn color information through deep feature mapping to ensure that the predicted color can accurately reflect the light color characteristics of the 3D volume. By using GAN, it is ensured that the deep network can still retain key local information and texture details, so that the final generated fusion image is closer to the real visible light image.

[0121] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for generating a space image based on visible light and thermal imaging, characterized in that: include: Acquire thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data, and perform unified coordinate system and spatial alignment; The collected data is segmented into patches and position codes are added. The ViT network is used to calculate the attention score matrix. Multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features. The convolutional block attention mechanism (CBAM) is used to calculate the channel attention weight and spatial attention weight of the multimodal features to obtain channel enhancement features. The 3D position is output based on the MLP neural network to predict the color. NeRF is used to calculate the light color value and generate a 3D feature matrix. The deep separable convolution is used to reduce the feature dimension and extract the multi-scale feature matrix. A generative adversarial network is used for the final mapping output of the fused image, which is losslessly compressed and encrypted for secure transmission.

2. The method for generating a space image based on visible light and thermal imaging according to claim 1, characterized in that: The collected data is segmented into patches, and position encoding is added. The ViT network is used to calculate the attention score matrix, multi-head attention calculation is performed to form a comprehensive feature matrix, and linear changes are performed to obtain multimodal features, including: The ViT network is used to segment the spatially aligned collected data into patches, and each patch is linearly transformed into a feature vector of fixed length. According to the feature vector sequence composed of all patches, the number of feature channels is determined; Calculate the position code for each patch using sine and cosine functions , add the position code to the corresponding Patch to form a Patch sequence with position information; Calculate the query, key, and value according to the Patch sequence, and calculate the dot product of the query and the key to obtain the attention score matrix, which represents the attention weight of each patch to all patches; Perform Softmax normalization on the attention score matrix so that the sum of the attention weights of each patch is 1, and perform weighted Value calculation on the normalized attention weight matrix to calculate the global features; Perform multi-head attention calculation, copy Query, Key and Value into h parts, calculate the attention score of each attention head respectively, and perform Softmax normalization to obtain the output after multiple attention weights , the outputs of all attention heads form a comprehensive feature matrix; Perform a linear transformation on the concatenated comprehensive feature matrix and map the feature dimension back to the original dimension to obtain the multimodal feature F that integrates global information, including the spatial size of the feature map, the batch size, and the number of channels.

3. The method for generating a space image based on visible light and thermal imaging according to claim 2, characterized in that: The convolutional block attention mechanism CBAM is used to calculate the channel attention weight and spatial attention weight of the multimodal features to obtain the channel enhancement features, including: Using the convolutional block attention mechanism (CBAM), global average pooling and global maximum pooling are calculated based on the multimodal features of each patch to obtain the average pooling feature and the maximum pooling feature, respectively. The channel attention weight is calculated through a network consisting of two fully connected layers. The calculated channel attention weights will act on the multimodal features calculated by ViT to obtain an enhanced feature matrix; The channel average pooling and channel maximum pooling are calculated according to the enhanced feature matrix corresponding to the number of feature channels, and the spatial attention weight is calculated through the convolution kernel; The calculated channel attention weights will be applied to the enhanced feature matrix to obtain channel enhanced features.

4. The method for generating a space image based on visible light and thermal imaging as claimed in claim 3, characterized in that: The method outputs 3D position prediction color based on MLP neural network, uses NeRF to calculate light color value, generates 3D feature matrix, uses deep separable convolution to reduce feature dimension, and extracts multi-scale feature matrix, including: Based on the depth map data, NeRF is used to perform 3D projection according to the camera's intrinsic parameter matrix K and extrinsic parameter matrix; Construct a pre-trained color prediction model based on the MLP neural network, including input layer, hidden layer and output layer; According to the channel enhancement features, a multi-layer perceptron MLP is used in the hidden layer to perform nonlinear transformation and output the 3D position prediction color; Based on the 3D projection data, for each light direction r, the channel enhancement feature calculates the volume density and predict the color to determine the color values ​​of different lights, and generate a 3D feature matrix through NeRF training; The deep separable convolution (DSConv) is used to perform channel-independent convolution on the 3D feature matrix generated by NeRF training in the spatial dimension, and feature dimensionality reduction is performed by performing point convolution to extract a multi-scale feature matrix.

5. The method for generating a space image based on visible light and thermal imaging according to claim 4, characterized in that: The use of a generative adversarial network for final mapping and outputting a fused image refers to using a generative adversarial network for final mapping based on a multi-scale feature matrix, wherein the generative adversarial network includes a generator G and a discriminator D, wherein the generator G uses a ResNet residual block to perform feature transformation on the multi-scale feature matrix, and uses bilinear interpolation to perform upsampling reconstruction to generate a fused image, and uses a pre-trained convolutional neural network through the discriminator D to perform authenticity prediction, and the generative adversarial network is trained with the goal of minimizing the error of the discriminator D. After the training is completed, the final fused image is output through the input multi-scale feature matrix.

6. The method for generating a space image based on visible light and thermal imaging as claimed in claim 5, characterized in that: The step of acquiring thermal imaging image data, visible light grayscale data, depth map data, and event camera data as collected data and performing unified coordinate system and spatial alignment includes: Based on the visible light camera to collect visible light image data, the red, green, and blue channel values ​​of each pixel are multiplied by fixed weighting coefficients to obtain the grayscale value of a single channel; The infrared thermal imaging image data is collected based on the thermal imaging sensor to remove noise, and the event camera data is collected based on the event camera, and time-aligned with the infrared thermal imaging image data and visible light grayscale data; Perform noise filtering based on the depth map data collected by the LiDAR sensor, and perform normalization processing on the collected data including thermal imaging image data, visible light grayscale data, depth map data, and event camera data; Perform perspective transformation on the collected data, obtain the corresponding homography transformation matrix through camera calibration, transform the coordinates of the collected data into a unified global reference coordinate system, calculate the field of view difference of the collected data, and use bilinear interpolation to adjust the data resolution to align the collected data in space.

7. The method for generating a space image based on visible light and thermal imaging according to claim 6, characterized in that: The lossless compression of the fused image and encrypted secure transmission refer to using the lossless compression technology TIFF LZW compression to compress the fused image file, using AES-256 to encrypt the compressed data, and using the key management system KMS for key protection, securely transmitting the data to the cloud storage through the SFTP protocol, generating a fused image storage log according to the storage location, and performing terminal local storage.

8. A system for generating a spatial image based on visible light and thermal imaging, based on the method for generating a spatial image based on visible light and thermal imaging according to any one of claims 1 to 7, characterized in that: include, The data processing module is responsible for acquiring thermal imaging image data, visible light grayscale data, depth map data, event camera data, and performing data processing and unified coordinate system alignment; The multimodal feature module is responsible for patch segmentation and position encoding of the aligned multimodal data, and calculates the attention matrix through ViT to obtain fusion features; The attention enhancement module optimizes the channel attention and spatial attention of the multimodal features calculated by ViT based on the convolutional block attention mechanism (CBAM); The multi-scale feature module uses NeRF to calculate the color value of light, generate a 3D volume feature matrix, and uses deep separable convolution DSConv to reduce the dimension and extract multi-scale features; The fusion image generation module uses GAN to synthesize the final image and maps the multi-scale feature matrix into a high-quality fusion image; The secure transmission module is used to store, losslessly compress, securely encrypt and transmit the fused image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating a spatial image based on visible light and thermal imaging according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a spatial image based on visible light and thermal imaging according to any one of claims 1 to 7 are implemented.

Images