Physical data dual-drive short-wave infrared image colorization band optimization method and device

Abstract

This application relates to the field of shortwave infrared imaging technology, and provides a method and apparatus for optimizing color bands in shortwave infrared images driven by both physical data and technical data. Addressing the problem of exponentially increasing combinations of hyperspectral / multispectral bands, this method abandons the computationally expensive exhaustive training method and utilizes a hierarchical screening framework. First, a physical model is used to determine candidate bands. Then, improved statistical indices are used to quickly perform unsupervised dimensionality reduction to obtain a small number of high-potential combinations. Finally, generative validation of these high-potential combinations yields the optimal band combination. This strategy ensures the physical interpretability of band selection while significantly reducing the time cost and computational consumption in finding the optimal solution.

Description

Technical Field

[0001] This application relates to the field of shortwave infrared imaging technology, and in particular to a method and apparatus for optimizing the color band of shortwave infrared images using dual-drive physical data. Background Technology

[0002] Short-wave infrared (SWIR) imaging (usually referring to the 900-1700 nm band) plays an irreplaceable role in remote sensing monitoring, camouflage identification, and all-weather assisted driving due to its superior penetration ability of smoke and haze and its unique material reflection characteristics.

[0003] However, limited by sensor technology, SWIR images are typically presented in single-channel grayscale, lacking color information that the human eye is sensitive to, severely restricting scene interpretation efficiency and target recognition accuracy. Therefore, converting single-band SWIR grayscale images into color images that conform to human visual habits (i.e., image colorization) has become a research hotspot in the field of optoelectronic imaging. To address this issue, infrared image colorization methods based on deep learning, particularly Generative Adversarial Networks (GANs), have emerged. In recent years, with the rapid development of deep learning technology, GAN-based infrared image colorization methods have gradually replaced traditional lookup table methods and color transfer methods, becoming the mainstream technical approach.

[0004] Current GAN-based infrared image colorization methods suffer from several problems, including mapping uncertainty caused by the inconsistency between panchromatic and red-green-blue (RGB) data dimensions, a lack of proactive data optimization mechanisms, and a disconnect between structural and physical properties. Therefore, to address the color mapping ambiguity caused by mapping uncertainty in single-band SWIR imaging, a band optimization method suitable for generative colorization tasks is needed. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and apparatus for optimizing the color band of a shortwave infrared image driven by physical data, in order to solve the problem of color mapping ambiguity caused by mapping uncertainty in single-band SWIR imaging in the prior art.

[0006] A first aspect of this application provides a method for optimizing the colorization band of a shortwave infrared image using dual-drive physical data, comprising:

[0007] Physical feasibility analysis was performed on each band within the shortwave infrared band range to obtain N candidate bands. Among them, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each candidate band is greater than the preset score threshold, and the material discrimination is greater than the preset discrimination threshold; N is a positive integer.

[0008] Any M different bands from the candidate bands are combined, and the characteristics of each band combination are statistically analyzed. Based on the statistical characteristics of each band combination, the candidate band combination is determined. The statistical characteristics include the improved optimal index factor and structural consistency. M is a positive integer less than N.

[0009] The pre-trained deep generative network is trained using each band combination from the candidate band combinations, and the target band combination is obtained based on the image colorization quality of each trained deep generative network.

[0010] A second aspect of this application provides a physical data dual-drive shortwave infrared image colorization band selection device, comprising:

[0011] The physical screening module is configured to perform physical feasibility analysis on each band within the shortwave infrared band range to obtain N candidate bands; among them, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each band in the candidate band is greater than a preset score threshold, and the material discrimination is greater than a preset discrimination threshold; N is a positive integer;

[0012] The data filtering module is configured to combine any M different bands from the candidate bands, perform feature statistics on each band combination, and determine the candidate band combination based on the statistical characteristics of each band combination; wherein, the statistical characteristics include the improved optimal index factor and structural consistency; M is a positive integer less than N;

[0013] The determination module is configured to train a pre-trained deep generative network using each band combination from the candidate band combinations, and obtain the target band combination based on the image colorization quality of each trained deep generative network.

[0014] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0015] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] The beneficial effects of the embodiments in this application compared with the prior art are:

[0017] This application addresses the problem of an exponential explosion in hyperspectral / multispectral band combinations. It abandons the computationally expensive exhaustive training method and utilizes a hierarchical screening framework. First, a physical model is used to determine candidate bands. Then, improved statistical indicators are used to quickly perform unsupervised dimensionality reduction to obtain a small number of high-potential combinations. Finally, generative validation of these high-potential combinations yields the optimal band combinations. This strategy ensures the physical interpretability of band selection while significantly reducing the time cost and computational consumption in finding the optimal solution. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for selecting the color band of a physical data dual-drive shortwave infrared image, as provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram showing the statistical results of the energy mean and energy variance across the entire shortwave infrared spectrum.

[0021] Figure 3 This is a schematic diagram of the experimental apparatus for data collection provided in this application embodiment.

[0022] Figure 4 These are schematic diagrams of different scenarios.

[0023] Figure 5 This is a schematic diagram of the statistical scores of candidate band combinations in some scene types.

[0024] Figure 6 This is a schematic diagram showing the results of an experiment on image colorization processing using different bands or combinations of bands in an urban architectural scene.

[0025] Figure 7 This is a schematic diagram showing the results of an experiment in an industrial setting where different bands or combinations of bands were used to perform image colorization.

[0026] Figure 8 This is a schematic diagram showing the results of an experiment on image colorization processing using different bands or combinations of bands in a traffic scene.

[0027] Figure 9 This is a schematic diagram showing the results of an experiment in colorizing images in a rural setting using different bands or combinations of bands.

[0028] Figure 10 This is a flowchart illustrating another method for optimizing the color band of a physical data dual-drive shortwave infrared image, as provided in an embodiment of this application.

[0029] Figure 11 This is a schematic diagram of a preferred device for colorizing shortwave infrared images using dual-drive physical data, provided in an embodiment of this application.

[0030] Figure 12 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for selecting the color band of a physical data dual-drive shortwave infrared image according to an embodiment of this application.

[0033] As mentioned above, GAN-based infrared image colorization methods have become the mainstream technical approach. For example, at the level of network structure improvement, some solutions have proposed improved algorithms based on recurrent generative adversarial networks (GANs). By integrating the Res-ASPP-Unet module with a deep bottleneck layer, the network's ability to recognize local region features is enhanced, effectively improving color distortion. Here, the Res module is a residual module in the neural network, the ASPP module is an Atrous Spatial Pyramid Pooling module, and Unet is a convolutional neural network architecture used for image segmentation.

[0034] Meanwhile, to address the issue of miscoloration in local image regions, some solutions propose introducing a hollow global attention mechanism into the generator and optimizing the normalization layer of the discriminant network, thereby further improving the ability to preserve texture details in near-infrared image colorization.

[0035] Regarding multimodal fusion, to address the insufficient information content of a single infrared image, some solutions propose transferring visible light color features to infrared images to significantly enhance the visual perception of night vision surveillance. Other studies attempt to utilize dual-discriminator structures or multi-scale loss functions to constrain the structural consistency of the generated images.

[0036] However, current GAN-based infrared image colorization methods still have the following problems:

[0037] 1) Mapping uncertainty caused by the inconsistency between panchromatic and RGB data dimensions: Existing studies mostly use broadband or single-band infrared images directly as input. In the SWIR band, different materials (such as green vegetation, camouflage nets, and asphalt pavements) often exhibit extremely similar gray values ​​(metachromatic phenomenon). Adjusting the network structure alone (such as attention mechanisms or multi-scale fusion) cannot fundamentally supplement the spectral distinguishability at the physical level, leading to severe semantic confusion and color "illusion" during network inference.

[0038] 2) Lack of proactive data optimization mechanism: Most existing solutions focus on the iteration of backend algorithms while neglecting the optimization of frontend data acquisition. The lack of a band selection strategy that combines atmospheric transport models and ground reflection characteristics results in low information entropy of the input data itself, limiting the upper limit of colorization effect.

[0039] 3) The disconnect between structure and physical properties: Although most end-to-end generated color images have acceptable visual effects, they often lose the physical reflection characteristics unique to short-wave infrared, resulting in a mismatch between the generated edge structure and the real visible light scene, such as a low Structural Similarity (SSIM) index.

[0040] In view of this, embodiments of this application provide a physical data dual-driven shortwave infrared image colorization band selection method. Through three stages of physical pre-selection, statistical coarse screening, and task fine selection, the method selects the three-band combination containing the maximum information content and the best consistency with the visible light structure from the wide-band SWIR, thereby significantly improving the peak signal to noise ratio (PSNR), SSIM, and learned perceptual image patch similarity (LPIPS) of the colorized image generated by the deep learning method.

[0041] Figure 1 This is a flowchart illustrating a method for selecting the color band of a physical data-driven shortwave infrared image, as provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0042] In step S101, a physical feasibility analysis is performed on each band within the shortwave infrared band range to obtain N candidate bands.

[0043] Among them, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each band in the candidate band is greater than the preset score threshold, and the material discrimination is greater than the preset discrimination threshold; N is a positive integer.

[0044] In step S102, any M different bands in the candidate bands are combined, the characteristics of each band combination are statistically analyzed, and the candidate band combination is determined based on the statistical characteristics of each band combination.

[0045] Among them, the statistical characteristics include the improved optimal index factor and structural consistency; M is a positive integer less than N.

[0046] In step S103, the pre-trained deep generation network is trained using each band combination in the candidate band combination, and the target band combination is obtained based on the image colorization quality of each deep generation network after training.

[0047] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0048] In some embodiments of this application, the range of shortwave infrared bands to be processed can be determined first. This range of shortwave infrared bands to be processed is the band range composed of all shortwave infrared bands corresponding to the shortwave infrared images acquired.

[0049] Physical feasibility analysis can be performed on each band within the shortwave infrared band range to be processed, resulting in N candidate bands. Each of these N candidate bands must satisfy a combined score of source-end radiation intensity, transmission path transmittance, and receiver detection efficiency greater than a preset score threshold, and the material discrimination threshold must also be greater than a preset discrimination threshold. The specific values ​​of the preset transmittance threshold and the preset discrimination threshold can be determined according to actual needs and are not restricted here.

[0050] In some embodiments of this application, any M different bands from the candidate bands can be combined, and feature statistics can be performed on each band combination. These feature statistics may include improved optimal index factor statistics and structural consistency statistics. Then, candidate band combinations can be determined based on the statistical characteristics of each band combination.

[0051] In one example, the characteristic statistical scores of each band combination can be determined based on the improved optimal index factor statistics and structural consistency statistics. Then, the K band combinations with the highest characteristic statistical scores are selected as candidate band combinations. K can be a positive integer less than N.

[0052] In some embodiments of this application, the pre-trained deep generation network can be trained using each band combination in the candidate band combination, and the target band combination can be obtained based on the image colorization quality of each trained deep generation network.

[0053] For example, trained deep generative networks can be used to generate colorized shortwave infrared images of the target. The image quality of each colorized image is then determined, and finally, the band combination corresponding to the deep generative network with the highest image quality is selected as the target band combination. The band combination corresponding to the deep generative network is the same band combination used to train this deep generative network.

[0054] According to the technical solution provided in the embodiments of this application, addressing the problem of the exponential explosion of hyperspectral / multispectral band combinations, this approach abandons the computationally expensive exhaustive training method and utilizes a hierarchical screening framework. First, a physical model is used to determine candidate bands. Then, improved statistical indicators are used to quickly perform unsupervised dimensionality reduction to obtain a small number of high-potential combinations. Finally, generative verification of these high-potential combinations yields the preferred band combinations. This strategy ensures the physical interpretability of band selection while significantly reducing the time cost and computational consumption in finding the optimal solution.

[0055] In some embodiments of this application, a physical feasibility analysis is performed on each band within the shortwave infrared band range to obtain N candidate bands. This may include: constructing an effective energy integral model based on the optical radiation transfer theory; using the effective energy integral model to determine the mean energy and variance of each band within the shortwave infrared band range; determining a set of pre-selected bands based on the mean energy and variance of each band; each pre-selected band in the set of pre-selected bands having a first wavelength range; obtaining the reflectance of ground objects in each band within the shortwave infrared band range from a ground object spectral reflectance database, and determining the distinguishability of each band within the shortwave infrared band range for different materials based on the ground object reflectance; determining N candidate bands in the set of pre-selected bands based on the material distinguishability; each of the N candidate bands having a second wavelength range, the second wavelength range being smaller than the first wavelength range.

[0056] In other words, physical feasibility can be pre-selected first, including constructing an effective energy integral model based on optical radiative transfer theory to screen out the spectral window where the source-end radiation intensity, transmission path transmittance, and receiver-end detection efficiency achieve optimal coupling. Then, within this spectral window, further screening can be performed using material discrimination to obtain candidate bands with high material discrimination. Specifically, achieving optimal coupling among source-end radiation intensity, transmission path transmittance, and receiver-end detection efficiency means that the combined score of these three factors exceeds a preset score threshold.

[0057] Since the primary task of band selection is to define a physically feasible effective search space, this application's embodiments, from the perspective of the entire photon energy transmission chain, construct an effective energy integration model based on optical radiation transmission theory. This model ensures that the imaging system achieves the highest signal-to-noise ratio benchmark at the physical level by finding the spectral window that achieves optimal coupling among the source-end radiation intensity, transmission path transmittance, and receiver-end detection efficiency.

[0058] Shortwave infrared imaging is essentially the response of a detector to solar photons reflected after atmospheric attenuation. To accurately quantify the effective signal intensity of each band, embodiments of this application characterize the integrated energy of each band using an energy response function. The band range is defined as the wavelength. To wavelength For example, its energy response function is The energy response function can represent the wavelength. To wavelength Band integral energy within the band range The wavelength is within the band range. Standard solar spectral irradiance, Atmospheric transmittance, This represents the quantum efficiency of the receiver detector. In one example, the receiver detector could be an InGaAs (indium gallium arsenide) detector.

[0059] To quantitatively evaluate the imaging potential of each potential window, this application embodiment used 100nm as the first wavelength range and statistically analyzed the mean energy and energy variance of the short-wave infrared full spectrum. The statistical results are as follows: Figure 2 As shown in Table 1, significant interval differences were observed. Among them, Figure 2 The values ​​in the table are slightly different from those in Table 1 because the number of decimal places retained is different.

[0060] Table 1. Results of energy mean and energy variance analysis across the entire spectrum.

[0061]

[0062] A pre-selected band set can be determined based on the mean energy and variance of each band. For example, a coarsely screened band set can be first determined where the mean energy is greater than a preset mean threshold and the variance energy is less than a preset variance threshold; then, bands in the coarsely screened band set whose combined evaluation value of mean energy and variance energy is greater than a preset evaluation threshold are determined as pre-selected bands, thus obtaining the pre-selected band set.

[0063] refer to Figure 2 Based on Table 1, the following high-energy windows can be selected as pre-selected bands:

[0064] 1) First high-energy window (1000-1100nm): The normalized average energy in this range is as high as 0.897, which is the region with the most abundant photon energy, and the variance is controlled at a low level of 0.0168, making it suitable as the main source of imaging information.

[0065] 2) Second high-energy window (1200-1300nm): The average energy value is maintained at 0.621, showing good transmission continuity.

[0066] 3) The third high-energy window (1500-1700nm): the average energy is 0.42. Thanks to the flatness of the atmospheric window in this band, the stability of this range is excellent.

[0067] Next, N candidate bands can be determined from the pre-selected band set based on material discrimination. Specifically, in each pre-selected band of the pre-selected band set, at least one candidate band with a material discrimination greater than a preset discrimination threshold can be determined, resulting in N candidate bands. The second wavelength range corresponding to each candidate band can be, for example, 50 nm.

[0068] The candidate bands obtained by further screening within the high-energy windows identified above are shown in Table 2:

[0069] Table 2 Candidate bands determined based on physical feasibility analysis

[0070]

[0071] It can be seen that the bandwidth of the five candidate bands, B1 to B5, is 50nm, and they all avoid the 1400nm wavelength, which has high water vapor absorption.

[0072] This approach enables initial band screening based on physical feasibility analysis, thereby reducing the workload of subsequent band combination optimization.

[0073] In some embodiments of this application, combining any M different bands from the candidate bands, performing feature statistics on each band combination, and determining the candidate band combination based on the statistical characteristics of each band combination may include:

[0074] First, obtain the image dataset; the image dataset includes n sets of image data, each set of image data corresponds to a scene type, and each set of image data includes N short-wave infrared image data and one visible light image data. The N short-wave infrared image data are acquired using different bands in the candidate bands; n is a positive integer.

[0075] Then, for each set of image data in each scene type, the optimal exponential factor and structural consistency of improvement for each band combination in any M different band combinations among N candidate bands are statistically analyzed.

[0076] Next, we calculate the comprehensive score of each band combination in all image data sets across all scenes, and determine the K band combinations with the highest comprehensive scores as candidate band combinations.

[0077] In other words, after the candidate bands are determined, the band combinations composed of the candidate bands can be coarsely screened based on data characteristics to obtain candidate band combinations. In some implementations, unsupervised statistical scoring can be performed on all possible band combinations of candidate bands based on an improved optimal index factor and structural consistency to quickly eliminate redundant combinations.

[0078] The improved optimal index factor introduces Shannon entropy instead of standard deviation to calculate the ratio of information richness of band combinations to inter-band correlation, maximizing information entropy. Structural consistency is obtained by calculating the Pearson correlation coefficient between the average edge map of the SWIR band combination and the edge map of the visible light image, ensuring that the geometric structure is consistent with that of the visible light.

[0079] Taking the five candidate bands B1 to B5 selected above as examples, this article details the process of band combination coarse screening for determining data characteristics.

[0080] First, you can prepare the dataset. You can use... Figure 3 The experimental setup shown is used for data collection. For example... Figure 3 As shown, the experimental setup may include an RGB camera to acquire visible light images, a SWIR camera to acquire short-wave infrared images, a filter wheel, and a beam splitter. The beam splitter can be a dichroic beam splitter to ensure that the visible light and SWIR optical paths are coaxial, thereby ensuring spatial pixel-level alignment between the visible light and short-wave infrared images. The filter wheel can be a motorized filter wheel, positioned in front of the SWIR camera, for mounting filters corresponding to different wavelengths.

[0081] This experimental device can be used to collect datasets in different scenarios, such as urban building scenarios, industrial scenarios, traffic road scenarios, and rural scenarios. Figure 4 These are schematic diagrams of different scenarios.

[0082] Data sets can also be collected at different locations in each type of scenario, with each scenario location corresponding to a scenario type, and the collected data sets can be obtained.

[0083] Image data for each scene type can be grouped. Each group of image data can contain one shortwave infrared image of the scene type acquired using the B1 to B5 bands, and one corresponding visible light image.

[0084] Next, for each set of image data in each scene type, all possible 3-band combinations (10 in total) are generated from the 5 candidate bands, and the optimal improvement index factor is calculated for each band combination. ;in, For the improved optimal index factor, To utilize the first band in this combination Shannon entropy of shortwave infrared images acquired in each band. This is the [number]th [band] in this band combination. The first band and the first The correlation coefficients of each band.

[0085] At the same time, structural consistency is calculated for each band combination. ;in, For structural consistency, For the Pearson correlation coefficient operator, For edge extraction operators, This is the [number]th [band] in this band combination. Shortwave infrared images acquired in each band, These are the visible light images in this image data set.

[0086] The optimal index factor and structural consistency of the improved band combination can be weighted and summed to obtain the statistical score of the band combination in this image dataset. The weighting coefficients can be set according to actual needs and are not restricted here.

[0087] In one example, a weighting factor of 0.5 can be set, then for each band combination, its statistical score can be obtained as follows: ,in, This involves statistical scoring of band combinations. Using this method, statistical scores can be obtained for different band combinations of image data sets within each scene type.

[0088] Finally, the comprehensive score of each band combination in all image data sets of all scenes can be calculated, and the three band combinations with the highest comprehensive scores can be determined as candidate band combinations. Figure 5 Partial statistical results are shown. Figure 5 It can be seen that the selected candidate band combinations are 1-2-4, 1-3-4, and 1-4-5, namely, the combination of bands B1-B2-B4, the combination of bands B1-B3 and B4, and the combination of bands B1-B4 and B5. Furthermore, it can be seen that the selected candidate band combinations also have high statistical scores in independent scene types.

[0089] This approach enables pre-screening of band combinations based on data features, further reducing the computational complexity of subsequent model training and screening, and improving the efficiency of band selection.

[0090] In some embodiments of this application, using a pre-trained deep generative network to perform colorization training on each band combination in the candidate band combination, and obtaining the target band combination based on the image colorization quality, may include: obtaining a pre-trained deep generative network; the deep generative network is a generative adversarial network; using each band combination in the candidate band combination as model input, and using generative adversarial loss and L1 pixel loss as loss functions, training to obtain K deep generative networks; determining the quality index of the colorized image generated by the K deep generative networks based on the target shortwave infrared image, and calculating the comprehensive score of each deep generative network based on the quality index; the quality index includes at least peak signal-to-noise ratio, structural similarity, and learnable perceptual image patch similarity; determining the band combination corresponding to the deep generative network with the highest comprehensive score as the target band combination.

[0091] In other words, a deep generative network can be constructed to perform actual colorization training on candidate combinations, and the final scheme can be determined based on the generation quality. The generation quality can include a comprehensive selection score calculated from Peak Signal-to-Noise Ratio (PSNR), SSIM, and Learned Perceptual Image Patch Similarity (LPIPS) metrics.

[0092] In one example, the pre-trained deep generative network could be a Pix2Pix network. Pix2Pix is ​​a conditional generative model based on generative adversarial networks (GANs) designed to solve image-to-image translation problems. By introducing conditional information, Pix2Pix can learn to generate one image from another.

[0093] The candidate band combinations 1-2-4, 1-3-4, and 1-4-5 selected above can be used as model inputs to train the Pix2Pix network, with the objective function set to generative adversarial loss and L1 pixel loss during training. Then, PSNR, SSIM, and LPIPS metrics are used to determine the best-performing trained Pix2Pix network, and the candidate band combination corresponding to this Pix2Pix network is selected as the target band combination. The results of this fine selection are shown in Table 3. As can be seen from Table 3, combination 1-2-4 (1000-1050nm, 1050-110nm, 1525-1575nm) achieves optimal results in PSNR, SSIM, and LPIPS, therefore it can be determined as the target band combination.

[0094] Table 3. Results of band combination fine screening based on Pix2Pix network

[0095]

[0096] Figures 6 to 9 In order to be in Figure 4 The diagram illustrates the results of image colorization experiments using different bands or combinations of bands in four different scenarios. Figures 6 to 9 In the image, each column from left to right represents a single-band input image, a visible light image, a single-band image colorization result, a low-score combination (3-4-5) image colorization result, a candidate band combination (1-4-5) image colorization result, and an optimal band combination (1-2-4) image colorization result.

[0097] Depend on Figures 6 to 9 As can be seen, the coloring results in the rightmost column of each scene are most similar to the color information of a color visible light image captured by a real visible light camera, exhibiting the least local detail distortion and edge blurring. Therefore, the optimal three-band combination (1-2-4) determined in this embodiment maximizes spectral discrimination at the physical level, transforming the colorization task from an ill-conditioned inverse problem into a well-posed problem, and significantly reducing color artifacts in the generated image.

[0098] Figure 10 This is a flowchart illustrating another method for selecting the color band of a shortwave infrared image using dual-drive physical data, provided in an embodiment of this application. Figure 10 As shown, all bands capable of acquiring shortwave infrared images can be divided into bands, and five candidate bands—Band1, Band2, Band3, Band4, and Band5—are selected through physical feasibility analysis. In one example, the acquired shortwave infrared image could be a single-band shortwave infrared image of 256×256×1 pixels.

[0099] Five candidate bands can be combined, and each band combination can be coarsely screened based on statistical characteristics to obtain K sets of candidate band combinations. Each set of candidate band combinations can generate a three-band 256×256×3 shortwave infrared image, thus obtaining K sets of 256×256×3 shortwave infrared images.

[0100] Then, a Pix2Pix network is trained using the K candidate band combinations. For each of the K candidate band combinations, the Pix2Pix network generates a 256×256×3 RGB image corresponding to the three shortwave infrared images, resulting in K sets of RGB images generated from these three shortwave infrared images. Each RGB image is compared with a 256×256×3 real visible light image to analyze the generated image quality and ultimately determine the optimal combination.

[0101] The technical solution provided in this application eliminates mapping ambiguity caused by "different images within the same spectrum" from the physical source. Existing single-band or wide-band colorization techniques often lead to color confusion (i.e., "one-to-many" pathological mapping) because different materials (such as vegetation and coatings) have similar gray levels in the SWIR band. This application constructs input data using three preferred specific narrow bands (1000-1050nm, 1050-1100nm, 1525-1575nm), and successfully introduces key complementary information to distinguish heterogeneous materials by utilizing the high signal-to-noise ratio texture information of the short band and the unique spectral reflectance characteristics of the long band. This transforms the colorization task into a suitable "one-to-one" mapping, fundamentally eliminating semantically erroneous color generation.

[0102] The technical solution provided in this application establishes an efficient and interpretable "physics-data" dual-driven screening system. Addressing the problem of the exponential explosion of hyperspectral / multispectral band combinations, this application abandons the computationally expensive exhaustive training method and innovatively proposes a physical data dual-driven hierarchical screening framework. Invalid bands are eliminated through a physical model, unsupervised dimensionality reduction is performed quickly using improved statistical indicators, and finally, generative verification is conducted only on a very small number of high-potential combinations. This strategy ensures the physical interpretability of band selection while significantly reducing the time cost and computational power consumption in finding the optimal solution.

[0103] The technical solution provided in this application establishes a high-precision reference data acquisition process. It proposes a complete dual-optical-path coaxial acquisition and homography registration scheme, resolving the inconsistency between the short-wave infrared and visible light fields of view, ensuring pixel-level alignment of training data, and providing a high-quality data foundation for multispectral colorization research.

[0104] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0105] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0106] Figure 11 This is a schematic diagram of a physical data dual-drive shortwave infrared image colorization band selection device provided in an embodiment of this application. Figure 11 As shown, the device includes:

[0107] The physical screening module 1101 is configured to perform physical feasibility analysis on each band within the shortwave infrared band range to obtain N candidate bands; wherein, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each band in the candidate band is greater than a preset score threshold, and the material discrimination is greater than a preset discrimination threshold; N is a positive integer.

[0108] The data filtering module 1102 is configured to combine any M different bands in the candidate bands, perform feature statistics on each band combination, and determine the candidate band combination based on the statistical characteristics of each band combination; wherein, the statistical characteristics include the improved optimal index factor and structural consistency; M is a positive integer less than N.

[0109] The determination module 1103 is configured to train a pre-trained deep generation network using each band combination in the candidate band combination, and obtain the target band combination based on the image colorization quality of each deep generation network after training.

[0110] According to the technical solution provided in the embodiments of this application, addressing the problem of the exponential explosion of hyperspectral / multispectral band combinations, this approach abandons the computationally expensive exhaustive training method and utilizes a hierarchical screening framework. First, a physical model is used to determine candidate bands. Then, improved statistical indicators are used to quickly perform unsupervised dimensionality reduction to obtain a small number of high-potential combinations. Finally, generative verification of these high-potential combinations yields the preferred band combinations. This strategy ensures the physical interpretability of band selection while significantly reducing the time cost and computational consumption in finding the optimal solution.

[0111] In some implementations, a physical feasibility analysis is performed on each band within the shortwave infrared band range to obtain N candidate bands, including: constructing an effective energy integral model based on the optical radiation transfer theory; using the effective energy integral model to determine the mean energy and variance of each band within the shortwave infrared band range; determining a set of pre-selected bands based on the mean energy and variance of each band; each pre-selected band in the set of pre-selected bands having a first wavelength range; obtaining the reflectance of ground objects in each band within the shortwave infrared band range from a ground object spectral reflectance database, and determining the distinguishability of each band within the shortwave infrared band range for different materials based on the ground object reflectance; determining N candidate bands in the set of pre-selected bands based on the material distinguishability; each of the N candidate bands having a second wavelength range, the second wavelength range being smaller than the first wavelength range.

[0112] In some implementations, the effective energy integral model is as follows: ;in, To from wavelength To wavelength Band integral energy within the band range The wavelength is within the band range. Standard solar spectral irradiance, Atmospheric transmittance, The quantum efficiency of the receiver detector.

[0113] In some implementations, the pre-selected band set is determined based on the mean energy and variance of each band, including:

[0114] A coarse-screen band set is determined, whose mean energy is greater than a preset mean threshold and whose variance energy is less than a preset variance threshold. Bands in the coarse-screen band set whose combined evaluation value of mean energy and variance energy is greater than a preset evaluation threshold are identified as pre-selected bands, thus obtaining a pre-selected band set. Based on material discrimination, N candidate bands are determined from the pre-selected band set, including: in each pre-selected band in the pre-selected band set, at least one candidate band whose material discrimination is greater than a preset discrimination threshold is identified, thus obtaining N candidate bands.

[0115] In some implementations, candidate band combinations are determined as follows: An image dataset is acquired; the image dataset includes n sets of image data, each set corresponding to a scene type, and each set includes N shortwave infrared image data and one visible light image data. The N shortwave infrared image data are acquired using different bands from the candidate bands; n is a positive integer; for each set of image data in each scene type, the optimal improvement index factor and structural consistency of each band combination in any M different band combinations from the N candidate bands are calculated; the optimal improvement index factor and structural consistency of each band combination are weighted and summed to obtain the statistical score of the band combination for this image data set; the comprehensive score of each band combination in all image data sets across all scenes is calculated, and the K band combinations with the highest comprehensive scores are determined as candidate band combinations; K is a positive integer less than N.

[0116] In some implementations, the improved optimal exponential factor is determined by the formula Calculated; where, For the improved optimal index factor, To utilize the first band in this combination Shannon entropy of shortwave infrared images acquired in each band. This is the [number]th [band] in this band combination. The first band and the first Correlation coefficients for each band; structural consistency is determined by the formula. Calculated; where, For structural consistency, For the Pearson correlation coefficient operator, For edge extraction operators, This is the [number]th [band] in this band combination. Shortwave infrared images acquired in each band, These are the visible light images in this image data set.

[0117] In some implementations, a pre-trained deep generative network is used to colorize each band combination in the candidate band combination, and the target band combination is obtained based on the image colorization quality. This includes: obtaining a pre-trained deep generative network; the deep generative network is a generative adversarial network; training K deep generative networks by using each band combination in the candidate band combination as model input, and using generative adversarial loss and L1 pixel loss as loss functions; K is a positive integer less than N; determining the quality index of the colorized image generated by the K deep generative networks based on the target shortwave infrared image, and calculating the comprehensive score of each deep generative network based on the quality index; the quality index includes at least peak signal-to-noise ratio, structural similarity, and learnable perceptual image patch similarity; and determining the band combination corresponding to the deep generative network with the highest comprehensive score as the target band combination.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Figure 12 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 12 As shown, the electronic device 12 of this embodiment includes: a processor 1201, a memory 1202, and a computer program 1203 stored in the memory 1202 and executable on the processor 1201. When the processor 1201 executes the computer program 1203, it implements the steps in the various method embodiments described above. Alternatively, when the processor 1201 executes the computer program 1203, it implements the functions of each module / unit in the various device embodiments described above.

[0120] Electronic device 12 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 12 may include, but is not limited to, processor 1201 and memory 1202. Those skilled in the art will understand that... Figure 12 This is merely an example of electronic device 12 and does not constitute a limitation on electronic device 12. It may include more or fewer components than shown, or different components.

[0121] The processor 1201 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0122] The memory 1202 can be an internal storage unit of the electronic device 12, such as a hard disk or RAM of the electronic device 12. The memory 1202 can also be an external storage device of the electronic device 12, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc., equipped on the electronic device 12. The memory 1202 can also include both internal and external storage units of the electronic device 12. The memory 1202 is used to store computer programs and other programs and data required by the electronic device.

[0123] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0125] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A physical data dual drive short wave infrared image colorization band optimization method, characterized in that, include: Physical feasibility analysis is performed on each band within the shortwave infrared band range to obtain N candidate bands; wherein, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each band in the candidate band is greater than a preset score threshold, and the material discrimination is greater than a preset discrimination threshold; N is a positive integer; Any M different bands from the candidate bands are combined, and the characteristics of each band combination are statistically analyzed. Based on the statistical characteristics of each band combination, the candidate band combination is determined. The statistical characteristics include the improved optimal index factor and structural consistency. M is a positive integer less than N. The pre-trained deep generative network is trained using each band combination in the candidate band combination, and the target band combination is obtained based on the image colorization quality of each trained deep generative network. Among them, a physical feasibility analysis was conducted on each band within the shortwave infrared band range, resulting in N candidate bands, including: An effective energy integral model is constructed based on the theory of optical radiative transfer. The effective energy integral model is used to determine the mean energy and variance of each band within the shortwave infrared band range. A set of preselected bands is determined based on the mean energy and variance of each band; each preselected band in the set of preselected bands has a first wavelength range; The reflectance of ground objects in each band of the shortwave infrared band is obtained from the ground object spectral reflectance library, and the distinguishability of different materials in each band of the shortwave infrared band is determined based on the ground object reflectance. Based on material discrimination, N candidate bands are determined from a pre-selected band set; each of the N candidate bands has a second wavelength range, which is smaller than the first wavelength range; Candidate band combinations are determined as follows: Obtain an image dataset; the image dataset includes n sets of image data, each set of image data corresponds to a scene type, and each set of image data includes N sets of shortwave infrared image data and one set of visible light image data. The N sets of shortwave infrared image data are acquired using different bands in the candidate bands; n is a positive integer. For each set of image data in each scene type, the optimal exponential factor for improvement and structural consistency of each band combination in any M different band combinations among N candidate bands are statistically analyzed. The optimal index factor and structural consistency of each band combination are weighted and summed to obtain the statistical score of this band combination in this image data set. The comprehensive score of each band combination in all image data sets of all scenes is calculated, and the K band combinations with the highest comprehensive scores are determined as candidate band combinations; K is a positive integer less than N; The improved optimal index factor introduces Shannon entropy to replace standard deviation, calculates the ratio of information richness of band combination to correlation between bands, and maximizes information entropy; structural consistency is obtained by calculating the Pearson correlation coefficient between the average edge map of SWIR band combination and the edge map of visible light image.

2. The physical data dual drive short wave infrared image colorization band preference method of claim 1, wherein, The effective energy integral model is as follows: ； in, To from wavelength To wavelength Band integral energy within the band range The wavelength is within the band range. Standard solar spectral irradiance, Atmospheric transmittance, The quantum efficiency of the receiver detector.

3. The physical data dual drive short wave infrared image colorization band preference method of claim 1, wherein, The pre-selected band set is determined based on the mean and variance of energy in each band, including: Determine a coarsely selected set of bands whose mean energy is greater than a preset mean threshold and whose variance energy is less than a preset variance threshold; Bands in the coarsely screened band set whose combined energy mean and energy variance evaluation values ​​are greater than a preset evaluation threshold are identified as pre-selected bands, thus obtaining the pre-selected band set; Based on material discrimination, N candidate bands are determined from the pre-selected band set, including: In each pre-selected band of the pre-selected band set, at least one candidate band with a material discrimination degree greater than a preset discrimination degree threshold is determined, resulting in N candidate bands.

4. The physical data dual drive short wave infrared image colorization band preference method of claim 1, wherein, The improved optimal exponential factor is obtained through the formula. Calculated; where, The optimal exponential factor for the aforementioned improvement. To utilize the first band in this combination Shannon entropy of shortwave infrared images acquired in each band. This is the [number]th [band combination] in this band combination The first band and the first Correlation coefficients for each band; The structural consistency is expressed by the formula Calculated; where, For the sake of structural consistency. For the Pearson correlation coefficient operator, For edge extraction operators, This is the [number]th [band] in this band combination. Shortwave infrared images acquired in each band, These are the visible light images in this image data set.

5. The physical data dual drive short wave infrared image colorization band preference method of claim 1, wherein, A pre-trained deep generative network is used to colorize each band combination in the candidate band combinations. Based on the image colorization quality, the target band combinations are obtained, including: Obtain a pre-trained deep generative network; the deep generative network is a generative adversarial network. Using each band combination in the candidate band combination as the model input, and using generative adversarial loss and L1 pixel loss as the loss functions, K deep generative networks are trained to obtain K networks; K is a positive integer less than N. The quality metrics of the colorized images generated by the K deep generation networks based on the target shortwave infrared image are determined respectively, and the comprehensive score of each deep generation network is calculated based on the quality metrics; the quality metrics include at least peak signal-to-noise ratio, structural similarity and learnable perceptual image patch similarity; The band combination corresponding to the deep generation network with the highest comprehensive score is determined as the target band combination.

6. A physical data dual drive short wave infrared image colorization band preference apparatus, characterized by, include: The physical screening module is configured to perform physical feasibility analysis on each band within the shortwave infrared band range to obtain N candidate bands; wherein, the combined score of the source end radiation intensity, transmission path transmittance and receiver detection efficiency of each band in the candidate band is greater than a preset score threshold, and the material discrimination is greater than a preset discrimination threshold; N is a positive integer; The data filtering module is configured to combine any M different bands from the candidate bands, perform feature statistics on each band combination, and determine the candidate band combination based on the statistical characteristics of each band combination; wherein, the statistical characteristics include the improved optimal index factor and structural consistency; M is a positive integer less than N; The determination module is configured to train a pre-trained deep generative network using each band combination in the candidate band combinations, and obtain the target band combination based on the image colorization quality of each trained deep generative network. Among them, a physical feasibility analysis was conducted on each band within the shortwave infrared band range, resulting in N candidate bands, including: An effective energy integral model is constructed based on the theory of optical radiative transfer. The effective energy integral model is used to determine the mean energy and variance of each band within the shortwave infrared band range. A set of preselected bands is determined based on the mean energy and variance of each band; each preselected band in the set of preselected bands has a first wavelength range; The reflectance of ground objects in each band of the shortwave infrared band is obtained from the ground object spectral reflectance library, and the distinguishability of different materials in each band of the shortwave infrared band is determined based on the ground object reflectance. Based on material discrimination, N candidate bands are determined from a pre-selected band set; each of the N candidate bands has a second wavelength range, which is smaller than the first wavelength range; Candidate band combinations are determined as follows: Obtain an image dataset; the image dataset includes n sets of image data, each set of image data corresponds to a scene type, and each set of image data includes N sets of shortwave infrared image data and one set of visible light image data. The N sets of shortwave infrared image data are acquired using different bands in the candidate bands; n is a positive integer. For each set of image data in each scene type, the optimal exponential factor for improvement and structural consistency of each band combination in any M different band combinations among N candidate bands are statistically analyzed. The optimal index factor and structural consistency of each band combination are weighted and summed to obtain the statistical score of this band combination in this image data set. The comprehensive score of each band combination in all image data sets of all scenes is calculated, and the K band combinations with the highest comprehensive scores are determined as candidate band combinations; K is a positive integer less than N; The improved optimal index factor introduces Shannon entropy to replace standard deviation, calculates the ratio of information richness of band combination to correlation between bands, and maximizes information entropy; structural consistency is obtained by calculating the Pearson correlation coefficient between the average edge map of SWIR band combination and the edge map of visible light image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the physical data dual-drive shortwave infrared image colorization band selection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: When the computer program is executed by the processor, it implements the steps of the physical data dual-drive shortwave infrared image colorization band selection method as described in any one of claims 1 to 5.

Images