A method, apparatus, device and medium for processing multispectral imaging data

Abstract

This invention discloses a multispectral imaging data processing method, apparatus, device, and medium, applied in the field of optical imaging detection technology. The method includes: acquiring target data to be identified from image data and confirming the full-band data of the target to be identified, and constructing a full-band database; obtaining prior information of the target to be identified and determining characteristic bands; acquiring multidimensional optical image data of the target to be identified; acquiring low-frequency and high-frequency information of the spectral image and fusing it with the RGB image respectively to obtain low-frequency fusion components and high-frequency fusion components; reconstructing the multidimensional optical image data to obtain high-resolution multidimensional image data; classifying and identifying the target to be identified, obtaining classification results; constructing a classification system to filter the classification results to obtain the correct identification result. This invention ensures the timeliness of high-resolution image acquisition while improving the identification efficiency and accuracy of the imaging area.

Description

Technical Field

[0001] This invention belongs to the field of optical imaging detection technology, and particularly relates to a multispectral imaging data processing method, apparatus, device and medium. Background Technology

[0002] Polarization multispectral imaging technology, relying on payload platforms such as UAVs and aircraft, is currently widely used in remote sensing operations such as agricultural monitoring and geological exploration. It leverages the acquired multidimensional image data to effectively and accurately identify and classify targets. Currently, polarization multispectral imaging devices mainly employ time-division, wavefront-division, and focal plane-division structures. These three types of polarization imaging devices have the following drawbacks:

[0003] Time-division imaging devices record information by rotating polarizers and changing filters, but the imaging time is relatively long.

[0004] Wavefront-splitting imaging devices achieve rapid imaging through beam splitting devices, but the imaging devices are relatively bulky and expensive.

[0005] The focal plane can acquire spectral and polarization information simultaneously, but it takes a long time to acquire high-resolution spectral images, has poor timeliness, and cannot meet the need for rapid acquisition of high-resolution spectral images. Summary of the Invention

[0006] In view of this, the present invention aims to provide a multispectral imaging data processing method, apparatus, device and medium, which enriches the data dimensions of the imaging area by jointly analyzing the polarization characteristics, spectral characteristics and spatial geometric characteristics of objects in the imaging area based on multidimensional data image data. While ensuring the timeliness of high-resolution image acquisition, it can also effectively improve the recognition efficiency and accuracy of the imaging area.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0008] In a first aspect, the present invention provides a multispectral imaging data processing method, comprising:

[0009] S1: The camera acquires image data, confirms the target to be identified in the image data, obtains full-band data of the target to be identified, and constructs a full-band database of the target to be identified based on the full-band data;

[0010] S2: Obtain prior information about the target to be identified, and determine the characteristic bands of the target to be identified in the full-band database based on the prior information;

[0011] S3: Select a filter that matches the characteristic band so that the camera can acquire multi-dimensional optical image data of the target to be identified. The multi-dimensional optical image data includes the RGB image, spectral image and polarization spectral image of the target to be identified. The filter is used to filter out invalid spectral components or interfering light in the RGB image, spectral image and polarization spectral image.

[0012] S4: Obtain the low-frequency and high-frequency information of the spectral image, fuse the high-frequency and low-frequency information of the spectral image with the RGB image respectively to obtain the first low-frequency fusion component and the first high-frequency fusion component of the spectral image, and perform high-resolution reconstruction of the spectral image based on the first low-frequency fusion component and the first high-frequency fusion component to obtain the first high-resolution multidimensional image data.

[0013] S5: Based on the first high-resolution multidimensional image data, classify and discriminate the target to be identified, generate the classification and discrimination results of the target to be identified, construct a classification system to filter the classification and discrimination results, and obtain the correct recognition result of the target to be identified.

[0014] Preferably, the spectral image is reconstructed at high resolution based on the first low-frequency fusion component and the first high-frequency fusion component to obtain first high-resolution multidimensional image data, including:

[0015] S41: Perform discrete wavelet transform on the spectral image to obtain the low-frequency and high-frequency information generated after the discrete wavelet transform of the spectral image;

[0016] S42: Perform discrete wavelet transform on the RGB image to extract high-frequency edge information of the target to be identified, and obtain the high-frequency subbands of the RGB image containing the geometric features of the target to be identified. Construct the full-size geometric features of the target to be identified based on the high-frequency subbands.

[0017] S43: Retain the low-frequency information generated by the discrete wavelet transform of the spectral image at each level, use the low-frequency information as the first low-frequency fusion component of the spectral image, and perform weighted fusion of the high-frequency information of the spectral image with the full-size geometric features to obtain the first high-frequency fusion component of the spectral image. After all the low-frequency and high-frequency information of the spectral image are converted into the first low-frequency fusion component and the first high-frequency fusion component, the spectral image is inversely transformed and reconstructed into the first high-resolution multidimensional image data.

[0018] Preferably, the inverse transformation of the spectral image is reconstructed into first high-resolution multidimensional image data, including:

[0019] S431: Obtain the geometric feature map of the spectral image, and the target high-resolution geometric features of the preset spectral image in the camera, obtain the Gaussian kernel and gradient magnitude of the target high-resolution geometric features, and quantize the geometric feature map of the spectral image based on the Gaussian kernel and gradient magnitude.

[0020] S432: Obtain the mean and standard deviation of the geometric feature map, and perform adaptive normalization on the quantized geometric feature map based on the mean and standard deviation;

[0021] S433: Obtain the empirical curvature coefficients and initial weights of the geometric feature map after adaptive normalization. Based on the geometric feature map after adaptive normalization, the empirical curvature coefficients, and the initial weights, obtain the weight factor of the spectral image. Fuse the weight factor, characteristic high-frequency subbands, and high-frequency information of the spectral image to obtain the first high-frequency fusion component of the spectral image.

[0022] S434: Perform inverse discrete wavelet transform on the spectral image to generate an approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform. Combine the approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform with the first high-frequency fusion component of the spectral image. Then perform inverse discrete wavelet transform on the combined approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform and the first high-frequency fusion component of the spectral image to recursively reconstruct the spectral image into the first high-resolution multidimensional image data.

[0023] Preferably, the classification system filters the classification results to obtain correct identification results, including:

[0024] S51: Obtain the low-frequency subband coefficients of the spectral image after the first-level discrete wavelet transform and the high-frequency subband coefficients of the RGB image after the first-level discrete wavelet transform. Obtain the first-level confidence level based on the high-frequency subband coefficients and the low-frequency subband coefficients. Determine the first-level confidence level based on the first-level confidence level. Confirm the confidence threshold of the first-level confidence level. Construct a first-level classification system based on the first-level confidence level and the first-level confidence level.

[0025] S52: Obtain the mean polarization angle of the target region and the mean polarization angle of the background region, the difference in polarization degree between the target region and the background region, the difference in reflectance, and the number of image blocks in the polarization spectral image within the k-th local window of the polarization spectral image. Based on the mean polarization angle of the target region, the mean polarization angle of the background region, the difference in polarization degree between the target region and the background region, the difference in reflectance, and the number of image blocks in the polarization spectral image, obtain the second-level confidence level.

[0026] S53: Obtain the joint weights of the polarization spectral image and the spectral image, obtain the second-level confidence level based on the first-level confidence level, the second-level confidence level, and the joint weights of the polarization spectral image and the spectral image, and construct a two-level classification system based on the second-level confidence level and the second-level confidence level;

[0027] S54: The first high-resolution multidimensional image data is fused with the spectral image to obtain a first fused image. The first fused image is input into a first-level classification system. The first-level classification system judges the first fused image according to a preset first confidence threshold to obtain the correct recognition result of the first fused image. When there are multiple correct recognition results in the first fused image, and the confidence of multiple correct recognition results is greater than the preset first confidence threshold, a second low-frequency fusion component and a second high-frequency fusion component are constructed. The spectral image is subjected to inverse discrete wavelet transform according to the second low-frequency fusion component and the second high-frequency fusion component to obtain a second high-resolution multidimensional image data. The second high-resolution multidimensional image data is fused with the polarization spectral image to obtain a second fused image. The second fused image is input into a second-level classification system. The second-level classification system judges the second fused image according to a preset second confidence threshold to obtain the correct recognition result of the second fused image.

[0028] S55: When the confidence level of the second fused image is greater than a preset second confidence threshold, the correct recognition result of the second fused image is used as the final predicted target of the target to be recognized. Preferably, a second low-frequency fusion component and a second high-frequency fusion component are constructed, and an inverse discrete wavelet transform is performed on the spectral image based on the second low-frequency fusion component and the second high-frequency fusion component to obtain second high-resolution multidimensional image data, including:

[0029] S541: Calculate the degree of polarization and polarization angle of the polarization spectral image, and generate a binary mask and polarization angle gradient of the polarization spectral image based on the degree of polarization and polarization angle.

[0030] S542: Obtain the target mean, background mean, and background standard deviation of the polarization spectral image. Based on the target mean, background mean, and background standard deviation, obtain the weight coefficient of the polarization degree of the polarization spectral image. Fuse the low-frequency information of the spectral image, the polarization degree of the polarization spectral image, the binary mask of the polarization spectral image, and the weight coefficient of the polarization degree of the polarization spectral image to obtain the second low-frequency fusion component of the spectral image.

[0031] S543: Obtain the first fusion weight factor of the spectral image based on the high-frequency information of the spectral image and the polarization angle gradient of the polarization spectral image, and obtain the second fusion weight factor of the spectral image based on the first fusion weight factor;

[0032] S544: Obtain the horizontal gradient fusion component and the vertical gradient fusion component of the polarization angle gradient, and fuse the first fusion weight factor, the second fusion weight factor, the high-frequency information of the spectral image, the horizontal gradient fusion component and the vertical gradient fusion component to obtain the second high-frequency fusion component of the spectral image.

[0033] S545: After the low-frequency and high-frequency information generated by the discrete wavelet transform of the spectral image are converted into the second low-frequency fusion component and the second high-frequency fusion component, the inverse discrete wavelet transform of the spectral image is performed to recursively reconstruct the spectral image into the second high-resolution multidimensional image data.

[0034] Preferably, the camera setup includes a first polarization spectral camera, a second polarization spectral camera, a third polarization spectral camera, and a fourth polarization spectral camera. The first polarization spectral camera is used to acquire images at a 0° polarization angle, the second polarization spectral camera is used to acquire images at a 45° polarization angle, the third polarization spectral camera is used to acquire images at a 90° polarization angle, and the fourth polarization spectral camera is used to acquire images at a 135° polarization angle. The degree of polarization and the polarization angle of the polarization spectral images are calculated, including:

[0035] S5411: The first polarization spectral camera acquires the first angle value when the angle between the incident light and the polarization spectral image is 0°; the second polarization spectral camera acquires the second angle value when the angle between the incident light and the polarization spectral image is 45°; the third polarization spectral camera acquires the third angle value when the angle between the incident light and the polarization spectral image is 90°; the fourth polarization spectral camera acquires the fourth angle value when the angle between the incident light and the polarization spectral image is 135°.

[0036] S5412: Calculate the first polarized light component of the polarization spectrum image based on the first angle value, the second angle value, the third angle value, and the fourth angle value; calculate the second polarized light component of the polarization spectrum image based on the first angle value and the third angle value; calculate the third polarized light component of the polarization spectrum image based on the second angle value and the fourth angle value.

[0037] S5413: Calculate the image polarization degree of the polarization spectrum image based on the first polarization component, the second polarization component, and the third polarization component, and calculate the polarization angle of the polarization spectrum image based on the second polarization component and the third polarization component.

[0038] Preferably, each camera is equipped with a detector array, and the first polarization spectral camera, the second polarization spectral camera, the third polarization spectral camera and the fourth polarization spectral camera are all equipped with four-quadrant polarizers and multilayer dielectric films.

[0039] In a second aspect, embodiments of the present invention provide a multispectral imaging data processing apparatus, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform the multispectral imaging data processing method as described in the first aspect above.

[0040] Thirdly, embodiments of the present invention provide an electronic device including the multispectral imaging data processing apparatus as described in the second aspect above.

[0041] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the multispectral imaging data processing method as described in the first aspect above.

[0042] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0043] This invention, by setting up one high-resolution camera, two spectral cameras with spectral channels, and four polarization spectral cameras with spectral channels, enables the image data processing module of a polarization multispectral imaging device to simultaneously extract polarization spectral images, RGB images, and spectral images acquired by the cameras, thereby enriching the data dimensions and improving the working efficiency of the polarization multispectral imaging device. Further, by acquiring multidimensional optical image data of the target to be identified through the cameras, the spectral image and RGB image in the multidimensional optical image data are fused to obtain a first low-frequency fusion component and a first high-frequency fusion component. By fusing the first low-frequency fusion component and the first high-frequency fusion component, high-resolution reconstruction is performed on the multidimensional optical image data to obtain first high-resolution multidimensional image data, enriching the data dimensions of the target to be identified in the image data. The first high-resolution multidimensional image data is then fused with the spectral image to obtain a first fused image. A first-level classification system is constructed to judge the first fused image. When the first fused image has only one correct identification result, that correct identification result is used as the final predicted target of the target to be identified. When the first fused image has multiple correct identification results, the correct identification result is used as the final predicted target of the target to be identified. When multiple correct identification results are obtained, the second high-resolution multidimensional image data is fused with polarization spectral data to obtain a second fused image. A two-level classification system is constructed to judge the second fused image to obtain the correct identification result of the second fused image. The correct identification result of the second fused image is used as the final predicted target of the target to be identified. This invention performs joint analysis on the polarization features, spectral features and spatial geometric features of the target to be identified in the image data. By constructing a first-level classification system, the correct identification results of the target to be identified are quickly screened, reducing invalid calculations in the multispectral imaging data processing. By constructing a second-level classification system, multiple correct identification results are further verified to obtain the final predicted target of the target to be identified, thereby effectively improving the identification efficiency and accuracy of the target to be identified in the image data. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0045] Figure 1 This is a schematic diagram of the overall structure of the polarization multispectral imaging device;

[0046] Figure 2 This is a flowchart of a multispectral imaging data processing method provided in one embodiment of the present invention;

[0047] Figure 3 It is a flowchart of high-resolution reconstruction of the spectral image based on the first low-frequency fusion component and the first high-frequency fusion component to obtain the first high-resolution multidimensional image data.

[0048] Figure 4 This is a flowchart of the process of reconstructing a first high-resolution multidimensional image data through inverse transformation of a spectral image;

[0049] Figure 5 It is a flowchart of constructing a classification system, filtering the classification results, and obtaining the correct identification result;

[0050] Figure 6 The flowchart describes how to construct a second low-frequency fusion component and a second high-frequency fusion component, and then perform an inverse discrete wavelet transform on the spectral image based on the second low-frequency fusion component and the second high-frequency fusion component to obtain the second high-resolution multidimensional image data.

[0051] Figure 7 This is a flowchart for calculating the degree of polarization and polarization angle of a polarization spectral image;

[0052] Figure 8 This is a structural diagram of a multispectral imaging data processing device provided in another embodiment of the present invention.

[0053] The reference numerals in the figures include:

[0054] Camera 100, high-resolution camera 110, spectroscopic camera 120, polarization spectroscopic camera 130, four-quadrant polarizer 131, detector array 140, filter 150. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0057] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0058] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0059] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0060] This invention provides a multispectral imaging data processing method, applied in a polarization multispectral imaging device. (Refer to...) Figure 1 The polarization multispectral imaging device includes a camera 100, an image data processing module, and a storage module. The image data processing module is connected to the camera 100 and the storage module, respectively. Specifically, the camera 100 includes a high-resolution camera 110 with a high-resolution channel, two spectroscopic cameras 120 with spectral channels, and four polarization spectroscopic cameras 130 with polarization spectral channels. Each of the high-resolution camera 110, the two spectroscopic cameras 120, and the four polarization spectroscopic cameras 130 is equipped with a high-sensitivity, low-noise complementary metal-oxide-semiconductor (CMOS) detector array 140 and a filter 150. In each of the four polarization spectroscopic cameras 130, a four-quadrant polarizer 131 and a multilayer dielectric film (not shown in the figure) are integrated in front of the complementary metal-oxide-semiconductor (CMOS) detector array 140. The image data processing module consists of a field-programmable gate array (FPGA) core processor, a high-speed digital signal processing (DSP) chip, and an intelligent recognition chip. The storage module consists of a hardware storage medium, a data cache unit, an interface unit, and a software management system.

[0061] For example, in the camera 100 of this embodiment, one high-resolution camera 110 and six spectral cameras 120 with spectral channels can adopt a refractive structure with multiple lenses. This refractive structure is provided with an achromatic lens group for acquiring clear images of the same target scene. The achromatic lens group can focus light in the 400nm-750nm wavelength band and reduce chromatic aberration, so that the light is incident on the subsequent complementary metal-oxide-semiconductor (CMOS) detector array 140 with a uniform light spot. Furthermore, the achromatic lens group is designed by combining multiple layers of lenses with different refractive index materials. The curvature, thickness and spacing parameters of the achromatic lenses can be optimized to control the chromatic aberration of the light in the image data acquired by the camera 100 within an acceptable range, ensuring the imaging clarity and color accuracy of the image data acquired by the camera 100, thereby improving the imaging quality and detection accuracy of the image data acquired by the camera 100 in different environments. For example, in this embodiment, the pixel size of the high-resolution camera 110 is 7200×6000.

[0062] It should be noted that, in this embodiment, the four polarization spectral cameras 130 are equipped with a complementary metal-oxide-semiconductor (CMOS) detector array 140, a four-quadrant polarizer 131, and a multilayer dielectric film manufactured using a back-illuminated process. The CMOS detector array 140 itself possesses high quantum efficiency, low dark current characteristics, and advantages in high spacing sensitivity and low noise. On one hand, the polarization spectral cameras 130 filter the acquired polarization spectral images through the CMOS detector array 140, the four-quadrant polarizer 131, and the multilayer dielectric film. On the other hand, the CMOS detector array 140, the four-quadrant polarizer 131, and the multilayer dielectric film allow the polarization spectral images to flexibly optimize pixel layout according to their own imaging characteristics and data processing requirements, thereby effectively enhancing the target detection capability and signal acquisition quality of the multiple polarization spectral cameras 130 in the camera 100 under low-light conditions. For example, in this embodiment, each polarization spectral camera 130 has a pixel size of 2400×2000 (single channel / single polarization state).

[0063] It should be noted that the four-quadrant polarizer 131 is a set of polarizers covering the surface of the detector array 140. The four-quadrant polarizer 131 is divided into four independent quadrant regions with fixed polarization directions. Each quadrant of the four-quadrant polarizer 131 only allows incident light with a specific vibration direction to pass through, ultimately realizing the acquisition of polarization spectral images when the targets to be identified are in the same scene. The filter 150 is used to filter out unwanted spectral components or interfering light in the RGB image acquired by the high-resolution camera 110, the spectral image acquired by the spectral camera 120, and the polarization spectral image acquired by the polarization spectral camera 130, thereby reducing interference components in the RGB image, spectral image, and polarization spectral image and improving the contrast of the target to be identified.

[0064] Furthermore, by setting up one high-resolution camera 110, two spectral cameras 120 with spectral channels, and four polarization spectral cameras 130 with spectral channels, the image data processing module can extract the polarization spectral image, RGB image, and spectral image acquired by the camera 100 at one time, thereby enriching the data dimensions and working efficiency of the polarization multispectral imaging device.

[0065] Four polarization spectral cameras 130 decompose the incident light of the acquired image data into polarization light components of 0°, 45°, 90° and 135°, so that the camera 100 of the polarization multispectral imaging device can simultaneously acquire polarized light of four different directions through the same complementary metal-oxide-semiconductor (CMOS) detector array 140.

[0066] The data processing module uses a Field-Programmable Gate Array (FPGA) as its core processor, combined with a high-speed digital signal processing (DSP) chip and an intelligent recognition chip. The FPGA is used to implement high-speed data acquisition, preliminary data preprocessing, and system logic for the Complementary Metal-Oxide-Semiconductor (CMOS) detector array 140. The DSP chip and intelligent recognition chip are used to execute target recognition algorithms based on band selection and polarization spectral feature extraction and matching algorithms to detect, locate, and classify the target to be identified in the image data acquired by the camera 100. Furthermore, the data processing module has abundant logic resources and a high-speed data transmission interface, and implements image data acquisition, buffering, format conversion, bad pixel correction, and preliminary feature extraction functions through hardware description language (HDL) programming.

[0067] In addition, the high-speed digital signal processing (DSP) chip and the intelligent recognition chip have powerful computing capabilities, use an optimized deep learning library to execute target recognition algorithms, and exchange data with the field programmable gate array through a high-speed serial bus to achieve efficient collaborative processing, thereby improving the real-time performance and accuracy of the polarization multispectral imaging device when acquiring data.

[0068] Furthermore, in this embodiment, among the six spectral cameras 120 equipped with spectral channels in the camera 100, the multilayer dielectric film used in each spectral camera 120 is designed and deposited according to the strict optical thin film interference theory. By optimizing the film material (such as SiO2, TiO2, etc.), the thickness and number of layers of the dielectric film, the incident light in the image data acquired by the camera 100 is filtered through the multilayer dielectric film, realizing the precise screening of specific spectral ranges in multiple spectral bands (such as 400-450nm, 450-550nm, 550-650nm, 650-750nm). This enables the camera 100 to achieve high diffraction efficiency (average higher than 70%) filtering in specific spectral ranges in multiple spectral bands, with a spectral resolution of 5nm, accurately distinguishing the target spectral features in the image data acquired by the camera 100, and improving the recognition accuracy of the target to be identified in the image data acquired by the camera 100.

[0069] The signal storage module is used to store the raw data acquired by the camera 100 or the identification data processed by the data processing module.

[0070] The multispectral imaging data processing method of the present invention will be further described below with reference to the accompanying drawings.

[0071] Reference Figure 2 , Figure 2 This is a flowchart of a multispectral imaging data processing method according to an embodiment of the present invention. The multispectral imaging data processing method includes the following steps:

[0072] S1: The camera acquires image data, confirms the target to be identified in the image data, obtains full-band data of the target to be identified, and constructs a full-band database of the target to be identified based on the full-band data;

[0073] S2: Obtain prior information about the target to be identified, and determine the characteristic bands of the target to be identified in the full-band database based on the prior information.

[0074] It should be noted that prior information characterizes the optical features of the target to be identified, such as spectral reflectance, absorption peaks, and emission peaks, acquired by the polarization multispectral imaging device. The spectral bands of the target to be identified acquired by the camera are compared with the preset spectral bands in the full-band database. When the spectral bands of the target to be identified acquired by the polarization multispectral imaging device are the same as the spectral bands set in the full-band database, the spectral bands of the target to be identified are used as feature bands, thereby improving the identification efficiency of the spectral bands of the target to be identified acquired by the polarization multispectral imaging device.

[0075] S3: Select a filter that matches the characteristic band so that the camera can acquire multidimensional optical image data of the target to be identified. The multidimensional optical image data includes the RGB image, spectral image and polarization spectral image of the target to be identified. The filter is used to filter out invalid spectral components or interfering light in the RGB image, spectral image and polarization spectral image.

[0076] It should be noted that the filter is used to perform spectral filtering on the image data acquired by the camera, thereby obtaining multi-dimensional optical image data of the target to be identified. For example, the filter in this embodiment is designed for the visible and near-infrared band (400nm-750nm). The filter performs efficient dispersion and precise spectral separation on the light component of the target to be identified in the image data acquired by the camera according to multiple preset spectral ranges, thereby obtaining multi-dimensional optical image data of the target to be identified.

[0077] S4: Obtain the low-frequency and high-frequency information of the spectral image, fuse the low-frequency and high-frequency information of the spectral image with the RGB image respectively, and obtain the first low-frequency fusion component and the first high-frequency fusion component of the spectral image. Perform high-resolution reconstruction of the spectral image based on the first low-frequency fusion component and the first high-frequency fusion component to obtain the first high-resolution multidimensional image data.

[0078] It should be noted that the first high-resolution multidimensional image data represents the fine spatial structural features of the target to be identified, such as texture, shape, or detailed contours; the spectral image represents the reflection features of the target to be identified at different wavelengths; the low-frequency information of the spectral image includes spectral information and structural information, and the high-frequency information of the spectral image includes horizontal sub-bands, vertical sub-bands, and diagonal sub-bands; in this embodiment, based on the acquired RGB image, the background of the target to be identified, and the image characteristics of the target to be identified, the low-frequency and high-frequency information of the spectral image are fused with the RGB image respectively to obtain the first low-frequency fusion component and the first high-frequency fusion component of the spectral image, and the spectral image is reconstructed in high resolution based on the first low-frequency fusion component and the first high-frequency fusion component to obtain the first high-resolution multidimensional image data.

[0079] S5: Based on the first high-resolution multidimensional image data, classify and discriminate the target to be identified, generate the classification and discrimination results of the target to be identified, construct a classification system to filter the classification and discrimination results, and obtain the correct recognition result of the target to be identified.

[0080] Additionally, in one embodiment, reference is made to Figure 3 ,exist Figure 2 In step S4 of the illustrated embodiment, the spectral image is reconstructed at high resolution based on the first low-frequency fusion component and the first high-frequency fusion component to obtain the first high-resolution multidimensional image data. The step S4 also includes the following steps:

[0081] S41: Perform discrete wavelet transform on the spectral image to obtain the low-frequency and high-frequency information of the spectral image generated after the discrete wavelet transform.

[0082] It should be noted that before processing the spectral, RGB, and polarization spectral images in the multidimensional optical image data, preprocessing is required. This preprocessing includes, but is not limited to, image denoising, non-uniformity correction, and image feature extraction. In this embodiment, after denoising and non-uniformity correction of the spectral image, discrete wavelet transform is performed on the spectral image under the characteristic bands to obtain the low-frequency and high-frequency information of the spectral image.

[0083] Furthermore, when performing discrete wavelet transform on the spectral image, in each transform layer, two parallel operations—low-frequency filtering and high-frequency filtering—are performed to decompose and transform the currently processed spectral image into two types of information: one type is low-frequency information that retains the overall contour and main features of the target to be identified in the spectral image, and the other type is high-frequency information that reflects the local details of the target to be identified in the spectral image. In this embodiment, the low-frequency information of the spectral image is one low-frequency sub-band, and the high-frequency information of the spectral image is three high-frequency sub-bands, namely, a horizontal sub-band, a vertical sub-band, and a diagonal sub-band. The specific process of performing discrete wavelet transform on the spectral image is represented by the following first formula:

[0084] ;

[0085] in, For the low-frequency subband of the spectral image, For the horizontal sub-bands of the spectral image, For the vertical sub-bands of the spectral image, The diagonal subband of the spectral image, L is the low-pass filter, and H is the high-pass filter. For convolution, For downsampling, For the first l The number of rows in the low-frequency subband of the spectral image after discrete wavelet transform. For the first l The number of columns in the low-frequency subband of the spectral image after discrete wavelet transform. For the first l The number of rows in the high-frequency subbands of the spectral image after discrete wavelet transform. For the first l The number of columns in the high-frequency subbands of the spectral image after discrete wavelet transform, T represents the transyscopy operation, and I represents the spectral image. l The number of transform layers when performing discrete wavelet transform on a spectral image.

[0086] S42: Perform discrete wavelet transform on the RGB image to extract high-frequency edge information of the target to be identified, and obtain the high-frequency sub-bands of the target containing the geometric features of the target in the RGB image. Construct the full-size geometric features of the target to be identified based on the high-frequency sub-bands.

[0087] It should be noted that RGB images are color images presented based on the mixing principle of the three primary colors (red, green, and blue); the high-frequency characteristic subband represents the high-frequency subband containing the full-size geometric features of the target to be identified.

[0088] Discrete wavelet transform is performed on the RGB image acquired by the camera to extract high-frequency edge information of the target to be identified, resulting in a high-frequency feature subband in the RGB image containing the geometric features of the target to be identified, as expressed by the following second formula:

[0089] ;

[0090] in, The first high-frequency subband of the RGB image containing the geometric features of the target to be identified. The second feature high-frequency subband of the RGB image containing the geometric features of the target to be identified. This refers to the third high-frequency subband of the RGB image containing the geometric features of the target to be identified. l L is the number of layers for performing discrete wavelet transform on the first high-frequency feature subband, and L is the total number of layers for discrete wavelet transform.

[0091] It should be noted that in this embodiment, discrete wavelet transform is performed on the spectral image of the multidimensional optical image data to extract as much low-frequency information such as spectral information and regional structure as possible, as well as high-frequency information such as geometric contours of the spectral image. Combined with the RGB image of the target to be identified acquired by the high-resolution camera with high-resolution channel, the high-frequency edge information of the RGB image is extracted by discrete wavelet transform to obtain the characteristic high-frequency subband of the target containing the geometric features of the target to be identified in the RGB image. The full-size geometric features of the target to be identified are constructed based on the characteristic high-frequency subband.

[0092] S43: Retain the low-frequency information generated by the discrete wavelet transform of the spectral image at each level, use the low-frequency information as the first low-frequency fusion component of the spectral image, and perform weighted fusion of the high-frequency information of the spectral image with the full-size geometric features to obtain the first high-frequency fusion component of the spectral image. After all the low-frequency and high-frequency information of the spectral image are converted into the first low-frequency fusion component and the first high-frequency fusion component, the spectral image is inversely transformed and reconstructed into the first high-resolution multidimensional image data.

[0093] It should be noted that, since the low-frequency information of the spectral image contains both spectral and structural information, in this embodiment, the low-frequency information in the spectral image is directly retained as the first low-frequency fusion component, expressed by the following third formula:

[0094] ;

[0095] in, For the first low-frequency fusion component, This is low-frequency information.

[0096] Additionally, in one embodiment, reference is made to Figure 4 ,exist Figure 3 In step S43 of the illustrated embodiment, the inverse transformation of the spectral image is reconstructed into first high-resolution multidimensional image data, and the following steps are also included:

[0097] S431: Obtain the geometric feature map of the spectral image, and the target high-resolution geometric features of the preset spectral image in the camera, obtain the Gaussian kernel and gradient magnitude of the target high-resolution geometric features, and quantize the geometric feature map based on the Gaussian kernel and gradient magnitude.

[0098] S432: Obtain the mean and standard deviation of the geometric feature map, and perform adaptive normalization on the quantized geometric feature map based on the mean and standard deviation of the geometric feature map.

[0099] S433: Obtain the empirical curvature coefficients and initial weights of the geometric feature map after adaptive normalization. Based on the geometric feature map after adaptive normalization, the empirical curvature coefficients, and the initial weights, obtain the weight factor of the spectral image. Fuse the weight factor, the characteristic high-frequency subband, and the high-frequency information of the spectral image to obtain the first high-frequency fusion component of the spectral image.

[0100] S434: Perform inverse discrete wavelet transform on the spectral image to generate an approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform. Combine the approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform with the first high-frequency fusion component of the spectral image. Then perform inverse discrete wavelet transform on the combined approximate component with one less layer than the total number of layers in the inverse discrete wavelet transform and the first high-frequency fusion component of the spectral image to recursively reconstruct the spectral image into the first high-resolution multidimensional image data.

[0101] It should be noted that the high-resolution camera contains the target's high-resolution geometric features in the spectral image. Therefore, by obtaining the Gaussian kernel and gradient magnitude of the target's high-resolution geometric features, the geometric feature map of the spectral image is quantized based on the Gaussian kernel and gradient magnitude to improve the structural saliency of the spectral image.

[0102] The geometric feature map of the spectral image is quantized based on the Gaussian kernel and gradient magnitude, as expressed by the following fourth formula:

[0103] ;

[0104] in, Here, N represents the geometric feature map of the spectral image, N is the Gaussian kernel of the geometric feature map, and M is the gradient magnitude of the geometric feature map. Non-zero constants This represents the size of the Gaussian kernel.

[0105] Adaptive normalization is performed on the defined geometric feature map to eliminate the influence of illumination and contrast differences in the spectral image data.

[0106] Adaptive normalization is performed on the quantized geometric feature map, as expressed by the following fifth formula:

[0107] ;

[0108] in, For the adaptively normalized geometric feature map, The standard deviation of the quantized geometric feature map, The mean of the quantized geometric feature map, Non-zero constants This is a geometric feature map of the spectral image.

[0109] The weighting factor of the spectral image is obtained based on the curvature coefficients of the quantized geometric feature map after adaptive normalization and the quantized geometric feature map, and is expressed by the following sixth formula:

[0110] ;

[0111] in, Weighting factors for spectral images, Here, k represents the initial weights of the geometric feature map, and k is the curvature coefficient of the quantized geometric feature map after adaptive normalization. For adaptively normalized geometric feature maps, l L is the number of layers when performing discrete wavelet transform on the first feature high-frequency subband of the RGB image, and L is the total number of layers when performing discrete wavelet transform on the first feature high-frequency subband of the RGB image. Among them, the higher the image complexity of the geometric feature map, the more sensitive the weighting factor is to the saliency changes of the geometric feature map.

[0112] The high-frequency subband and the first feature high-frequency subband are fused according to the weighting factor to obtain the first high-frequency fusion component of the spectral image, which is expressed by the following seventh formula:

[0113] ;

[0114] in, The first high-frequency fusion component of the spectral image, Weighting factors for spectral images, For the characteristic high-frequency subbands of RGB images, This represents the high-frequency information of the spectral image.

[0115] It should be noted that the weighting factors of the spectral image It needs to be based on the number of discrete wavelet transform layers of the spectral image. l Adaptive intensity adjustment.

[0116] After fusing the components at each level in the spectral image, a layer-by-layer inverse discrete wavelet transform is used to reconstruct the spectral image. Taking the Lth level of the spectral image during the discrete wavelet transform as the initial level, the inverse discrete wavelet transform is performed on the spectral image to generate approximate components of the spectral image at level L (the total number of levels during the inverse discrete wavelet transform) - 1. Approximate components of the spectral image The first high-frequency fusion component of the L-th (total number of discrete wavelet transform layers of the spectral image)-1th layer The data is combined and inverse discrete wavelet transform is performed to recursively reconstruct the spectral image into the first high-resolution multidimensional image data.

[0117] Furthermore, since artifacts may exist in the first high-resolution multidimensional image data after the spectral image is recursively reconstructed into the first high-resolution multidimensional image data, weighted least squares filtering can be used to eliminate the artifacts.

[0118] Additionally, in one embodiment, referring to the figure, in Figure 2 In step S50 of the illustrated embodiment, which involves constructing a classification system to filter the classification results and obtain correct identification results, the following steps are also included:

[0119] S51: Obtain the low-frequency subband coefficients of the spectral image after the first-level discrete wavelet transform and the high-frequency subband coefficients of the RGB image after the first-level discrete wavelet transform. Obtain the first-level confidence score based on the high-frequency subband coefficients and the low-frequency subband coefficients. Obtain the first-level confidence score level based on the first-level confidence score. Confirm the confidence score threshold of the first-level confidence score. Construct a first-level classification system based on the first-level confidence score and the first-level confidence score level.

[0120] S52: Obtain the mean polarization angle of the target region and the mean polarization angle of the background region, the difference in polarization degree between the target region and the background region, the difference in reflectance, and the number of image blocks in the polarization spectral image within the k-th local window of the polarization spectral image. Based on the mean polarization angle of the target region, the mean polarization angle of the background region, the difference in polarization degree between the target region and the background region, the difference in reflectance, and the number of image blocks in the polarization spectral image, obtain the second-level confidence level.

[0121] S53: Obtain the joint weights of the polarization multispectral image and the spectral image, obtain the second-level confidence level based on the first-level confidence level, the second-level confidence level, and the joint weights of the polarization multispectral image and the spectral image, and construct a second-level classification system based on the second-level confidence level and the second-level confidence level;

[0122] S54: The first high-resolution multidimensional image data is fused with the spectral image to obtain a first fused image. The first fused image is input into a primary classification system. The primary classification system judges the first fused image according to a preset first confidence threshold to obtain the correct recognition result of the first fused image. When there are multiple correct recognition results in the first fused image, and the confidence of multiple correct recognition results is greater than the preset first confidence threshold, a second low-frequency fusion component and a second high-frequency fusion component are constructed. The spectral image is subjected to inverse discrete wavelet transform according to the second low-frequency fusion component and the second high-frequency fusion component to obtain a second high-resolution multidimensional image data. The second high-resolution multidimensional image data is fused with the polarization spectral image to obtain a second fused image. The second fused image is input into a secondary classification system. The secondary classification system judges the second fused image according to a preset second confidence threshold to obtain the correct recognition result of the second fused image.

[0123] S55: When the confidence level of the second fused image is greater than the preset second confidence level threshold, the correct recognition result of the second fused image is used as the final predicted target of the target to be recognized.

[0124] For inverse discrete wavelet transform of spectral images, it is necessary to make optimal selection based on the background information and characteristics of the target to be identified in the image data acquired by the camera, including but not limited to the fusion processing of spectral images and the fusion processing of polarization spectral images.

[0125] The first-level confidence level is obtained based on the high-frequency subband coefficient and the low-frequency subband coefficient, and is expressed by the following eighth formula:

[0126] ;

[0127] in, Indicates the first l Low-frequency subband coefficients of the spectral image obtained by layer discrete wavelet transform. Indicates the firstl High-frequency subband coefficients of RGB images obtained by layer discrete wavelet transform.

[0128] It should be noted that a first-level classification system is constructed based on the low-frequency subband coefficients of the spectral image and the high-frequency subband coefficients of the RGB image. A consistency index between the spectral image and the RGB image is defined to measure the correlation between the spectral image and the RGB image in spatial distribution, thereby enhancing the robustness of the first-level classification system.

[0129] It should be noted that the closer the first-level confidence value is to 1, the more synergistic the features of the spectral image and the high-frequency geometric features are in the matching region.

[0130] The first-level confidence level is obtained from the first-level confidence level using the following ninth formula:

[0131] ;

[0132] in, For the first confidence level, The confidence level is 1.

[0133] In this embodiment, after the confidence threshold of the first-level confidence level is confirmed, the correct recognition result of the first fused image is obtained by combining the image information of the first fused image with the prior information of the full-band database. The correct recognition result of the first fused image is then filtered to obtain the confidence level of the correct recognition result of the first fused image. When the confidence level of the correct recognition result of the first fused image is greater than the preset first confidence threshold and there is only one correct recognition result, the classification and recognition of the target to be identified is completed. When the confidence level of the correct recognition result of the first fused image is less than or equal to the preset first confidence threshold, all correct recognition results of the first fused image with a confidence level less than or equal to the first confidence threshold are discarded. For example, in this embodiment, the first confidence threshold of the first-level classification system is 0.8.

[0134] It should be noted that when there are multiple correct recognition results in the first fused image, these multiple correct recognition results are input into the secondary classification system to further classify and recognize the high-resolution multidimensional image data.

[0135] The secondary classification system needs to be determined in conjunction with the polarization phase consistency index, which is used to evaluate the consistency of the surface reflection direction of the target region in high-resolution multidimensional image data. The secondary confidence level is obtained based on the mean polarization angle of the target region, the mean polarization angle of the background region, the difference in polarization degree and reflectance between the target and background regions, and the number of image blocks in the spectral image, expressed by the following formula (number ten):

[0136] ;

[0137] in, The mean polarization angle of the target region in the k-th local window, The mean polarization angle of the background region in the k-th local window, The difference in image polarization degree between the target region and the background region. The reflectance difference between the target region and the background region is denoted as N, and the number of image blocks in the spectral image is denoted as N.

[0138] The second-level confidence level is obtained based on the first-level confidence level, the second-level confidence level, and the joint weight, and is expressed by the following eleventh formula:

[0139] ;

[0140] in, For the second confidence level, For the first confidence level, The weights are the combined weights of the polarization spectral image and the spectral image.

[0141] The image information of the second fused image is input into the secondary classification system to obtain the correct recognition result of the second fused image. The confidence level of the correct recognition result of the second fused image is obtained. The secondary classification system filters the correct recognition result of the second fused image based on the confidence level of the second fused image according to a preset second confidence level threshold. When the confidence level of the correct recognition result of the second fused image is greater than the preset second confidence level, the correct recognition result is used as the final predicted target of the target to be identified. When the confidence level of the correct recognition result of the second fused image is less than or equal to the preset second confidence level, all correct recognition results of the second fused image with a confidence level less than or equal to the second confidence level threshold are discarded. For example, in this embodiment, the second confidence level threshold of the secondary classification system is 0.6.

[0142] Additionally, in one embodiment, reference is made to Figure 6 ,exist Figure 5 In the embodiment shown, step S54, which constructs a second low-frequency fusion component and a second high-frequency fusion component, and performs an inverse discrete wavelet transform on the spectral image based on the second low-frequency fusion component and the second high-frequency fusion component to obtain the second high-resolution multidimensional image data, also includes the following steps:

[0143] S541: Calculate the degree of polarization and polarization angle of the polarization spectral image, and generate a binary mask and polarization angle gradient of the polarization spectral image based on the degree of polarization and polarization angle.

[0144] S542: Obtain the target mean, background mean, and background standard deviation of the polarization spectral image. Based on the target mean, background mean, and background standard deviation of the polarization spectral image, obtain the weight coefficient of the polarization degree of the polarization spectral image. Fuse the low-frequency information of the spectral image, the polarization degree of the polarization spectral image, the binary mask of the polarization spectral image, and the weight coefficient of the polarization degree of the polarization spectral image to obtain the second low-frequency fusion component of the spectral image.

[0145] S543: The first fusion weight factor of the spectral image is obtained based on the high-frequency information of the spectral image and the polarization angle gradient of the polarization spectral image, and the second fusion weight factor of the spectral image is obtained based on the first fusion weight factor of the spectral image.

[0146] S544: Obtain the horizontal gradient fusion component and the vertical gradient fusion component of the polarization angle gradient, and fuse the first fusion weight factor, the second fusion weight factor, the high-frequency information of the spectral image, the horizontal gradient fusion component and the vertical gradient fusion component to obtain the second high-frequency fusion component of the spectral image.

[0147] S545: After the low-frequency and high-frequency information generated by the discrete wavelet transform of the spectral image are converted into the second low-frequency fusion component and the second high-frequency fusion component, the inverse discrete wavelet transform of the spectral image is performed to recursively reconstruct the spectral image into the second high-resolution multidimensional image data.

[0148] It should be noted that polarization spectral images characterize the physical properties of the target surface, such as roughness, refractive index, and scattering characteristics. Therefore, information such as the degree of polarization and polarization angle of a polarization spectral image can reflect detailed information such as surface roughness and reflection angle. In the process of constructing the second low-frequency fusion component in this embodiment, since the polarization intensity of the light reflected from the target surface is closely related to the microstructure and roughness of the polarization spectral image, this embodiment uses the degree of polarization to reflect the polarization intensity of the light reflected from the material surface in the polarization spectral image.

[0149] It should be noted that the physical meaning of the polarization angle is used to describe the polarization direction of reflected light in a polarized spectral image. In this embodiment, the polarization angle gradient is used to reflect the local consistency of surface microstructures in the polarized spectral image. Therefore, this embodiment constructs the second low-frequency fusion component of the spectral image by introducing a binary mask of the polarized spectral image, and constructs the second high-frequency fusion component of the spectral image by introducing the polarization angle gradient of the polarized spectral image, thereby enhancing the high-resolution detail information and noise resistance of the spectral image.

[0150] It should be noted that the first fusion weight factor represents the weight coefficient of the first fused image, and the second fusion weight factor represents the weight coefficient of the second fused image.

[0151] The weighting coefficients for the degree of polarization of the polarization spectral image are obtained based on the target mean, background mean, and background standard deviation, and are expressed by the following twelfth formula:

[0152] ;

[0153] in, The weighting coefficients for the degree of polarization of the polarization spectrum image. For the target mean, For background mean, The standard deviation is the background value.

[0154] The low-frequency information of the spectral image, the weighting coefficient of the polarization degree of the polarization spectral image, the polarization degree of the polarization spectral image, and the binary mask of the polarization spectral image are fused to obtain the first low-frequency fusion component, which is expressed by the following formula (thirteenth equation):

[0155] ;

[0156] in, For the first low-frequency fusion component, For low-frequency information in spectral images, DoP is the weighting coefficient for the degree of polarization of the polarization spectrum image. It is a binary mask for polarization spectrum images.

[0157] In the polarization spectral image, the second angle value (45°) and the fourth angle value (135°) of the polarization angle gradient in the diagonal direction are obtained. Based on the second angle value, the fourth angle value, and the polarization angle gradient of the polarization spectral image, it is expressed by the following fourteenth formula:

[0158] ;

[0159] in, The polarization angle gradient of the polarization spectrum image, For the second angle value, This is the fourth angle value.

[0160] The first fusion weighting factor of the spectral image is obtained based on the high-frequency information of the spectral image and the polarization angle gradient of the polarization spectral image, and is expressed by the following fifteenth formula:

[0161] ;

[0162] in, The first fusion weight factor for the spectral image. For high-frequency information in spectral images, This represents the polarization angle gradient of the polarization spectrum image.

[0163] The second fusion weighting factor of the spectral image is obtained based on the first fusion weighting factor of the spectral image, and is expressed by the following sixteenth formula:

[0164] ;

[0165] in, The second fusion weighting factor for the spectral image. This is the first fusion weighting factor for the spectral image.

[0166] The first fusion weight factor of the spectral image, the second fusion weight factor of the spectral image, the high-frequency information of the spectral image, and the horizontal gradient fusion component and vertical gradient fusion component of the polarization spectral image are fused to obtain the second high-frequency fusion component of the spectral image, which is expressed by the following formula (number seventeen):

[0167] ;

[0168] in, The second high-frequency fusion component of the spectral image, As the first fusion weight factor, For high-frequency information in spectral images, For the second fusion weight factor, These are the horizontal gradient fusion component and the vertical gradient fusion component.

[0169] In discrete wavelet transform, the spectral image is subjected to a layer-by-layer inverse discrete wavelet transform based on the second low-frequency fusion component and the second high-frequency fusion component, as expressed by the following eighteenth formula:

[0170] ;

[0171] in, This is the second high-resolution multidimensional image data. The second low-frequency fusion component of the spectral image, For the high-frequency horizontal subband in the second high-frequency fusion component of the spectral image, For the high-frequency vertical subband in the second high-frequency fusion component of the spectral image, This refers to the high-frequency diagonal subband in the second high-frequency fusion component of the spectral image.

[0172] Additionally, in one embodiment, reference is made to Figure 7 ,exist Figure 6 In step S541 of the illustrated embodiment, calculating the degree of polarization and the polarization angle of the polarization spectral image further includes the following steps:

[0173] S5411: The first polarization spectral camera acquires the first angle value when the angle between the incident light and the polarization spectral image is 0°; the second polarization spectral camera acquires the second angle value when the angle between the incident light and the polarization spectral image is 45°; the third polarization spectral camera acquires the third angle value when the angle between the incident light and the polarization spectral image is 90°; the fourth polarization spectral camera acquires the fourth angle value when the angle between the incident light and the polarization spectral image is 135°.

[0174] S5412: Calculate the first polarized light component of the polarization spectrum image based on the first angle value, the second angle value, the third angle value, and the fourth angle value; calculate the second polarized light component of the polarization spectrum image based on the first angle value and the third angle value; and calculate the third polarized light component of the polarization spectrum image based on the second angle value and the fourth angle value.

[0175] S5413: Calculate the image polarization degree of the polarization spectrum image based on the first polarization component, the second polarization component, and the third polarization component, and calculate the polarization angle of the polarization spectrum image based on the second polarization component and the third polarization component.

[0176] It should be noted that the degree of polarization and the polarization angle of a polarization spectral image are calculated using the following nineteenth formula:

[0177] ;

[0178] ;

[0179] in, The first angle value when the angle between the incident light and the polarization spectrum image is 0°. The second angle value is the angle between the incident light and the polarization spectrum image when the angle is 45°. The third angle value is the angle between the incident light and the polarization spectrum image when the angle is 90°. The fourth angle value is the angle between the incident light and the polarization spectrum image when the angle is 135°. The first polarization component of the polarization spectrum image, The second polarization component of the polarization spectrum image, denoted as the third polarization component of the polarization spectrum image, DoP as the degree of polarization of the polarization spectrum image, and AoP as the polarization angle of the polarization spectrum image.

[0180] like Figure 8 As shown, Figure 8This is a structural diagram of a multispectral imaging data processing apparatus provided in one embodiment of the present invention. The present invention also provides a multispectral imaging data processing apparatus, comprising:

[0181] The processor 801 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the multispectral imaging data processing method provided in the above embodiments of the present invention.

[0182] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the multispectral imaging data processing method provided in the above embodiments of the present invention is implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801.

[0183] The 803 input / output interface is used to implement information input and output.

[0184] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0185] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0186] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0187] Memory 802, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 802 may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 802 may optionally include remotely located memories 802 relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative; the units described as separate components may or may not be physically separate, allowing them to be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] This invention also provides an electronic device, including the multispectral imaging data processing apparatus described above.

[0189] This invention also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described multispectral imaging data processing method.

[0190] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0191] The systems, apparatuses, modules, or units described in one or more of the above embodiments may be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, a computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0194] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A multispectral imaging data processing method, characterized in that, The method comprises the following steps: S1: a camera collects image data, identifies a target to be identified in the image data, obtains full-band data of the target to be identified, and constructs a full-band database of the target to be identified according to the full-band data; S2: prior information of the target to be identified is obtained, and a characteristic band of the target to be identified in the full-band database is determined according to the prior information; S3: a filter matched with the characteristic band is selected to enable the camera to collect multi-dimensional optical image data of the target to be identified, wherein the multi-dimensional optical image data comprises an RGB image, a spectral image and a polarized spectral image of the target to be identified, and the filter is used to filter out invalid spectral components or interfering light rays in the RGB image, the spectral image and the polarized spectral image; S4: low-frequency information and high-frequency information of the spectral image are obtained, the high-frequency information and the low-frequency information of the spectral image are fused with the RGB image respectively to obtain a first low-frequency fusion component and a first high-frequency fusion component of the spectral image, and the spectral image is high-resolution reconstructed according to the first low-frequency fusion component and the first high-frequency fusion component to obtain first high-resolution multi-dimensional image data; S5: the target to be identified is classified and discriminated according to the first high-resolution multi-dimensional image data to generate a classification and discrimination result of the target to be identified, a classification system is constructed to screen the classification and discrimination result, and a correct identification result of the target to be identified is obtained.

2. The multispectral imaging data processing method of claim 1, wherein, The method of high-resolution reconstructing the spectral image according to the first low-frequency fusion component and the first high-frequency fusion component to obtain first high-resolution multi-dimensional image data comprises the following steps: S41: discrete wavelet transform is performed on the spectral image to obtain low-frequency information and high-frequency information generated after the discrete wavelet transform of the spectral image; S42: discrete wavelet transform is performed on the RGB image to extract feature information of high-frequency edge information of the target to be identified, obtain a feature high-frequency sub-band containing target geometric features of the target to be identified in the RGB image, and construct geometric full-size features of the target to be identified according to the feature high-frequency sub-band; S43: each level of the low-frequency information generated after the discrete wavelet transform of the spectral image is retained, the low-frequency information is taken as a first low-frequency fusion component of the spectral image, the high-frequency information of the spectral image is weighted and fused with the geometric full-size features to obtain a first high-frequency fusion component of the spectral image, and after all the low-frequency information and the high-frequency information of the spectral image are converted into the first low-frequency fusion component and the first high-frequency fusion component, the spectral image is inverse transformed and reconstructed into first high-resolution multi-dimensional image data.

3. The multispectral imaging data processing method of claim 2, wherein, The method of inverse transforming and reconstructing the spectral image into first high-resolution multi-dimensional image data comprises the following steps: S431: Obtain a geometric feature map of the spectral image, and a target high-resolution geometric feature of the spectral image preset in the camera, obtain a Gaussian kernel and a gradient amplitude of the target high-resolution geometric feature, and quantize the geometric feature map of the spectral image according to the Gaussian kernel and the gradient amplitude; S432: Obtain a mean value and a standard deviation of the geometric feature map, and perform adaptive normalization processing on the quantized geometric feature map according to the mean value and the standard deviation; S433: Obtain an empirical curvature coefficient and an initial weight of the geometric feature map after adaptive normalization processing, obtain a weight factor of the spectral image according to the geometric feature map after adaptive normalization processing, the empirical curvature coefficient and the initial weight, fuse the weight factor of the spectral image, a feature high-frequency sub-band and high-frequency information, and obtain the first high-frequency fusion component of the spectral image; S434: Perform inverse discrete wavelet transform on the spectral image, generate an approximate component of the inverse discrete wavelet transform of the total number of layers minus one layer when the spectral image is subjected to inverse discrete wavelet transform, combine the approximate component of the inverse discrete wavelet transform of the total number of layers minus one layer and the first high-frequency fusion component of the spectral image, and perform inverse discrete wavelet transform on the combined approximate component of the inverse discrete wavelet transform of the total number of layers minus one layer and the first high-frequency fusion component of the spectral image to recursively reconstruct the spectral image into the first high-resolution multi-dimensional image data.

4. The multispectral imaging data processing method of claim 1, wherein, The constructed classification system filters the classification discrimination result to obtain a correct recognition result, including: S51: Obtain low-frequency sub-band coefficients of the spectral image after first-layer discrete wavelet transform and high-frequency sub-band coefficients of the RGB image after first-layer discrete wavelet transform, obtain a first-level confidence according to the high-frequency sub-band coefficients and the low-frequency sub-band coefficients, obtain a first-level confidence level according to the first-level confidence, confirm a confidence threshold of the first-level confidence, and construct a first-level classification system according to the first-level confidence and the first-level confidence level; S52: Obtain a polarization angle mean value of a target region and a polarization angle mean value of a background region, a polarization degree difference value between the target region and the background region, a reflectivity difference value, and an image block quantity of the polarized spectral image in a kth local window of the polarized spectral image, obtain a second-level confidence according to the polarization angle mean value of the target region, the polarization angle mean value of the background region, the polarization degree difference value between the target region and the background region, the reflectivity difference value, and the image block quantity of the polarized spectral image; S53: Obtain a joint weight of the polarized spectral image and the spectral image, obtain a second-level confidence level according to the first-level confidence, the second-level confidence, and the joint weight of the polarized spectral image and the spectral image, and construct a second-level classification system according to the second-level confidence and the second-level confidence level; S54: fuse the first high-resolution multi-dimensional image data and the spectral image to obtain a first fused image, input the first fused image into the primary classification system, and determine the first fused image according to a preset first confidence threshold to obtain a correct recognition result of the first fused image; when there are multiple correct recognition results of the first fused image and the confidence of the multiple correct recognition results is greater than the preset first confidence threshold, a second low-frequency fusion component and a second high-frequency fusion component are constructed, inverse discrete wavelet transform is performed on the spectral image according to the second low-frequency fusion component and the second high-frequency fusion component to obtain second high-resolution multi-dimensional image data, the second high-resolution multi-dimensional image data is fused with the polarized spectral image to obtain a second fused image, and the second fused image is input into the secondary classification system; the secondary classification system determines the second fused image according to a preset second confidence threshold to obtain a correct recognition result of the second fused image; S55: when the confidence of the second fused image is greater than the preset second confidence threshold, the correct recognition result of the second fused image is taken as the final prediction target of the to-be-recognized target.

5. The multispectral imaging data processing method of claim 4, wherein, The construction of the second low-frequency fusion component and the second high-frequency fusion component, the inverse discrete wavelet transform of the spectral image according to the second low-frequency fusion component and the second high-frequency fusion component, and the obtaining of the second high-resolution multi-dimensional image data include: S541: calculate a polarization degree and a polarization angle of the polarized spectral image, and generate a binary mask and a polarization angle gradient of the polarized spectral image according to the polarization degree and the polarization angle; S542: obtain a target mean value, a background mean value, and a background standard deviation of the polarized spectral image, obtain a weight coefficient of the polarization degree of the polarized spectral image according to the target mean value, the background mean value, and the background standard deviation, fuse low-frequency information of the spectral image, the polarization degree of the polarized spectral image, the binary mask of the polarized spectral image, and the weight coefficient of the polarization degree of the polarized spectral image to obtain a second low-frequency fusion component of the spectral image; S543: obtain a first fusion weight factor of the spectral image according to high-frequency information of the spectral image and a polarization angle gradient of the polarized spectral image, and obtain a second fusion weight factor of the spectral image according to the first fusion weight factor; S544: obtain a horizontal gradient fusion component and a vertical gradient fusion component of the polarization angle gradient, fuse the first fusion weight factor, the second fusion weight factor, the high-frequency information of the spectral image, the horizontal gradient fusion component, and the vertical gradient fusion component to obtain a second high-frequency fusion component of the spectral image; S545: After the low-frequency information and the high-frequency information generated after the spectral image is subjected to the discrete wavelet transform are both converted into the second low-frequency fusion component and the second high-frequency fusion component, inverse discrete wavelet transform is performed on the spectral image to recursively reconstruct the spectral image into second high-resolution multi-dimensional image data.

6. The multispectral imaging data processing method of claim 5, wherein, The camera is provided with a first polarized spectral camera, a second polarized spectral camera, a third polarized spectral camera, and a fourth polarized spectral camera, the first polarized spectral camera is used to collect a 0° polarization angle image, the second polarized spectral camera is used to collect a 45° polarization angle image, the third polarized spectral camera is used to collect a 90° polarized spectral image, and the fourth polarized spectral camera is used to collect a 135° polarized spectral image, and the polarization degree and polarization angle of the polarized spectral image are calculated, including: S5411: The first polarized spectral camera collects a first angle value when the included angle between the incident light of the polarized spectral image and the polarized spectral image is 0°; the second polarized spectral camera collects a second angle value when the included angle between the incident light of the polarized spectral image and the polarized spectral image is 45°; the third polarized spectral camera collects a third angle value when the included angle between the incident light of the polarized spectral image and the polarized spectral image is 90°; and the fourth polarized spectral camera collects a fourth angle value when the included angle between the incident light of the polarized spectral image and the polarized spectral image is 135°; S5412: A first polarized light component of the polarized spectral image is calculated according to the first angle value, the second angle value, the third angle value, and the fourth angle value, a second polarized light component of the polarized spectral image is calculated according to the first angle value and the third angle value, and a third polarized light component of the polarized spectral image is calculated according to the second angle value and the fourth angle value; S5413: An image polarization degree of the polarized spectral image is calculated according to the first polarized light component, the second polarized light component, and the third polarized light component, and a polarization angle of the polarized spectral image is calculated according to the second polarized light component and the third polarized light component.

7. The multispectral imaging data processing method of claim 6, wherein, Each camera of the camera is provided with a detector array, and the first polarized spectral camera, the second polarized spectral camera, the third polarized spectral camera, and the fourth polarized spectral camera are each provided with a four-quadrant polarizer and a multilayer dielectric film.

8. A multispectral imaging data processing apparatus, characterized by The memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the multi-spectral imaging data processing method according to any one of claims 1 to 7.

9. An electronic device, comprising: The multi-spectral imaging data processing device according to claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the multi-spectral imaging data processing method according to any one of claims 1 to 7.

Images