Weak light equipment inspection method based on adaptive multispectral fusion

Abstract

The invention provides a weak light equipment inspection method based on adaptive multispectral fusion, and relates to the technical field of equipment inspection, and the method comprises the steps: firstly obtaining a plurality of spectral image sequences of an equipment surface region through a multispectral collection unit, and combining the spectral image sequences into an initial multispectral collection data set; carrying out adaptive decomposition processing on the initial multispectral acquisition data set and recombining the initial multispectral acquisition data set into a decomposed multispectral component data set, and carrying out cross-component feature association and enhancement by utilizing a spatial-spectral joint perception and reconstruction network to generate an enhanced multispectral joint feature tensor; an equipment abnormal state space distribution diagram is generated based on the enhanced multispectral joint feature tensor, finally, a composite inspection task instruction is generated according to the equipment abnormal state space distribution diagram, the unmanned inspection aircraft is triggered to execute differentiated re-check shooting operation, and the weak light equipment inspection efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of equipment inspection technology, and more specifically, to a method for inspecting low-light equipment based on adaptive multispectral fusion. Background Technology

[0002] In the inspection of large wind tunnel equipment, the wind tunnel, as a rectangular closed passage, has large guide vanes stacked upwards at its four corners. These guide vanes play a key role in guiding airflow and reducing wind resistance, and their condition directly affects the operating efficiency and safety of the wind tunnel equipment.

[0003] However, the internal environment of a wind tunnel is unique, often characterized by complete darkness or extremely poor lighting. Traditional equipment inspection methods are ineffective in such environments. For example, inspection methods based on single visible light are insufficient to obtain clear and complete images of the blade surface due to the lack of illumination. This makes it difficult to accurately determine whether there are any abnormalities such as loose metal plates or fixing screws on the blade surface, which can easily lead to missed inspections and pose a threat to the safe operation of wind tunnel equipment.

[0004] Meanwhile, existing inspection methods are mostly quite limited, relying on a single spectrum or a single type of sensor for detection, failing to comprehensively acquire diverse information about the blade surface. For example, using laser detection alone may only obtain information about the geometric structure of the blade surface, making it difficult to accurately identify changes in material composition; while relying solely on infrared light detection can acquire thermal radiation information, it is not precise enough in judging anomalies in surface geometry. Moreover, the lack of methods for effectively fusing and processing multispectral data prevents the full utilization of the advantages of different spectra in detection, making it difficult to achieve accurate inspection of the blade surface condition. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, the present invention provides a method for inspecting low-light devices based on adaptive multispectral fusion, the method comprising:

[0006] The visible light spectrum image sequence, near-infrared spectrum image sequence, and thermal infrared spectrum image sequence of the equipment surface area are acquired by the multispectral acquisition unit deployed in the equipment inspection area under low light environment, and combined into an initial multispectral acquisition data set; The initial multispectral acquisition data set is subjected to adaptive decomposition processing based on image content features, which decomposes the initial multispectral acquisition data set into a spatial structure-dominant component layer, a spectral material-dominant component layer, and a thermal radiation-dominant component layer, and then reassembles it into a decomposed multispectral component data set. The pre-built spatial-spectral joint sensing and reconstruction network is invoked to perform cross-component feature association and enhancement processing on the decomposed multispectral component data set, generating an enhanced multispectral joint feature tensor that integrates spatial structure, spectral material and thermal radiation information; Based on the enhanced multispectral joint feature tensor, the abnormal state of the surface region of the equipment is jointly determined and spatially located to generate a spatial distribution map of the abnormal state of the equipment. The spatial distribution map of the abnormal state of the equipment includes the geometric boundary coordinates of the abnormal surface geometric structure region, the abnormal surface material composition region, and the abnormal surface thermal field distribution region. Based on the spatial distribution map of the abnormal state of the equipment, a composite inspection task instruction is generated and sent to the unmanned inspection aircraft that is connected to the multispectral acquisition unit to trigger the unmanned inspection aircraft to fly to the abnormal surface geometry area, the abnormal surface material composition area, and the abnormal surface thermal field distribution area to perform differentiated spectral mode verification and imaging operations.

[0007] Furthermore, this invention also provides a low-light equipment inspection system based on adaptive multispectral fusion, comprising: A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the aforementioned low-light device inspection method based on adaptive multispectral fusion by executing the machine-executable instructions.

[0008] Based on the above, by deploying multispectral acquisition units in the equipment inspection area under low-light conditions, visible light, near-infrared, and thermal infrared spectral image sequences of the equipment surface area are acquired and combined into an initial multispectral acquisition data set. This initial multispectral acquisition data set undergoes adaptive decomposition processing, breaking it down into spatial structure-dominant, spectral material-dominant, and thermal radiation-dominant component layers, which are then recombined. A pre-constructed spatial-spectral joint sensing and reconstruction network is invoked to perform cross-component feature association and enhancement processing on the decomposed multispectral component data set, generating an enhanced multispectral joint feature tensor that integrates multiple information sources. Based on this enhanced multispectral joint feature tensor, the abnormal state of the equipment surface area is jointly determined and spatially located, generating a spatial distribution map of equipment abnormal states containing the geometric boundary coordinates of various abnormal regions. This map can identify various abnormal conditions on the equipment surface and achieve precise positioning. Finally, a composite inspection task command is generated based on this spatial distribution map of equipment abnormal states and sent to the unmanned inspection drone, triggering it to perform differentiated spectral mode verification and imaging operations. This achieves an automated and intelligent process from anomaly detection to verification, significantly improving the efficiency and accuracy of equipment inspection in low-light environments. Attached Figure Description

[0009] Figure 1This is a schematic diagram of the execution flow of the weak light equipment inspection method based on adaptive multispectral fusion provided in an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of exemplary hardware and software components of a low-light equipment inspection system based on adaptive multispectral fusion provided in an embodiment of the present invention. Detailed Implementation

[0011] Figure 1 This is a flowchart illustrating a low-light equipment inspection method based on adaptive multispectral fusion provided in one embodiment of the present invention, which will be described in detail below.

[0012] Step S110: Obtain visible light spectral image sequences, near-infrared spectral image sequences, and thermal infrared spectral image sequences of the equipment surface area through the multispectral acquisition unit deployed in the equipment inspection area under low light conditions, and combine them into an initial multispectral acquisition data set.

[0013] In this embodiment, the equipment inspection area is the corner of the rectangular enclosed passage of the wind tunnel, where the guide vanes installed need to be inspected. The multispectral acquisition unit is pre-installed on a fixed bracket inside the wind tunnel. This multispectral acquisition unit includes a visible light camera, a near-infrared camera, and a thermal infrared camera, all of which have their field of view adjusted to completely cover the surface area of ​​the guide vanes.

[0014] In low-light environments, the visible light camera operates in high-sensitivity mode, with the exposure time set to the minimum required for a clear image to avoid blurring. The near-infrared camera activates its built-in near-infrared supplementary lighting module, adjusting the intensity to a level that does not interfere with normal equipment operation. The thermal infrared camera is set to high-sensitivity mode to capture subtle temperature differences on the blade surface. At the start of the inspection, the multispectral acquisition unit simultaneously activates all three cameras at a preset sampling frequency, continuously acquiring N frames to form a visible light spectral image sequence, a near-infrared spectral image sequence, and a thermal infrared spectral image sequence. In the visible light spectral image sequence, each frame has a resolution of W×H pixels, with each pixel value representing the visible light intensity at that location. The near-infrared spectral image sequence also has a resolution of W×H pixels, with pixel values ​​corresponding to the intensity of the near-infrared band. The thermal infrared spectral image sequence has a resolution of W×H pixels, with pixel values ​​representing the temperature at that location. Subsequently, the three spectral image sequences are combined according to time order and spectral channels to generate an initial multispectral acquisition data set. The structure of this initial multispectral acquisition data set is a four-dimensional array with dimensions of time frame, height, width, and spectral channels, namely [number of time frames, W, H, 3]. The three spectral channels correspond to visible light, near-infrared light, and thermal infrared light, respectively.

[0015] Step S120: Perform adaptive decomposition processing based on image content features on the initial multispectral acquisition data set, decompose the initial multispectral acquisition data set into a spatial structure-dominant component layer, a spectral material-dominant component layer, and a thermal radiation-dominant component layer, and reassemble them into a decomposed multispectral component data set.

[0016] In this embodiment, the initial multispectral acquisition data set is decomposed to separate different types of features, enabling more accurate feature analysis and anomaly detection in subsequent steps. Since the structural, material, and thermal radiation characteristics of the guide vane surface exhibit varying degrees of saliency in different spectral images, adaptive decomposition can highlight the dominant features of each component. The specific processing procedure will be described in detail in subsequent sub-steps.

[0017] Step S121: Extract the grayscale matrix of each frame of the visible light spectrum image in the visible light spectrum image sequence, calculate the grayscale difference between each pixel and its eight neighboring pixels in the grayscale matrix, synthesize the grayscale difference vectors in all directions to obtain the local gradient magnitude of the pixel, and arrange the local gradient magnitudes of each pixel according to the pixel coordinates to form a visible light gradient magnitude distribution map.

[0018] In this embodiment, for each frame of the visible light spectrum image sequence, it is first converted into a grayscale matrix, where each element corresponds to the grayscale value of a pixel in the image. For each pixel (i, j) in the grayscale matrix, its eight neighboring pixels are considered: (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1). The grayscale difference between this pixel and each of its eight neighboring pixels is calculated, resulting in grayscale differences in eight directions. These grayscale differences are used as vector components, and the local gradient magnitude of the pixel is synthesized by calculating the magnitude of the vectors. Specifically, for the grayscale difference in each direction, the squares are summed, and the square root of the sum is taken to obtain the local gradient magnitude of the pixel. The local gradient magnitudes of all pixels are arranged according to their coordinate positions in the image, forming a visible light gradient magnitude distribution map. This visible light gradient amplitude distribution map can reflect the edge and texture changes in different regions of the image, which is of great significance for extracting structural features of the guide vane surface.

[0019] Step S122: Perform threshold segmentation on the visible light gradient magnitude distribution map, mark the pixels with local gradient magnitude greater than the first gradient threshold as strong edge candidate points, mark the pixels with local gradient magnitude between the first gradient threshold and the second gradient threshold as weak texture candidate points, and mark the pixels with local gradient magnitude less than the second gradient threshold as smooth region candidate points.

[0020] In this embodiment, after obtaining the visible light gradient amplitude distribution map, it is necessary to classify and label pixels with different gradient amplitudes through threshold segmentation. The first and second gradient thresholds are preset based on extensive historical data and experimental experience, with the first threshold being greater than the second. For each pixel in the visible light gradient amplitude distribution map, its local gradient amplitude is compared with these two thresholds. If the local gradient amplitude is greater than the first gradient threshold, it indicates that the pixel location may have a significant edge, and it is marked as a strong edge candidate point; if the local gradient amplitude is between the first and second gradient thresholds, it indicates that the region has some texture variation, but the edge is not obvious, and it is marked as a weak texture candidate point; while pixels with a local gradient amplitude less than the second threshold are located in relatively smooth regions, and are marked as smooth region candidate points. Through the above classification and labeling, different structural regions on the surface of the guide vane can be preliminarily distinguished.

[0021] Step S123: Extract the pixel intensity matrix of each frame of the near-infrared spectral image in the near-infrared spectral image sequence. For each local sliding window centered on a pixel, calculate the gray-level co-occurrence matrix of all pixel intensity values ​​within the local sliding window. Calculate the entropy value from the gray-level co-occurrence matrix as the local gray-level co-occurrence matrix entropy value of that pixel. Arrange the local gray-level co-occurrence matrix entropy values ​​of each pixel according to the pixel coordinates to form a near-infrared texture complexity distribution map.

[0022] In this embodiment, for each frame of the near-infrared spectral image sequence, its pixel intensity matrix is ​​extracted. Each element of the pixel intensity matrix represents the near-infrared intensity value of the corresponding pixel. Then, a local sliding window is set, the size of which is selected according to the texture feature scale of the guide vane surface, for example, a window of size M×M. The window is slid across the pixel intensity matrix with each pixel as the center. For the pixel intensity value within each window, its gray-level co-occurrence matrix is ​​calculated. The gray-level co-occurrence matrix is ​​obtained by calculating the probability that two pixel gray-level values ​​appear simultaneously in a certain direction and distance. After calculating the gray-level co-occurrence matrix, the entropy value of the matrix is ​​calculated according to the definition of entropy. The magnitude of the entropy value reflects the complexity of the image texture; the larger the entropy value, the more complex the texture. The local gray-level co-occurrence matrix entropy values ​​corresponding to each pixel are arranged according to their pixel coordinates to form a near-infrared texture complexity distribution map. This near-infrared texture complexity distribution map can reflect the material texture features of different regions on the guide vane surface.

[0023] Step S124: Perform cluster analysis on the near-infrared texture complexity distribution map, mark the pixels with local gray-level co-occurrence matrix entropy values ​​greater than the first entropy value cluster center as first material candidate points, and mark the pixels with local gray-level co-occurrence matrix entropy values ​​less than the second entropy value cluster center as second material candidate points.

[0024] In this embodiment, after obtaining the near-infrared texture complexity distribution map, cluster analysis is used to classify the pixels. First, a clustering algorithm (such as K-means clustering) is used to cluster the local gray-level co-occurrence matrix entropy values ​​in the near-infrared texture complexity distribution map, resulting in multiple cluster centers. Based on the possible material types on the guide vane surface, two main cluster centers are selected: a first entropy value cluster center and a second entropy value cluster center, where the first entropy value cluster center is greater than the second entropy value cluster center. Pixels with local gray-level co-occurrence matrix entropy values ​​greater than the first entropy value cluster center are marked as first material candidate points; the texture of these pixels is relatively complex and may belong to a specific material. Pixels with local gray-level co-occurrence matrix entropy values ​​less than the second entropy value cluster center are marked as second material candidate points; the texture of these pixels is relatively simple and may belong to another material. Through the above clustering and marking, regions of different materials on the guide vane surface can be preliminarily distinguished.

[0025] Step S125: Extract the temperature matrix of each frame of the thermal infrared spectrum image in the thermal infrared spectrum image sequence. Calculate the temperature change rate of each pixel in the temperature matrix in the horizontal and vertical directions. Combine the temperature change rate vectors in the horizontal and vertical directions to obtain the local temperature gradient vector of the pixel. Arrange the magnitude of the local temperature gradient vector of each pixel according to the pixel coordinates to form a thermal infrared temperature gradient amplitude distribution map. Record the direction angle of the local temperature gradient vector of each pixel to form a thermal infrared temperature gradient direction distribution map.

[0026] In this embodiment, for each frame of the thermal infrared spectral image sequence, its temperature matrix is ​​extracted, where each element represents the temperature value of the corresponding pixel. For each pixel (i, j) in the temperature matrix, its temperature change rate in the horizontal and vertical directions is calculated. The horizontal temperature change rate is obtained by dividing the temperature difference between the pixel and its right-hand neighbor (i, j+1) by the horizontal pixel distance, and the vertical temperature change rate is obtained by dividing the temperature difference between the pixel and its lower-hand neighbor (i+1, j) by the vertical pixel distance. The horizontal and vertical temperature change rates are used as two components of a vector to synthesize the local temperature gradient vector of the pixel. The magnitude of this local temperature gradient vector is calculated by taking the square root of the sum of the squares of the horizontal and vertical temperature change rates. The magnitudes of the local temperature gradient vectors of all pixels are arranged according to their pixel coordinates to form a thermal infrared temperature gradient amplitude distribution map. Simultaneously, the direction angle of the local temperature gradient vector at each pixel is recorded. This direction angle is obtained by calculating the ratio of the horizontal temperature change rate to the vertical temperature change rate using the arctangent function, forming a thermal infrared temperature gradient direction distribution map. These two distribution maps can reflect the temperature distribution changes on the surface of the guide vanes, which is crucial for the extraction of the dominant thermal radiation components.

[0027] Step S126: Based on the thermal infrared temperature gradient amplitude distribution map and the thermal infrared temperature gradient direction distribution map, the pixel areas with thermal infrared temperature gradient amplitude greater than the temperature gradient threshold and continuous thermal infrared temperature gradient direction angles are marked as candidate heat source centers, and the pixel areas with thermal infrared temperature gradient amplitude less than the temperature gradient threshold and absolute temperature values ​​greater than the ambient temperature threshold are marked as candidate thermal diffusion influence areas.

[0028] In this embodiment, the temperature gradient threshold and ambient temperature threshold are set based on the temperature characteristics of the guide vanes during normal wind tunnel operation. First, in the thermal infrared temperature gradient amplitude distribution map, pixels with thermal infrared temperature gradient amplitudes greater than the temperature gradient threshold are identified. Then, combined with the thermal infrared temperature gradient direction distribution map, it is determined whether the thermal infrared temperature gradient direction angles of these pixels are continuous and consistent. If pixels in a certain region satisfy the condition that the thermal infrared temperature gradient amplitude is greater than the temperature gradient threshold and the direction angles are continuous and consistent, then this region is marked as a candidate region for the heat source center, and this region may contain localized heating points. For pixel regions where the thermal infrared temperature gradient amplitude is less than the temperature gradient threshold but the absolute temperature value is greater than the ambient temperature threshold, since their temperature is higher than the ambient temperature but the rate of temperature change is small, they may be affected by thermal diffusion from the heat source center, and therefore are marked as candidate regions for thermal diffusion influence. Through the above marking, the thermal radiation-related regions on the surface of the guide vanes can be preliminarily determined.

[0029] Step S127: Based on the labeling results of each spectrum, perform joint decision-making on the composition of the pixels and generate a set of decomposed multispectral composition data.

[0030] In this embodiment, the labeling results of the visible light, near-infrared, and thermal infrared spectral images from the preceding steps are combined to make a joint decision on the component assignment of each pixel. This is because the labeling results of a single spectrum may have certain limitations. By jointly analyzing the multispectral labeling results, the dominant component of each pixel can be determined more accurately. The specific decision fusion rules will be explained in detail in subsequent sub-steps. Finally, based on the decision results, the initial multispectral acquisition data set is decomposed into a spatial structure dominant component layer, a spectral material dominant component layer, and a thermal radiation dominant component layer, and then reassembled into a decomposed multispectral component data set.

[0031] Step S1271: For each spatial pixel location, the marking results of strong edge candidate points, weak texture candidate points, and smooth region candidate points in the visible light gradient amplitude distribution map, the clustering results of the first material candidate points and the second material candidate points in the near-infrared texture complexity distribution map, and the marking results of the heat source center candidate area and the heat diffusion influence candidate area in the thermal infrared temperature gradient amplitude distribution map and the thermal infrared temperature gradient direction distribution map are jointly decided. According to the preset decision fusion rules, the pixel point is determined to belong to the spatial structure dominant component, the spectral material dominant component, or the thermal radiation dominant component.

[0032] In this embodiment, the preset decision fusion rule is formulated based on the weights of each spectral labeling result. For each spatial pixel location, its labeling status in the visible, near-infrared, and thermal infrared spectra is comprehensively considered. For example, if a pixel is labeled as a strong edge candidate point in the visible gradient amplitude distribution map, and the labeling results in other spectra are not obvious, then the pixel is more likely to belong to the dominant spatial structure component; if a pixel is labeled as a first material candidate point or a second material candidate point in the near-infrared texture complexity distribution map, and the labeling results in the visible and thermal infrared spectra are weak indicative of structure and thermal radiation, then the pixel belongs to the dominant spectral material component; if a pixel is labeled as a heat source center candidate region or a thermal diffusion influence candidate region in the thermal infrared correlation labeling, and the labeling results in other spectra are not prominent, then it belongs to the dominant thermal radiation component. The decision fusion rule also sets priorities, for example, when a pixel simultaneously meets the labeling conditions of spatial structure and spectral material, its affiliation is determined according to the degree of attention paid to structure or material in actual applications. Through the above joint decision, a unique component affiliation is determined for each pixel.

[0033] Step S1272: Based on the compositional affiliation of each pixel, extract the original pixel values ​​of the pixels belonging to the dominant spatial structure component from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence, retain them according to the original spectral channels and spatial positions, and assign the position of the pixels belonging to the other two components to 0, thereby generating a spatial structure dominant component layer composed of visible light spatial structure image, near-infrared spatial structure image, and thermal infrared spatial structure image.

[0034] In this embodiment, after determining the component affiliation of each pixel, for the spatial structure dominant component layer, the original pixel values ​​of all pixels belonging to the spatial structure dominant component are extracted from the initial multispectral acquisition data set. These pixel values ​​come from the visible light, near-infrared, and thermal infrared spectral channels and are preserved according to their spectral channels and spatial positions in the original image. For pixel positions belonging to the spectral material dominant component and the thermal radiation dominant component, their corresponding pixel values ​​in the spatial structure dominant component layer are assigned a value of 0. After this processing, the non-zero pixel values ​​in the spatial structure dominant component layer centrally reflect the spatial structure characteristics of the guide vane surface, while interference from other components is eliminated. This component layer consists of visible light spatial structure images, near-infrared spatial structure images, and thermal infrared spatial structure images, each corresponding to a spectral channel, collectively constituting a multispectral representation of the spatial structure dominant component.

[0035] Step S1273: Based on the composition of each pixel, extract the original pixel values ​​of the pixels belonging to the dominant component of the spectral material from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence, retain them according to the original spectral channels and spatial positions, and assign the position of the pixels belonging to the other two components to 0, thereby generating a spectral material dominant component layer composed of visible light spectral material images, near-infrared spectral material images, and thermal infrared spectral material images.

[0036] In this embodiment, similar to generating the spatial structure dominant component layer, for the spectral material dominant component layer, the original pixel values ​​of pixels belonging to the spectral material dominant component are extracted from the initial multispectral acquisition data set, retaining their original spectral channels and spatial positions. The positions of pixels belonging to the spatial structure dominant component and the thermal radiation dominant component are assigned a value of 0. Thus, the non-zero pixel values ​​in the spectral material dominant component layer mainly reflect the material characteristics of the guide vane surface. This spectral material dominant component layer consists of visible light spectral material images, near-infrared spectral material images, and thermal infrared spectral material images.

[0037] Step S1274: Based on the compositional affiliation of each pixel, extract the original pixel values ​​of the pixels belonging to the dominant thermal radiation component from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence, retain them according to the original spectral channels and spatial positions, and assign the position of the pixels belonging to the other two components to 0, thereby generating a thermal radiation dominant component layer composed of visible light thermal radiation image, near-infrared thermal radiation image, and thermal infrared thermal radiation image.

[0038] In this embodiment, the generation method of the thermal radiation dominant component layer is similar to that of the two component layers mentioned above. The original pixel values ​​of pixels belonging to the thermal radiation dominant component are extracted, their spectral channels and spatial positions are preserved, and the pixel positions of other components are assigned a value of 0. The thermal radiation dominant component layer consists of visible light thermal radiation images, near-infrared thermal radiation images, and thermal infrared thermal radiation images. Among these, the non-zero pixel values ​​of the thermal infrared thermal radiation images are particularly important, directly reflecting the temperature distribution and thermal radiation characteristics of the guide vane surface.

[0039] Step S1275: The spatial structure dominant component layer, the spectral material dominant component layer, and the thermal radiation dominant component layer are superimposed and recombined in the spectral channel dimension to generate a multi-channel data volume with a three-level structure as a decomposed multispectral component data set. Each spatial pixel position in the decomposed multispectral component data set is associated with three spectral vectors from the spatial structure dominant component layer, the spectral material dominant component layer, and the thermal radiation dominant component layer, respectively. Each spectral vector contains a visible light band intensity value, a near-infrared band intensity value, and a thermal infrared band intensity value.

[0040] In this embodiment, after obtaining the spatial structure-dominant component layer, the spectral material-dominant component layer, and the thermal radiation-dominant component layer, these three component layers are superimposed and recombined along the spectral channel dimension. Each component layer contains three spectral channels: visible light, near-infrared, and thermal infrared. After superposition and recombination, a multi-channel data volume with a three-level structure is formed, namely, the decomposed multispectral component data set. In this data set, each spatial pixel location is associated with three spectral vectors, corresponding to the three component layers respectively. Each spectral vector contains the intensity values ​​of the visible light band, the near-infrared band, and the thermal infrared band, which enriches the information of each pixel and facilitates subsequent feature association and enhancement processing.

[0041] Step S130: Call the pre-built spatial-spectral joint sensing and reconstruction network to perform cross-component feature association and enhancement processing on the decomposed multispectral component data set, and generate an enhanced multispectral joint feature tensor that integrates spatial structure, spectral material and thermal radiation information.

[0042] In this embodiment, the pre-constructed spatial-spectral joint sensing and reconstruction network is a deep learning network specifically designed for processing multispectral component data. This network can fully explore the intrinsic relationships between spatial structure, spectral material, and thermal radiation components in the decomposed multispectral component data set, and enhance the features to generate a more discriminative joint feature tensor. The specific structure and processing flow of the network will be described in detail in subsequent sub-steps.

[0043] Step S131: Input the spatial structure dominant component layer in the decomposed multispectral component data set into the spatial structure perception subnet of the spatial-spectral joint sensing and reconstruction network. Convolve the visible light spatial structure image in the spatial structure dominant component layer through the preset multi-directional Gabor filter bank in the spatial structure perception subnet, extract edge responses in different directions, perform non-maximum suppression and double threshold hysteresis processing on the edge responses in all directions, and generate an edge probability map.

[0044] In this embodiment, the spatial structure perception subnet is used to extract edge and structural features from the spatial structure dominant component layers. First, the spatial structure dominant component layers are input into this subnet, with a focus on processing the visible light spatial structure image, as visible light images have advantages in representing spatial details. The multi-directional Gabor filter bank is a key component of this subnet, used to extract edge responses with different orientations; the specific processing steps will be described in detail in subsequent sub-steps.

[0045] Step S1311: Initialize the multi-directional Gabor filter bank and set the orientation angle parameters of multiple Gabor filter cores with different orientations. The orientation angle parameters uniformly cover the range from 0 degrees to 180 degrees.

[0046] In this embodiment, the multi-directional Gabor filter bank includes multiple Gabor filter cores with different orientations, for example, eight filter cores with orientation angle parameters of 0 degrees, 22.5 degrees, 45 degrees, 67.5 degrees, 90 degrees, 112.5 degrees, 135 degrees, and 157.5 degrees, respectively. This can uniformly cover the range from 0 degrees to 180 degrees to capture edge features of the guide vane surface in different directions. Other parameters of each Gabor filter core, such as wavelength, bandwidth, and phase, are set according to the characteristics of the visible light spatial structure image to ensure effective extraction of edge information.

[0047] Step S1312: Perform a two-dimensional convolution operation between the Gabor filter kernel of each orientation and the visible light spatial structure image to obtain the edge response intensity map of the pixel in each orientation.

[0048] In this embodiment, for each orientation of the Gabor filter kernel, a two-dimensional convolution operation is performed with the visible light spatial structure image. During the convolution process, the filter kernel slides across the image, and the convolution result at each position is the edge response intensity at that position in the corresponding orientation. Through the above operation, edge response intensity maps for each orientation are obtained, and these intensity maps reflect the edge features of the image in different directions.

[0049] Step S1313: For each pixel, traverse the edge response intensity map of all orientations, select the maximum edge response intensity of the pixel in all orientations as the initial edge intensity of the pixel, and record the orientation corresponding to the maximum value as the main edge direction of the pixel, and generate the initial edge intensity map and the main edge direction map.

[0050] In this embodiment, for each pixel in the visible light spatial structure image, its response intensity value in the edge response intensity map across all orientations is examined. The maximum value is selected as the initial edge intensity of that pixel, reflecting its response intensity in the most probable edge direction. Simultaneously, the orientation corresponding to this maximum value is recorded as the principal edge direction of that pixel. The initial edge intensities of all pixels are arranged according to their coordinates to form an initial edge intensity map, and the principal edge directions are arranged to form a principal edge direction map.

[0051] Step S1314: Perform non-maximum suppression processing on the initial edge intensity map. For each pixel, compare the initial edge intensity of the pixel with the initial edge intensity of the two adjacent pixels in the direction indicated by the main edge direction map. If the initial edge intensity of the pixel is not the maximum value, set the initial edge intensity of the pixel to 0 and generate the edge intensity map after non-maximum suppression.

[0052] In this embodiment, the purpose of non-maximum suppression (NMS) is to refine wide edges in the edge intensity map to single-pixel-wide edges. For each pixel in the initial edge intensity map, two adjacent pixels in that direction are found according to the direction indicated by the main edge direction map. The initial edge intensity of the current pixel is compared with the initial edge intensity of these two adjacent pixels. If the initial edge intensity of the current pixel is not the maximum value, it means that the pixel is not at the center of the edge, and its initial edge intensity is set to 0. After the above processing, an edge intensity map with NMS is generated, in which the edges are clearer and more refined.

[0053] Step S1315: Apply dual threshold processing to the edge intensity map after non-maximum suppression, set a first edge threshold and a second edge threshold, mark the pixels with edge intensity greater than the first edge threshold as first edge pixels, mark the pixels with edge intensity between the second edge threshold and the first edge threshold as second edge pixels, and mark the pixels with edge intensity less than the second edge threshold as third edge pixels.

[0054] In this embodiment, the first edge threshold and the second edge threshold are determined based on the statistical characteristics of the edge intensity map after non-maximum suppression, with the first edge threshold being greater than the second edge threshold. Through dual-threshold processing, the pixels in the edge intensity map are divided into three categories: the first edge pixel is a strong edge point with obvious edge features; the second edge pixel is a weak edge point with relatively weak edge features; and the third edge pixel is a non-edge point.

[0055] Step S1316: Mark the first edge pixel as a connected component. Starting from the first edge pixel, search for second edge pixels connected to the first edge pixel in the eight-neighborhood. Mark all the searched connected second edge pixels as first edge pixels. Finally, all the positions marked as first edge pixels constitute an edge probability map with a single-pixel wide continuous edge.

[0056] In this embodiment, firstly, connected component labeling is performed on the first edge pixels to determine the connected regions of strong edges. Then, starting from each first edge pixel, a search is conducted within its eight-neighborhood for connected second edge pixels. Since the second edge pixels may be extensions of strong edges, these connected second edge pixels are also labeled as first edge pixels. Through this process, broken edges can be connected to form an edge probability map with continuous edges of single-pixel width. This edge probability map can accurately reflect the spatial structural edge features of the guide vane surface.

[0057] Step S132: Multiply the edge probability map pixel-by-pixel with the near-infrared spatial structure image in the dominant spatial structure layer to obtain a weighted near-infrared edge enhancement image; simultaneously, multiply the edge probability map pixel-by-pixel with the thermal infrared spatial structure image in the dominant spatial structure layer to obtain a weighted thermal infrared edge enhancement image; concatenate the weighted near-infrared edge enhancement image and the weighted thermal infrared edge enhancement image along the channel dimension to generate a preliminary multi-channel fusion feature map; input the preliminary multi-channel fusion feature map into the first spatial convolution module of the spatial structure perception subnet, and gradually expand the spatial receptive field through multiple cascaded convolutional layers to extract geometric structure primitive feature maps.

[0058] In this embodiment, the pixel values ​​in the edge probability map represent the probability that the location is an edge. The edge probability map is multiplied pixel-by-pixel by the near-infrared spatial structure image to obtain a weighted near-infrared edge enhancement image. This enhances edge-related regions in the near-infrared image, highlighting edge features in the near-infrared band. Similarly, the edge probability map is multiplied pixel-by-pixel by the thermal infrared spatial structure image to obtain a weighted thermal infrared edge enhancement image. Then, these two weighted enhancement images are stitched together along the channel dimension to form a preliminary multi-channel fusion feature map. This preliminary multi-channel fusion feature map contains information on visible light edge probability, near-infrared edge enhancement, and thermal infrared edge enhancement. The preliminary multi-channel fusion feature map is input into a first spatial convolution module, which consists of multiple cascaded convolutional layers. The kernel size and stride of each convolutional layer are designed to gradually expand the spatial receptive field, enabling the capture of geometric structural features at different scales. Finally, a geometric structural primitive feature map is extracted, which contains information on the basic geometric structural units of the guide vane surface.

[0059] Step S133: Input the geometric structure primitive feature map into the spatial attention module of the spatial structure perception subnet, perform global max pooling and global average pooling on the geometric structure primitive feature map to generate two spatial context descriptors, concatenate them, calculate the spatial attention weight map through a shared fully connected layer, multiply the spatial attention weight map with the geometric structure primitive feature map element by element, and output the spatial structure enhancement feature map.

[0060] In this embodiment, the spatial attention module is used to highlight important spatial regions in the geometric structure primitive feature map. First, global max pooling and global average pooling operations are performed on the geometric structure primitive feature map. Global max pooling extracts the most salient response regions in the feature map, while global average pooling reflects the overall average response of the feature map. The results of these two pooling operations are used as spatial context descriptors and concatenated to form a feature vector that integrates local salient information and global average information. Then, this feature vector is nonlinearly transformed through a shared fully connected layer to generate a spatial attention weight map, where each pixel value represents the importance of the corresponding position in the geometric structure features. Finally, the spatial attention weight map is multiplied element-wise with the geometric structure primitive feature map, which enhances the features of important regions and suppresses the features of unimportant regions, thereby outputting a spatial structure enhanced feature map.

[0061] Step S134: Input the spectral material dominant component layer in the decomposed multispectral component data set into the spectral material perception subnet of the spatial-spectral joint perception and reconstruction network, and perform spectral dimension vectorization on the visible light spectral material image, the near-infrared spectral material image and the thermal infrared spectral material image in the spectral material dominant component layer. The visible light, near-infrared and thermal infrared intensity values ​​of each pixel position are used to form the original spectral vector. The original spectral vectors of all pixels are arranged in space to form the original spectral feature cube.

[0062] In this embodiment, the spectral material perception subnetwork is used to extract material features from the spectral material dominant component layer. First, the three spectral images (visible light spectral material image, near-infrared spectral material image, and thermal infrared spectral material image) in the spectral material dominant component layer are vectorized according to their spectral dimensions. For each pixel location, its intensity values ​​in the three spectral images are arranged sequentially to form an original spectral vector, which contains the material information of the pixel in the three spectral bands. The original spectral vectors of all pixels are arranged according to their spatial positions in the image to form a three-dimensional original spectral feature cube with dimensions [height, width, number of spectral channels], and the number of spectral channels is 3, corresponding to the three spectral bands.

[0063] Step S135: Input the original spectral feature cube into the non-negative matrix decomposition module of the spectral material perception subnet. According to the preset number of material endmembers, the original spectral feature cube is decomposed into a material endmember spectral matrix and a material abundance distribution matrix through iterative optimization. The material abundance distribution matrix is ​​reshaped according to the spatial dimension to generate a set of abundance distribution maps of each material endmember.

[0064] In this embodiment, the nonnegative matrix factorization module is used to unmix the original spectral feature cube, separating different material endmembers and their abundance distributions. The preset number of material endmembers is determined based on the possible types of materials present on the guide vane surface, for example, set to K types. Through an iterative optimization algorithm, the original spectral feature cube is decomposed into a material endmember spectral matrix and a material abundance distribution matrix. Each row of the material endmember spectral matrix represents the spectral feature of a material endmember, and each column of the material abundance distribution matrix represents the abundance value of the corresponding material endmember at each pixel in space. The material abundance distribution matrix is ​​reshaped according to the spatial dimension, that is, the abundance values ​​of each material endmember are arranged according to the spatial coordinates of the pixels, generating a set of abundance distribution maps for each material endmember. Each abundance distribution map reflects the spatial distribution of the corresponding material endmember on the guide vane surface.

[0065] For example, step S1351: reshape the original spectral feature cube into a two-dimensional matrix, where the number of rows of the two-dimensional matrix corresponds to the total number of pixels, and the number of columns corresponds to the number of spectral channels composed of visible light band intensity values, near-infrared band intensity values, and thermal infrared band intensity values, to obtain the observed spectral matrix.

[0066] In this embodiment, the original spectral feature cube has dimensions [height, width, 3]. When it is reshaped into a two-dimensional matrix, the number of rows is height × width, which is the total number of pixels, and the number of columns is 3, corresponding to the intensity values ​​of the three spectral channels. The resulting two-dimensional matrix is ​​the observed spectral matrix, where each row represents the spectral information of one pixel.

[0067] Step S1352: Set the number of material endmembers, and initialize the material endmember spectral matrix and material abundance distribution matrix as random non-negative matrices.

[0068] In this embodiment, based on prior knowledge of the surface material of the guide vane, the number of material endmembers is set to K. Then, the material endmember spectral matrix is ​​initialized as a K×3 random non-negative matrix, where each element represents the intensity value of the corresponding material endmember in a certain spectral channel; the material abundance distribution matrix is ​​initialized as a (height×width)×K random non-negative matrix, where each element represents the abundance of a certain material endmember in the corresponding pixel.

[0069] Step S1353: The material endmember spectral matrix and the material abundance distribution matrix are alternately updated using a multiplicative iterative update rule. In each iteration, the material abundance distribution matrix is ​​updated while the current material endmember spectral matrix is ​​fixed, and then the updated material abundance distribution matrix is ​​updated while the material endmember spectral matrix is ​​fixed.

[0070] In this embodiment, the multiplicative iterative update rule is an effective method to ensure matrix non-negativity. During each iteration, the material endmember spectral matrix is ​​first fixed, and the material abundance distribution matrix is ​​updated using a multiplicative update formula based on the objective function of minimizing the reconstruction error. Then, the updated material abundance distribution matrix is ​​fixed, and the material endmember spectral matrix is ​​updated again using the same multiplicative update formula. This process is repeated alternately until the reconstruction error converges.

[0071] Step S1354: Calculate the reconstruction error after each update. The reconstruction error is the Euclidean distance between the observed spectral matrix and the reconstruction matrix obtained by multiplying the updated material endmember spectral matrix by the updated material abundance distribution matrix. Repeat the alternating update process until the reconstruction error is less than a preset convergence threshold.

[0072] In this embodiment, the reconstruction error is used to measure the accuracy of the decomposition results. The Euclidean distance is calculated by summing the squared differences between corresponding elements of the observed spectral matrix and the reconstructed matrix, and then taking the square root. A preset convergence threshold is set according to the actual application requirements. When the reconstruction error is less than this threshold, the decomposition process is considered to have converged, and the iteration stops.

[0073] Step S1355: Use the material endmember spectral matrix obtained after iterative convergence as the final material endmember spectral matrix, and use the material abundance distribution matrix obtained after iterative convergence as the final material abundance distribution matrix.

[0074] In this embodiment, after multiple iterations and updates, when the reconstruction error is less than the convergence threshold, the material endmember spectral matrix and material abundance distribution matrix can better represent the information of the original spectral feature cube, and are used as the final decomposition result.

[0075] Step S136: Input the material endmember spectral matrix into the pre-trained material category discrimination network, perform nonlinear mapping on each spectral feature curve through multiple fully connected layers, output the matching probability of each curve with the standard material spectrum in the preset material library, assign the standard material category label with the highest matching probability to the corresponding material endmember, and combine all material endmembers with category labels and their corresponding abundance distribution maps into a spectral material unmixing feature tensor.

[0076] In this embodiment, the pre-trained material category discrimination network is a classification network. Its input is each spectral feature curve in the spectral matrix of the material endmembers. Multiple fully connected layers perform nonlinear mapping and dimensionality reduction on the spectral features. Finally, a softmax activation function is used to output the matching probability of each spectral curve with the spectra of various standard materials in a preset material library. The preset material library contains standard spectral data of various materials that may be used in the guide vanes. The standard material category label with the highest matching probability is assigned to the corresponding material endmember, thereby achieving the classification of the material endmembers. Then, all material endmembers with category labels and their corresponding abundance distribution maps are combined to form a spectral material unmixing feature tensor. This spectral material unmixing feature tensor contains both material category information and spatial distribution information.

[0077] Step S137: Input the thermal radiation dominant component layer in the decomposed multispectral component data set into the thermal radiation sensing subnet of the spatial-spectral joint sensing and reconstruction network, extract the thermal infrared thermal radiation image in the thermal radiation dominant component layer, normalize the temperature value of each pixel to generate a normalized temperature distribution map, and simultaneously extract the thermal infrared temperature gradient direction distribution map of the thermal infrared thermal radiation image. Then, stitch the channels of the normalized temperature distribution map and the thermal infrared temperature gradient direction distribution map together to obtain the initial thermal radiation feature map.

[0078] In this embodiment, the thermal radiation sensing subnet is used to extract thermal radiation features from the thermal radiation dominant component layer. First, a thermal infrared thermal radiation image is extracted from the thermal radiation dominant component layer, where the pixel values ​​are temperature values. The temperature value of each pixel is normalized, mapping it to a range of 0 to 1, generating a normalized temperature distribution map. This eliminates the influence of absolute temperature values, facilitating subsequent processing. Simultaneously, the thermal infrared temperature gradient direction distribution map obtained in the previous step is extracted. The normalized temperature distribution map and the thermal infrared temperature gradient direction distribution map are concatenated along the channel dimension to obtain an initial thermal radiation feature map, which contains information on both temperature magnitude and temperature gradient direction.

[0079] Step S138: Input the initial thermal radiation feature map into the optical flow estimation module of the thermal radiation sensing subnet. Using the initial thermal radiation feature maps of two consecutive frames in the thermal infrared spectrum image sequence as input, calculate the displacement vector of each pixel between the two frames to generate a thermal radiation optical flow field containing horizontal and vertical motion components.

[0080] In this embodiment, the optical flow estimation module is used to capture the dynamic changes in thermal radiation. Using the initial thermal radiation feature maps of two consecutive frames in a thermal infrared spectral image sequence as input, the displacement vector of each pixel between the two frames is calculated using an optical flow estimation algorithm. The displacement vector includes horizontal and vertical motion components, representing the distance the pixel moves in the horizontal and vertical directions, respectively. The displacement vectors of all pixels constitute the thermal radiation optical flow field, which can reflect the dynamic changes in thermal radiation on the surface of the guide vane, such as the movement or diffusion of heat sources.

[0081] Step S139: Generate a dynamic feature field of thermal radiation based on the thermal radiation optical flow field, and perform feature synergistic fusion and reconstruction on the features output by each sensing subnet to obtain an enhanced multispectral joint feature tensor.

[0082] In this embodiment, the generation of the dynamic feature field of thermal radiation is based on the thermal radiation optical flow field, which can more comprehensively reflect the dynamic characteristics of thermal radiation. Then, the features output by the spatial structure sensing subnet, the spectral material sensing subnet, and the thermal radiation sensing subnet are synergistically fused and reconstructed. By comprehensively utilizing the feature information of each component, an enhanced multispectral joint feature tensor is generated. This enhanced multispectral joint feature tensor will serve as the basis for subsequent abnormal state determination.

[0083] Step S1391: Calculate the divergence and curl of the thermal radiation optical flow field to obtain a divergence distribution map reflecting the degree of heat source divergence and a curl distribution map reflecting the degree of heat source rotation and aggregation. Merge the divergence distribution map and the curl distribution map with the normalized temperature distribution map to generate a dynamic feature field of thermal radiation containing information on the location of the heat source, the direction of thermal diffusion, and the thermal aggregation mode.

[0084] In this embodiment, divergence and curl are important physical quantities describing the characteristics of a vector field. Divergence is calculated for the thermal radiation optical flow field; a positive divergence value indicates that the point is a heat source divergence center, and a negative divergence value indicates a heat source convergence center. The divergence values ​​of all pixels are arranged to form a divergence distribution map. Curl is calculated; the curl value reflects the degree of rotation of the heat source. The larger the curl value, the more intense the rotation, forming a curl distribution map. Then, the divergence distribution map, curl distribution map, and normalized temperature distribution map are fused. The feature of each pixel is composed of the corresponding pixel values ​​from these three distribution maps, forming a dynamic thermal radiation feature field. This dynamic thermal radiation feature field contains rich dynamic information such as the location of the heat source, the direction of heat diffusion, and the heat accumulation mode.

[0085] For example, step S1391-1: Perform spatial gradient calculations on the components of the thermal radiation optical flow field in the horizontal and vertical directions respectively to obtain the horizontal partial derivative of the horizontal component, the vertical partial derivative of the horizontal component, the horizontal partial derivative of the vertical component, and the vertical partial derivative of the vertical component.

[0086] In this embodiment, the thermal radiation optical flow field includes a horizontal motion component and a vertical motion component. The spatial gradient of the horizontal motion component is calculated in the horizontal direction to obtain its horizontal partial derivative; similarly, the spatial gradient is calculated in the vertical direction to obtain its vertical partial derivative. Likewise, the spatial gradients of the vertical motion component are calculated in both the horizontal and vertical directions to obtain its horizontal and vertical partial derivatives. These partial derivatives reflect the rate of change of the optical flow field in different directions.

[0087] Step S1391-2: Calculate the divergence value at each pixel based on the horizontal partial derivative of the horizontal component and the vertical partial derivative of the vertical component. The divergence value is the sum of the horizontal partial derivative of the horizontal component and the vertical partial derivative of the vertical component. Arrange the divergence values ​​of all pixels according to spatial coordinates to form a divergence distribution map.

[0088] In this embodiment, the divergence value is calculated by adding the horizontal partial derivative of the horizontal component to the vertical partial derivative of the vertical component. By performing this calculation on each pixel, the divergence value is obtained. Then, all divergence values ​​are arranged according to spatial coordinates to form a divergence distribution map.

[0089] Step S1391-3: Calculate the curl value at each pixel point based on the horizontal partial derivative of the vertical component and the vertical partial derivative of the horizontal component. The curl value is the horizontal partial derivative of the vertical component minus the vertical partial derivative of the horizontal component. Arrange the curl values ​​of all pixels according to spatial coordinates to form a curl distribution map.

[0090] In this embodiment, the curl value is calculated by subtracting the partial derivative of the horizontal direction from the partial derivative of the vertical direction component. After calculating the curl value for each pixel, they are arranged according to spatial coordinates to form a curl distribution map.

[0091] Step S1391-4: Perform threshold segmentation on the divergence distribution map, extract pixels with divergence values ​​greater than the positive divergence threshold as candidate points for heat source divergence centers, extract pixels with divergence values ​​less than the negative divergence threshold as candidate points for heat source convergence centers, perform connected component analysis on the candidate points for heat source divergence centers and the candidate points for heat source convergence centers, and determine the boundary coordinates of the heat source divergence region and the heat source convergence region.

[0092] In this embodiment, the positive and negative divergence thresholds are set based on the characteristics of the thermal radiation optical flow field. Through threshold segmentation, pixels with divergence values ​​greater than the positive divergence threshold in the divergence distribution map are marked as candidate heat source divergence centers, and pixels with divergence values ​​less than the negative divergence threshold are marked as candidate heat source convergence centers. Then, connected component analysis is performed on these candidate points to form regions, thus determining the boundary coordinates of the heat source divergence region and the heat source convergence region.

[0093] Step S1391-5: Perform threshold segmentation on the curl distribution map, extract pixels with absolute curl values ​​greater than the curl threshold as candidate points for the heat source rotation region, perform connected component analysis on the candidate points for the heat source rotation region, and determine the boundary coordinates of the heat source rotation region.

[0094] In this embodiment, the curl threshold is set according to the actual situation to filter out regions with obvious rotational characteristics. The curl distribution map is segmented by thresholding, and pixels with absolute curl values ​​greater than the curl threshold are extracted as candidate points for the heat source rotation region. Then, the boundary coordinates of the heat source rotation region are determined through connected component analysis.

[0095] Step S1391-6: The divergence distribution map, the curl distribution map, and the normalized temperature distribution map are fused pixel by pixel. For each pixel, its divergence value, curl value, and normalized temperature value are combined into a three-dimensional feature vector. The three-dimensional feature vectors of all pixels are arranged according to spatial coordinates to generate a dynamic feature field of thermal radiation.

[0096] In this embodiment, for each pixel, its divergence value in the divergence distribution map, curl value in the curl distribution map, and normalized temperature value in the normalized temperature distribution map are combined into a three-dimensional feature vector. The three-dimensional feature vectors of all pixels are arranged according to spatial coordinates to form a dynamic feature field of thermal radiation. This dynamic feature field of thermal radiation integrates the static temperature information of thermal radiation and the dynamic divergence and curl information.

[0097] Step S1392: Input the spatial structure enhancement feature map, the spectral material unmixing feature tensor, and the thermal radiation dynamic feature field into the feature collaborative fusion module of the spatial-spectral joint sensing and reconstruction network. Adjust the spatial size of the spectral material unmixing feature tensor and the thermal radiation dynamic feature field to make them consistent with the spatial resolution of the spatial structure enhancement feature map. Then, concatenate the adjusted spectral material unmixing feature tensor and the thermal radiation dynamic feature field with the spatial structure enhancement feature map in the channel dimension to generate a preliminary joint feature tensor.

[0098] In this embodiment, the primary task of the feature collaborative fusion module is to unify the spatial resolution of the output features of each subnet. Since the spatial structure enhancement feature map, the spectral material unmixing feature tensor, and the thermal radiation dynamic feature field may have different spatial dimensions, it is necessary to adjust the spatial dimensions of the spectral material unmixing feature tensor and the thermal radiation dynamic feature field, for example, through interpolation or pooling, to make them have the same spatial resolution as the spatial structure enhancement feature map. Then, the adjusted three features are concatenated along the channel dimension, that is, the spatial structure feature, spectral material feature, and thermal radiation dynamic feature of each pixel are combined to form a preliminary joint feature tensor. This preliminary joint feature tensor contains multi-source heterogeneous feature information.

[0099] Step S1393: Input the preliminary joint feature tensor into the multi-head cross-attention unit of the feature collaborative fusion module, use the spatial structure enhancement feature map as the query matrix, and use the joint feature tensor after splicing the spectral material unmixing feature tensor and the thermal radiation dynamic feature field as the key matrix and value matrix. Calculate the similarity score between the query and the key to generate cross-attention weights. Use the cross-attention weights to perform a weighted summation of the value matrix to realize the spatial alignment and information injection of spectral material information and thermal radiation information into the spatial structure features, and output the structure-guided enhancement feature tensor.

[0100] In this embodiment, a multi-head cross-attention unit is used to achieve information interaction between different features. First, the spatial structure enhancement feature map is used as the query matrix, representing the spatial location information that needs attention. The joint feature tensor obtained by concatenating the spectral material unmixing feature tensor and the thermal radiation dynamic feature field is used as the key matrix and value matrix. The key matrix is ​​used to calculate the similarity with the query matrix, and the value matrix is ​​used to provide the information to be fused. By calculating the similarity score between the query matrix and the key matrix, cross-attention weights are generated, which represent the degree of contribution of different features to each position in the spatial structure feature. Then, the value matrix is ​​weighted and summed using the cross-attention weights, aligning the spectral material information and thermal radiation information according to their spatial positions and injecting them into the spatial structure feature, outputting a structure-guided enhancement feature tensor.

[0101] Step S1394: Input the structure-guided enhanced feature tensor into the multi-head cross-attention unit again, using the spectral material unmixing feature tensor as the query matrix, and the joint feature tensor after splicing the structure-guided enhanced feature tensor and the thermal radiation dynamic feature field as the key matrix and value matrix, to perform a second cross-attention calculation, thereby realizing the alignment and information complementarity of spatial structure information and thermal radiation information to the feature space of spectral material features, and outputting the material-guided secondary enhanced feature tensor.

[0102] In this embodiment, the second cross-attention calculation uses the spectral material unmixing feature tensor as the query matrix, focusing on the spatial distribution of material features. The joint feature tensor obtained by concatenating the structure-guided enhanced feature tensor and the thermal radiation dynamic feature field is used as the key matrix and value matrix. Attention weights are generated by calculating similarity scores, injecting spatial structure information and thermal radiation information into the spectral material features. Thus, through two cross-attention calculations, bidirectional information interaction and complementarity between different features are achieved, outputting a material-guided secondary enhanced feature tensor.

[0103] Step S1395: Input the secondary enhanced feature tensor into the feature reconstruction layer of the feature collaborative fusion module, perform convolution operations on the spatial dimension and channel dimension simultaneously through multiple three-dimensional convolution kernels, perform deep fusion and dimensional compression on the multi-source features after two cross-attention interactions, and output the enhanced multispectral joint feature tensor.

[0104] In this embodiment, the feature reconstruction layer consists of multiple three-dimensional convolutional layers. The three-dimensional convolutional kernels can simultaneously perform convolution operations on the spatial dimension (height and width) and channel dimension of the feature tensor. Through these convolutional operations, multi-source features in the secondary enhanced feature tensor can be deeply fused to extract higher-level joint features, while dimensionality compression is performed to reduce redundant information. The final output enhanced multispectral joint feature tensor integrates information from spatial structure, spectral material, and thermal radiation, exhibiting stronger discriminative capabilities.

[0105] Step S140: Based on the enhanced multispectral joint feature tensor, perform joint determination and spatial positioning of the abnormal state of the device surface region, and generate a spatial distribution map of the abnormal state of the device. The spatial distribution map of the abnormal state of the device includes the geometric boundary coordinates of the abnormal surface geometric structure region, the abnormal surface material composition region, and the abnormal surface thermal field distribution region.

[0106] In this embodiment, the abnormal state determination is achieved by comparing the enhanced multispectral joint feature tensor with the feature template under normal operating conditions of the equipment. By calculating the feature difference, abnormal regions are identified, classified, and located, ultimately generating a spatial distribution map of the equipment's abnormal state. This spatial distribution map details the geometric boundary coordinates of different types of abnormal regions.

[0107] Step S141: Retrieve the standard operating condition joint feature template corresponding to the current equipment inspection area from the pre-stored equipment standard operating condition joint feature template library. The standard operating condition joint feature template includes the standard values ​​of the spatial structure feature vector, the standard values ​​of the spectral material feature vector, and the standard values ​​of the thermal radiation feature vector for each spatial location on the equipment surface under normal operating conditions.

[0108] In this embodiment, the equipment standard operating condition joint feature template library stores feature templates for different equipment inspection areas under normal operating conditions. For the current wind tunnel guide vane inspection area, the corresponding standard operating condition joint feature template is retrieved from the template library. This standard operating condition joint feature template contains standard values ​​for the spatial structure feature vector, spectral material feature vector, and thermal radiation feature vector of each spatial location on the guide vane surface under normal conditions. These standard values ​​are obtained through the collection and analysis of a large amount of multispectral data under normal operating conditions.

[0109] Step S142: Compare the enhanced multispectral joint feature tensor with the standard operating condition joint feature template spatially. For the feature vector of each spatial location in the enhanced multispectral joint feature tensor, calculate the Mahalanobis distance between it and the standard feature vector of the same spatial location in the standard operating condition joint feature template. The Mahalanobis distance is calculated using a pre-estimated covariance matrix, which characterizes the correlation between different feature dimensions and the fluctuation range of the feature itself under normal operating conditions. The normalized statistical distance obtained by dividing the feature vector difference by the correlation scale in the feature space is used as the feature difference degree of that spatial location.

[0110] In this embodiment, the enhanced multispectral joint feature tensor is compared with the standard operating condition joint feature template at each spatial location. For each spatial location, the feature vector at that location is compared with the standard feature vector, and the Mahalanobis distance between them is calculated. The Mahalanobis distance takes into account the correlation between feature dimensions and can more accurately measure the difference between feature vectors. The covariance matrix is ​​pre-estimated based on feature data under normal operating conditions, reflecting the correlation between different feature dimensions and the fluctuation range of the features themselves. By dividing the difference between feature vectors by the correlation scale represented by the covariance matrix, the normalized statistical distance, i.e., the feature difference degree, is obtained, which can reflect the degree of deviation between the current feature and the standard feature.

[0111] Step S143: Arrange the feature difference degrees of all spatial locations according to spatial coordinates to generate a feature difference degree distribution map, and statistically analyze the feature difference degree values ​​of all non-zero pixels in the feature difference degree distribution map to calculate the global statistical mean and global statistical standard deviation.

[0112] In this embodiment, the feature difference degree of each spatial location is arranged according to its coordinates to form a feature difference degree distribution map. This feature difference degree distribution map visually displays the feature deviation at each position on the guide vane surface. Then, statistical analysis is performed on the feature difference degree values ​​of all non-zero pixels in the feature difference degree distribution map to calculate the global statistical mean and global statistical standard deviation. The global statistical mean reflects the overall feature difference level, and the global statistical standard deviation reflects the dispersion of feature differences.

[0113] Step S144: Dynamically set a dynamic adaptive threshold based on the global statistical mean and the global statistical standard deviation, mark the pixels in the feature difference distribution map whose feature difference value exceeds the dynamic adaptive threshold as initial anomaly candidate points, and combine all the initial anomaly candidate points according to spatial coordinates to form a set of anomaly candidate points on the device surface.

[0114] In this embodiment, the dynamic adaptive threshold is set considering the statistical characteristics of feature difference. For example, the dynamic adaptive threshold can be set as the global statistical mean plus a certain multiple of the global statistical standard deviation, with the specific multiple determined based on the sensitivity requirements for anomaly detection. Pixels with feature difference values ​​exceeding this threshold are marked as initial anomaly candidate points; these pixels may correspond to abnormal areas on the device surface. The spatial coordinates of all initial anomaly candidate points are combined to form a set of anomaly candidate points on the device surface.

[0115] Step S145: Perform morphological dilation on the set of candidate abnormal points on the device surface, connect the initial candidate abnormal points that are spatially adjacent to each other to fill the corresponding gaps, generate a mask of the candidate abnormal region after dilation, perform connected component analysis on the mask of the candidate abnormal region after dilation, identify all independent connected regions as abnormal regions to be classified, and assign a unique identifier to each abnormal region to be classified.

[0116] In this embodiment, morphological dilation is used to connect spatially adjacent initial anomaly candidate points, filling any gaps and creating continuous anomaly regions. The dilation process uses a structuring element of a preset size to dilate the set of anomaly candidate points on the device surface, generating a dilated anomaly candidate region mask. Then, connected component analysis is performed on this mask, grouping interconnected pixels into independent connected regions. Each connected region serves as an anomaly region to be classified, and a unique identifier is assigned to each region for subsequent anomaly type determination.

[0117] Step S146: Based on the abnormal area to be classified, determine the abnormal type and generate a spatial distribution map of the abnormal state of the equipment.

[0118] In this embodiment, each abnormal region to be classified is subjected to anomaly type discrimination to determine whether it belongs to surface geometric structure anomaly, surface material composition anomaly, or surface thermal field distribution anomaly. Based on the discrimination results, the geometric boundary coordinates of each abnormal region are extracted to generate a spatial distribution map of the equipment abnormal state.

[0119] Step S1461: For each abnormal region to be classified, extract the feature vectors of all spatial locations covered by the abnormal region to be classified from the enhanced multispectral joint feature tensor to form a local feature tensor block, and extract the standard feature vectors of the same spatial location from the standard working condition joint feature template to form a local standard feature tensor block.

[0120] In this embodiment, for each abnormal region to be classified, feature vectors of all pixels within that region are extracted from the enhanced multispectral joint feature tensor based on its spatial location range, forming a local feature tensor block. Simultaneously, standard feature vectors at the same spatial location are extracted from the standard operating condition joint feature template, forming a local standard feature tensor block. These two tensor blocks will be used to calculate the feature differences of the abnormal region.

[0121] Step S1462: Calculate the channel-wise difference map between the local feature tensor block and the local standard feature tensor block of each anomaly region to be classified, and perform average pooling on the channel-wise difference map in the spatial dimension to obtain the average difference vector of the anomaly region to be classified in the spatial structure feature channel, spectral material feature channel and thermal radiation feature channel.

[0122] In this embodiment, the channel-by-channel difference map is obtained by calculating the difference between the corresponding channels of the local feature tensor block and the local standard feature tensor block. Average pooling is performed on the difference map of each channel in the spatial dimension, that is, the difference values ​​of all pixels in the difference map are averaged to obtain the average difference value for that channel. The average difference values ​​of the spatial structure feature channel, the spectral material feature channel, and the thermal radiation feature channel are combined to form an average difference vector. This average difference vector reflects the average deviation of the anomaly region to be classified across different feature channels.

[0123] Step S1463: Input the average difference vector into the pre-trained region anomaly type discrimination network. The region anomaly type discrimination network takes the average difference vector as input, and uses a fully connected layer and a classification output layer to discriminate the anomaly type of the anomaly region to be classified. It outputs the structural anomaly confidence score of the anomaly region to be classified as belonging to the surface geometric structure anomaly type, the material anomaly confidence score of the anomaly region to be classified as belonging to the surface material composition anomaly type, and the thermal field anomaly confidence score of the anomaly region to be classified as belonging to the surface thermal field distribution anomaly type.

[0124] In this embodiment, the pre-trained region anomaly type discrimination network is a multi-classification network. The input to this network is the average difference vector. Multiple fully connected layers perform nonlinear transformations and feature extraction on the input vector. Finally, a classification output layer (such as a softmax layer) outputs three confidence scores, corresponding to three types: surface geometric structure anomalies, surface material composition anomalies, and surface thermal field distribution anomalies. A higher confidence score indicates a greater likelihood that the anomaly region belongs to the corresponding anomaly type.

[0125] Step S1464: Determine the final anomaly type of each anomaly region to be classified based on the comparison results of the structural anomaly confidence score, the material anomaly confidence score, and the thermal field anomaly confidence score. Select the anomaly type corresponding to the maximum value among the three confidence scores as the anomaly type of the anomaly region to be classified. If the maximum value corresponds to the structural anomaly confidence score, then the anomaly region to be classified is marked as a surface geometric structure anomaly region. If the maximum value corresponds to the material anomaly confidence score, then the anomaly region to be classified is marked as a surface material composition anomaly region. If the maximum value corresponds to the thermal field anomaly confidence score, then the anomaly region to be classified is marked as a surface thermal field distribution anomaly region.

[0126] In this embodiment, the three confidence scores of each anomaly region to be classified are compared, and the one with the highest score is selected as the final anomaly type of the region. For example, if the structural anomaly confidence score is the highest, the region is marked as a surface geometric structure anomaly region; if the material anomaly confidence score is the highest, it is marked as a surface material composition anomaly region; if the thermal field anomaly confidence score is the highest, it is marked as a surface thermal field distribution anomaly region.

[0127] Step S1465: Extract the geometric boundary coordinates of each region marked as an abnormal surface geometry region, including the coordinate sequence of the vertex of the smallest bounding rectangle of the abnormal region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all abnormal surface geometry regions to form a list of geometric boundary coordinates of abnormal surface geometry regions.

[0128] In this embodiment, for each surface geometric anomaly region, its geometric boundary coordinates are extracted using an image processing algorithm. The sequence of vertex coordinates of the minimum bounding rectangle is obtained by calculating the coordinates of the four vertices of the minimum rectangle that can completely contain the anomaly region; the sequence of region contour point coordinates is obtained by extracting the contour pixel coordinates of the anomaly region using an edge detection algorithm. These coordinate information for all surface geometric anomaly regions are summarized to form a list of geometric boundary coordinates for surface geometric anomaly regions.

[0129] Step S1466: Extract the geometric boundary coordinates of each region marked as an abnormal surface material composition region, including the coordinate sequence of the vertex of the minimum bounding rectangle of the abnormal region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all abnormal surface material composition regions to form a list of geometric boundary coordinates of abnormal surface material composition regions.

[0130] In this embodiment, similar to extracting the geometric boundary coordinates of abnormal surface geometric structures, the minimum bounding rectangle vertex coordinate sequence and the region contour point coordinate sequence are extracted for each abnormal surface material composition region, and then summarized to form a list of geometric boundary coordinates of abnormal surface material composition regions.

[0131] Step S1467: Extract the geometric boundary coordinates of each region marked as having an abnormal surface thermal field distribution, including the coordinate sequence of the vertex of the smallest bounding rectangle of the region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all regions with abnormal surface thermal field distribution to form a list of geometric boundary coordinates of regions with abnormal surface thermal field distribution.

[0132] In this embodiment, similarly, the geometric boundary coordinates of each abnormal surface thermal field distribution region are extracted and summarized to form a list of geometric boundary coordinates of the abnormal surface thermal field distribution regions.

[0133] Step S1468: Superimpose the list of geometric boundary coordinates of the abnormal surface geometry region, the list of geometric boundary coordinates of the abnormal surface material composition region, and the list of geometric boundary coordinates of the abnormal surface thermal field distribution region onto the spatial coordinate system of the equipment surface to generate a spatial distribution map of the abnormal state of the equipment.

[0134] In this embodiment, a spatial coordinate system for the device surface is pre-established to accurately locate various positions on the device surface. The coordinate information from the three geometric boundary coordinate lists is superimposed onto this coordinate system. Different types of abnormal regions can be distinguished using different colors or markers, thereby generating a spatial distribution map of the device's abnormal state. This spatial distribution map of the device's abnormal state displays the location and extent of various abnormal regions on the guide vane surface.

[0135] Step S150: Generate a composite inspection task instruction based on the spatial distribution map of the abnormal state of the equipment, and send it to the unmanned inspection aircraft that is connected to the multispectral acquisition unit to trigger the unmanned inspection aircraft to fly to the abnormal area of ​​the surface geometry, the abnormal area of ​​the surface material composition and the abnormal area of ​​the surface thermal field distribution to perform a differentiated spectral mode verification and shooting operation.

[0136] In this embodiment, the generation of composite inspection task instructions is based on abnormal area information in the spatial distribution map of equipment abnormal states. According to the location and type of the abnormal area, the flight path and shooting parameters of the unmanned inspection drone are planned, detailed task instructions are generated, and sent to the unmanned inspection drone, enabling it to accurately fly to the abnormal area for verification and further confirmation of the abnormal situation.

[0137] Step S151: Analyze the spatial distribution map of the abnormal state of the equipment, extract the geometric boundary coordinates of all abnormal surface geometric structure areas, the geometric boundary coordinates of abnormal surface material composition areas, and the geometric boundary coordinates of abnormal surface thermal field distribution areas, and convert the geometric boundary coordinates of each abnormal area into a spatial polygon representation of that abnormal area. Each spatial polygon is composed of continuous vertex coordinates.

[0138] In this embodiment, the spatial distribution map of the device's abnormal state is first analyzed to extract the geometric boundary coordinates of three types of abnormal regions. Then, these boundary coordinates are converted into spatial polygon representations. Each polygon consists of a series of continuous vertex coordinates, which are connected in sequence to form a closed polygon, accurately representing the shape and extent of the abnormal region.

[0139] Step S152: Calculate the coordinates of the geometric center point of each abnormal region based on the spatial polygon representation of each abnormal region. Cluster the coordinates of the geometric center points of all abnormal regions according to their spatial proximity. Use a density-based spatial clustering algorithm to group the coordinates of the geometric center points that are less than the preset clustering radius into the same inspection target point cluster, generating multiple inspection target point clusters.

[0140] In this embodiment, for each spatial polygon of an abnormal region, the coordinates of its geometric center point are obtained by calculating the geometric center of the polygon. Then, a density-based spatial clustering algorithm (such as the DBSCAN algorithm) is used to cluster all the coordinates of the geometric center points. According to the preset clustering radius, center points whose distance is less than the clustering radius are grouped into the same inspection target point cluster, thereby combining spatially adjacent abnormal regions together to generate multiple inspection target point clusters. Each point cluster contains the coordinates of the geometric center points of several spatially adjacent abnormal regions.

[0141] Step S153: For each cluster of inspection target points, based on the types of abnormal regions contained in the cluster, count the number of abnormal regions in surface geometry, surface material composition, and surface thermal field distribution, and generate an abnormal type composition vector for the cluster of inspection target points.

[0142] In this embodiment, for each inspection target cluster, the number of different types of abnormal regions contained therein is counted. For example, a cluster may contain 2 abnormal surface geometry regions, 1 abnormal surface material composition region, and 0 abnormal surface thermal field distribution regions. Then, the abnormality type composition vector of this cluster is [2, 1, 0]. This abnormality type composition vector reflects the distribution of abnormality types within the cluster.

[0143] Step S154: Obtain the endurance parameters, maximum flight speed parameters, and current docking position coordinates of the unmanned inspection aircraft. Using the geometric center points of the multiple inspection target point clusters as waypoints, use a path planning algorithm to calculate the shortest flight path that starts from the current docking position, sequentially traverses the geometric center points of all inspection target point clusters, and returns. Generate a preliminary inspection track containing a sequence of flight path points.

[0144] In this embodiment, the endurance parameter of the unmanned inspection drone determines its maximum flight distance, while the maximum flight speed parameter affects the flight time. The current docking position coordinates are the start and end points of the path planning. Using the geometric center point of multiple inspection target point clusters as waypoints, a path planning algorithm (such as an approximate algorithm for the traveling salesman problem) is used to calculate the shortest flight path starting from the current docking position, passing through all waypoints in sequence, and returning to the starting point. This shortest flight path consists of a series of flight path points, forming a preliminary inspection track.

[0145] Step S155: Input the preliminary inspection trajectory into the three-dimensional spatial obstacle avoidance optimization model. The three-dimensional spatial obstacle avoidance optimization model, combined with the pre-stored three-dimensional spatial obstacle distribution map of the equipment inspection area, performs spatial position fine-tuning on each flight path point in the preliminary inspection trajectory, so that the distance between the adjusted path point and the surface of the adjacent obstacle meets the preset safe distance threshold, and generates the obstacle avoidance optimized inspection trajectory.

[0146] In this embodiment, a three-dimensional obstacle avoidance optimization model is used to ensure the flight safety of the unmanned inspection drone. A pre-stored three-dimensional obstacle distribution map of the equipment inspection area contains information on various obstacles inside the wind tunnel. Based on this map, the model fine-tunes the spatial position of each flight path point in the initial inspection trajectory to avoid obstacles, ensuring that the distance between the adjusted path point and the surface of adjacent obstacles is not less than a preset safe distance threshold. After adjustment, an obstacle-avoidance-optimized inspection trajectory is generated, which consists of a series of path points with three-dimensional spatial coordinates.

[0147] Step S156: For each track point in the obstacle avoidance optimized inspection track, configure the verification imaging parameters of the track point according to the anomaly type composition vector of the inspection target point cluster corresponding to the track point. If the number of abnormal surface geometric structure regions in the inspection target point cluster is dominant, configure the verification imaging mode of the track point as a joint detection mode of visible light imaging mode and structured light scanning mode. If the number of abnormal surface material composition regions in the inspection target point cluster is dominant, configure the verification imaging mode of the track point as a joint detection mode of near-infrared spectral analysis mode and multi-angle polarization imaging mode. If the number of abnormal surface thermal field distribution regions in the inspection target point cluster is dominant, configure the verification imaging mode of the track point as a joint detection mode of thermal infrared imaging mode and mid-infrared spectral detection mode.

[0148] In this embodiment, the dominant anomaly type of the inspected target point cluster is determined based on the anomaly type composition vector of the point cluster. If the number of surface geometric anomaly regions is the largest, then the dominant anomaly type of the point cluster is surface geometric anomaly, and a joint detection mode combining visible light imaging mode and structured light scanning mode is configured to obtain detailed structural information. If the number of surface material composition anomaly regions is the largest, then a joint detection mode combining near-infrared spectral analysis mode and multi-angle polarization imaging mode is configured to analyze the material composition. If the number of surface thermal field distribution anomaly regions is the largest, then a joint detection mode combining thermal infrared imaging mode and mid-infrared spectral detection mode is configured to further monitor temperature and thermal radiation characteristics.

[0149] Step S157: Encode and encapsulate the three-dimensional spatial coordinates of each track point in the obstacle avoidance optimized inspection track and the corresponding verification shooting mode parameters of each track point to generate a formatted composite inspection task instruction.

[0150] In this embodiment, the three-dimensional spatial coordinates of each track point in the obstacle avoidance optimized inspection track and the corresponding verification shooting mode parameters are encoded and encapsulated according to a preset format. The composite inspection task instruction contains a complete sequence of flight track point coordinates, as well as the verification shooting task type identifier and corresponding sensor parameter configuration, such as exposure time and spectral resolution, to be performed at each track point.

[0151] Step S158: Establish a wireless communication link with the unmanned inspection aircraft, and send the composite inspection task command to the airborne flight control system of the unmanned inspection aircraft through the wireless communication link to trigger the airborne flight control system to perform flight and verification shooting operations.

[0152] In this embodiment, a communication link is established with the unmanned inspection aircraft via a wireless communication module (such as Wi-Fi, Bluetooth, or a dedicated communication protocol). The generated composite inspection task command is sent to the airborne flight control system through this link. After receiving the command, the airborne system controls the aircraft to perform flight and review photography operations according to the command content.

[0153] Step S1581: After receiving the composite inspection mission instruction, the airborne flight control system parses the flight track point coordinate sequence and the verification and shooting mission type identifier of each track point, and controls the power system of the unmanned inspection aircraft to perform take-off, cruise, hovering and landing operations according to the flight track point coordinate sequence.

[0154] In this embodiment, the airborne flight control system parses the received composite inspection mission instructions, extracts the flight path point coordinate sequence and the verification and shooting mission type identifier for each path point. Then, based on the coordinate sequence, it plans the flight path and controls the power system (such as motors and propellers) to perform takeoff, cruise along the path, hover at each path point, and landing operations after mission completion.

[0155] Step S1582: When the unmanned inspection aircraft flies to the three-dimensional spatial position corresponding to a certain track point in the flight track point coordinate sequence, the airborne flight control system automatically triggers the multispectral verification and imaging unit mounted on the unmanned inspection aircraft to perform the corresponding imaging action according to the verification and imaging task type identifier corresponding to the track point.

[0156] In this embodiment, after the unmanned inspection drone flies to a certain waypoint and hovers stably, the onboard flight control system activates the corresponding sensors in the multispectral verification and imaging unit to take pictures according to the verification and imaging task type identifier of that waypoint. Different task type identifiers correspond to different imaging modes and sensor combinations.

[0157] For example, in step S15821: the airborne flight control system acquires the current three-dimensional spatial coordinates output by the global positioning system module and the current flight attitude angle output by the inertial measurement unit in real time, compares the current three-dimensional spatial coordinates with the target track point coordinates in the flight track point coordinate sequence to calculate the position deviation vector, and compares the current flight attitude angle with the preset stable hovering attitude angle to calculate the attitude deviation vector.

[0158] In this embodiment, the Global Positioning System (GPS) module is used to acquire the real-time three-dimensional spatial coordinates of the aircraft, and the Inertial Measurement Unit (IMU) is used to acquire flight attitude angles (such as pitch angle, roll angle, and yaw angle). The current coordinates are compared with the target track point coordinates to obtain the position deviation vector; the current attitude angles are compared with the preset stable hovering attitude angles to obtain the attitude deviation vector.

[0159] Step S15822: When the magnitude of the position deviation vector is less than the preset position locking threshold and the magnitude of the attitude deviation vector is less than the preset attitude locking threshold, the airborne flight control system determines that the unmanned inspection aircraft has stably arrived at the target waypoint and sends a hovering command to the flight control motor.

[0160] In this embodiment, the position lock threshold and attitude lock threshold are key parameters to ensure stable hovering of the aircraft. When the magnitudes of the position deviation vector and the attitude deviation vector are both less than their respective thresholds, it indicates that the aircraft has accurately reached the target waypoint and is in a stable attitude. At this time, a hovering command is sent to keep the aircraft in that position.

[0161] Step S15823: The airborne flight control system parses the verification shooting task type identifier corresponding to the target track point, and retrieves the sensor start-up timing table and sensor parameter configuration file corresponding to the verification shooting task type identifier from the local memory.

[0162] In this embodiment, the local memory stores sensor startup timing tables and parameter configuration files corresponding to different review and imaging task type identifiers. The timing tables specify the startup order and time intervals of each sensor, and the parameter configuration files contain the sensor's operating parameters, such as exposure time and spectral range.

[0163] Step S15824: If the verification shooting task type is identified as a joint detection mode with both visible light imaging mode and structured light scanning mode, the airborne flight control system sends a shooting trigger signal to the visible light camera according to the sensor start-up timing table. After receiving the trigger signal, the visible light camera acquires a visible light image of the abnormal area according to the exposure time parameter and aperture size parameter in the sensor parameter configuration file. Subsequently, the airborne flight control system sends a scanning trigger signal to the structured light scanner. After receiving the trigger signal, the structured light scanner projects a preset encoded structured light pattern onto the abnormal area and simultaneously acquires a structured light image modulated by the object surface. The three-dimensional point cloud data of the abnormal area is reconstructed from the structured light image using the built-in phase calculation algorithm.

[0164] In this embodiment, for the joint detection mode of visible light imaging and structured light scanning, the visible light camera is first activated to acquire images, and then the structured light scanner is activated. The structured light scanner projects a pre-coded structured light pattern onto the surface of the abnormal region, acquires the modulated structured light image, and then calculates the three-dimensional point cloud data through a phase resolution algorithm, thereby obtaining the three-dimensional structural information of the abnormal region.

[0165] Step S15825: If the verification shooting task type is identified as a joint detection mode that combines near-infrared spectral analysis mode and multi-angle polarization imaging mode, the airborne flight control system sends a spectral acquisition trigger signal to the near-infrared spectral sensor according to the sensor start-up timing table. After receiving the trigger signal, the near-infrared spectral sensor acquires near-infrared spectral data of the abnormal area according to the spectral resolution parameters and scanning band range parameters in the sensor parameter configuration file. Subsequently, the airborne flight control system sends a multi-angle shooting trigger signal to the polarization camera. After receiving the trigger signal, the polarization camera rotates the polarizer to each preset polarization angle according to the polarization angle sequence parameters in the sensor parameter configuration file, and acquires a polarization image of the abnormal area at each polarization angle, generating a multi-angle polarization image sequence.

[0166] In this embodiment, in near-infrared spectroscopy analysis mode, the near-infrared spectral sensor collects spectral data of abnormal areas for material composition analysis. In multi-angle polarization imaging mode, the polarization camera acquires polarization images at different angles by rotating a polarizer, generating a multi-angle polarization image sequence to obtain the polarization characteristics of the material.

[0167] Step S15826: If the verification shooting task type is identified as a joint detection mode where thermal infrared imaging mode and mid-infrared spectral detection mode coexist, the airborne flight control system sends a thermal imaging trigger signal to the thermal infrared imaging sensor according to the sensor activation sequence table. After receiving the trigger signal, the thermal infrared imaging sensor acquires the thermal infrared image of the abnormal area according to the integration time parameter and gain parameter in the sensor parameter configuration file. Subsequently, the airborne flight control system sends a spectral detection trigger signal to the mid-infrared spectral sensor. After receiving the trigger signal, the mid-infrared spectral sensor acquires the mid-infrared spectral data of the abnormal area according to the wavenumber resolution parameter and scanning speed parameter in the sensor parameter configuration file.

[0168] In this embodiment, a thermal infrared imaging sensor acquires thermal infrared images of the abnormal area, reflecting the temperature distribution. A mid-infrared spectral sensor acquires mid-infrared spectral data to analyze the molecular vibration information of the substance, further determining the cause of the thermal field anomaly.

[0169] Step S1583: All visible light images, structured light 3D point cloud data, near-infrared spectral data, multi-angle polarization images, thermal infrared images, and mid-infrared spectral data collected by the unmanned inspection aircraft during the inspection mission, together with the aircraft attitude parameters and GPS coordinates corresponding to the collection time, are packaged into an inspection mission execution result data package and transmitted back to the ground control station.

[0170] In this embodiment, the unmanned inspection vehicle associates the collected data with corresponding attitude parameters and positioning coordinates, packaging them into an inspection mission execution result data packet. This data packet is then transmitted back to the ground control station via a wireless communication link for further analysis and processing by technicians to confirm the specific details of any anomalies on the guide vane surface.

[0171] In one exemplary embodiment, a low-light device inspection system based on adaptive multispectral fusion is provided. This system can be a terminal, server, etc., and its internal structure diagram can be as follows: Figure 2As shown, this low-light equipment inspection system based on adaptive multispectral fusion includes a processor, memory, input / output interface, communication interface, display unit, system bus, and input device. The processor, memory, and input / output interface are connected via the system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements a low-light equipment inspection method based on adaptive multispectral fusion. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the casing of a low-light equipment inspection system based on adaptive multispectral fusion, or an external keyboard, touchpad, or mouse, etc.

[0172] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for inspecting low-light equipment based on adaptive multispectral fusion, characterized in that, The method includes: The visible light spectrum image sequence, near-infrared spectrum image sequence, and thermal infrared spectrum image sequence of the equipment surface area are acquired by the multispectral acquisition unit deployed in the equipment inspection area under low light environment, and combined into an initial multispectral acquisition data set; The initial multispectral acquisition data set is subjected to adaptive decomposition processing based on image content features, which decomposes the initial multispectral acquisition data set into a spatial structure-dominant component layer, a spectral material-dominant component layer, and a thermal radiation-dominant component layer, and then reassembles it into a decomposed multispectral component data set. The pre-built spatial-spectral joint sensing and reconstruction network is invoked to perform cross-component feature association and enhancement processing on the decomposed multispectral component data set, generating an enhanced multispectral joint feature tensor that integrates spatial structure, spectral material and thermal radiation information; Based on the enhanced multispectral joint feature tensor, the abnormal state of the surface region of the equipment is jointly determined and spatially located to generate a spatial distribution map of the abnormal state of the equipment. The spatial distribution map of the abnormal state of the equipment includes the geometric boundary coordinates of the abnormal surface geometric structure region, the abnormal surface material composition region, and the abnormal surface thermal field distribution region. Based on the spatial distribution map of the abnormal state of the equipment, a composite inspection task instruction is generated and sent to the unmanned inspection aircraft that is connected to the multispectral acquisition unit to trigger the unmanned inspection aircraft to fly to the abnormal surface geometry area, the abnormal surface material composition area, and the abnormal surface thermal field distribution area to perform differentiated spectral mode verification and imaging operations.

2. The method for inspecting low-light equipment based on adaptive multispectral fusion according to claim 1, characterized in that, The initial multispectral acquisition data set undergoes adaptive decomposition processing based on image content features, decomposing it into a spatial structure-dominant component layer, a spectral material-dominant component layer, and a thermal radiation-dominant component layer, and then recombining them into a decomposed multispectral component data set, including: Extract the grayscale matrix of each frame of the visible light spectrum image in the visible light spectrum image sequence, calculate the grayscale difference between each pixel and its eight neighboring pixels in the grayscale matrix, synthesize the grayscale difference vectors in all directions to obtain the local gradient magnitude of the pixel, and arrange the local gradient magnitudes of each pixel according to the pixel coordinates to form a visible light gradient magnitude distribution map. The visible light gradient magnitude distribution map is segmented by thresholding. Pixels with local gradient magnitudes greater than the first gradient threshold are marked as strong edge candidate points, pixels with local gradient magnitudes between the first and second gradient thresholds are marked as weak texture candidate points, and pixels with local gradient magnitudes less than the second gradient threshold are marked as smooth region candidate points. Extract the pixel intensity matrix of each frame of the near-infrared spectral image in the near-infrared spectral image sequence. For each local sliding window centered on a pixel, calculate the gray-level co-occurrence matrix of all pixel intensity values ​​within the local sliding window. Calculate the entropy value from the gray-level co-occurrence matrix as the local gray-level co-occurrence matrix entropy value of that pixel. Arrange the local gray-level co-occurrence matrix entropy values ​​of each pixel according to the pixel coordinates to form a near-infrared texture complexity distribution map. Cluster analysis is performed on the near-infrared texture complexity distribution map. Pixels whose local gray-level co-occurrence matrix entropy value is greater than the first entropy value are marked as first material candidate points, and pixels whose local gray-level co-occurrence matrix entropy value is less than the second entropy value are marked as second material candidate points. Extract the temperature matrix of each frame of the thermal infrared spectrum image sequence. Calculate the temperature change rate of each pixel in the temperature matrix in the horizontal and vertical directions. Combine the temperature change rate vectors in the horizontal and vertical directions to obtain the local temperature gradient vector of the pixel. Arrange the magnitude of the local temperature gradient vector of each pixel according to the pixel coordinates to form a thermal infrared temperature gradient amplitude distribution map. Record the direction angle of the local temperature gradient vector of each pixel to form a thermal infrared temperature gradient direction distribution map. Based on the thermal infrared temperature gradient amplitude distribution map and the thermal infrared temperature gradient direction distribution map, the pixel areas with thermal infrared temperature gradient amplitude greater than the temperature gradient threshold and thermal infrared temperature gradient direction angles are marked as heat source center candidate areas, and the pixel areas with thermal infrared temperature gradient amplitude less than the temperature gradient threshold and absolute temperature values ​​greater than the ambient temperature threshold are marked as thermal diffusion influence candidate areas. Based on the labeling results of each spectrum, joint decision-making on the composition of pixels is performed, and a set of decomposed multispectral composition data is generated.

3. The method for inspecting low-light equipment based on adaptive multispectral fusion according to claim 2, characterized in that, The process of jointly determining the composition of pixels based on the labeling results of each spectrum and generating a decomposed multispectral component data set includes: For each spatial pixel location, the marking results of strong edge candidate points, weak texture candidate points, and smooth region candidate points in the visible light gradient amplitude distribution map, the clustering results of first material candidate points and second material candidate points in the near-infrared texture complexity distribution map, and the marking results of heat source center candidate areas and heat diffusion influence candidate areas in the thermal infrared temperature gradient amplitude distribution map and thermal infrared temperature gradient direction distribution map are jointly decided. According to the preset decision fusion rules, the pixel point is determined to belong to the spatial structure dominant component, the spectral material dominant component, or the thermal radiation dominant component. Based on the composition of each pixel, the original pixel values ​​of the pixels belonging to the dominant spatial structure component are extracted from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence. The original spectral channels and spatial positions are retained, and the positions of the pixels belonging to the other two components are assigned a value of 0, thereby generating a spatial structure dominant component layer composed of visible light spatial structure image, near-infrared spatial structure image, and thermal infrared spatial structure image. Based on the composition of each pixel, the original pixel values ​​of the pixels belonging to the dominant component of the spectral material are extracted from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence. The original spectral channels and spatial positions are retained, and the positions of the pixels belonging to the other two components are assigned to 0, thereby generating a spectral material dominant component layer composed of visible light spectral material images, near-infrared spectral material images, and thermal infrared spectral material images. Based on the composition of each pixel, the original pixel values ​​of the pixels belonging to the thermal radiation dominant component are extracted from the initial multispectral acquisition data set in the visible light spectral image sequence, near-infrared spectral image sequence, and thermal infrared spectral image sequence. The original spectral channels and spatial positions are retained, and the positions of the pixels belonging to the other two components are assigned to 0, thereby generating a thermal radiation dominant component layer composed of visible light thermal radiation image, near-infrared thermal radiation image, and thermal infrared thermal radiation image. The spatial structure dominant component layer, the spectral material dominant component layer, and the thermal radiation dominant component layer are superimposed and recombined in the spectral channel dimension to generate a multi-channel data volume with a three-level structure as the decomposed multispectral component data set. Each spatial pixel position in the decomposed multispectral component data set is associated with three spectral vectors from the spatial structure dominant component layer, the spectral material dominant component layer, and the thermal radiation dominant component layer, respectively. Each spectral vector contains a visible light band intensity value, a near-infrared band intensity value, and a thermal infrared band intensity value.

4. The method for inspecting low-light equipment based on adaptive multispectral fusion according to claim 1, characterized in that, The method involves invoking a pre-constructed spatial-spectral joint sensing and reconstruction network to perform cross-component feature association and enhancement processing on the decomposed multispectral component data set, generating an enhanced multispectral joint feature tensor that integrates spatial structure, spectral material, and thermal radiation information, including: The spatial structure dominant component layer in the decomposed multispectral component data set is input into the spatial structure perception subnet of the spatial-spectral joint perception and reconstruction network. The visible light spatial structure image in the spatial structure dominant component layer is convolved by the preset multi-directional Gabor filter bank in the spatial structure perception subnet to extract edge responses in different directions. Non-maximum suppression and double threshold hysteresis processing are performed on the edge responses in all directions to generate an edge probability map. The edge probability map is multiplied pixel-by-pixel with the near-infrared spatial structure image in the dominant spatial structure layer to obtain a weighted near-infrared edge enhancement image. Simultaneously, the edge probability map is multiplied pixel-by-pixel with the thermal infrared spatial structure image in the dominant spatial structure layer to obtain a weighted thermal infrared edge enhancement image. The weighted near-infrared edge enhancement image and the weighted thermal infrared edge enhancement image are then concatenated along the channel dimension to generate a preliminary multi-channel fusion feature map. This preliminary multi-channel fusion feature map is input into the first spatial convolutional module of the spatial structure perception subnet, and the spatial receptive field is gradually expanded through multiple cascaded convolutional layers to extract geometric structure primitive feature maps. The geometric structure primitive feature map is input into the spatial attention module of the spatial structure perception subnet. Global max pooling and global average pooling are performed on the geometric structure primitive feature map to generate two spatial context descriptors. After concatenation, the spatial attention weight map is calculated through a shared fully connected layer. The spatial attention weight map is multiplied element-wise with the geometric structure primitive feature map to output the spatial structure enhancement feature map. The spectral material dominant component layer in the decomposed multispectral component data set is input into the spectral material perception subnet of the spatial-spectral joint perception and reconstruction network. The visible light spectral material image, the near-infrared spectral material image and the thermal infrared spectral material image in the spectral material dominant component layer are vectorized into spectral dimensions. The visible light, near-infrared and thermal infrared intensity values ​​of each pixel position are used to form the original spectral vector. The original spectral vectors of all pixels are arranged in space to form the original spectral feature cube. The original spectral feature cube is input into the non-negative matrix decomposition module of the spectral material perception subnet. Based on the preset number of material endmembers, the original spectral feature cube is decomposed into a material endmember spectral matrix and a material abundance distribution matrix through iterative optimization. The material abundance distribution matrix is ​​then reshaped according to the spatial dimension to generate a set of abundance distribution maps for each material endmember. The material end-member spectral matrix is ​​input into a pre-trained material category discrimination network. Each spectral feature curve is nonlinearly mapped through multiple fully connected layers. The matching probability of each curve with the standard material spectrum in the preset material library is output. The standard material category label with the highest matching probability is assigned to the corresponding material end-member. All material end-members with category labels and their corresponding abundance distribution maps are combined into a spectral material unmixing feature tensor. The thermal radiation dominant component layer in the decomposed multispectral component data set is input into the thermal radiation sensing subnetwork of the spatial-spectral joint sensing and reconstruction network. The thermal infrared thermal radiation image in the thermal radiation dominant component layer is extracted. The temperature value of each pixel is normalized to generate a normalized temperature distribution map. At the same time, the thermal infrared temperature gradient direction distribution map of the thermal infrared thermal radiation image is extracted. The channels of the normalized temperature distribution map and the thermal infrared temperature gradient direction distribution map are stitched together to obtain the initial thermal radiation feature map. The initial thermal radiation feature map is input into the optical flow estimation module of the thermal radiation sensing subnet. Using the initial thermal radiation feature maps of two consecutive frames in the thermal infrared spectrum image sequence as input, the displacement vector of each pixel between the two frames is calculated to generate a thermal radiation optical flow field containing horizontal and vertical motion components. Based on the aforementioned thermal radiation optical flow field, a dynamic feature field of thermal radiation is generated, and the features output by each sensing subnet are synergistically fused and reconstructed to obtain an enhanced multispectral joint feature tensor.

5. The method for inspecting low-light equipment based on adaptive multispectral fusion according to claim 4, characterized in that, The process involves generating a dynamic feature field based on the thermal radiation optical flow field, and performing feature synergistic fusion and reconstruction on the features output by each sensing subnetwork to obtain an enhanced multispectral joint feature tensor, including: The divergence and curl of the thermal radiation optical flow field are calculated to obtain a divergence distribution map reflecting the degree of heat source divergence and a curl distribution map reflecting the degree of heat source rotation and aggregation. The divergence distribution map and the curl distribution map are fused with the normalized temperature distribution map to generate a dynamic feature field of thermal radiation containing information on the location of the heat source, the direction of thermal diffusion, and the mode of thermal aggregation. The spatial structure enhancement feature map, the spectral material unmixing feature tensor, and the thermal radiation dynamic feature field are input into the feature collaborative fusion module of the spatial-spectral joint perception and reconstruction network. The spatial size of the spectral material unmixing feature tensor and the thermal radiation dynamic feature field is adjusted to match the spatial resolution of the spatial structure enhancement feature map. Then, the adjusted spectral material unmixing feature tensor and the thermal radiation dynamic feature field are concatenated with the spatial structure enhancement feature map in the channel dimension to generate a preliminary joint feature tensor. The preliminary joint feature tensor is input into the multi-head cross-attention unit of the feature collaborative fusion module. The spatial structure enhancement feature map is used as the query matrix, and the joint feature tensor after splicing the spectral material unmixing feature tensor and the thermal radiation dynamic feature field is used as the key matrix and value matrix. The similarity score between the query and the key is calculated to generate cross-attention weights. The value matrix is ​​weighted and summed using the cross-attention weights to realize the spatial alignment and information injection of spectral material information and thermal radiation information into the spatial structure features, and outputs the structure-guided enhancement feature tensor. The structure-guided enhanced feature tensor is input again into the multi-head cross-attention unit. The spectral material unmixing feature tensor is used as the query matrix, and the joint feature tensor after splicing the structure-guided enhanced feature tensor and the thermal radiation dynamic feature field is used as the key matrix and value matrix. A second cross-attention calculation is performed to realize the alignment and information complementarity of spatial structure information and thermal radiation information to the feature space of spectral material features, and outputs the material-guided secondary enhanced feature tensor. The enhanced feature tensor is input into the feature reconstruction layer of the feature collaborative fusion module. Multiple three-dimensional convolution kernels are used to perform convolution operations on the spatial and channel dimensions simultaneously. The multi-source features after two cross-attention interactions are deeply fused and dimensionally compressed, and the enhanced multispectral joint feature tensor is output.

6. The method for inspecting weak light equipment based on adaptive multispectral fusion according to claim 1, characterized in that, The step of jointly determining and spatially locating the abnormal state of the device surface region based on the enhanced multispectral joint feature tensor, and generating a spatial distribution map of the device's abnormal state, includes: The standard operating condition joint feature template corresponding to the current equipment inspection area is called from the pre-stored equipment standard operating condition joint feature template library. The standard operating condition joint feature template includes the standard values ​​of the spatial structure feature vector, the standard values ​​of the spectral material feature vector, and the standard values ​​of the thermal radiation feature vector for each spatial location on the equipment surface under normal operating conditions. The enhanced multispectral joint feature tensor is compared spatially with the standard operating condition joint feature template. For the feature vector of each spatial location in the enhanced multispectral joint feature tensor, the Mahalanobis distance between it and the standard feature vector of the same spatial location in the standard operating condition joint feature template is calculated. The Mahalanobis distance is calculated using a pre-estimated covariance matrix, which represents the correlation between different feature dimensions and the fluctuation range of the feature itself under normal operating conditions. The normalized statistical distance obtained by dividing the feature vector difference by the correlation scale in the feature space is used as the feature difference degree of that spatial location. A feature difference distribution map is generated by arranging the feature difference degrees of all spatial locations according to spatial coordinates. The feature difference degree values ​​of all non-zero pixels in the feature difference distribution map are statistically analyzed, and the global statistical mean and global statistical standard deviation are calculated. Dynamic adaptive thresholds are dynamically set based on the global statistical mean and the global statistical standard deviation. Pixels whose feature difference values ​​in the feature difference distribution map exceed the dynamic adaptive thresholds are marked as initial anomaly candidate points. All initial anomaly candidate points are combined according to spatial coordinates to form a set of anomaly candidate points on the device surface. Morphological dilation is performed on the set of candidate points for anomalies on the surface of the device to connect the initial candidate points that are spatially adjacent to each other to fill the corresponding gaps, thereby generating a mask of the candidate anomaly region after dilation. Connectivity analysis is performed on the mask of the candidate anomaly region after dilation to identify all independent connected regions as anomaly regions to be classified and to assign a unique identifier to each anomaly region to be classified. Anomaly type is determined based on the anomaly region to be classified, and a spatial distribution map of the equipment's abnormal status is generated.

7. The method for inspecting low-light equipment based on adaptive multispectral fusion according to claim 6, characterized in that, The step of determining the anomaly type based on the anomaly region to be classified and generating a spatial distribution map of the equipment's abnormal state includes: For each abnormal region to be classified, feature vectors of all spatial locations covered by the abnormal region to be classified are extracted from the enhanced multispectral joint feature tensor to form a local feature tensor block, and standard feature vectors of the same spatial location are extracted from the standard working condition joint feature template to form a local standard feature tensor block. Calculate the channel-wise difference map between the local feature tensor block and the local standard feature tensor block of each anomaly region to be classified, and perform average pooling on the channel-wise difference map in the spatial dimension to obtain the average difference vector of the anomaly region to be classified in the spatial structure feature channel, spectral material feature channel and thermal radiation feature channel. The average difference vector is input into a pre-trained region anomaly type discrimination network. The region anomaly type discrimination network takes the average difference vector as input and uses a fully connected layer and a classification output layer to discriminate the anomaly type of the anomaly region to be classified. It outputs the structural anomaly confidence score of the anomaly region to be classified as belonging to the surface geometric structure anomaly type, the material anomaly confidence score of the surface material composition anomaly type, and the thermal field anomaly confidence score of the surface thermal field distribution anomaly type. Based on the comparison results of the structural anomaly confidence score, material anomaly confidence score, and thermal field anomaly confidence score, the final anomaly type of each anomaly region to be classified is determined. The anomaly type corresponding to the maximum value among the three confidence scores is selected as the anomaly type of the anomaly region to be classified. If the maximum value corresponds to the structural anomaly confidence score, the anomaly region to be classified is marked as a surface geometric structure anomaly region. If the maximum value corresponds to the material anomaly confidence score, the anomaly region to be classified is marked as a surface material composition anomaly region. If the maximum value corresponds to the thermal field anomaly confidence score, the anomaly region to be classified is marked as a surface thermal field distribution anomaly region. Extract the geometric boundary coordinates of each region marked as an abnormal surface geometry region, including the coordinate sequence of the vertex of the minimum bounding rectangle of the region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all abnormal surface geometry regions to form a list of geometric boundary coordinates of abnormal surface geometry regions. Extract the geometric boundary coordinates of each region marked as an abnormal surface material composition region, including the coordinate sequence of the vertex of the minimum bounding rectangle of the region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all abnormal surface material composition regions to form a list of geometric boundary coordinates of abnormal surface material composition regions. Extract the geometric boundary coordinates of each region marked as an anomalous surface thermal field distribution region, including the coordinate sequence of the vertex of the minimum bounding rectangle of the region to be classified and the coordinate sequence of the region contour points. Summarize the geometric boundary coordinates of all anomalous surface thermal field distribution regions to form a list of geometric boundary coordinates of anomalous surface thermal field distribution regions. The list of geometric boundary coordinates of the abnormal surface geometry region, the list of geometric boundary coordinates of the abnormal surface material composition region, and the list of geometric boundary coordinates of the abnormal surface thermal field distribution region are superimposed onto the spatial coordinate system of the equipment surface to generate a spatial distribution map of the abnormal equipment state.

8. The method for inspecting weak light equipment based on adaptive multispectral fusion according to claim 1, characterized in that, The step of generating a composite inspection task instruction based on the spatial distribution map of the abnormal equipment status and sending it to the unmanned inspection drone communicatively connected to the multispectral acquisition unit, thereby triggering the unmanned inspection drone to fly to the abnormal surface geometry region, the abnormal surface material composition region, and the abnormal surface thermal field distribution region to perform a differentiated spectral mode verification and imaging operation, includes: The spatial distribution map of the abnormal state of the equipment is analyzed, and the geometric boundary coordinates of all abnormal surface geometric structure areas, abnormal surface material composition areas, and abnormal surface thermal field distribution areas are extracted. The geometric boundary coordinates of each abnormal area are converted into a spatial polygon representation of the abnormal area, and each spatial polygon is composed of continuous vertex coordinates. The geometric center coordinates of each abnormal region are calculated based on the spatial polygon representation of the abnormal region. The geometric center coordinates of all abnormal regions are clustered according to their spatial proximity. A density-based spatial clustering algorithm is used to group the geometric center coordinates that are less than the preset clustering radius into the same inspection target point cluster, generating multiple inspection target point clusters. Each inspection target point cluster contains the geometric center coordinates of several spatially adjacent abnormal regions. For each cluster of inspection target points, the number of abnormal regions in surface geometry, surface material composition, and surface thermal field distribution is counted according to the types of abnormal regions contained in the cluster, and an anomaly type composition vector of the cluster is generated. The system obtains the endurance parameters, maximum flight speed parameters, and current docking position coordinates of the unmanned inspection aircraft. Using the geometric center points of the multiple inspection target point clusters as waypoints, it employs a path planning algorithm to calculate the shortest flight path that starts from the current docking position, sequentially traverses the geometric center points of all inspection target point clusters, and returns. This generates a preliminary inspection track containing a sequence of flight path points, where each path point in the sequence corresponds to the geometric center point coordinates of an inspection target point cluster. The preliminary inspection trajectory is input into a three-dimensional obstacle avoidance optimization model. The three-dimensional obstacle avoidance optimization model, combined with a pre-stored three-dimensional obstacle distribution map of the equipment inspection area, performs spatial position fine-tuning on each flight path point in the preliminary inspection trajectory, so that the distance between the adjusted path point and the surface of the adjacent obstacle meets the preset safe distance threshold, and generates an obstacle-avoidance-optimized inspection trajectory. The obstacle-avoidance-optimized inspection trajectory consists of a series of trajectory points with three-dimensional spatial coordinates, and each trajectory point corresponds to the coordinates of the geometric center point of an adjusted inspection target point cluster. For each track point in the obstacle avoidance optimized inspection track, the verification imaging parameters of the track point are configured according to the anomaly type composition vector of the inspection target point cluster corresponding to the track point. If the number of abnormal surface geometric regions in the inspection target point cluster is dominant, the verification imaging mode of the track point is configured as a joint detection mode of visible light imaging mode and structured light scanning mode. If the number of abnormal surface material composition regions in the inspection target point cluster is dominant, the verification imaging mode of the track point is configured as a joint detection mode of near-infrared spectral analysis mode and multi-angle polarization imaging mode. If the number of abnormal surface thermal field distribution regions in the inspection target point cluster is dominant, the verification imaging mode of the track point is configured as a joint detection mode of thermal infrared imaging mode and mid-infrared spectral detection mode. The three-dimensional spatial coordinates of each track point in the obstacle avoidance optimized inspection track and the corresponding verification shooting mode parameters of each track point are encoded and encapsulated to generate a formatted composite inspection task instruction. The composite inspection task instruction contains a complete sequence of flight track point coordinates, as well as the verification shooting task type identifier and corresponding sensor parameter configuration to be executed at each track point. A wireless communication link is established with the unmanned inspection aircraft, and the data is transmitted through the wireless communication link to the airborne flight control system of the unmanned inspection aircraft to trigger the airborne flight control system to perform flight and verification shooting operations.

9. The method for inspecting weak light equipment based on adaptive multispectral fusion according to claim 8, characterized in that, The establishment of a wireless communication link with the unmanned inspection aircraft, and the transmission of data through the wireless communication link to the onboard flight control system of the unmanned inspection aircraft to trigger the onboard flight control system to execute flight and verification photography operations, includes: After receiving the composite inspection mission command, the airborne flight control system parses the flight track point coordinate sequence and the verification and shooting mission type identifier of each track point, and controls the power system of the unmanned inspection aircraft to perform take-off, cruise, hovering and landing operations according to the flight track point coordinate sequence. When the unmanned inspection vehicle flies to the three-dimensional spatial position corresponding to a certain waypoint in the flight track point coordinate sequence, the airborne flight control system automatically triggers the multispectral verification and imaging unit mounted on the unmanned inspection vehicle to perform corresponding imaging actions according to the verification and imaging task type identifier corresponding to the waypoint. If the verification and imaging task type identifier is a joint detection mode with both visible light imaging mode and structured light scanning mode, then the visible light camera is activated to collect the visible light image of the abnormal area and the structured light scanner is activated to collect the three-dimensional point cloud data of the abnormal area. If the verification and imaging task type identifier is a joint detection mode with both near-infrared spectral analysis mode and multi-angle polarization imaging mode, then the near-infrared spectral sensor is activated to collect the near-infrared spectral data of the abnormal area and the polarization camera is activated to collect the polarization image of the abnormal area at multiple polarization angles. If the verification and imaging task type identifier is a joint detection mode with both thermal infrared imaging mode and mid-infrared spectral detection mode, then the thermal infrared imaging sensor is activated to collect the thermal infrared image of the abnormal area and the mid-infrared spectral sensor is activated to collect the mid-infrared spectral data of the abnormal area. All visible light images, structured light 3D point cloud data, near-infrared spectral data, multi-angle polarization images, thermal infrared images, and mid-infrared spectral data collected by the unmanned inspection aircraft during the inspection mission, along with the aircraft attitude parameters and GPS coordinates corresponding to the time of collection, are packaged together to form an inspection mission execution result data package and transmitted back to the ground control station.

10. The method for inspecting weak light equipment based on adaptive multispectral fusion according to claim 4, characterized in that, The process involves convolving the visible light spatial structure image in the dominant component layer of the spatial structure using a pre-set multi-directional Gabor filter bank in the spatial structure perception subnet, extracting edge responses in different directions, performing non-maximum suppression and double threshold hysteresis processing on the edge responses in all directions, and generating an edge probability map, including: Initialize the multi-directional Gabor filter bank by setting the orientation angle parameters of multiple Gabor filter cores with different orientations, wherein the orientation angle parameters uniformly cover the range from 0 degrees to 180 degrees. Perform a two-dimensional convolution operation between the Gabor filter kernel for each orientation and the visible light spatial structure image to obtain the edge response intensity map of the pixel for each orientation; For each pixel, iterate through the edge response intensity maps of all orientations, select the maximum edge response intensity of the pixel in all orientations as the initial edge intensity of the pixel, and record the orientation corresponding to the maximum value as the main edge direction of the pixel, thus generating the initial edge intensity map and the main edge direction map; Non-maximum suppression processing is performed on the initial edge intensity map. For each pixel, the initial edge intensity of the pixel is compared with the initial edge intensity of the two adjacent pixels in the direction indicated by the main edge direction map. If the initial edge intensity of the pixel is not the maximum value, the initial edge intensity of the pixel is set to 0, and a non-maximum suppressed edge intensity map is generated. A dual-threshold processing method is applied to the edge intensity map after non-maximum suppression. A first edge threshold and a second edge threshold are set. Pixels with edge intensity greater than the first edge threshold are marked as first edge pixels, pixels with edge intensity between the second edge threshold and the first edge threshold are marked as second edge pixels, and pixels with edge intensity less than the second edge threshold are marked as third edge pixels. The first edge pixel is marked as a connected component. Starting from the first edge pixel, the second edge pixel connected to the first edge pixel is searched in the eight-neighbor area. All the searched connected second edge pixels are marked as first edge pixels. Finally, all the positions marked as first edge pixels constitute an edge probability map with a continuous edge of single pixel width.

Images