Photovoltaic hot spot detection method and system based on cross-modal feature deep fusion

Abstract

The application discloses a photovoltaic hot spot detection method and system based on cross-modal feature deep fusion, relates to the technical field of image processing, and comprises the following steps: performing bidirectional sensing pairing on a photovoltaic module to construct a dual-modal image dataset; performing parallel feature set extraction through double-branch parallel analysis; performing registration splicing according to the channel dimension to obtain joint features, performing attention weight distribution, calculating infrared modal adaptive importance coefficients and visible light modal adaptive importance coefficients, and performing element-level weighting to obtain dual-modal fusion weighted features; generating fusion features to perform hot spot detection and classification on the photovoltaic module and generating a hot spot detection result. The application solves the technical problems that the prior art ignores the attribute differences between multiple modalities, the fusion weight lacks adaptability, and the feature fusion is fragmented, resulting in poor anti-interference ability and accuracy of photovoltaic hot spot detection in a complex environment, and achieves the technical effects of improving the discriminability, robustness of cross-modal feature fusion and the accuracy of photovoltaic hot spot detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a photovoltaic hot spot detection method and system based on cross-modal feature deep fusion. Background Technology

[0002] The safe and stable operation of photovoltaic power plants is directly related to grid reliability and return on investment. Photovoltaic modules, when exposed to wind, sand, snow, shade, and internal cracks for extended periods, are highly susceptible to "hot spot effects." These effects not only cause a sharp drop in power generation efficiency but can also lead to a rapid increase in local temperature, resulting in aging of encapsulation materials, cell burnout, and even fires. Currently, the detection of photovoltaic hot spots mainly relies on manual infrared thermal imaging inspections or periodic scanning using drones equipped with infrared cameras. However, in complex real-world scenarios, these methods have increasingly limited effectiveness. Infrared images are easily affected by changes in ambient temperature, solar irradiance, and cloud reflections, resulting in numerous false hot spots or missed detections. Furthermore, infrared images lack textural detail, making it difficult to distinguish whether hot spots are caused by genuine physical defects (such as cracks or poor soldering) or non-destructive factors like external shadows or dirt. While visible light images possess rich color and edge information, they cannot directly detect temperature anomalies and are easily affected by the angle of illumination and shadows. Existing methods for fusing visible and infrared images ignore the essential differences in physical properties between infrared and visible modes, leading to shallow mutual contamination of mode-specific information. Furthermore, they lack the ability to dynamically evaluate the confidence level of modes at different regions and scales. When a mode is severely affected by noise, the quality of the joint features after fusion drops sharply.

[0003] Therefore, current technologies suffer from several technical problems, including ignoring the differences in attributes between multiple modes, lacking adaptability in fusion weights, and fragmented feature fusion, resulting in poor anti-interference capability and accuracy of photovoltaic hot spot detection in complex environments. Summary of the Invention

[0004] This application provides a photovoltaic hot spot detection method and system based on deep fusion of cross-modal features, which solves the technical problems in the prior art, such as ignoring the differences in attributes between multiple modes, lack of adaptability of fusion weights, and fragmented feature fusion, resulting in poor anti-interference ability and accuracy of photovoltaic hot spot detection in complex environments. It achieves the technical effect of improving the discriminativeness and robustness of cross-modal feature fusion and the accuracy of photovoltaic hot spot detection.

[0005] This application provides a photovoltaic hotspot detection method based on deep fusion of cross-modal features. The method includes: performing bidirectional sensing pairing based on photovoltaic modules to construct a dual-modal image dataset, the dual-modal image dataset including infrared images and visible light images; performing dual-branch parallel analysis on the infrared images and the visible light images to extract a parallel feature set, the parallel feature set including infrared modal feature maps and visible light modal feature maps; registering and stitching the infrared modal feature maps and the visible light modal feature maps according to the channel dimension to obtain joint features, assigning attention weights, calculating and generating adaptive importance coefficients for the infrared modality and the visible light modality, and performing element-level weighting on the infrared modal feature maps and the visible light modal feature maps to obtain dual-modal fusion weighted features, the dual-modal fusion weighted features including infrared modal weighted features and visible light modal weighted features; performing multi-layer fusion of the infrared modal weighted features and the visible light modal weighted features to generate fusion features for hotspot detection and classification of photovoltaic modules, generating hotspot detection results.

[0006] This application also provides a photovoltaic hot spot detection system with cross-modal feature deep fusion. The system includes: a bidirectional sensing pairing module for bidirectional sensing pairing based on photovoltaic modules to construct a bimodal image dataset, the bimodal image dataset containing infrared images and visible light images; an image feature extraction module for performing bidirectional parallel analysis on the infrared images and the visible light images to extract a parallel feature set, the parallel feature set containing infrared modal feature maps and visible light modal feature maps; and a fusion weighted feature acquisition module for fusion weighted features based on the infrared modal feature maps and the visible light modal feature maps. The system performs registration and stitching along the channel dimension to obtain joint features. Attention weights are then assigned, and adaptive importance coefficients for infrared and visible light modes are calculated. Element-level weighting is applied to the infrared and visible light mode feature maps to obtain dual-modal fusion weighted features. These dual-modal fusion weighted features include both infrared and visible light mode weighted features. A hot spot detection result generation module is used to perform multi-layer fusion of the infrared and visible light mode weighted features to generate fused features for hot spot detection and classification of photovoltaic modules, generating hot spot detection results.

[0007] This application proposes a photovoltaic hotspot detection method and system based on deep cross-modal feature fusion. The method involves bidirectional sensing pairing of photovoltaic modules to construct a dual-modal image dataset; dual-branch parallel analysis to extract parallel feature sets; registration and stitching according to channel dimensions to obtain joint features; attention weight allocation; calculation and element-level weighting of adaptive importance coefficients for infrared and visible light modes to obtain dual-modal fused weighted features; and classification of photovoltaic modules for hotspot detection using the generated fused features to produce hotspot detection results. This method addresses the technical problems in existing technologies, such as neglecting intermodal attribute differences, lack of adaptability in fusion weights, and fragmented feature fusion, leading to poor anti-interference capability and accuracy of photovoltaic hotspot detection in complex environments. It achieves the technical effect of improving the discriminative power, robustness, and accuracy of cross-modal feature fusion in photovoltaic hotspot detection. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments of this disclosure will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0009] Figure 1 This is a schematic diagram of the photovoltaic hot spot detection method based on deep fusion of cross-modal features provided in an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of the photovoltaic hot spot detection system with deep fusion of cross-modal features provided in an embodiment of this application.

[0011] Figure labeling: 10 for bidirectional sensing pairing module, 20 for image feature extraction module, 30 for fusion weighted feature acquisition module, and 40 for hot spot detection result generation module. Detailed Implementation

[0012] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on the specific implementation methods, structures, features and effects of the present invention.

[0013] This application provides a photovoltaic hot spot detection method based on deep fusion of cross-modal features, such as... Figure 1 As shown, the photovoltaic hotspot detection method based on deep fusion of cross-modal features includes: Step S100: Perform bidirectional sensing pairing based on photovoltaic modules to construct a bimodal image dataset, which includes infrared images and visible light images.

[0014] Preferably, photovoltaic modules (i.e., solar panels) are used as the detection object and data acquisition target. At the same time and for the same area of ​​the photovoltaic module, two different types of sensors are used to simultaneously acquire images. The two sets of sensors are established in spatial and temporal correspondence. Specifically, the infrared sensor and the visible light sensor are installed with parallel or coaxial optical axes and consistent field of view to ensure that the two sensors capture the same physical area on the photovoltaic module, with pixel-level spatial alignment. The infrared sensor and the visible light sensor are synchronously triggered for exposure to ensure that the two sets of images are acquired at the same time, avoiding changes in the state of the photovoltaic module due to time differences, such as temperature fluctuations and changes in illumination angle. A one-to-one data association is established between each frame of infrared image and the corresponding visible light image at the same time, forming image pairs, which are then organized, labeled, and stored to obtain a dual-modal image dataset, including infrared images and visible light images. The infrared images contain temperature field distribution data on the surface of the photovoltaic module, with temperature anomaly areas (hot spots) appearing as high-temperature bright spots. The visible light images contain structural information such as the surface texture of the photovoltaic module, cell arrangement, busbar direction, dirt obstruction, cracks, and scratches.

[0015] Step S200: Perform dual-branch parallel analysis on the infrared image and the visible light image to extract a parallel feature set, which includes infrared modal feature maps and visible light modal feature maps.

[0016] Step 200 further includes: constructing an infrared branch feature extraction network, performing detail enhancement convolution on the infrared image through the infrared branch feature extraction network to obtain an infrared modal feature map; constructing a visible light branch feature extraction network, performing surface structure analysis on the visible light image through the visible light branch feature extraction network to obtain a visible light modal feature map; and integrating the infrared modal feature map and the visible light modal feature map in parallel to construct a parallel feature set.

[0017] Preferably, an infrared branch feature extraction network is constructed based on a deep convolutional neural network. Specifically, the parameters of the infrared branch feature extraction network are trained and optimized only on infrared images and do not participate in the processing of visible light images. It is used to extract temperature distribution features related to hot spots from infrared images. The network consists of an input layer, multiple convolutional layers, pooling layers, and activation function layers, which are used to receive single-channel or pseudo-three-channel infrared image data, extract thermal radiation features at different levels of abstraction, reduce the spatial resolution of feature maps, and introduce nonlinear transformation capabilities, respectively. After the infrared image is input into the infrared branch feature extraction network, a detail enhancement convolution operation is performed. This includes using a small-sized convolution kernel (such as 3×3 or 5×5) to perform sliding window convolution on the infrared image, preserving detailed information about local temperature changes. During the convolution process, by setting specific convolution kernel weights or using second-order differential operators such as the Laplacian operator, the edge regions with drastic temperature gradient changes in the infrared image (i.e., the boundary between the hot spot and the normal region) are enhanced, making the boundaries sharper. After the convolution operation, each pixel position of the infrared image is mapped to a new feature value, forming a feature map. The response value of the temperature boundary position in the feature map is higher than that of the uniform temperature region, thereby amplifying the boundary information of the temperature anomaly region in the infrared image, making the edge of the hot spot more clearly distinguishable in the feature space. Finally, an infrared modal feature map with spatial and channel dimensions is obtained. The high response value region indicates that there is a significant temperature anomaly at that location, corresponding to the hot spot region on the surface of the photovoltaic module.

[0018] Preferably, a visible light branch feature extraction network is constructed based on a deep convolutional neural network. Specifically, the parameters of the visible light branch feature extraction network are trained and optimized only on visible light images and do not participate in the processing of infrared images. It is used to extract texture and geometric features related to the surface structure of photovoltaic modules from visible light images. The network consists of an input layer, multiple convolutional layers, pooling layers, and activation function layers, which are used to receive three-channel RGB visible light image data, extract information such as texture, edge, and color of the photovoltaic module surface, reduce the spatial resolution of the feature map, and introduce nonlinear transformation capabilities. After inputting the visible light image into the visible light branch feature extraction network, the network performs surface structure analysis operations. This includes capturing microscopic texture information such as the grid arrangement of solar cells, the direction and density of fine grid lines, and the position of busbars on the surface of the photovoltaic module through convolutional kernels. It also identifies the position and shape of geometric structures such as cell edges, module frames, and crack boundaries within the photovoltaic module. Furthermore, it extracts regions in the visible light image that differ from the normal module surface, such as structural changes caused by dirt coverage, scratches, cracks, and shading, thereby obtaining physical structure information of the photovoltaic module surface. This information can be used to distinguish between "temperature rise but structural integrity" (e.g., hot spots caused by external shading) and "temperature rise and structural damage" (e.g., hot spots caused by cell cracks). Finally, a visible light modal feature map is generated. High response value regions indicate significant surface structural anomalies at that location, corresponding to physical defect areas on the photovoltaic module surface. Finally, the infrared modal feature map and the visible light modal feature map are integrated and stored in parallel, and indexed according to a preset data structure to construct a parallel feature set.

[0019] Furthermore, step S200 also includes: extracting multi-scale feature responses from the infrared image to capture local temperature gradient change information; stitching the local temperature gradient change information according to the channel dimension to obtain a multi-scale thermal response feature map; performing hybrid convolution on the multi-scale thermal response feature map to generate an infrared intermediate feature map; aggregating the infrared intermediate feature maps; and performing information interaction on the multi-scale thermal response feature maps according to the aggregation result to generate an infrared modal feature map.

[0020] Preferably, for the input infrared image, multiple convolutional kernels of different sizes or feature extraction operators with different receptive fields are used simultaneously to extract feature responses at the same image location. This includes deploying multiple convolutional kernels of different sizes (such as 3×3, 5×5, 7×7) on the infrared image, scanning the same pixel location by multiple convolutional kernels respectively, generating response values ​​of multiple different receptive fields, and sampling the same image region using convolutional kernels with different dilatancy rates to obtain feature responses with different spatial spans. In the multi-scale feature response extraction process, after the convolutional kernels of each scale act on the infrared image, the response value reflects the spatial rate of change of the temperature difference between the pixel location and its surrounding neighboring pixels, thereby capturing local temperature gradient change information. The temperature gradient value of a pixel location is the rate of change of the difference between the temperature value of that pixel and the temperature value of its neighboring pixels in the spatial direction. In the infrared image, the temperature distribution in the normal area of ​​the photovoltaic module is relatively uniform, and the temperature gradient value is small. There is an obvious temperature jump at the boundary of the hot spot area, and the temperature gradient value increases significantly. The temperature gradient feature maps generated in the multi-scale feature response extraction stage are arranged and concatenated along the channel dimension to form a multi-scale thermal response feature map. Assuming there are three convolutional kernels of different scales, each scale outputs a feature map with a spatial size of H×W (consistent with the spatial size of the input infrared image) and C channels. Concatenating the feature maps of the three scales along the channel dimension results in a new feature map with 3×C channels. Each pixel at a spatial location now has 3×C channel feature values, derived from the responses of the convolutional kernels at different scales to the temperature gradient at that location. The multi-scale thermal response feature map contains multiple channels at the same spatial location, with different channel groups encoding the temperature gradient response intensity at different spatial scales, while maintaining the spatial dimension's correspondence with the original infrared image's location.

[0021] Preferably, on the multi-scale thermal response feature map, a multi-branch convolutional structure is used to perform convolutional operations with different configurations simultaneously. Specifically, a conventional 3×3 convolution is used to extract the neighborhood comprehensive response in the multi-scale thermal response feature map. A 1×1 pointwise convolution is used to linearly recombine the cross responses between channels. A depthwise separable convolution is used to reduce the number of parameters while maintaining spatial resolution. The output feature maps of each branch are spliced ​​or fused in the channel dimension, allowing the information of each scale in the multi-scale thermal response feature map to cross-penetrate and recombine under different configurations of convolutional operations, generating a more information-rich intermediate representation and obtaining an infrared intermediate feature map. Its spatial resolution is consistent with the input infrared image, containing both the original temperature gradient multi-scale information and the cross-scale interaction information after hybrid convolution recombination.

[0022] Preferably, global or local statistical aggregation operations are performed on the infrared intermediate feature map, including global average pooling for each channel of the infrared intermediate feature map to obtain the average response intensity of each channel at all spatial locations, global max pooling for each channel of the infrared intermediate feature map to obtain the maximum response intensity of each channel at all spatial locations, and concatenating the average pooling result and the max pooling result to form an aggregated vector that reflects the overall response intensity distribution of each channel of the infrared intermediate feature map. Then, based on the aggregation results, information interaction is performed on the multi-scale thermal response feature map. Specifically, the aggregation vector is mapped through a fully connected layer to generate a weight vector with the same number of channels as the multi-scale thermal response feature map. This weight vector is then multiplied channel by channel in the multi-scale thermal response feature map, that is, each channel of the multi-scale thermal response feature map is multiplied by its corresponding weight. Channels with larger weight values ​​are enhanced in the multi-scale thermal response feature map, while channels with smaller weight values ​​are suppressed. Thus, by utilizing the global aggregation information of the infrared intermediate feature map, the channel response intensity of the multi-scale thermal response feature map is adjusted in turn, so that channels that contribute more to hot spot detection receive higher response weights. Finally, an infrared modal feature map is generated, which enhances the temperature gradient feature channels that are beneficial to hot spot boundary discrimination and region localization, while suppressing channels that are unfavorable or redundant.

[0023] Furthermore, step S200 also includes: performing global content perception based on the visible light image, extracting global content data and local salient data; concatenating and convolving the global content data and the local salient data according to the channel dimension to construct a content-guided attention weight map; extracting the original features of the visible light image and multiplying them with the content-guided attention weight map according to data elements to generate content enhancement features; and performing residual convolution depth analysis based on the content enhancement features to construct the visible light modal feature map.

[0024] Preferably, global content perception is performed on the visible light image to obtain the overall semantic information of the photovoltaic module's visible light image, and global content data reflecting the overall information of the entire visible light image is generated. This data includes at least the overall light intensity, global color distribution, background environment category, and overall orientation of the module. High-contrast, high-abruptness, or high-attention pixel blocks appearing in local areas of the visible light image are located and their features are extracted to generate local salient data. For example, a sliding window method is used to scan the entire visible light image region by region, and the pixel value difference between each local region and the surrounding region is calculated. Regions with values ​​higher than a preset threshold are marked as salient regions. The content includes at least the location coordinates of the salient region, the morphological features of the salient region, the intensity value of the salient region, and the category of the salient region.

[0025] Preferably, global content data and local salient data are combined and arranged along the channel dimension. This includes spatially copying and expanding the global content data tensor, expanding the 1×1×C1 tensor to H×W×C1 in the spatial dimension, with each spatial location carrying the same global content data. The expanded global content data tensor is then concatenated with the local salient data tensor along the channel dimension to form a new tensor with a spatial size of H×W and the number of channels C1+C2. A convolution operation is then performed on this tensor, using a convolution kernel of size 3×3 or 1×1, which slides simultaneously along the spatial and channel dimensions of the concatenated tensor. The input channels of the convolution kernel are C1+C2, covering the channels of both global content and local salient data. Each convolution kernel outputs 1 channel, and multiple convolution kernels output multiple channels in parallel. Finally, a content-guided attention weight map is constructed, with attributes including spatial size, number of channels, and the value at each position.

[0026] Preferably, the original feature map of the visible light image and the content-guided attention weight map are multiplied element-wise to generate content-enhanced features. In this process, spatial locations with high attention weight values ​​in the original feature map are amplified, while locations with low weight values ​​are suppressed. Significant areas on the photovoltaic module surface (such as cracked areas or dirt areas) receive higher feature response values, while non-significant uniform areas or pure background areas have lower feature response values. The content-enhanced features are then input into a residual convolution module for multiple convolution operations with skip connections. Specifically, the main path consists of multiple convolutional layers (such as 3×3 convolutions), batch normalization layers, and activation function layers connected in series to perform deep feature extraction on the content-enhanced features. The bypass path is used to enhance the content input of the residual module through 1×1 convolution or direct identity mapping, and then connects it to the output of the module. The output of the main path and the output of the bypass connection are added element-wise to form the final output of the residual module, which is the visible light modal feature map. Its attributes include channel dimension, spatial dimension and enhancement effect. For example, the content-guided attention mechanism completes the key enhancement of salient areas, and the residual convolution further extracts the enhanced deep features, so that the feature extraction process of the visible light branch actively focuses on the abnormal areas on the surface of the photovoltaic module. At the same time, the global information of the whole image provides contextual constraints for local judgment, enhancing the detection sensitivity of visible light features for surface anomalies such as cracks, shading, and dirt.

[0027] Step S300: The infrared modal feature map and the visible light modal feature map are registered and stitched together according to the channel dimension to obtain joint features. Attention weights are assigned, and adaptive importance coefficients for infrared modality and visible light modality are calculated and generated. The infrared modal feature map and the visible light modal feature map are then weighted element-wise to obtain dual-modal fusion weighted features. The dual-modal fusion weighted features include infrared modal weighted features and visible light modal weighted features.

[0028] Step S300 further includes: compressing the spatial dimension of the joint features to obtain a channel description vector of the global response intensity; performing weight mapping based on the channel description vector to generate an original channel attention vector; splitting the original channel attention weight vector according to the channel source, forming infrared channel attention sub-weights from the source of the infrared modal feature map, and forming visible light channel attention sub-weights from the source of the visible light modal feature map; aggregating and compressing the channel dimension of the infrared channel attention sub-weights to obtain infrared modal aggregated weights; and aggregating the channel dimension of the visible light channel attention sub-weights. The infrared modal aggregation weights and visible light modal aggregation weights are obtained by performing aggregation compression on the infrared modal aggregation weights and visible light modal aggregation weights. Normalization constraint processing is applied to the infrared modal aggregation weights and visible light modal aggregation weights to generate infrared modal adaptive importance coefficients and visible light modal adaptive importance coefficients. The infrared modal adaptive importance coefficients are multiplied element-wise with the infrared modal feature map to obtain infrared modal weighted features. The visible light modal adaptive importance coefficients are multiplied element-wise with the visible light modal feature map to obtain visible light modal weighted features. The infrared modal weighted features and visible light modal weighted features are combined and output to form the dual-modal fusion weighted features.

[0029] Preferably, the infrared modal feature map and the visible light modal feature map are registered and stitched together according to the channel dimension to obtain joint features. Each channel value at each spatial location represents the feature response intensity of that location in the corresponding modality. The joint features are aggregated in the spatial dimension (H×W) to compress the response values ​​of all spatial locations in each channel into a single value. This includes calculating the average value of the channel at all spatial locations (H×W pixels), i.e., global average pooling, or taking the maximum value of each channel in the joint features at all spatial locations, i.e., global max pooling. The results of the two are then stitched together or added to obtain the channel description vector of the global response intensity. Each element corresponds to the global response intensity value of one channel in the joint features. Channels with high response intensity values ​​indicate that the feature type encoded by that channel is prevalent or strongly activated in the entire image. Channels with low response intensity values ​​indicate that the feature type encoded by that channel is weakly activated in the entire image.

[0030] Preferably, the channel description vector is input into the mapping function to convert the global response intensity value of each channel into the attention weight value of that channel. Specifically, a fully connected layer is used to perform a linear transformation on the channel description vector, followed by a Sigmoid activation function to constrain the output value to the (0, 1) interval. Alternatively, a ReLU activation function is used in the middle of a two-layer fully connected structure, with the output connected to a Sigmoid activation function, allowing the network to autonomously learn the nonlinear mapping relationship from the channel global response intensity to the channel attention weight, thereby generating the original channel attention vector. Each element corresponds to the attention weight value of a channel in the joint features, and the larger the weight value, the more important the channel is.

[0031] Preferably, the elements in the original channel attention vector are split into two independent sub-vectors according to their corresponding feature map sources. The first C_IR elements of the original channel attention vector correspond to the C_IR channels of the infrared modal feature map before splicing, and these are extracted to form the infrared channel attention sub-weights. The last C_RGB elements of the original channel attention vector correspond to the C_RGB channels of the visible light modal feature map before splicing, and these are extracted to form the visible light channel attention sub-weights. The weight values ​​in the infrared channel attention sub-weights are aggregated and compressed to generate the infrared modal aggregated weights. For example, the arithmetic mean or maximum value of all weight values ​​can be calculated as the infrared modal aggregated weights. Similarly, the visible light channel attention sub-weights are aggregated and compressed along the channel dimension to obtain the visible light modal aggregated weights, which reflect the overall importance of the visible light modality in the joint features. Then, the two values ​​of infrared mode aggregation weight and visible light mode aggregation weight are transformed by the Softmax function or the normalized exponential function so that the sum of the two transformed values ​​equals 1, thereby generating two scalar values ​​in the interval of 0 and 1, which are the infrared mode adaptive importance coefficient and the visible light mode adaptive importance coefficient. The infrared mode adaptive importance coefficient represents the importance ratio of the infrared mode feature map in the fusion, and the visible light mode adaptive importance coefficient represents the importance ratio of the visible light mode feature map in the fusion.

[0032] Preferably, the infrared modal adaptive importance coefficient is multiplied element-wise with the infrared modal feature map to obtain the infrared modal weighted feature. Similarly, the visible light modal adaptive importance coefficient is multiplied element-wise with the visible light modal feature map to obtain the visible light modal weighted feature. A higher importance coefficient retains more of the overall response amplitude of the infrared modal feature map; a lower importance coefficient compresses the overall response amplitude. The infrared modal weighted feature and the visible light modal weighted feature are then concatenated a second time along the channel dimension to form the final dual-modal fusion weighted feature, ensuring global consistency of the weights obtained during fusion between the two modes.

[0033] Furthermore, step S300 also includes that both the infrared mode adaptive importance coefficient and the visible light mode adaptive importance coefficient are dynamically adjustable values, wherein the sum of the infrared mode adaptive importance coefficient and the visible light mode adaptive importance coefficient is 1; when the infrared mode adaptive importance coefficient is greater than 0.5, attention is biased towards the infrared mode feature map; when the visible light mode adaptive importance coefficient is greater than 0.5, attention is biased towards the visible light mode feature map.

[0034] Preferably, the adaptive importance coefficients for the infrared mode and the visible light mode are calculated and generated in real time based on the specific content of the input image. For different input image pairs (different photovoltaic modules, different lighting conditions, different hot spot states), the importance coefficients are different. When the infrared temperature characteristics of the photovoltaic module hot spot are significant and clear (such as temperature difference, sharp boundary), the network tends to assign a higher importance coefficient to the infrared mode. When the infrared image is severely affected by environmental thermal noise (such as global heating caused by direct sunlight), while the structural features such as cracks and occlusions in the visible light image are more obvious, the network tends to assign a higher importance coefficient to the visible light mode. When the information of both modes is clear and effective, the two coefficients approach an equilibrium state of 0.5, and the sum of the two coefficients is strictly equal to 1, ensuring that the two coefficients are compared and assigned on the same numerical scale. When the adaptive importance coefficient of the infrared mode is greater than 0.5, attention is biased towards the infrared mode feature map. For example, the original response amplitude of the infrared mode feature map is preserved by more than 0.5 times, while the original response amplitude of the visible light mode feature map is compressed to less than 0.5 times. The values ​​of the infrared mode feature channels are generally higher than those of the visible light mode feature channels, and infrared information dominates in the fusion features. When the adaptive importance coefficient of the visible light mode is greater than 0.5, attention is biased towards the visible light mode feature map. For example, the original response amplitude of the visible light mode feature map is preserved by more than 0.5 times, while the original response amplitude of the infrared mode feature map is compressed to less than 0.5 times. The values ​​of the visible light mode feature channels are generally higher than those of the infrared mode feature channels, and visible light information dominates in the fusion features. This allows the system to adaptively tilt the fusion weights toward modes that are more reliable and effective, based on the actual quality distribution of the input data. When the infrared mode is affected by environmental interference or the visible light mode is affected by illumination interference, the system can automatically switch to the dominant mode, thus avoiding a decrease in hot spot detection performance when the fixed fusion weights fail in a certain mode.

[0035] Step S400: The infrared mode weighted features and the visible light mode weighted features are fused in multiple layers to generate fused features for hot spot detection and classification of photovoltaic modules, and hot spot detection results are generated.

[0036] Step S400 further includes: performing shallow fusion of the infrared modal weighted features and the visible light modal weighted features to generate shallow fused features; performing mid-level fusion of the infrared modal weighted features and the visible light modal weighted features to generate mid-level fused features; performing deep fusion of the infrared modal weighted features and the visible light modal weighted features to generate deep fused features; splicing the shallow fused features, the mid-level fused features, and the deep fused features according to the channel dimension to obtain cross-layer spliced ​​features; performing association learning based on the cross-layer spliced ​​features, and performing cross-layer association response on the shallow fused features, the mid-level fused features, and the deep fused features according to the association learning results to generate the fused features.

[0037] Preferably, in the shallow stage of the feature extraction network, the infrared modal weighted features and the visible light modal weighted features are initially fused. This includes concatenating the infrared modal weighted features and the visible light modal weighted features in the channel dimension, and then performing one or two small-size convolutions (such as 3×3 convolutions) on the concatenated features. The number of convolution kernels is small (usually 64 or 128), the stride is 1, and the spatial resolution remains unchanged. After the convolution, a ReLU activation function is applied to introduce a nonlinear transformation to obtain shallow fused features, which contain fine spatial information of the photovoltaic module surface, such as the precise boundary position of hot spots and the fine edges of cracks. In the middle layer stage of the feature extraction network, the weighted features of the two modalities are fused at a deeper level. This includes passing the infrared modal weighted features and the visible light modal weighted features through max pooling layers or convolutional layers with a stride of 2 and concatenating them along the channel dimension. Then, multiple convolutions (usually 3-5 layers) are performed on the concatenated features, with a moderate number of convolution kernels (usually 256 or 512). Batch normalization and ReLU activation functions are interspersed in between. The concatenated features are nonlinearly transformed through multiple convolutions to extract information at a higher level of abstraction, generating middle-layer fused features to encode the fusion information of infrared and visible light at the middle layer (medium level of abstraction). In the deep stage of the feature extraction network, the weighted features of the two modalities are fused at the deepest level. This includes downsampling the infrared modal weighted features and the visible light modal weighted features twice with a stride of 2, and then concatenating them along the channel dimension. The concatenated features are then subjected to multi-layer deep convolution (usually 6-10 layers) with a large number of convolution kernels (usually 512 or 1024). Through the stacking of a large number of convolutional layers, the high-level semantic features required for hot spot detection of photovoltaic modules are extracted, generating deep fusion features, such as the overall temperature distribution of hot spots, the geometric structure of cracks, and the shape features of obstructions, which are used to encode the deep fusion information of infrared and visible light.

[0038] Preferably, since the spatial resolutions of shallow, mid-level, and deep fusion features are different, their spatial dimensions are aligned before channel-dimensional stitching. Then, the shallow, mid-level, and deep fusion features are concatenated along the channel dimension to obtain cross-layer stitched features, which simultaneously contain fusion information from the shallow, mid-level, and deep layers at the same spatial location. These cross-layer stitched features are then input into the association learning module, allowing the network to automatically learn the interdependencies and interaction patterns between the shallow, mid-level, and deep features. Specifically, multi-head self-attention operations are performed on the cross-layer stitched features to calculate the attention weights between each spatial location and all other spatial locations. Gated convolutional layers are used to control the shallow and mid-level features through learnable gating signals. The system determines the direction and intensity of information flow between deep features. Shallow features are transformed through convolution and added to mid-level features, then transformed again and added to deep features, forming a bottom-up cross-layer information transmission path. Based on the association learning results, cross-layer association responses are applied to the three fused features, including element-wise multiplication of the response coefficients of shallow, mid-, and deep features with their corresponding hierarchical features. This achieves mutual enhancement or suppression of cross-layer information, ultimately generating fused features. These features include the spatial detail precision of shallow fused features, enabling accurate hotspot boundary location; structural combination information from mid-level fused features, enabling perception of hotspot geometry; and semantic judgment capabilities from deep fused features, accurately distinguishing hotspots from other fault categories. This ensures both high-precision localization and high-accuracy classification capabilities.

[0039] Furthermore, step S400 also includes: performing global feature aggregation based on the fused features, obtaining the global feature aggregation result for multi-task classification, and calculating the fault category probability distribution parameters; performing probability detection based on the fault category probability distribution parameters to obtain hot spot detection classification results; performing spatial location analysis based on the fused features, extracting spatial location features for mapping, and determining hot spot region location regression parameters; and associating the hot spot detection classification results with the hot spot region location regression parameters to generate the hot spot detection results.

[0040] Preferably, for each channel of the fused feature, the average value of that channel at all spatial locations (H×W pixels) is calculated, or the maximum value of that channel at all spatial locations is taken. The global feature aggregation result is output, with each element corresponding to the overall response intensity of a channel across the entire image. The global feature aggregation result is then input into a multi-task classifier, which outputs probability distributions for multiple categories. Specifically, the first classification task determines whether the photovoltaic module has hot spot faults, the second classification task determines the specific category of the fault, such as hot spot, crack, shadow occlusion, dirt coverage, normal, etc., and the third classification task determines the severity level of the hot spot, such as mild, moderate, and severe. Furthermore, a probability distribution vector is output for each classification task, where each element represents the probability value of the input image belonging to that category, such as the probability of hot spot presence, the probability vector of fault category, and the probability vector of severity. Multiple probabilities are obtained after processing by the Softmax function, and the sum of the probabilities of each category is 1. Next, a threshold judgment or maximum probability selection operation is performed on the fault category probability distribution parameters. For example, the category with the highest probability value is selected from the fault category probability vector as the preliminary classification result. When the probability of hot spot existence is greater than a preset threshold (such as 0.5 or 0.7), it is judged as a hot spot fault; otherwise, it is judged as a non-hot spot fault. Finally, the hot spot detection classification result is obtained, which may include hot spot fault / non-hot spot fault, specific fault category name and severity level label, such as hot spot fault, hot spot caused by crack, and moderate hot spot. Then, a position-by-position analysis is performed on the spatial dimension of the fused features to evaluate the probability or confidence of each spatial position belonging to the hot spot region. This includes extracting feature information related to spatial position from the fused features and converting it into specific position coordinate parameters through regression mapping to determine the hot spot region position regression parameters, which are used to quantitatively describe the specific position and range of the hot spot in the original image, including at least the bounding box center coordinates, bounding box size, and confidence score.

[0041] Furthermore, step S400 also includes: when the hot spot detection classification result is a hot spot fault category, extracting bounding box parameters based on the hot spot region location regression parameters, mapping the bounding box parameters to the bimodal image dataset to identify the hot spot fault category, and generating the hot spot detection result; when the hot spot detection classification result is a non-hot spot fault category, determining the target fault category, mapping the target fault category in combination with the hot spot region location regression parameters to the bimodal image dataset to identify the fault category, and generating the hot spot detection result.

[0042] Preferably, the category information output by the classification branch is integrated with the location information output by the regression branch. When the hot spot detection classification result is a hot spot fault category, the center coordinates, width, height, and confidence score of the hot spot region bounding box are extracted from the hot spot region location regression parameters and transformed to form bounding box parameters for image region localization. Then, the bounding box coordinate parameters are synchronously mapped onto the registered and aligned infrared and visible light images. In the infrared image, the bounding box selects the hot spot region with abnormally high temperature, and in the visible light image, the bounding box at the same location selects the surface structure region corresponding to the hot spot region. Visual annotation information is then added to clearly indicate that there is a hot spot fault in the region. For example, a red or colored rectangle is drawn at the bounding box position in the infrared image, and a text label is added next to the bounding box with the content "hot spot fault" and the corresponding probability confidence. The annotation is also completed synchronously on the dual-modal image dataset (infrared image and visible light image), and the annotation content includes the bounding box, category label, and confidence information.

[0043] Preferably, if the hot spot detection classification result is a non-hot spot fault category, such as crack fault, shadow occlusion fault, dirt coverage fault, circuit fault, and aging fault, the specific fault type with the highest probability value is determined as the target fault category based on the fault category probability distribution parameters. For example, the category with the highest probability value is selected from the fault category probability vector (e.g., hot spot 0.05, crack 0.82, shadow occlusion 0.08, dirt coverage 0.03, normal 0.02). Then, the determined target fault category name is bound and associated with the position regression parameters output by the regression branch to form a complete information pair indicating that the anomaly at that location belongs to that fault category. The bounding box coordinates after the position regression parameters are converted are then synchronously mapped onto the registered and aligned infrared and visible light images. Visual annotation information matching the target fault category is added to the area selected by the bounding box, including drawing a rectangle at the bounding box position in the infrared image, with different colors set according to the fault category, such as yellow for cracks and blue for shadow occlusion; and adding a text label next to the bounding box, the content of which is the target fault category name and the corresponding probability confidence level, such as crack fault 82.0%. By presenting hot spot detection results simultaneously in the form of visual annotations, maintenance personnel can identify the location of abnormal temperatures in infrared images and view the structural status of the corresponding location in visible light images, which facilitates comprehensive judgment and on-site handling, thereby ensuring the accuracy of hot spot detection.

[0044] In the above text, refer to Figure 1 A photovoltaic hotspot detection method based on cross-modal feature deep fusion according to embodiments of the present invention is described in detail. Next, reference will be made to... Figure 2 A photovoltaic hot spot detection system based on cross-modal feature deep fusion according to an embodiment of the present invention is described.

[0045] The photovoltaic hot spot detection system based on cross-modal feature deep fusion according to embodiments of the present invention addresses the technical problems in existing technologies, such as ignoring differences in attributes between multiple modes, lack of adaptability in fusion weights, and fragmented feature fusion, leading to poor anti-interference capability and accuracy of photovoltaic hot spot detection in complex environments. It achieves the technical effect of improving the discriminative power and robustness of cross-modal feature fusion, as well as the accuracy of photovoltaic hot spot detection. Figure 2 As shown, the photovoltaic hot spot detection system with cross-modal feature deep fusion includes: a bidirectional sensing pairing module 10, an image feature extraction module 20, a fusion weighted feature acquisition module 30, and a hot spot detection result generation module 40.

[0046] A bidirectional sensing pairing module 10 is used to perform bidirectional sensing pairing based on photovoltaic modules to construct a bimodal image dataset, which includes infrared images and visible light images. An image feature extraction module 20 is used to perform bi-branch parallel analysis on the infrared and visible light images to extract a parallel feature set, which includes infrared modal feature maps and visible light modal feature maps. A fusion weighted feature acquisition module 30 is used to register and stitch the infrared and visible light modal feature maps according to channel dimensions to obtain a joint feature map. The features are weighted by attention allocation, and the infrared modality adaptive importance coefficient and the visible light modality adaptive importance coefficient are calculated and generated. The infrared modality feature map and the visible light modality feature map are then element-wise weighted to obtain the dual-modality fused weighted features. The dual-modality fused weighted features include infrared modality weighted features and visible light modality weighted features. The hot spot detection result generation module 40 is used to perform multi-layer fusion of the infrared modality weighted features and the visible light modality weighted features to generate fused features for hot spot detection and classification of photovoltaic modules, and to generate hot spot detection results.

[0047] The specific configuration of the image feature extraction module 20 will be described in detail below. The image feature extraction module 20 further includes: constructing an infrared branch feature extraction network, performing detail enhancement convolution on the infrared image through the infrared branch feature extraction network to obtain an infrared modal feature map; constructing a visible light branch feature extraction network, performing surface structure analysis on the visible light image through the visible light branch feature extraction network to obtain a visible light modal feature map; and integrating the infrared modal feature map and the visible light modal feature map in parallel to construct a parallel feature set.

[0048] The specific configuration of the image feature extraction module 20 will be described in detail below. The image feature extraction module 20 further includes: extracting multi-scale feature responses from the infrared image to capture local temperature gradient change information; stitching the local temperature gradient change information according to channel dimensions to obtain a multi-scale thermal response feature map; performing hybrid convolution based on the multi-scale thermal response feature map to generate an infrared intermediate feature map; aggregating the infrared intermediate feature maps; and performing information interaction on the multi-scale thermal response feature maps based on the aggregation result to generate an infrared modal feature map.

[0049] The specific configuration of the image feature extraction module 20 will be described in detail below. The image feature extraction module 20 further includes: performing global content perception based on the visible light image to extract global content data and local salient data; concatenating and convolving the global content data and the local salient data according to the channel dimension to construct a content-guided attention weight map; multiplying the original features of the visible light image with the content-guided attention weight map element by element to generate content enhancement features; and performing residual convolution depth analysis based on the content enhancement features to construct the visible light modal feature map.

[0050] The specific configuration of the fusion weighted feature acquisition module 30 will be described in detail below. The fusion weighted feature acquisition module 30 further includes: compressing the spatial dimension of the joint features to obtain a channel description vector of the global response intensity; performing weight mapping based on the channel description vector to generate an original channel attention vector; splitting the original channel attention weight vector according to its channel source, forming infrared channel attention sub-weights from the infrared modal feature map source and forming visible light channel attention sub-weights from the visible light modal feature map source; aggregating and compressing the channel dimension of the infrared channel attention sub-weights to obtain infrared modal aggregated weights; and performing... Channel-level aggregation and compression are performed to obtain visible light mode aggregation weights. Normalization constraints are applied to the infrared mode aggregation weights and the visible light mode aggregation weights to generate adaptive importance coefficients for both infrared and visible light modes. The adaptive importance coefficients for infrared modes are multiplied element-wise by the infrared mode feature map to obtain infrared mode weighted features. The adaptive importance coefficients for visible light modes are also multiplied element-wise by the visible light mode feature map to obtain visible light mode weighted features. Finally, the infrared mode weighted features and the visible light mode weighted features are combined and output to form the dual-modal fusion weighted features.

[0051] The specific configuration of the fusion weighted feature acquisition module 30 will be described in detail below. The fusion weighted feature acquisition module 30 further includes: both the infrared modality adaptive importance coefficient and the visible light modality adaptive importance coefficient are dynamically adjustable values, wherein the sum of the infrared modality adaptive importance coefficient and the visible light modality adaptive importance coefficient is 1; when the infrared modality adaptive importance coefficient is greater than 0.5, attention is biased towards the infrared modality feature map; when the visible light modality adaptive importance coefficient is greater than 0.5, attention is biased towards the visible light modality feature map.

[0052] The specific configuration of the hot spot detection result generation module 40 will be described in detail below. The hot spot detection result generation module 40 further includes: shallowly fusing the infrared modal weighted features and the visible light modal weighted features to generate shallow fused features; performing intermediate-level fusing the infrared modal weighted features and the visible light modal weighted features to generate intermediate-level fused features; performing deep-level fusing the infrared modal weighted features and the visible light modal weighted features to generate deep fused features; stitching the shallow fused features, the intermediate fused features, and the deep fused features according to the channel dimension to obtain cross-layer stitched features; performing association learning based on the cross-layer stitched features, and performing cross-layer association responses on the shallow fused features, the intermediate fused features, and the deep fused features according to the association learning results to generate the fused features.

[0053] The specific configuration of the hot spot detection result generation module 40 will be described in detail below. The hot spot detection result generation module 40 further includes: performing global feature aggregation based on the fused features, obtaining the global feature aggregation result for multi-task classification, and calculating fault category probability distribution parameters; performing probability detection based on the fault category probability distribution parameters to obtain hot spot detection classification results; performing spatial location analysis based on the fused features, extracting spatial location features for mapping, and determining hot spot region location regression parameters; and associating the hot spot detection classification results with the hot spot region location regression parameters to generate the hot spot detection result.

[0054] The specific configuration of the hot spot detection result generation module 40 will be described in detail below. The hot spot detection result generation module 40 further includes: when the hot spot detection classification result is a hot spot fault category, extracting bounding box parameters based on the hot spot region location regression parameters, mapping the bounding box parameters to the bimodal image dataset to identify the hot spot fault category, and generating the hot spot detection result; when the hot spot detection classification result is a non-hot spot fault category, determining the target fault category, mapping the target fault category in conjunction with the hot spot region location regression parameters to the bimodal image dataset to identify the fault category, and generating the hot spot detection result.

[0055] The photovoltaic hot spot detection system with cross-modal feature deep fusion provided in this embodiment of the invention can execute the photovoltaic hot spot detection method with cross-modal feature deep fusion provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A photovoltaic hot spot detection method based on deep fusion of cross-modal features, characterized in that, The method includes: Based on photovoltaic modules, bidirectional sensing pairing is performed to construct a bimodal image dataset, which includes infrared images and visible light images; The infrared image and the visible light image are subjected to bi-branch parallel analysis to extract a parallel feature set, which includes an infrared modal feature map and a visible light modal feature map. The infrared modal feature map and the visible light modal feature map are registered and stitched together according to the channel dimension to obtain joint features. Attention weights are then assigned to these joint features, and adaptive importance coefficients for the infrared modality and the visible light modality are calculated and generated. Element-wise weighting is then applied to the infrared modal feature map and the visible light modal feature map to obtain dual-modal fusion weighted features. The dual-modal fusion weighted features include both infrared modal weighted features and visible light modal weighted features. The infrared mode weighted features and the visible light mode weighted features are fused in multiple layers to generate fused features for hot spot detection and classification of photovoltaic modules, and hot spot detection results are generated.

2. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 1, characterized in that, The infrared image and the visible light image are subjected to dual-branch parallel analysis to extract a parallel feature set, which includes infrared modal feature maps and visible light modal feature maps. The method includes: An infrared branch feature extraction network is constructed, and the infrared image is subjected to detail enhancement convolution through the infrared branch feature extraction network to obtain an infrared modal feature map; A visible light branch feature extraction network is constructed, and the visible light image is analyzed by the visible light branch feature extraction network to obtain a visible light modal feature map; The infrared modal feature map and the visible light modal feature map are integrated in parallel to construct a parallel feature set.

3. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 2, characterized in that, The infrared image is subjected to detail enhancement convolution through the infrared branch feature extraction network to obtain an infrared modal feature map. The method includes: Multi-scale feature response extraction is performed on the infrared image to capture local temperature gradient change information; The local temperature gradient change information is stitched together according to the channel dimension to obtain a multi-scale thermal response feature map; Based on the multi-scale thermal response feature map, a hybrid convolution is performed to generate an infrared intermediate feature map; The infrared intermediate feature maps are aggregated, and information is exchanged between the multi-scale thermal response feature maps based on the aggregation results to generate infrared modal feature maps.

4. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 2, characterized in that, The visible light image is analyzed for surface structure using the visible light branch feature extraction network to obtain a visible light modal feature map. The method includes: Global content perception is performed based on the visible light image to extract global content data and local salient data. The global content data and the local salient data are concatenated and convolved according to the channel dimension to construct a content-guided attention weight map. The original features of the visible light image are extracted and multiplied with the content-guided attention weight map according to the data elements to generate content enhancement features; Based on the enhanced features described above, residual convolution depth analysis is performed to construct the visible light modal feature map.

5. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 1, characterized in that, The method involves obtaining joint features, assigning attention weights, calculating and generating adaptive importance coefficients for infrared and visible light modes, and then performing element-wise weighting on the infrared and visible light mode feature maps to obtain dual-modal fusion weighted features. The joint features are spatially compressed to obtain the channel description vector of the global response intensity; Weight mapping is performed based on the channel description vector to generate the original channel attention vector; The original channel attention weight vector is split according to the channel source. The channel weights belonging to the infrared modal feature map source are combined into infrared channel attention sub-weights, and the channel weights belonging to the visible light modal feature map source are combined into visible light channel attention sub-weights. The infrared channel attention sub-weights are aggregated and compressed along the channel dimension to obtain the infrared modal aggregated weights; The channel attention subweights of the visible light channels are aggregated and compressed along the channel dimension to obtain the visible light mode aggregated weights; The infrared mode aggregation weights and the visible light mode aggregation weights are subjected to normalization constraint processing to generate infrared mode adaptive importance coefficients and visible light mode adaptive importance coefficients; The infrared modal adaptive importance coefficient is multiplied element-wise with the infrared modal feature map to obtain the infrared modal weighted features; The visible light mode adaptive importance coefficient is multiplied element-wise with the visible light mode feature map to obtain the visible light mode weighted feature; The infrared modal weighted features and the visible light modal weighted features are combined and output to form the dual-modal fusion weighted features.

6. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 1, characterized in that, After generating the adaptive importance coefficients for the infrared mode and the visible light mode, the method includes: Both the infrared mode adaptive importance coefficient and the visible light mode adaptive importance coefficient are dynamically adjustable values, wherein the sum of the infrared mode adaptive importance coefficient and the visible light mode adaptive importance coefficient is 1; When the infrared modality adaptive importance coefficient is greater than 0.5, attention is biased towards the infrared modality feature map; When the adaptive importance coefficient of the visible light mode is greater than 0.5, attention is biased towards the visible light mode feature map.

7. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 1, characterized in that, The infrared mode-weighted features and the visible light mode-weighted features are fused in multiple layers to generate fused features. The method includes: The infrared mode weighted features and the visible light mode weighted features are shallowly fused to generate shallow fused features; The infrared mode weighted features and the visible light mode weighted features are fused at the middle layer to generate the middle layer fused features; The infrared mode weighted features and the visible light mode weighted features are deeply fused to generate deep fused features; The shallow fusion feature, the middle fusion feature, and the deep fusion feature are spliced ​​together according to the channel dimension to obtain cross-layer splicing features; Based on the cross-layer splicing features, association learning is performed, and cross-layer association responses are made to the shallow fusion features, the mid-layer fusion features, and the deep fusion features according to the association learning results to generate the fusion features.

8. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 1, characterized in that, The method for generating fusion features to classify hot spots in photovoltaic modules and generate hot spot detection results includes: Based on the fused features, global feature aggregation is performed, and the global feature aggregation result is used for multi-task classification to calculate the fault category probability distribution parameters. Probability detection is performed based on the fault category probability distribution parameters to obtain hot spot detection classification results; Based on the fusion features, spatial location analysis is performed, spatial location features are extracted and mapped, and hot spot region location regression parameters are determined. The hot spot detection classification result is correlated with the hot spot region location regression parameter to generate the hot spot detection result.

9. The photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in claim 8, characterized in that, The hot spot detection classification result is correlated with the hot spot region location regression parameter to generate the hot spot detection result. The method includes: When the hot spot detection classification result is a hot spot fault category, the bounding box parameters are extracted based on the hot spot region location regression parameters, and the bounding box parameters are mapped to the dual-modal image dataset to identify the hot spot fault category, thereby generating the hot spot detection result; When the hot spot detection classification result is a non-hot spot fault category, the target fault category is determined, and the target fault category is combined with the hot spot region location regression parameters and mapped to the dual-modal image dataset for fault category identification, thereby generating the hot spot detection result.

10. A photovoltaic hot spot detection system with deep fusion of cross-modal features, characterized in that, The system is used to implement the photovoltaic hot spot detection method based on cross-modal feature deep fusion as described in any one of claims 1 to 9, and the system comprises: A bidirectional sensing pairing module is used to perform bidirectional sensing pairing based on photovoltaic modules to construct a bimodal image dataset, which includes infrared images and visible light images. The image feature extraction module is used to perform dual-branch parallel analysis on the infrared image and the visible light image to extract a parallel feature set, which includes an infrared modal feature map and a visible light modal feature map. The fusion weighted feature acquisition module is used to register and stitch the infrared modal feature map and the visible light modal feature map according to the channel dimension to obtain joint features, perform attention weight allocation, calculate and generate infrared modal adaptive importance coefficients and visible light modal adaptive importance coefficients, and perform element-level weighting on the infrared modal feature map and the visible light modal feature map to obtain dual-modal fusion weighted features, wherein the dual-modal fusion weighted features include infrared modal weighted features and visible light modal weighted features; The hot spot detection result generation module is used to perform multi-layer fusion of the infrared mode weighted features and the visible light mode weighted features to generate fused features for hot spot detection and classification of photovoltaic modules, and generate hot spot detection results.

Images