A fire source positioning method and system fusing multi-modal sensing and photoelectric turntable

Abstract

The application provides a fire source positioning method and system fusing multi-modal sensing and photoelectric turntable, and relates to the field of photovoltaic power station image monitoring. First, infrared images and visible light images are acquired, and the information dislocation problem caused by the difference of acquisition equipment is eliminated through space-time alignment, and then the thermal feature map and the morphological feature map are extracted, and then the thermal feature map and the morphological feature map are fused and the candidate region is screened according to the temperature threshold, and the suspected fire is preliminarily locked. Then, the control processing unit screens the effective fire source area by calculating the minimum distance and the overlap degree of the high-temperature area contour information and the edge contour of the photovoltaic equipment, and excludes the false high temperature, and then uses the double loss term weighted regression to optimize the bounding box, and considers the matching degree of the thermal feature and the morphological feature, and improves the fire source positioning accuracy. Finally, the final fire source is determined combined with the confidence, and the photoelectric turntable is controlled to track. The method effectively alleviates the environmental interference, and significantly improves the accuracy of fire source positioning in complex outdoor scenes.

Description

Technical Field

[0001] This application relates to the field of image monitoring of photovoltaic power plants, and in particular to a method and system for fire source localization that integrates multimodal sensing and photoelectric turntable. Background Technology

[0002] With the rapid development and widespread application of photovoltaic power generation technology, the scale of photovoltaic power plants is constantly expanding. However, photovoltaic modules may experience problems such as hot spot effects and welding failures during long-term operation, which can easily lead to fire hazards, posing a significant challenge to the safe operation of photovoltaic power plants.

[0003] In related technologies, photovoltaic power plant fire monitoring systems employ image recognition technology for fire detection. These systems collect image information from photovoltaic modules by deploying surveillance cameras, and then analyze and process the images using computer vision algorithms to identify potential fires. When the system detects an anomaly, it quickly locates the relevant area and issues an early warning signal, providing crucial information for maintenance personnel to respond promptly to the fire.

[0004] However, in complex outdoor environments, relying solely on image data for fire identification and location may be affected by factors such as changes in lighting and weather conditions. For example, strong light can cause images to be overexposed, making it difficult to extract flame features; rainy weather can reduce image contrast, easily generating false high-temperature interference, affecting the identification of the fire source edge, and thus affecting the accuracy of determining the fire source location. Summary of the Invention

[0005] This application provides a fire source localization method and system that integrates multimodal sensing and photoelectric turntable, which is used to alleviate the technical problem of inaccurate fire source localization caused by environmental influences and improve the accuracy of fire source localization.

[0006] In a first aspect, this application provides a fire source localization method integrating multimodal sensing and a photoelectric turntable, applied to the control processing unit of a fire source localization system. The fire source localization system further includes a photoelectric turntable, which is connected to the control processing unit. The method includes: Infrared and visible light images of the photovoltaic (PV) device are acquired, and the pixel coordinates in the infrared image are mapped to the visible light image coordinate system using a transformation matrix, combined with timestamps, to obtain time- and space-aligned infrared and visible light images. Target thermal feature maps are extracted from the infrared images, and target morphological feature maps are extracted from the visible light images. The thermal feature map includes the contour information of the high-temperature region, and the morphological feature map includes the edge contour features of the PV device, smoke texture features, and the spatial shape features of the flame. The thermal and morphological feature maps are fused to generate a fused feature map, and regions in the fused feature map whose temperature exceeds a preset temperature threshold are identified as candidate fire source regions. For each candidate fire source region, the minimum distance and overlap between the high-temperature region contour information and the edge contour features of the PV device are calculated. Based on the minimum distance and overlap... The system calculates the regional matching degree between the photovoltaic equipment and the candidate fire source areas, and identifies candidate fire source areas with a regional matching degree higher than a preset matching degree threshold as valid fire source areas. A weighted regression algorithm incorporating infrared temperature loss and visible light morphology loss terms is used to calculate the bounding box positions of the valid fire source areas, obtaining the bounding box parameters. The infrared temperature loss term measures the difference in thermal features between the predicted bounding box and the contour information of the high-temperature area, while the visible light morphology loss term measures the difference in morphology between the predicted bounding box and the smoke texture features and the spatial shape features of the flame. Based on the bounding box parameters, the bounding box coordinates and confidence scores of the valid fire source areas are generated, and valid fire source areas with confidence scores reaching a preset confidence threshold are identified as the final fire source. Based on the position of the final fire source, the photoelectric turntable is controlled to track the final fire source.

[0007] By employing the above technical solution, the control processing unit first acquires infrared and visible light images and achieves spatiotemporal alignment, eliminating information misalignment caused by differences in the acquisition devices. Then, it extracts thermal feature maps containing the contours of high-temperature areas and morphological feature maps containing smoke textures, flame shapes, and device edges. Subsequently, the thermal and morphological feature maps are fused, and candidate regions are filtered according to temperature thresholds to initially identify suspected fires. Next, the control processing unit filters effective fire source areas by calculating distance and overlap, eliminating false high temperatures unrelated to the device. Then, it uses double-loss term weighted regression to optimize the bounding box, taking into account the matching degree of thermal and morphological features, thus improving the accuracy of fire source localization. Finally, the control processing unit combines confidence levels to determine the final fire source and controls the photoelectric turntable to track it. This method effectively mitigates environmental interference and significantly improves the accuracy of fire source localization in complex outdoor scenarios.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, a target thermal feature map is extracted from an infrared image, and a target morphological feature map is extracted from a visible light image. Specifically, this includes: extracting a target thermal feature map from an infrared image and a target morphological feature map from a visible light image; the target thermal feature map includes contour information of a high-temperature region, and the target morphological feature map includes edge contour features of a photovoltaic device, smoke texture features, and spatial shape features of a flame; concatenating the target thermal feature map and the target morphological feature map by channel, and applying multiple sets of one-dimensional convolutions of different sizes in parallel to the concatenated feature image to obtain correlation information representing local interactions between channels; calculating infrared modulation coefficients and visible light modulation coefficients based on the correlation information; applying the infrared modulation coefficients to the target thermal feature map and the visible light modulation coefficients to the target morphological feature map to generate a feature-enhanced target thermal feature map and a target morphological feature map.

[0009] By employing the above technical solution, the control processing unit first extracts basic thermal and morphological feature maps, preserving the core features of the fire source. Then, it concatenates the two feature map channels and applies multi-size one-dimensional convolutions in parallel to capture local interactive correlation information between different channels. Next, the control processing unit applies infrared modulation coefficients to the target thermal feature map and visible light modulation coefficients to the target morphological feature map. This specifically enhances key features related to fire source identification (such as the correlation between high-temperature contours and smoke textures) while suppressing irrelevant interference features. The resulting enhanced feature map information is more focused, improving the fire identification accuracy.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the target thermal feature map and the target morphological feature map are fused to generate a fused feature map. Specifically, this includes: using the target thermal feature map as the semantic feature of the feature pyramid network, and using the target morphological feature map as the spatial feature of the feature pyramid network. The semantic feature is used for the overall judgment of the fire source area, and the spatial feature is used for the location of the fire source details. The channels of the semantic feature are compressed through convolution operations, and multi-size convolutions are applied in parallel to the compressed semantic features to extract multi-scale semantic features. The multi-scale semantic features are concatenated along the channels to generate enhanced semantic features. The enhanced semantic features are upsampled through sub-pixel convolutions, and based on the feature pyramid network, the upsampled enhanced semantic features and spatial features are fused layer by layer from top to bottom to obtain the fused feature map.

[0011] By adopting the above technical solution, the control processing unit uses thermal feature maps as semantic features and morphological feature maps as spatial features. First, it compresses the semantic feature channels and extracts multi-scale semantic features, reducing computational load while capturing global fire information across different ranges. Then, it upsamples the enhanced semantic features through sub-pixel convolution to avoid blurring details. Next, the control processing unit combines the feature pyramid network to fuse semantic and spatial features layer by layer from top to bottom. This results in a fused feature map that possesses both accurate global fire situation judgment criteria and clear fire source location details, significantly improving the fused feature map's ability to represent fire source areas and providing more accurate feature support for candidate fire source area selection.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, for each candidate fire source region, the minimum distance and overlap between the high-temperature region contour information and the photovoltaic device edge contour feature are calculated. Specifically, this includes: for each candidate fire source region, extracting the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour feature; calculating the shortest Euclidean distance between the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour feature, and using the shortest Euclidean distance as the minimum distance between the high-temperature region contour information and the photovoltaic device edge contour feature; calculating the area of ​​the first region enclosed by the high-temperature region contour information and the area of ​​the second region enclosed by the photovoltaic device edge contour feature, and calculating the overlap between the high-temperature region contour information and the photovoltaic device edge contour feature based on the intersection area and the union area of ​​the first region area and the second region area.

[0013] By employing the above technical solution, the control processing unit first extracts the pixel sets of the high-temperature region and the edge contour of the equipment, calculates the shortest Euclidean distance between the pixel sets to determine whether the fire source is close to the equipment, and then calculates the degree of overlap between the high-temperature region and the edge contour of the equipment. The spatial correlation range between the fire source and the equipment is quantified by the ratio of the intersection to the union area. This method effectively eliminates false high-temperature regions that are not spatially related to the equipment, improving the accuracy and reliability of effective fire source area screening.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, a weighted regression algorithm that integrates infrared temperature loss and visible light morphology loss is used to calculate the bounding box position of the effective fire source region, thereby obtaining the bounding box parameters of the effective fire source region. Specifically, this includes: calculating the infrared temperature loss term based on the difference in thermal features between the predicted bounding box of the effective fire source region and the corresponding high-temperature region contour information; calculating the visible light morphology loss term based on the difference in morphology between the predicted bounding box of the effective fire source region and the corresponding smoke texture features and the corresponding flame spatial shape features; and using a weighted regression algorithm based on the infrared temperature loss term and the visible light morphology loss term to calculate the predicted bounding box of the effective fire source region, thereby obtaining the optimized bounding box parameters of the effective fire source region. The bounding box parameters include the offset of the center point position of the bounding box corresponding to the effective fire source region, as well as the width scaling factor and height scaling factor used to determine the size of the bounding box corresponding to the effective fire source region.

[0015] By adopting the above technical solution, the control processing unit first calculates the infrared temperature loss term based on the thermal feature difference between the predicted bounding box and the high-temperature region contour, ensuring that the bounding box can accurately match the temperature distribution range of the fire source. Then, it calculates the visible light morphology loss term based on the morphological differences between the predicted bounding box and the smoke texture and flame shape, ensuring that the bounding box fits the appearance of the fire source. Next, the control processing unit uses a weighted regression algorithm to fuse the two loss terms, dynamically adjusting the weights according to the actual environment (e.g., increasing the weight of the temperature loss term when there is poor illumination). By minimizing the comprehensive loss, the bounding box parameters are continuously optimized, resulting in more accurate center point offset and size scaling factor of the final bounding box, significantly improving the spatial accuracy of fire source positioning.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the bounding box coordinates and confidence levels of the effective fire source region are generated based on the bounding box parameters, and the effective fire source region whose confidence level reaches a preset confidence threshold is determined as the final fire source. Specifically, this includes: calculating the bounding box coordinates and confidence levels of the effective fire source region in the visible light image based on the bounding box parameters; calculating the three-dimensional geographic coordinates of the effective fire source region through coordinate system transformation based on the bounding box coordinates, the camera parameters of the photoelectric turntable, the angle parameters of the photoelectric turntable, and the geographic coordinates of the photoelectric turntable; determining the effective fire source region whose confidence level reaches a preset confidence threshold as the final fire source, and associating the final fire source with the corresponding three-dimensional geographic coordinates.

[0017] By adopting the above technical solution, the control processing unit first calculates the coordinates and confidence level in the image coordinate system based on the bounding box parameters. Then, combined with the camera parameters, angle parameters, and geographic coordinates of the photoelectric turntable, the image coordinates are transformed into three-dimensional geographic coordinates in the real physical space through coordinate system transformation. This realizes the mapping of the fire source from image positioning to the actual geographic location. Finally, the area with a confidence level reaching the threshold is determined as the final fire source and associated with the three-dimensional coordinates. This not only ensures the reliability of the final fire source but also accurately determines its specific location in the real scene, avoiding the limitations of positioning at the image level and improving the practicality and accuracy of the positioning results.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after determining the effective fire source area with a confidence level reaching a preset confidence threshold as the final fire source and associating the final fire source with the corresponding three-dimensional geographic coordinates, the method further includes: acquiring paired images captured by a binocular camera, and calculating the verification three-dimensional coordinates of the final fire source based on the paired images using binocular vision, wherein the paired images include a left view and a right view located on the same horizontal line; calculating the coordinate deviation value between the verification three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates, wherein the coordinate deviation value is the spatial straight-line distance between the verification three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates; when the coordinate deviation value exceeds the deviation range threshold, recalculating the three-dimensional geographic coordinates of the final fire source based on the paired images and the fused feature map.

[0019] By adopting the above technical solution, the control processing unit first acquires paired images from the binocular cameras, and then calculates and verifies the three-dimensional coordinates based on binocular vision. Next, the control processing unit calculates the spatial straight-line distance (deviation value) between the verified coordinates and the original three-dimensional coordinates. This allows for a direct quantification of the error in the original positioning result. When the deviation exceeds a threshold, the three-dimensional coordinates are recalculated by combining the stereo information of the paired images with the precise fire characteristics of the fused feature map, correcting the deviation in the original positioning and ensuring that the final obtained three-dimensional geographic coordinates of the fire source are more accurate, thus improving the accuracy, stability, and reliability of the entire positioning scheme.

[0020] In a second aspect, this application provides a fire source locating system, including a control processing unit and a photoelectric turntable, wherein the control processing unit is connected to the photoelectric turntable; the control processing unit includes one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors call the computer instructions to cause the control processing unit to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium including instructions that, when executed on a control processing unit, cause the control processing unit to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product that, when run on a control processing unit, causes the control processing unit to perform the method described in the first aspect and any possible implementation thereof.

[0023] Compared with the prior art, this application has at least the following beneficial effects: The fire source localization method and system integrating multimodal sensing and photoelectric turntable provided in this application eliminates the spatiotemporal bias in dual-sensor information acquisition at its source by acquiring infrared and visible light images and performing spatiotemporal alignment, providing a reliable data foundation for subsequent accurate fusion. By extracting thermal feature maps containing the contours of high-temperature regions and morphological feature maps containing device edges, smoke textures, and flame shapes, a comprehensive perception of the essential attributes (temperature) and external manifestations (morphology) of the fire source is achieved, significantly enhancing the ability to identify fire sources in complex environments.

[0024] In the initial screening stage, this application integrates thermal and morphological features and determines candidate regions based on temperature thresholds. This leverages the sensitivity of infrared to temperature while incorporating detailed information from visible light. More importantly, by calculating the minimum distance and overlap between the high-temperature contour of the candidate region and the edge contour of the photovoltaic device, and utilizing the physical principle that a fire source must be attached to a combustible material (photovoltaic device), it effectively eliminates false high-temperature interference unrelated to the device, such as solar reflection and other high-temperature objects, significantly improving the accuracy of the early warning. In the final positioning optimization stage, a weighted regression algorithm that integrates infrared temperature loss terms and visible light morphological loss terms is introduced. This ensures that the optimization of the bounding box is simultaneously constrained by both temperature distribution and visual morphology, dynamically balancing the contributions of the two modes based on ambient lighting, weather, and other conditions (e.g., relying more on infrared information in poor lighting conditions and more on thermal radiation information when obscured by smoke). This allows for high-precision fire source location in various complex outdoor scenarios. In summary, this application, through deep fusion and intelligent screening of multimodal information, significantly mitigates environmental interference and greatly improves the accuracy, reliability, and robustness of fire source positioning in complex outdoor scenarios. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of a fire source location system in an embodiment of this application; Figure 2 This is a flowchart illustrating a fire source localization method integrating multimodal sensing and photoelectric turntable in an embodiment of this application. Figure 3This is another flowchart illustrating the fire source localization method integrating multimodal sensing and photoelectric turntable in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the control processing unit in an embodiment of this application. Detailed Implementation

[0026] This application provides a fire source location system, including a control processing unit, a photoelectric turntable, and a binocular camera, wherein the photoelectric turntable and the binocular camera are respectively connected to the control processing unit. For ease of understanding, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the fire source location system in an embodiment of this application. The following is a detailed description of the hardware devices in the system: 1. Control Processing Unit: The control processing unit is the core of the fire source location system. It consists of a high-performance processor, storage module, etc., and has data processing, logical judgment, and command output functions. The control processing unit is responsible for receiving image data from the photoelectric turntable and binocular camera, performing spatiotemporal alignment, feature extraction and fusion, fire source screening, bounding box optimization, and 3D coordinate calculation, while generating commands to drive the photoelectric turntable to track the fire source.

[0027] 2. Photovoltaic Turntable: The photovoltaic turntable needs to be installed in a location within the photovoltaic power station with a wide field of view and no obstructions, preferably at the highest point in the center of the photovoltaic array area. The photovoltaic turntable achieves full-area image acquisition without blind spots through 360° horizontal rotation and ±90° vertical tilt, providing multimodal raw data to the control and processing unit; after receiving control commands, it adjusts the angle to keep the fire source always in the center of the camera's field of view, achieving dynamic tracking. Optional models: HM-TD71C8-55 / W, HM-TD71C8-75 / W.

[0028] 3. Binocular Camera: The binocular camera needs to be installed in conjunction with the photoelectric turntable using an integrated installation method. The binocular camera is fixed to the camera mounting bracket of the photoelectric turntable, aligned with the infrared camera and visible light camera, and located on the same horizontal plane. The binocular camera is mainly used to simultaneously acquire two test views (left and right views). The control processing unit calculates the three-dimensional coordinates of the fire source verification based on these views and compares them with the monocular positioning coordinates of the photoelectric turntable. If the deviation exceeds a threshold, the coordinates are corrected to improve the fire source positioning accuracy and compensate for monocular positioning errors. Optional models: DS2CD3R446WD-LT, DS-2CD3T447WD-LPT.

[0029] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a fire source localization method integrating multimodal sensing and a photoelectric turntable in an embodiment of this application.

[0030] 101. Acquire infrared and visible light images of the photovoltaic equipment, and combine them with timestamps to map the pixel coordinates in the infrared image to the visible light image coordinate system through a transformation matrix to obtain time- and space-aligned infrared and visible light images.

[0031] Infrared images are thermal imaging images that reflect the surface temperature distribution of photovoltaic equipment; visible light images are real-world images of photovoltaic equipment taken under natural light conditions; a transformation matrix is ​​a mathematical matrix used to realize the mapping relationship between different coordinate systems; pixel coordinates represent the position information of each pixel in the image; temporal and spatial alignment means synchronizing infrared and visible light images in time and space dimensions.

[0032] Specifically, during the normal operation of the photovoltaic equipment, the control and processing unit can acquire infrared and visible light images of the photovoltaic equipment using infrared and visible light cameras, respectively. Due to differences in the installation location, imaging characteristics, and resolution of the two cameras, image registration is required. The control and processing unit first compares the timestamps of the two images to ensure that their acquisition times are close. Then, using a pre-calibrated transformation matrix, it maps the pixels in the infrared image to the visible light image coordinate system, achieving spatial alignment of the two images and ultimately obtaining an image pair that is aligned both temporally and spatially.

[0033] 102. Extract target thermal feature map from infrared image and target morphological feature map from visible light image. Target thermal feature map includes high temperature area contour information, and target morphological feature map includes photovoltaic device edge contour features, smoke texture features, and flame spatial shape features.

[0034] The target thermal feature map represents a feature map reflecting the temperature distribution characteristics; the high temperature region contour information refers to the boundary and shape characteristics of the temperature anomaly region; the target morphology feature map represents a feature map reflecting the appearance morphology; the photovoltaic equipment edge contour features refer to the physical boundary characteristics of the photovoltaic module; the smoke texture features refer to the texture and grayscale variation characteristics of the smoke in the image; the spatial shape features of the flame refer to the geometric shape and dynamic change characteristics of the flame in the image.

[0035] Specifically, the control processing unit performs feature extraction on the spatiotemporally aligned infrared and visible light images respectively. For the infrared image, the control processing unit first performs temperature threshold segmentation to identify areas of temperature anomalies, and then extracts the contour information of these areas to generate a target thermal feature map. For the visible light image, the control processing unit uses a multi-scale feature extraction algorithm to extract the edge contour of the photovoltaic device, the texture features of the smoke, and the shape features of the flame, and integrates these features into a target morphology feature map. During the feature extraction process, the influence of different weather conditions and lighting environments on feature extraction needs to be considered, and an adaptive feature extraction method is adopted.

[0036] In some embodiments, feature extraction can be achieved in multiple ways: Optionally, the control processing unit can use traditional image processing methods for feature extraction. For infrared images, the Otsu adaptive thresholding method is used for temperature segmentation, and then an edge detection operator is used to extract the contour of the high-temperature region; for visible light images, Canny edge detection, gray-level co-occurrence matrix, and optical flow methods are combined to extract edge contours, smoke textures, and flame shape features, respectively.

[0037] It is understandable that other feature extraction methods can also be used to generate the target feature map, and this is not limited here. In practical applications, it may be necessary to adjust the parameters and methods of feature extraction according to the specific scenario. In addition, to improve the robustness of feature extraction, a combination of multiple feature extraction methods can be used, and temporal information can be introduced to constrain the spatiotemporal consistency of features.

[0038] 103. The target thermal feature map and the target morphological feature map are fused to generate a fused feature map, and the areas in the fused feature map whose temperature exceeds the preset temperature threshold are identified as candidate fire source areas.

[0039] The fused feature map represents the feature map obtained by combining the target thermal feature map and the target morphological feature map; the preset temperature threshold refers to the standard temperature value used to determine whether a region is abnormally high temperature; the candidate fire source region represents the suspected area where a fire may occur; feature fusion refers to the process of combining feature information from different sources.

[0040] Specifically, the control processing unit first performs feature alignment and normalization on the target thermal feature map and the target morphological feature map to ensure consistency in scale and dimension. Then, the control processing unit uses a feature fusion algorithm to fuse the two features, generating a fused feature map containing both temperature and morphological information. Based on the photovoltaic equipment's safety operation specifications and historical data, the control processing unit sets a reasonable temperature threshold and identifies regions exceeding the preset temperature threshold in the fused feature map; these regions are marked as candidate fire source areas. The control processing unit records the location, area, and other information of the candidate fire source areas, providing a basis for subsequent precise positioning. During the feature fusion process, the importance weights of different features need to be considered and dynamically adjusted according to the actual application scenario.

[0041] 104. For each candidate fire source region, calculate the minimum distance and degree of overlap between the high-temperature region contour information and the edge contour features of the photovoltaic equipment.

[0042] Minimum distance represents the shortest Euclidean distance from the candidate fire source region outline to the photovoltaic device edge outline; overlap refers to the ratio of the intersection area to the union area of ​​the candidate fire source region and the photovoltaic device edge region; high temperature region outline information represents the boundary features of the temperature anomaly region extracted from the infrared image; photovoltaic device edge outline features represent the physical boundary information of the photovoltaic module in the visible light image.

[0043] Specifically, the control processing unit performs positional relationship analysis on each candidate fire source region. First, it registers the high-temperature profile of the candidate fire source region with the edge profile of the photovoltaic device to ensure they are in the same coordinate system. Then, the control processing unit calculates the minimum Euclidean distance between the high-temperature profile of the candidate fire source region and the edge profile of the photovoltaic device. This minimum Euclidean distance reflects the spatial proximity of the fire source and the device. Simultaneously, the control processing unit calculates the overlap between the candidate fire source region and the edge region of the photovoltaic device to assess whether the fire source actually occurred on the photovoltaic device. The control processing unit needs to consider errors under different viewing angles and imaging conditions, employing a robust distance calculation method. During the calculation process, it also needs to address practical issues such as incomplete profiles and noise interference.

[0044] In some embodiments, distance and overlap calculations can be implemented in various ways: Optionally, the control processing unit can employ a calculation method based on contour point sets. First, the contour is sampled to obtain a discrete point set, and then a point set distance calculation algorithm is used. During the calculation process, an adaptive sampling strategy can be employed to increase the sampling density in key areas. Optionally, the control processing unit can use a distance transformation-based method. A distance field is generated for the edge contour of the photovoltaic device, and then the distance values ​​of the contour points in the high-temperature region are calculated within the distance field.

[0045] It is understandable that other distance calculation methods can be used to measure the spatial relationship between contours, and this is not limited here. In practical applications, it may be necessary to consider the effects of image resolution and distortion and correct the calculation results. In addition, temporal information can be introduced, and the stability of the calculation can be improved through statistical analysis of multi-frame results. To accommodate photovoltaic equipment of different specifications, the distance calculation can use a relative scale, that is, normalize the actual distance by dividing it by the feature size of the equipment.

[0046] 105. Calculate the regional matching degree between photovoltaic equipment and candidate fire source areas based on minimum distance and degree of overlap, and determine the candidate fire source areas with regional matching degree higher than the preset matching degree threshold as valid fire source areas.

[0047] Regional matching degree represents the quantitative value of the similarity between the candidate fire source area and the spatial location of the photovoltaic equipment; the preset matching degree threshold is the standard value used to determine whether a candidate area is a valid fire source; the valid fire source area refers to the area that is confirmed to be a possible fire area after the matching degree verification.

[0048] Specifically, the control processing unit first normalizes the obtained minimum distance and overlap degree, unifying their value range to the [0, 1] interval. Then, based on these two indicators, the control processing unit designs a matching degree calculation formula, which needs to consider the relative importance of the two indicators. The control processing unit sets a reasonable matching degree threshold based on historical data, compares the calculated matching degree with the threshold, and selects candidate areas with higher matching degrees as effective fire source areas. During the matching degree calculation process, the characteristics of different types of photovoltaic equipment need to be considered, and an adaptive weight allocation strategy is adopted. For the selected effective fire source areas, their location, area, matching degree, and other information need to be recorded to provide a basis for subsequent positioning.

[0049] In some embodiments, matching degree calculation and effective region determination can be implemented in multiple ways: Optionally, the control processing unit can adopt a weighted fusion matching degree calculation method: by setting distance weights and overlap weights, a weighted average value is calculated as the final matching degree. Optionally, the control processing unit can use a fuzzy logic-based matching degree calculation method: constructing a fuzzy rule set, using minimum distance and overlap as input variables, and obtaining the matching degree through fuzzy inference.

[0050] It is understandable that other matching degree calculation methods can also be used to determine the effective fire source area, and this is not limited here. In practical applications, time-series information can be introduced to improve the reliability of the judgment through statistical analysis of multi-frame results. In addition, the matching degree threshold can be set using an adaptive strategy, dynamically adjusted according to environmental conditions and system status. To improve system robustness, multi-level matching degree thresholds can be set to achieve hierarchical screening. At the same time, different matching degree requirements can be set for special areas to improve the adaptability of the system.

[0051] Furthermore, in outdoor scenarios such as photovoltaic power plants, there are numerous high-temperature targets that are not fire sources, such as strong reflections of sunlight on the surface of photovoltaic panels, hot vehicles in the distance, and heat-generating components of auxiliary equipment in the power plant. These interfering objects also appear as high-temperature areas in infrared images, and temperature thresholds alone cannot distinguish them from real fire sources. By calculating the minimum distance between the outline of the high-temperature area and the edge outline of the photovoltaic equipment, it is possible to quantitatively determine whether the high-temperature area occurs on or near the equipment itself. If the minimum distance is too large, it indicates that the high-temperature area is far from the photovoltaic equipment and is likely an interfering object in the background, thus being effectively eliminated. By calculating the degree of overlap (cross-over-union ratio), the degree of overlap between the high-temperature area and the equipment itself can be further quantified. Real fire sources are usually attached to the surface of photovoltaic panels, and their high-temperature outlines should have a large area of ​​overlap with the equipment outline. While sunlight reflections appear on the equipment surface, their outline shape is not strongly correlated with the edge of the equipment, and the degree of overlap may be low; background high temperatures do not overlap at all. This indicator can effectively solve the problem of accurately locating fire sources that are "on the equipment" but are not the equipment itself (such as floating flames above the equipment). This method utilizes the fundamental physical principle that "an ignition source must be attached to a combustible material (photovoltaic equipment)" as prior knowledge. By transforming this strong physical constraint into geometric calculations in image space (minimum distance, overlap), the decision-making logic for ignition source selection becomes clearer and more objective, independent of complex and environmentally sensitive feature learning. Even when slight noise is introduced into the extraction of equipment edge contours in visible light images due to illumination or smoke, the method maintains high selection accuracy through a comprehensive judgment of distance and overlap, significantly improving its robustness. Transforming fuzzy spatial relationships into specific numerical values ​​(minimum distance and overlap) facilitates the setting of uniform thresholds for selection and provides clear quantitative indicators for subsequent calculations of "regional matching degree." This makes the entire selection process objective and reproducible, and allows for flexible adjustment of thresholds based on the layout of different power plants or seasonal changes, demonstrating strong practicality and adaptability.

[0052] 106. Using a weighted regression algorithm that integrates infrared temperature loss terms and visible light morphology loss terms, the bounding box position of the effective fire source region is calculated, and the bounding box parameters of the effective fire source region are obtained.

[0053] The infrared temperature loss term represents a quantitative indicator used to measure the difference in thermal features between the predicted bounding box and the contour information of the high-temperature region; the visible light morphology loss term refers to a quantitative indicator used to measure the difference in morphology between the predicted bounding box and the spatial shape features of smoke texture and flame; the weighted regression algorithm is an algorithm that constructs a comprehensive loss function by assigning different weights to the infrared temperature loss term and the visible light morphology loss term, and solves the optimal bounding box by minimizing this function; the bounding box parameters represent parameters describing the position and size of the bounding box of the effective fire source region, which usually include the pixel coordinates of the upper left and lower right corners of the bounding box.

[0054] Specifically, after determining the effective fire source area, the control processing unit needs to further refine the calculation of its bounding box position. First, the control processing unit determines an initial bounding box, which is typically based on the approximate range of the effective fire source area. Then, the control processing unit constructs a comprehensive loss function that integrates infrared temperature loss and visible light morphology loss terms. The infrared temperature loss term is obtained by calculating the temperature feature difference between the temperature distribution within the predicted bounding box and the temperature feature of the high-temperature region contour information. The visible light morphology loss term is obtained by calculating the morphological difference between the predicted bounding box and the smoke texture features and flame spatial shape features. The control processing unit assigns appropriate weights to the infrared temperature loss term and the visible light morphology loss term. The weights can be adjusted according to the actual environmental conditions; for example, the weight of the infrared temperature loss term can be increased when lighting conditions are poor. Next, the control processing unit uses a weighted regression algorithm to optimize the comprehensive loss function. By continuously adjusting the bounding box parameters to minimize the comprehensive loss function value, the final bounding box parameters are the precise bounding box parameters of the effective fire source area. During the calculation process, the control processing unit needs to consider the impact of factors such as image resolution and feature extraction accuracy on the calculation of the loss terms to ensure the accuracy of the bounding box parameters.

[0055] In some embodiments, the calculation of the bounding box parameters of the effective fire source region can be achieved in several ways: Optionally, the control processing unit first determines the initial bounding box based on the contour of the effective fire source region, and then calculates the infrared temperature loss term and the visible light morphology loss term respectively. The infrared temperature loss term uses the mean square error between the pixel temperature within the predicted bounding box and the average temperature of the high-temperature region, while the visible light morphology loss term uses the intersection-union ratio (IUU) loss between the predicted bounding box and the contours of smoke and flame. Weights are then assigned to the two losses to construct a comprehensive loss function. Finally, the gradient descent algorithm is used to minimize the comprehensive loss function to obtain the bounding box parameters. It is understood that other weighted regression algorithms can also be used to calculate the bounding box parameters, which is not limited here. In practical applications, to improve computational efficiency, the adjustment range of the bounding box can be limited, and the weight values ​​can be calibrated periodically to adapt to different environmental changes.

[0056] 107. Based on the bounding box parameters, generate the bounding box coordinates and confidence levels of the effective fire source areas, and determine the effective fire source areas whose confidence levels reach the preset confidence threshold as the final fire sources.

[0057] Bounding box coordinates refer to the coordinate information in the image that identifies the specific location of the effective fire source area, determined by the bounding box parameters, and are usually expressed in pixels; the final fire source refers to the fire source area that has been determined to be real after confidence screening.

[0058] Specifically, after obtaining the bounding box parameters of the effective fire source region, the control processing unit generates the corresponding bounding box coordinates based on these parameters. For example, it directly determines the coordinates in the image based on the pixel coordinates of the top-left and bottom-right corners of the bounding box. Subsequently, the control processing unit calculates the confidence level of the effective fire source region. This calculation considers multiple factors, including the degree of matching between the bounding box and the contours of the high-temperature area, smoke and flame morphology features, as well as the magnitude of the infrared temperature and visible light morphology loss terms. The higher the matching degree and the smaller the loss term, the higher the confidence level. The control processing unit compares the calculated confidence level with a preset confidence threshold. If the confidence level reaches or exceeds the threshold, the effective fire source region is identified as the final fire source; otherwise, the region is excluded and not considered the final fire source. Throughout this process, the control processing unit must ensure the objectivity and accuracy of the confidence level calculation, and the preset confidence threshold must be set in conjunction with the actual conditions of the photovoltaic power station to balance the risks of missed detections and false detections.

[0059] 108. Based on the location of the final fire source, control the photoelectric turntable to track the final fire source.

[0060] The final location of the fire source refers to the location information of the area identified as the real fire source in the image coordinate system or actual physical space; the photoelectric turntable is a device that can rotate horizontally and vertically through motor drive, thereby driving the camera and other equipment installed on it to change the observation direction; tracking refers to the process by which the photoelectric turntable adjusts its own rotation angle so that the imaging equipment installed on it is always aligned with the target object and maintains clear observation.

[0061] Specifically, after determining the final fire source, the control processing unit needs to control the photoelectric turntable to track it. First, the control processing unit converts the coordinates of the final fire source in the image into control parameters for the photoelectric turntable. This conversion needs to consider the installation relationship between the photoelectric turntable and the imaging equipment, the focal length of the imaging equipment, and other parameters to determine the horizontal and vertical angles the photoelectric turntable needs to rotate. Then, the control processing unit sends control commands to the photoelectric turntable, which include the calculated horizontal and vertical rotation angles. Upon receiving the commands, the photoelectric turntable drives its internal motors to rotate according to the commands, adjusting the pointing of the infrared and visible light cameras mounted on it to ensure the final fire source remains centered in the camera's field of view. During the tracking process, the control processing unit continuously acquires images captured by the cameras, updates the position of the final fire source in real time, and continuously adjusts the control commands sent to the photoelectric turntable based on position changes to achieve continuous and stable tracking of the final fire source.

[0062] The fire source localization method integrating multimodal sensing and a photoelectric turntable as described in this application involves a control processing unit first acquiring infrared and visible light images of the photovoltaic equipment. This is combined with timestamps and a transformation matrix to achieve spatiotemporal alignment, resolving information misalignment caused by equipment differences. Next, the control processing unit extracts thermal feature maps from the infrared images and morphological feature maps from the visible light images, comprehensively capturing key fire source attributes and avoiding information omissions due to single features. The thermal and morphological feature maps are then fused to generate a fused feature map, which is used to filter candidate fire source areas according to a preset temperature threshold, initially identifying suspected fires and reducing interference from non-fire source high temperatures. Then, the control processing unit calculates the distance and overlap between the candidate areas and the equipment edges to obtain the area matching degree, filtering out effective fire source areas and eliminating false high temperatures. A weighted regression algorithm with double loss terms is then used to optimize the bounding box parameters, taking into account both thermal and morphological feature matching degrees to improve positioning accuracy. Finally, the control processing unit combines confidence levels to determine the final fire source and associates it with three-dimensional coordinates, controlling the photoelectric turntable to track it, mitigating environmental interference and improving fire source localization accuracy.

[0063] Specifically, in complex outdoor environments, the reliability of infrared and visible light images is dynamically changing. For example, under strong light, visible light images are rich in detail but flame features may be obscured; at night or when obscured by smoke, infrared images have clear temperature features but blurred morphology. This method does not pre-fix the weights of the two modalities, but instead uses a weighted regression algorithm to allow the weights of the infrared temperature loss term and the visible light morphology loss term to be dynamically adjusted or optimized according to the current environment. This enables the localization process to "adaptively" select the more reliable modality in the current environment as the primary basis (e.g., increasing the weight of the infrared loss term when the illumination is poor, and increasing the weight of the visible light loss term when the temperature distribution is blurred), ensuring optimal localization results in any environment. Traditional bounding box regression usually only focuses on the geometric differences from the "ground truth box" (such as center point, width and height errors). This application introduces two heterogeneous loss terms to constrain the predicted box from different dimensions. The infrared temperature loss term ensures that the predicted bounding box closely fits the true temperature distribution range of the fire source, avoiding truncating the edges of high-temperature areas or including low-temperature backgrounds within the box. The visible light morphology loss term ensures a high degree of match between the predicted bounding box and the visual appearance of the fire source under visible light (such as the diffuse shape of smoke and the flickering shape of flames), guaranteeing that the bounding box geometrically accurately frames the fire source entity. The weighted fusion of the two is equivalent to subjecting the predicted box to simultaneous testing by a "thermometer" and a "camera," resulting in a bounding box that is both "temperature-controlled" and "morphologically accurate," achieving a dual precise match between physical properties and visual morphology. Its positioning accuracy far exceeds that of methods relying solely on a single modality.

[0064] Furthermore, the optimized bounding box parameters (center point offset, width and height scaling factors) directly determine the final position of the fire source in the image. Through this refined regression optimization, the resulting bounding box can more accurately reflect the true boundary and center of the fire source. A more accurate two-dimensional image coordinate is a prerequisite for subsequently obtaining high-precision three-dimensional geographic coordinates through coordinate system transformation. For photoelectric turntable tracking, more accurate two-dimensional coordinates mean smaller rotation angle errors, enabling more stable and smoother tracking results, providing crucial positional information for subsequent firefighting or emergency response.

[0065] Based on the above, the following is a more detailed description of the process provided in this implementation. Please refer to [link / reference]. Figure 3 This is another flowchart illustrating the fire source localization method integrating multimodal sensing and photoelectric turntable in this application embodiment.

[0066] 201. Acquire the infrared and visible light images of the photovoltaic equipment, and combine them with the timestamp to map the pixel coordinates in the infrared image to the visible light image coordinate system using a transformation matrix, thus obtaining time- and space-aligned infrared and visible light images. (This step has been explained in 101 and will not be repeated here.) 202. Extract target thermal feature map from infrared image and target morphological feature map from visible light image. Target thermal feature map includes high temperature area contour information, and target morphological feature map includes photovoltaic device edge contour features, smoke texture features, and flame spatial shape features.

[0067] Step 202 includes steps 2021 to 2024, which are not shown in the figure.

[0068] 2021. Extract target thermal feature maps from infrared images and target morphological feature maps from visible light images. Target thermal feature maps include high-temperature region contour information, and target morphological feature maps include photovoltaic device edge contour features, smoke texture features, and flame spatial shape features.

[0069] Specifically, after completing the temporal and spatial alignment of the infrared and visible light images, the control processing unit immediately executes this step. The aim is to extract core features related to fire source detection from both images, laying the foundation for subsequent feature fusion and fire source identification. For the infrared image, the control processing unit first performs preprocessing, such as removing image noise through Gaussian filtering, and then uses an adaptive threshold segmentation algorithm (such as the Otsu algorithm) to segment areas with temperatures higher than the normal operating temperature of the photovoltaic equipment based on the temperature grayscale differences in different regions of the infrared image. Subsequently, the Canny edge detection algorithm is used to extract the boundary contours of the segmented regions, integrating them to form a target thermal feature map containing the contour information of the high-temperature regions. For visible light images, the control processing unit also performs preprocessing first, enhancing image contrast through histogram equalization (to address uneven brightness caused by changes in illumination), and then extracting morphological features in three categories: For the edge contour features of photovoltaic equipment, the Sobel edge detection algorithm is used, combined with the preset size of the photovoltaic equipment (e.g., the standard photovoltaic panel size 1640mm×992mm) and shape features, to filter out edge interference from non-photovoltaic equipment (e.g., weeds, gravel contours), and extract the edge contour of the photovoltaic equipment; For the smoke texture features, the gray-level co-occurrence matrix algorithm is used to calculate the texture contrast, correlation and other parameters of different regions in the visible light image, and match them with the preset smoke texture template (obtained through training with a large number of smoke samples) to select regions that conform to the smoke texture features; For the spatial shape features of flames, a contour tracking algorithm (e.g., eight-neighbor contour tracking) is used to extract the contours of regions in the image with dynamic irregular shapes and colors that conform to flame features (e.g., red-orange gradient), and finally, the three types of morphological features are integrated to form the target morphological feature map.

[0070] 2022. The target thermal feature map and the target morphological feature map are concatenated by channels, and multiple sets of one-dimensional convolutions of different sizes are applied in parallel to the concatenated feature image to obtain the association information representing the local interaction between channels.

[0071] Channel stitching refers to the operation by which the control processing unit merges the target thermal feature map and the target morphological feature map in the feature channel dimension, that is, superimposing the channels of the two feature maps to form a single multi-channel feature map; one-dimensional convolution refers to the operation by which the control processing unit uses a convolution kernel that slides only in the channel dimension to perform convolution operation on the stitched feature image; multiple sets of one-dimensional convolution with different sizes refers to the control processing unit using multiple one-dimensional convolution kernels of different sizes (such as 1×1, 1×3, 1×5) to process the stitched feature image in parallel; correlation information refers to the data obtained by the control processing unit through multiple sets of one-dimensional convolution operations that can reflect the local feature dependency relationship between the target thermal feature channel and the target morphological feature channel.

[0072] Specifically, after obtaining the target thermal feature map and the target morphological feature map, the control processing unit immediately executes this step. This aims to break the independence between the two feature maps, establish a local correlation between thermal and morphological features, and provide a basis for subsequent feature enhancement. First, the control processing unit performs channel stitching. Assuming the target thermal feature map has C1 channels (e.g., 1 channel, containing only high-temperature contour features) and the target morphological feature map has C2 channels (e.g., 3 channels, corresponding to the photovoltaic device edge, smoke texture, and flame shape features respectively), the stitched feature image will have 4 channels. During stitching, the spatial resolution of the feature maps is kept consistent to ensure a one-to-one correspondence between thermal and morphological features at each spatial location. Subsequently, the control processing unit applies three sets of one-dimensional convolutions of different sizes in parallel to the stitched 4-channel feature image: the first set uses a 1×1 one-dimensional convolution kernel (number set to 4) to capture local features within a single channel (such as the local continuity of the high-temperature contour); the second set uses a 1×3 one-dimensional convolution kernel (number set to 4) to capture feature associations between three adjacent channels (such as the adjacency association between the high-temperature contour channel and the photovoltaic device edge channel); the third set uses a 1×5 one-dimensional convolution kernel (number set to 4) to capture feature associations between interval channels (such as the cross-channel association between the high-temperature contour channel and the flame shape channel). Each convolution operation uses the ReLU activation function to enhance the nonlinear representation of features. Finally, the outputs of the three convolution operations are stitched together by channel (total number of channels is 4+4+4=12 channels) to obtain association information that can represent the local interaction relationships between different channels.

[0073] 2023. Calculate the infrared modulation coefficient and the visible light modulation coefficient based on the correlation information.

[0074] The infrared modulation coefficient refers to the weighting coefficient calculated by the control processing unit based on the association information and used to enhance the features of the target thermal feature map. The magnitude of the value reflects the influence of the correlation strength between thermal features and morphological features in the association information on the thermal features. The visible light modulation coefficient refers to the weighting coefficient calculated by the control processing unit based on the association information and used to enhance the features of the target morphological feature map. The magnitude of the value reflects the influence of the correlation strength between thermal features and morphological features in the association information on the morphological features.

[0075] Specifically, the control processing unit executes this step immediately after obtaining the associated information. First, the control processing unit performs channel attention calculation on the associated information: using global average pooling, each channel of the associated information (such as a 640×480×12 feature map) is compressed into a single value (i.e., the global average gray value of each channel), resulting in a 12-dimensional channel feature vector. This channel feature vector is then input into an attention network containing two fully connected layers (the first layer has 6 neurons, and the second layer has 12 neurons). The output value is mapped to the [0,1] interval using the Sigmoid activation function, resulting in a 12-dimensional channel attention weight. This channel attention weight reflects the importance of each associated information channel. Next, the control processing unit groups the channel attention weights: channels associated with target thermal features in the 12-dimensional weights (e.g., the first 4 dimensions, corresponding to the thermal feature associated channels in the 1×1 convolution in step 2022) are grouped into the infrared association group, and the average weight of this group is calculated as the infrared modulation coefficient (if the weights of the first 4 dimensions are 0.8, 0.7, 0.9, and 0.8 respectively, the average is 0.8, i.e., the infrared modulation coefficient is 0.8); channels associated with target morphological features (e.g., the last 8 dimensions, corresponding to the morphological feature associated channels in the 1×3 and 1×5 convolutions in step 2022) are grouped into the visible light association group, and the average weight of the visible light association group is calculated as the visible light modulation coefficient (if the average weight of the last 8 dimensions is 0.75, the visible light modulation coefficient is 0.75). If there is overlap in the grouping of associated information channels (e.g., some channels are associated with both thermal and morphological features), a weighted allocation method is used (weights are allocated according to the contribution ratio of the channel to the thermal and morphological features) to ensure that the modulation coefficient accurately reflects the association strength.

[0076] 2024. Apply the infrared modulation coefficient to the target thermal feature map and the visible light modulation coefficient to the target morphological feature map to generate a feature-enhanced target thermal feature map and target morphological feature map.

[0077] Feature-enhanced target thermal feature map refers to a feature map obtained by applying infrared modulation coefficients to the original target thermal feature map by the control processing unit, resulting in a more prominent thermal feature (especially the high-temperature region contour, which is strongly correlated with morphological features); feature-enhanced target morphological feature map refers to a feature map obtained by applying visible light modulation coefficients to the original target morphological feature map by the control processing unit, resulting in a more prominent morphological feature (especially the edge of photovoltaic equipment, smoke, and flame features, which are strongly correlated with thermal features).

[0078] Specifically, for the enhancement of the target thermal feature map: the control processing unit uses a pixel-by-pixel multiplication method, multiplying the infrared modulation coefficient by each pixel value of the original target thermal feature map. If the grayscale value of a pixel in the original thermal feature map is 200 (indicating high confidence in the high-temperature contour) and the infrared modulation coefficient is 0.8, then the enhanced pixel value is 200 × 0.8 = 160 (if the coefficient is greater than 1, such as 1.2, then the pixel value is 200 × 1.2 = 240, further strengthening the high-confidence feature); at the same time, the enhanced thermal feature map is subjected to threshold truncation processing to limit the pixel value to 0-255 (to avoid exceeding the grayscale value range), and morphological dilation operation (3×3 structuring element) is used to expand the clarity of the high-temperature region contour, finally generating a feature-enhanced target thermal feature map. For enhancing the target morphological feature map: the control processing unit also uses pixel-by-pixel multiplication, multiplying the visible light modulation coefficient by the pixel value of each channel of the original target morphological feature map. For example, if a pixel value of a certain channel of the photovoltaic device edge in the original morphological feature map is 180 and the visible light modulation coefficient is 0.73, then the enhanced pixel value is 180 × 0.73 ≈ 131. Subsequently, contrast enhancement (such as the CLAHE algorithm) is performed on each channel to further improve the distinguishability of the morphological features. Then, the channels are integrated to generate the enhanced target morphological feature map. If there are local interference areas in the original feature map (such as light spots in infrared images or shadows in visible light images), the modulation coefficient will suppress the interference features by reducing the weight of the pixels in that area (e.g., weak correlation information in the interference area leads to a smaller modulation coefficient).

[0079] 203. The target thermal feature map and the target morphological feature map are fused to generate a fused feature map, and the areas in the fused feature map whose temperature exceeds the preset temperature threshold are identified as candidate fire source areas.

[0080] The process of generating the fused feature map in step 203 includes steps 2031 to 2034, which are not shown in the figure.

[0081] 2031. The target thermal feature map is used as the semantic feature of the feature pyramid network, and the target morphological feature map is used as the spatial feature of the feature pyramid network. The semantic feature is used to make an overall judgment on the fire source area, and the spatial feature is used to locate the detailed position of the fire source.

[0082] Feature Pyramid Network (FPN) refers to a deep learning network structure where the control processing unit is used for multi-scale feature extraction and fusion. Through top-down feature propagation and lateral connections, it achieves effective combination of features at different resolutions. Semantic features refer to the features with global contextual information extracted by the control processing unit after inputting the target thermal feature map into the Feature Pyramid Network, which focuses on reflecting the category attributes of the region (such as "whether it is a high-temperature fire source region"). Spatial features refer to the features with fine location information extracted by the control processing unit after inputting the target morphological feature map into the Feature Pyramid Network, which focuses on reflecting the pixel-level spatial coordinates of the region (such as "the specific pixel position of the fire source edge").

[0083] Specifically, the control processing unit preprocesses the input target thermal feature map and target morphological feature map: the resolution of the target thermal feature map and target morphological feature map is uniformly adjusted to the input resolution of the feature pyramid network, and normalization processing is performed to avoid the feature extraction effect being affected by differences in numerical range. Subsequently, the control processing unit inputs the target thermal feature map into the semantic feature branch of the feature pyramid network. This semantic feature branch gradually reduces the feature map resolution through 4 layers of downsampling convolution (each layer uses a 3×3 convolution kernel and a stride of 2), while increasing the number of channels to enhance the capture of global temperature distribution patterns. Finally, it outputs semantic features with global contextual information, which are used to determine which areas in the image belong to the macroscopic range of potential fire sources (e.g., "the overall temperature of a certain area is relatively high, which may be a fire source"). Simultaneously, the control processing unit inputs the target morphology feature map into the spatial feature branch of the feature pyramid network. This spatial feature branch performs only one layer of downsampling (maintaining a high resolution, such as 256×256), and adjusts the number of channels to 128. It extracts the edge and texture details of local pixels through small-sized convolutional kernels (1×1, 3×3), and finally outputs spatial features that retain fine positional information. These features are used to locate the specific boundary of the fire source (such as "the edge of the fire source is at a certain pixel coordinate"), the texture distribution details of the smoke, etc.

[0084] 2032. Compress the channels of semantic features through convolution operations, and apply multi-size convolutions in parallel to the compressed semantic features to extract multi-scale semantic features.

[0085] Convolution operation compression refers to the operation of the control processing unit to perform convolution operation on semantic features using a 1×1 convolution kernel to reduce the number of semantic feature channels; semantic features after channel compression refer to semantic features that have reduced channel number and retain core global information after 1×1 convolution processing; multi-scale semantic features refer to a set of semantic features that contain global information of different ranges (local, mid-range, global) extracted through multiple sets of one-dimensional convolution.

[0086] Specifically, semantic feature channel compression is performed first: the control processing unit uses a 1×1 convolution kernel to process the semantic features. If the original semantic features are 128×128×256 (resolution 128×128, number of channels 256), the number of 1×1 convolution kernels is selected as 64 (channels are compressed to 64), the convolution stride is set to 1, and the padding is set to "same" (to ensure that the resolution remains unchanged). At the same time, BN (batch normalization) and ReLU activation functions are added to accelerate training convergence and enhance nonlinear expression. Finally, a 128×128×64 compressed channel semantic feature is obtained, which reduces the number of channels by 4 / 5 (reducing the amount of computation) while retaining the core high-temperature global semantic information. Subsequently, the control processing unit applies multiple sets of one-dimensional convolutions in parallel to the compressed semantic features: the control processing unit constructs three sets of parallel one-dimensional convolutional layers. The first set uses 1×1 convolutional kernels (64 in total) to extract local semantic information within a single channel (such as local associations within a small high-temperature region); the second set uses 1×3 convolutional kernels (64 in total) to extract mid-range semantic information between three adjacent channels (such as associations between medium-range high-temperature regions); and the third set uses 1×5 convolutional kernels (64 in total) to extract global semantic information spaced five channels apart (such as overall associations between large-range high-temperature regions). Each convolutional layer uses a stride of 1 and padding "same" to ensure that the output feature resolution is consistent with the input (128×128), ultimately resulting in three sets of 128×128×64 feature maps, which together constitute a multi-scale semantic feature set.

[0087] 2033. Concatenate the multi-scale semantic features along the channels to generate enhanced semantic features.

[0088] Channel-based stitching refers to the operation of the control processing unit to superimpose and merge multiple sets of multi-scale semantic features along the channel dimension, that is, to add the number of channels of semantic features at different scales while keeping the spatial resolution unchanged; enhanced semantic features refer to the feature map obtained by channel stitching, which integrates local, mid-range, and global multi-scale semantic information.

[0089] Specifically, the control processing unit performs spatial consistency checks on multiple sets of multi-scale semantic features: it checks whether the resolution of each set of feature maps is completely consistent. If there are resolution differences, it adjusts them to a uniform resolution using interpolation methods (such as bilinear interpolation) to ensure that the spatial positions correspond one-to-one during stitching. Subsequently, the control processing unit performs channel stitching: assuming that the multi-scale semantic features contain 3 sets of feature maps, each with 64 channels and a resolution of 128×128, the three sets of feature maps are stitched along the channel dimension. The number of channels after stitching is 64+64+64=192, and the resolution remains 128×128, resulting in an initial enhanced semantic feature of 128×128×192. Finally, the control processing unit performs channel attention optimization on the initial enhanced semantic features: the 192-channel feature map is compressed into a 192-dimensional vector through global average pooling, and input into an attention network containing two fully connected layers (96 neurons in the first layer and 192 neurons in the second layer). After passing through the Sigmoid activation function, 192-dimensional channel attention weights are obtained. The weights are multiplied with the initial enhanced semantic features channel by channel to strengthen the semantic information at key scales (such as strengthening the local semantic weights corresponding to the 1×1 convolution in the small fire source scenario and strengthening the global semantic weights corresponding to the 1×5 convolution in the large fire source scenario), and finally generating the optimized enhanced semantic features.

[0090] 2034. Upsample the enhanced semantic features through subpixel convolution, and based on the feature pyramid network, fuse the upsampled enhanced semantic features and spatial features layer by layer from top to bottom to obtain the fused feature map.

[0091] Subpixel convolution refers to a method where the control processing unit uses pixel rearrangement to convert the channel information of the semantic enhancement features into spatial resolution upsampling, avoiding the detail blurring caused by traditional interpolation upsampling. Upsampled semantic enhancement features refer to feature maps that have improved resolution, reduced channel number, and retain multi-scale semantic information after subpixel convolution processing. Top-down layer-by-layer fusion refers to the operation where the control processing unit, based on the structure of the feature pyramid network, starts from the top-level features with high resolution and fuses them layer by layer with the corresponding spatial features. The fused feature map refers to a feature map obtained by fusing semantic features and spatial features, which simultaneously contains global fire source judgment information and fine location information.

[0092] Specifically, the control processing unit first performs sub-pixel convolutional upsampling on the enhanced semantic features: if the enhanced semantic features are 128×128×192, the upsampling factor is set to 2 (matching the spatial feature resolution), the pixel rearrangement parameter r=2, and the number of output channels is 192 / (2²)=48, resulting in an upsampled semantic feature of 256×256×48. Then, it is fused layer by layer based on the feature pyramid network: the upsampled semantic features are concatenated with the 256×256×128 spatial feature channels to form 256×256×176, compressed to 64 channels by 1×1 convolution and activated to obtain the first layer of fused features; if there is a 64×64×256 semantic branch, it is first upsampled to 128×128×64, concatenated and compressed with the corresponding spatial features to obtain the second layer of fused features; finally, the fused features of each layer are horizontally connected and the channels are unified to generate a fused feature map.

[0093] 204. For each candidate fire source region, calculate the minimum distance and degree of overlap between the high-temperature region contour information and the edge contour features of the photovoltaic equipment.

[0094] Step 204 includes steps 2041 to 2043, which are not shown in the figure.

[0095] 2041. For each candidate fire source region, extract the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour features.

[0096] The pixel set of high-temperature region contour information refers to the set of coordinates of all pixels that constitute the boundary contour of the high-temperature region, extracted by the control processing unit from the target thermal feature map of the candidate fire source region; the pixel set of photovoltaic device edge contour features refers to the set of coordinates of all pixels that constitute the physical boundary contour of the photovoltaic device, extracted by the control processing unit from the target morphological feature map of the visible light image.

[0097] Specifically, for each candidate fire source region, the control processing unit first locates the candidate fire source region in the target thermal feature map, and then uses a contour tracking algorithm (such as the eight-neighborhood tracking method) to traverse the region boundary, recording the (x, y) coordinates of each boundary pixel to form a pixel set of high-temperature region contour information. Simultaneously, the control processing unit locates the corresponding photovoltaic device in the target morphological feature map, and similarly uses a contour tracking algorithm to extract the (x, y) coordinates of all pixels on the device edge to form a pixel set of photovoltaic device edge contour features, ensuring that the two sets of point sets are in the same image coordinate system.

[0098] 2042. Calculate the shortest Euclidean distance between the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour features, and use the shortest Euclidean distance as the minimum distance between the high-temperature region contour information and the photovoltaic device edge contour features.

[0099] The shortest Euclidean distance refers to the minimum Euclidean distance between any two pixels in the set of pixels representing the high-temperature region contour and the set of pixels representing the edge contour of the photovoltaic device, calculated by the control processing unit. Euclidean distance is the straight-line distance between two points (x1, y1) and (x2, y2) in a Cartesian coordinate system, calculated using the following formula: Minimum distance refers to an indicator used to characterize the spatial proximity between the outline of a high-temperature region and the edge outline of a photovoltaic device.

[0100] Specifically, for each pixel in the high-temperature region contour pixel set, the control processing unit calculates the Euclidean distance between each pixel and all pixels in the photovoltaic device edge contour pixel set, recording the minimum Euclidean distance for each high-temperature region contour pixel. Then, the control processing unit selects the smallest Euclidean distance from all the minimum Euclidean distances between the high-temperature region contour pixels and all pixels in the photovoltaic device edge contour pixel set. This smallest value is the shortest Euclidean distance between the two pixel sets, and the control processing unit determines it as the minimum distance between the high-temperature region contour information and the photovoltaic device edge contour features.

[0101] 2043. Calculate the area of ​​the first region surrounded by the high-temperature region contour information and the area of ​​the second region surrounded by the edge contour features of the photovoltaic device, respectively. Based on the intersection area and the union area of ​​the first region area and the second region area, calculate the degree of overlap between the high-temperature region contour information and the edge contour features of the photovoltaic device.

[0102] The first region area refers to the area (in pixels) of the closed region formed by the contour information of the high-temperature region, calculated by the control processing unit; the second region area refers to the area (in pixels) of the closed region formed by the edge contour features of the photovoltaic device, calculated by the control processing unit.

[0103] Specifically, the control processing unit uses a polygon area calculation algorithm (such as the Shoelace formula) to calculate the area of ​​the first region enclosed by the high-temperature region outline and the area of ​​the second region enclosed by the edge outline of the photovoltaic device, respectively; then, through a polygon intersection and union calculation algorithm, the intersection area and union area of ​​the two regions are obtained; finally, the intersection area is divided by the union area to obtain the degree of overlap (intersection-union ratio).

[0104] 205. Calculate the regional matching degree between the photovoltaic equipment and the candidate fire source areas based on the minimum distance and overlap, and determine the candidate fire source areas with regional matching degrees higher than the preset matching degree threshold as valid fire source areas. (This step has been explained in 105 and will not be repeated here.) 206. Calculate the infrared temperature loss term based on the difference in thermal characteristics between the predicted bounding box of the effective fire source area and the corresponding high-temperature area contour information.

[0105] The predicted bounding box represents the preliminary prediction result of the control processing unit on the location of the effective fire source area, and is represented by a rectangle; the high-temperature region contour information refers to the boundary features of the temperature anomaly region extracted from the infrared image; the difference in thermal features refers to the degree of inconsistency between the predicted bounding box and the actual high-temperature region in terms of temperature distribution; the infrared temperature loss term is a numerical index used to quantify the degree of matching between the predicted bounding box and the high-temperature region contour.

[0106] Specifically, the control processing unit executes this step after determining the effective fire source area, when its bounding box needs to be precisely located. The control processing unit first generates an initial predicted bounding box based on the approximate location of the effective fire source area. Then, it compares this bounding box with the outline of the corresponding high-temperature region, calculating the difference in temperature characteristics between the two. By calculating indicators such as the temperature gradient inside and outside the predicted bounding box and the overlap between the bounding box and the high-temperature region, the degree of matching between the predicted result and the actual high-temperature region is quantified, generating an infrared temperature loss term. The smaller the value of the infrared temperature loss term, the higher the degree of matching between the predicted bounding box and the high-temperature region.

[0107] In some embodiments, this can be achieved in several ways: Optionally, the mean square error between the average temperature within the predicted bounding box and the average temperature within the high-temperature region contour can be calculated and used as the infrared temperature loss term. Optionally, the temperature variance of the overlapping area between the predicted bounding box and the high-temperature region contour can be statistically analyzed, and combined with the temperature deviation of the non-overlapping area, a weighted calculation can be performed to obtain the infrared temperature loss term. It is understood that other calculation methods can also be used, and are not limited here. In practical applications, it is necessary to ensure accurate temperature value conversion based on the infrared camera calibration parameters.

[0108] 207. Based on the morphological differences between the predicted bounding box of the effective fire source area and the corresponding smoke texture features and the corresponding flame spatial shape features, calculate the visible light morphological loss term.

[0109] The visible light morphology loss term is used as a numerical indicator to quantify the degree of matching between the predicted bounding box and morphological features.

[0110] Specifically, the control processing unit executes this step immediately after calculating the infrared temperature loss term. The control processing unit compares the predicted bounding box with the corresponding smoke texture features and flame shape features. By calculating indices such as texture consistency and shape similarity within the bounding box region, it quantifies the degree of matching between the predicted result and the actual fire source morphology features, generating a visible light morphology loss term. The smaller the visible light morphology loss term, the higher the degree of matching between the predicted bounding box and the fire source morphology features.

[0111] In some embodiments, this can be achieved in several ways: Optionally, the intersection-union ratio (IU) of the predicted bounding box with the smoke texture feature region and the flame shape feature region is calculated, and (1 - IU) is used as the visible light morphology loss term. Optionally, the difference between the texture within the predicted bounding box and the smoke texture feature is calculated using a histogram comparison method, and the difference with the flame shape is calculated using a shape matching algorithm. The weighted sum is then used to obtain the visible light morphology loss term. It is understood that other calculation methods can also be used, and this is not limited here. In practical applications, the visible light image needs to be preprocessed to reduce the interference of illumination and noise on the morphological features.

[0112] 208. Based on the infrared temperature loss term and the visible light morphology loss term, a weighted regression algorithm is used to calculate the predicted bounding box of the effective fire source region, and the optimized bounding box parameters of the effective fire source region are obtained. The bounding box parameters include the offset of the center point position of the bounding box corresponding to the effective fire source region, as well as the width scaling factor and height scaling factor used to determine the size of the bounding box corresponding to the effective fire source region.

[0113] The offset of the center point position represents the horizontal and vertical displacement that the predicted bounding box center needs to be adjusted; the width scaling factor and height scaling factor are used to adjust the width and height dimensions of the bounding box.

[0114] Specifically, the control processing unit executes this step immediately after obtaining the two loss terms. The control processing unit constructs a comprehensive loss function by assigning weight coefficients to the infrared temperature loss term and the visible light morphology loss term. A weighted regression algorithm is then used to iteratively optimize the bounding box parameters until the loss function converges to its minimum, yielding the optimal bounding box parameters, including the center point offset and size scaling factor.

[0115] In some embodiments, this can be achieved in several ways: Optionally, gradient descent can be used, with the comprehensive loss function as the objective, to iteratively update the center point offset and scaling factor until the loss converges. Optionally, a linear regression model can be constructed, using the loss term as input, to directly solve for the optimal bounding box parameters. It is understood that other weighted regression methods can also be used, which are not limited here. In practical applications, the parameter adjustment range needs to be limited to avoid excessive offsetting or scaling of the bounding box.

[0116] 209. Based on the bounding box parameters, calculate the bounding box coordinates and confidence level of the effective fire source region in the visible light image.

[0117] The bounding box coordinates represent the specific location of the optimized bounding box in the visible light image pixel coordinate system; the confidence score is used to indicate the reliability of the bounding box localization result.

[0118] Specifically, the control processing unit calculates the exact pixel coordinates of the bounding box in the visible light image based on the center point offset and the size scaling factor. Simultaneously, based on the values ​​of the infrared temperature loss term and the visible light morphology loss term, it calculates the confidence level of the localization result; a higher confidence level indicates a more reliable localization result.

[0119] In some embodiments, coordinate location and confidence score calculation can be implemented in multiple ways: Optionally, the control processing unit first normalizes the bounding box parameters, then substitutes them into the initial coordinates for calculation, while introducing temporal consistency verification, and using the average confidence score of multiple frames as the final result. Optionally, a machine learning model is used, with bounding box parameters and loss terms as input features, directly outputting optimized coordinates and confidence scores; the model is trained and optimized using historical fire samples. It is understood that other calculation methods can also be used, and are not limited here. In practical applications, the coordinates need to be integerized to ensure they meet pixel coordinate requirements, and the confidence score calculation weights need to be adjusted according to lighting and weather conditions.

[0120] 210. Based on the bounding box coordinates, the camera parameters of the photoelectric turntable, the angle parameters of the photoelectric turntable, and the geographic coordinates of the photoelectric turntable, calculate the three-dimensional geographic coordinates of the effective fire source area through coordinate system transformation.

[0121] Camera parameters refer to the intrinsic parameters (such as focal length and pixel size) and extrinsic parameters (such as the offset of the camera's installation position relative to the turntable) of the camera on the photoelectric turntable; angle parameters refer to the current horizontal rotation angle and vertical pitch angle of the photoelectric turntable; geographic coordinates of the photoelectric turntable refer to the latitude, longitude, and altitude of the turntable installation position; three-dimensional geographic coordinates refer to the latitude, longitude, and altitude coordinates of the effective fire source area in real physical space, used to locate the actual location of the fire source.

[0122] Specifically, the control processing unit first performs a transformation from the image coordinate system to the camera coordinate system: based on the camera's intrinsic parameters, the image coordinates (u, v) of the bounding box center point are converted into three-dimensional coordinates (X_c, Y_c, Z_c) in the camera coordinate system, using the formulas (u-u0) / f_x=X_c / Z_c and (v-v0) / f_y=Y_c / Z_c (u0, v0 are the image center coordinates, f_x, f_y are the camera focal lengths). The control processing unit then combines the angle parameters of the photoelectric turntable (horizontal angle α, pitch angle β) to convert the camera coordinate system coordinates into geodetic coordinate system coordinates. The direction is adjusted using a rotation matrix, and the geographic coordinates (Lon0, Lat0, H0) of the turntable are superimposed to finally calculate the three-dimensional geographic coordinates (Lon, Lat, H) of the effective fire source area.

[0123] In some embodiments, coordinate system transformation and 3D coordinate calculation can be achieved in various ways: Optionally, the control processing unit adopts a step-by-step transformation method, first transforming the image to the camera coordinate system, then converting it to the geodetic coordinate system using turntable angle parameters, and finally combining the turntable's geographic coordinates to obtain the result. It is understood that other transformation methods can also be used, and are not limited here. In practical applications, it is necessary to periodically calibrate the camera parameters and turntable angle accuracy to reduce transformation errors and ensure the accuracy of the 3D coordinates.

[0124] 211. Valid fire source areas with confidence levels reaching the preset confidence threshold are identified as the final fire sources, and the final fire sources are associated with their corresponding three-dimensional geographic coordinates.

[0125] The preset confidence threshold refers to the critical value set in advance by the control processing unit to determine whether the effective fire source area is a real fire source. It is determined based on the historical fire data of the photovoltaic power station and safety regulations, and the value range is usually [0.7, 0.9].

[0126] Specifically, the control processing unit compares the confidence level of each valid fire source area with a preset confidence threshold. If the confidence level is greater than or equal to the threshold, the area is determined to be the final fire source; if the confidence level is less than the threshold, it is determined to be a suspected interference area and is removed. Afterward, the control processing unit establishes a mapping relationship between the final fire source and its three-dimensional geographic coordinates, storing the final fire source's number, bounding box information, confidence level, and corresponding three-dimensional geographic coordinates (Lon, Lat, H) in the same data structure. This facilitates subsequent data retrieval to control the photoelectric turntable for tracking or to send early warning information to the operation and maintenance system.

[0127] 212. Acquire paired images captured by binocular cameras, and calculate the final verification three-dimensional coordinates of the fire source based on the paired images using binocular vision. The paired images include a left view and a right view located on the same horizontal line.

[0128] A binocular camera is a stereo vision device consisting of two cameras placed in parallel; paired images represent the two images captured simultaneously by the binocular cameras; the horizontal line refers to the same horizontal plane where the optical axes of the left and right cameras are located; the verified three-dimensional coordinates refer to the spatial coordinates of the fire source obtained through binocular vision measurement.

[0129] Specifically, the control processing unit executes this step immediately after determining the final fire source. First, the control processing unit controls the binocular cameras to simultaneously acquire left and right views, ensuring that the two images are synchronized in time and located on the same horizontal plane. Then, the control processing unit performs stereo matching on the left and right views. By calculating the parallax (the pixel position difference between corresponding points in the left and right views) and combining it with the intrinsic and extrinsic parameters of the binocular cameras (focal length, baseline length, etc.), the three-dimensional spatial coordinates of the final fire source are calculated using the principle of triangulation. These three-dimensional spatial coordinates serve as verification data to evaluate the accuracy of the three-dimensional geographic coordinates previously obtained using a monocular camera.

[0130] 213. Calculate the coordinate deviation between the verification three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates. The coordinate deviation is the spatial straight-line distance between the verification three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates.

[0131] The coordinate deviation value represents the Euclidean distance between the spatial coordinates measured by the two methods.

[0132] Specifically, the control processing unit executes this step immediately after obtaining the verification 3D coordinates. The control processing unit compares the verification 3D coordinates measured by binocular vision with the previously obtained 3D geographic coordinates, and calculates the Euclidean distance between the verification 3D coordinates and the corresponding 3D geographic coordinates in 3D space as the coordinate deviation value. This coordinate deviation value reflects the degree of consistency between the measurement results of the two positioning methods; the smaller the deviation value, the more reliable the positioning result.

[0133] In some embodiments, coordinate deviation values ​​can be calculated in multiple ways: Optionally, the control processing unit first converts the verification three-dimensional coordinates and three-dimensional geographic coordinates into rectangular coordinates in a geodetic coordinate system, and then substitutes them into the formula "deviation value = "Calculation. It is understood that other three-dimensional distance calculation methods can also be used to calculate coordinate deviation values, which are not limited here."

[0134] 214. When the coordinate deviation value exceeds the deviation range threshold, the three-dimensional geographic coordinates of the final fire source are recalculated based on the paired image and the fused feature map.

[0135] The deviation range threshold is a critical value that is pre-set by the control processing unit to determine whether the coordinate deviation value is acceptable. It is determined based on the accuracy requirements for fire source positioning in photovoltaic power plants (such as usually set to 5 meters).

[0136] Specifically, the control processing unit executes this step when it detects that the coordinate deviation exceeds a preset threshold. The control processing unit combines the stereo vision information provided by the paired binocular images with multimodal features from the fused feature map, employing a more complex 3D reconstruction algorithm to recalculate the spatial location of the fire source. This method, combining multi-source information, can improve positioning accuracy and obtain more accurate 3D geographic coordinates.

[0137] In some embodiments, the recalculation of the final 3D geographic coordinates of the fire source can be achieved in several ways: Optionally, the control processing unit first accurately extracts the bounding box coordinates of the final fire source from the fused feature map, maps them to the left view of the paired image, and obtains the corrected image coordinates; it then converts them to camera coordinate system coordinates using the intrinsic parameters of the binocular camera, and then uses the turntable angle parameters and geographic coordinates to convert them to geodetic coordinate system coordinates, which are used as the recalculated 3D geographic coordinates. For example, if the original coordinate deviation is 8 meters, the deviation is reduced to 2 meters after correction. Optionally, the control processing unit combines the parallax information of the paired image with the thermal features of the fused feature map to determine the precise depth information of the final fire source; it then substitutes the coordinate system transformation model used to calculate the original 3D geographic coordinates, corrects the model parameters, and re-inputs the bounding box coordinates to calculate the 3D geographic coordinates. It is understood that other methods combining multi-source information can also be used to recalculate the 3D geographic coordinates, which are not limited here.

[0138] 215. Based on the location of the final fire source, control the photoelectric turntable to track the final fire source. (This step has been explained in 108 and will not be repeated here.) The fire source localization method integrating multimodal sensing and photoelectric turntable in this embodiment first acquires infrared and visible light images of photovoltaic equipment and aligns them spatiotemporally. Then, it extracts and enhances thermal and morphological features. Subsequently, it generates a fused feature map through feature pyramid network fusion to screen candidate fire source areas. Next, it calculates the distance and overlap between the candidate areas and the edge of the photovoltaic equipment to determine the effective fire source area. Then, it optimizes the bounding box parameters using a weighted regression algorithm with double loss terms. Finally, it combines the parameters to generate coordinates and confidence scores to determine the final fire source and associates it with three-dimensional geographic coordinates. It also verifies and corrects coordinate deviations through binocular vision. Finally, it controls the photoelectric turntable to track the fire source throughout the entire process. This method achieves accurate localization and dynamic monitoring of fire sources in photovoltaic power plants under complex environments. It not only effectively alleviates the interference of environmental factors such as sunlight and rain on localization, but also significantly improves the accuracy and reliability of fire source localization. Furthermore, it provides accurate location information for early warning and timely handling of fires.

[0139] The method provided in the above embodiments can be executed by the control processing unit of the fire source locating system. The control processing unit in the embodiments of this invention is described below from a hardware processing perspective; please refer to [link / reference]. Figure 4 This is a schematic diagram of the physical device structure of the control processing unit in the embodiments of this application.

[0140] It should be noted that, Figure 4 The structure of the control processing unit shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0141] like Figure 4As shown, the control processing unit includes a Central Processing Unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 402 or programs loaded from storage section 408 into Random Access Memory (RAM) 403, such as executing the fire source localization method integrating multimodal sensing and photoelectric turntable described in the above embodiment. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0142] The following components are connected to I / O interface 405: input section 406 including audio input devices, push-button switches, etc.; output section 407 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 408 including a hard disk, etc.; and communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0143] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the various functions defined in the present invention.

[0144] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0146] Specifically, the control processing unit in this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the control processing unit provided in the above embodiment.

[0147] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the control processing unit described in the above embodiments; or it may exist independently and not assembled into the control processing unit. The storage medium carries one or more computer programs that, when executed by a processor of the control processing unit, cause the control processing unit to implement the control processing unit provided in the above embodiments.

[0148] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0149] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0150] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for fire source localization integrating multimodal sensing and a photoelectric turntable, characterized in that, include: Infrared and visible light images of photovoltaic equipment are acquired, and the pixel coordinates in the infrared image are mapped to the visible light image coordinate system through a transformation matrix using a timestamp, to obtain time- and space-aligned infrared and visible light images; The target thermal feature map is extracted from the infrared image, and the target morphological feature map is extracted from the visible light image. The target thermal feature map includes the contour information of the high temperature area, and the target morphological feature map includes the edge contour features of the photovoltaic device, the smoke texture features, and the spatial shape features of the flame. The target thermal feature map and the target morphological feature map are fused to generate a fused feature map, and the areas in the fused feature map whose temperature exceeds a preset temperature threshold are identified as candidate fire source areas; For each of the candidate fire source regions, calculate the minimum distance and degree of overlap between the high-temperature region contour information and the edge contour features of the photovoltaic device; Based on the minimum distance and the degree of overlap, the regional matching degree between the photovoltaic device and the candidate fire source area is calculated, and the candidate fire source area with the regional matching degree higher than the preset matching degree threshold is determined as the effective fire source area. A weighted regression algorithm that integrates infrared temperature loss term and visible light morphology loss term is used to calculate the bounding box position of the effective fire source region and obtain the bounding box parameters of the effective fire source region. The infrared temperature loss term is used to measure the difference in thermal features between the predicted bounding box and the contour information of the high-temperature region, and the visible light morphology loss term is used to measure the difference in morphology between the predicted bounding box and the smoke texture features and the spatial shape features of the flame. Based on the bounding box parameters, the bounding box coordinates and confidence levels of the effective fire source region are generated, and the effective fire source region whose confidence level reaches the preset confidence threshold is determined as the final fire source. Based on the location of the final fire source, the photoelectric turntable is controlled to track the final fire source.

2. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 1, characterized in that, Extracting target thermal feature maps from the infrared image and target morphological feature maps from the visible light image specifically includes: The target thermal feature map is extracted from the infrared image, and the target morphological feature map is extracted from the visible light image. The target thermal feature map includes the contour information of the high temperature area, and the target morphological feature map includes the edge contour features of the photovoltaic device, the smoke texture features, and the spatial shape features of the flame. The target thermal feature map and the target morphological feature map are concatenated by channels, and multiple sets of one-dimensional convolutions of different sizes are applied in parallel to the concatenated feature image to obtain the association information representing the local interaction between channels. Based on the aforementioned correlation information, the infrared modulation coefficient and the visible light modulation coefficient are calculated; The infrared modulation coefficient is applied to the target thermal feature map, and the visible light modulation coefficient is applied to the target morphological feature map to generate a feature-enhanced target thermal feature map and a target morphological feature map.

3. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 1, characterized in that, The target thermal feature map and the target morphological feature map are fused to generate a fused feature map, specifically including: The target thermal feature map is used as the semantic feature of the feature pyramid network, and the target morphological feature map is used as the spatial feature of the feature pyramid network. The semantic feature is used to make an overall judgment on the fire source area, and the spatial feature is used to locate the detailed location of the fire source. The channels of the semantic features are compressed by convolution operations, and multi-size convolutions are applied in parallel to the compressed semantic features to extract multi-scale semantic features. The multi-scale semantic features are concatenated along the channels to generate enhanced semantic features; The enhanced semantic features are upsampled by subpixel convolution, and based on the feature pyramid network, the upsampled enhanced semantic features and the spatial features are fused layer by layer from top to bottom to obtain a fused feature map.

4. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 1, characterized in that, For each candidate fire source region, calculating the minimum distance and overlap between the high-temperature region contour information and the edge contour features of the photovoltaic device specifically includes: For each of the candidate fire source regions, the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour features are extracted respectively; Calculate the shortest Euclidean distance between the pixel set of the high-temperature region contour information and the pixel set of the photovoltaic device edge contour features, and use the shortest Euclidean distance as the minimum distance between the high-temperature region contour information and the photovoltaic device edge contour features; Calculate the area of ​​the first region surrounded by the high-temperature region contour information and the area of ​​the second region surrounded by the edge contour features of the photovoltaic device, respectively. Based on the intersection area and the union area of ​​the first region area and the second region area, calculate the degree of overlap between the high-temperature region contour information and the edge contour features of the photovoltaic device.

5. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 1, characterized in that, The weighted regression algorithm, which integrates infrared temperature loss and visible light morphology loss terms, is used to calculate the bounding box position of the effective fire source region, obtaining the bounding box parameters of the effective fire source region, specifically including: Based on the difference in thermal features between the predicted bounding box of the effective fire source area and the corresponding high-temperature area contour information, the infrared temperature loss term is calculated. Based on the morphological differences between the predicted bounding box of the effective fire source area and the corresponding smoke texture features and the corresponding flame spatial shape features, the visible light morphological loss term is calculated. Based on the infrared temperature loss term and the visible light morphology loss term, a weighted regression algorithm is used to calculate the predicted bounding box of the effective fire source region to obtain the optimized bounding box parameters of the effective fire source region. The bounding box parameters include the offset of the center point position of the bounding box corresponding to the effective fire source region, as well as the width scaling factor and height scaling factor used to determine the size of the bounding box corresponding to the effective fire source region.

6. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 1, characterized in that, Based on the bounding box parameters, the bounding box coordinates and confidence levels of the effective fire source regions are generated, and the effective fire source regions whose confidence levels reach a preset confidence threshold are determined as the final fire sources. Specifically, this includes: Based on the bounding box parameters, calculate the bounding box coordinates and confidence level of the effective fire source region in the visible light image; Based on the bounding box coordinates, the camera parameters of the photoelectric turntable, the angle parameters of the photoelectric turntable, and the geographic coordinates of the photoelectric turntable, the three-dimensional geographic coordinates of the effective fire source area are calculated through coordinate system transformation. The effective fire source area with a confidence level reaching a preset confidence threshold is determined as the final fire source, and the final fire source is associated with the corresponding three-dimensional geographic coordinates.

7. The fire source localization method integrating multimodal sensing and photoelectric turntable according to claim 6, characterized in that, After determining the effective fire source area with a confidence level reaching a preset confidence threshold as the final fire source and associating the final fire source with the corresponding three-dimensional geographic coordinates, the method further includes: Acquire paired images captured by a binocular camera, and calculate the verification three-dimensional coordinates of the final fire source based on the paired images using binocular vision. The paired images include a left view and a right view located on the same horizontal line. Calculate the coordinate deviation between the verified three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates. The coordinate deviation is the spatial straight-line distance between the verified three-dimensional coordinates of the final fire source and the corresponding three-dimensional geographic coordinates. When the coordinate deviation value exceeds the deviation range threshold, the three-dimensional geographic coordinates of the final fire source are recalculated based on the paired image and the fused feature map.

8. A fire source location system, characterized in that, It includes a control processing unit and a photoelectric turntable, wherein the control processing unit is connected to the photoelectric turntable; The control processing unit includes one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the control processing unit to execute the fire source positioning method of fusion multimodal sensing and photoelectric turntable as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the control processing unit, the control processing unit performs the fire source positioning method integrating multimodal sensing and photoelectric turntable as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the control processing unit, the control processing unit performs the method as claimed in any one of claims 1-7.

Images