A method, device, system and medium for wildfire alarm in a multimodal power transmission corridor

Abstract

This invention discloses a method, device, system, and medium for wildfire alarm in a multimodal power transmission corridor. The method includes: acquiring two-dimensional image data of a monitoring area in the power transmission corridor using a visual acquisition device, and determining a wildfire identification result for the monitoring area based on the two-dimensional image data; if the wildfire identification result indicates a wildfire has occurred, obtaining an extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area; determining the flame height of the wildfire based on the extrinsic parameter calibration matrix and the two-dimensional image data, and determining the wildfire alarm level for the monitoring area based on the flame height. This method solves the problem of high equipment cost for lidar-based wildfire monitoring, and improves the response speed of wildfire alarms while ensuring accuracy.

Description

Technical Field

[0001] This invention relates to the field of power line inspection technology, and in particular to a method, device, system and medium for wildfire alarm in a multimodal power transmission corridor. Background Technology

[0002] With the continuous development of the power system, the coverage of transmission corridors is becoming increasingly wider, and a large number of transmission lines pass through mountainous areas, forests, and other areas prone to wildfires. Wildfires pose a serious threat to the safe operation of transmission corridors, potentially causing accidents such as power line tripping and damage, thereby affecting the stability and reliability of power supply.

[0003] Currently, the main method for wildfire monitoring is to use lidar to collect three-dimensional point cloud data. However, the amount of three-dimensional point cloud data is enormous, and high-performance hardware is required in multiple technical aspects such as data storage, data transmission, and data analysis. As a result, the equipment cost for wildfire monitoring is high and the real-time monitoring performance is poor. Summary of the Invention

[0004] This invention provides a method, device, system, and medium for wildfire alarm in a multimodal power transmission corridor, in order to solve the problem of high equipment cost for lidar monitoring of wildfires and improve the response speed of wildfire alarms.

[0005] According to an embodiment of the present invention, a wildfire alarm method for a multimodal power transmission corridor is provided, the method comprising:

[0006] Two-dimensional image data of the monitoring area in the power transmission corridor is collected by a visual acquisition device, and the wildfire identification result of the monitoring area is determined based on the two-dimensional image data;

[0007] If the wildfire identification result indicates that a wildfire has occurred, obtain the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area;

[0008] Based on the external parameter calibration matrix and the two-dimensional image data, the flame height of the wildfire is determined, and based on the flame height, the wildfire alarm level of the monitoring area is determined.

[0009] According to another embodiment of the present invention, a wildfire alarm device for a multimode power transmission corridor is provided, the device comprising:

[0010] The wildfire identification result determination module is used to collect two-dimensional image data of the monitoring area in the power transmission corridor through a visual acquisition device, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data;

[0011] The extrinsic parameter calibration matrix acquisition module is used to acquire the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area when the wildfire identification result indicates that a wildfire has occurred.

[0012] The wildfire alarm level determination module is used to determine the flame height of the wildfire based on the external parameter calibration matrix and the two-dimensional image data, and to determine the wildfire alarm level of the monitoring area based on the flame height.

[0013] According to another embodiment of the present invention, a wildfire alarm system for a multimodal power transmission corridor is provided. The wildfire alarm system for a multimodal power transmission corridor includes a visual acquisition device and a terminal device with communication connection.

[0014] The visual acquisition device is used to acquire two-dimensional image data of the monitoring area in the power transmission corridor;

[0015] The terminal device includes: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the wildfire alarm method for the multimodal power transmission corridor according to any embodiment of the present invention.

[0016] According to another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the wildfire alarm method for a multimodal power transmission corridor as described in any embodiment of the present invention.

[0017] According to another embodiment of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the wildfire alarm method for multimodal power transmission corridors as described in any embodiment of the present invention.

[0018] One embodiment of the above invention has at least the following advantages or beneficial effects:

[0019] By using two-dimensional image data acquired by visual acquisition equipment, the wildfire identification result in the monitoring area of ​​the power transmission corridor is determined. When the wildfire identification result indicates that a wildfire has occurred, the extrinsic parameter calibration matrix between the visual acquisition equipment and the corresponding three-dimensional point cloud system of the monitoring area is obtained. Based on the extrinsic parameter calibration matrix and two-dimensional image data, the flame height of the wildfire is determined. Based on the flame height, the wildfire alarm level of the monitoring area is determined. The use of visual acquisition equipment combined with a three-dimensional point cloud system to realize daily monitoring of wildfire alarms in the power transmission corridor solves the problem of high equipment cost for LiDAR monitoring of wildfires. While ensuring the accuracy of wildfire alarms, the response speed of wildfire alarms is improved, thereby ensuring the safety of the power transmission corridor.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a wildfire alarm method for a multimodal power transmission corridor according to an embodiment of the present invention;

[0023] Figure 2 A flowchart illustrating another wildfire alarm method for a multimodal power transmission corridor provided in an embodiment of the present invention;

[0024] Figure 3 A flowchart illustrating another wildfire alarm method for a multimodal power transmission corridor provided in an embodiment of the present invention;

[0025] Figure 4 A flowchart illustrating a specific example of a wildfire alarm method for a multimodal power transmission corridor provided in an embodiment of the present invention;

[0026] Figure 5 A flowchart illustrating a wildfire alarm device for a multimodal power transmission corridor according to an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of a wildfire alarm system for a multimodal power transmission corridor according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that, in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0031] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features specified as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] Figure 1 This is a flowchart illustrating a wildfire alarm method for a multimodal power transmission corridor according to an embodiment of the present invention. This embodiment is applicable to monitoring and alarming wildfires in power transmission corridors. The method can be executed by a wildfire alarm device for the multimodal power transmission corridor, which can be implemented in hardware and / or software and can be configured in a terminal device. Figure 1 As shown, the method includes:

[0033] S110. Collect two-dimensional image data of the monitoring area in the power transmission corridor through visual acquisition equipment, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data.

[0034] Specifically, a visual acquisition device refers to a device that acquires two-dimensional image data, which may contain a single two-dimensional image or multiple two-dimensional images in a time series. For example, a visual acquisition device may be a camera or a video camera, but is not limited to the example scenario.

[0035] For example, the visual acquisition device can be fixed on a fixed device in the monitoring area, or it can be mounted on a mobile device, such as a pole or tower, or a drone, vehicle-mounted monitoring device, or mobile weather control device, but is not limited to the example scenario.

[0036] In an optional embodiment, determining the wildfire identification result of the monitoring area based on the two-dimensional image data includes: using a histogram equalization algorithm to determine the pixel value distribution data of the two-dimensional image data; if the pixel distribution data meets the preset distribution conditions, the wildfire identification result of the monitoring area is set to indicate that a wildfire has occurred; if the pixel distribution data does not meet the preset distribution conditions, the wildfire identification result of the monitoring area is set to indicate that no wildfire has occurred.

[0037] Specifically, the preset distribution condition is that the pixel ratio corresponding to the preset pixel range reaches a ratio threshold, where the pixel ratio represents the proportion of the number of pixels within the preset pixel range to the total number of pixels in the two-dimensional image data.

[0038] The preset pixel range is related to the color system used, which may include, but is not limited to, grayscale, RGB, CMYK, HSV, and HSL systems. For example, when the color system used is grayscale, the preset pixel range can be 200-255.

[0039] In another optional embodiment, determining the wildfire identification result of the monitoring area based on the two-dimensional image data includes: inputting the two-dimensional image data into a pre-trained wildfire identification model; wherein the wildfire identification model includes a feature extraction network and a wildfire identification network; using the feature extraction network, extracting features from the two-dimensional image data to obtain visual feature vectors and motion feature vectors; using the wildfire identification network, outputting a predicted wildfire result based on the visual feature vectors and motion feature vectors; and determining the wildfire identification result of the monitoring area based on the predicted wildfire result.

[0040] Specifically, visual feature vectors are used to describe visually meaningful information in two-dimensional image data. For example, visual feature vectors include, but are not limited to, color feature vectors, texture feature vectors, and spatial feature vectors. Spatial feature vectors are used to describe the relative positional information between multiple objects in two-dimensional image data, and can also be used to describe the shape information of each object in the two-dimensional image data.

[0041] Specifically, motion feature vectors are used to describe the motion characteristics of objects in two-dimensional image data. For example, motion feature vectors include, but are not limited to, displacement feature vectors, velocity feature vectors, rotation feature vectors, and trajectory feature vectors, etc.

[0042] In one optional embodiment, the feature extraction network includes a visual feature module for extracting visual feature vectors and a motion feature module for extracting motion feature vectors. Exemplarily, the visual feature module may be a convolutional neural network, a deep residual shrinking network, a generative adversarial network, or a capsule network, etc., and the motion feature module may be a recurrent neural network, an optical flow algorithm, or a motion detection algorithm, etc., but is not limited to the examples described above.

[0043] In another alternative embodiment, the wildfire identification model is a two-stream convolutional neural network or a three-dimensional convolutional neural network.

[0044] The advantage of extracting visual feature vectors and motion feature vectors from two-dimensional image data is that, since flames have unique visual and motion characteristics, such as the color of flames being composed of orange, red, and yellow, and flames exhibiting irregular shaking and jumping, the accuracy of wildfire identification results can be further improved.

[0045] In one optional embodiment, determining the wildfire identification result of the monitoring area based on the predicted wildfire result includes: using the predicted wildfire result as the wildfire identification result of the monitoring area.

[0046] In another optional embodiment, the wildfire identification result of the monitoring area is determined based on the predicted wildfire result, including: acquiring temperature data collected by temperature sensors in the monitoring area; if the temperature data exceeds the temperature threshold and the predicted wildfire result is that a wildfire has occurred, then the wildfire identification result of the monitoring area is set to that a wildfire has occurred.

[0047] Based on the above embodiments, optionally, determining the wildfire identification result of the monitoring area according to the wildfire prediction result further includes: if the temperature data does not exceed the temperature threshold, or the wildfire prediction result is that no wildfire has occurred, then setting the wildfire identification result of the monitoring area to no wildfire has occurred.

[0048] The advantage of setting up joint identification with temperature sensors is that by fusing two-dimensional image data with temperature data in a multimodal manner, the accuracy and reliability of wildfire identification results are further improved.

[0049] S120. If the wildfire identification result indicates that a wildfire has occurred, obtain the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0050] Specifically, the three-dimensional point cloud system corresponding to the monitoring area represents the three-dimensional spatial system to which the discrete points on the surface of the object within the monitoring area belong. The extrinsic parameter calibration matrix represents the spatial transformation relationship between the visual image system and the three-dimensional point cloud system when the visual acquisition device acquires data in the monitoring area, specifically including rotational transformation relationship and translational transformation relationship.

[0051] Specifically, the extrinsic parameter calibration matrix T is represented as:

[0052]

[0053] Where R represents the rotation transformation matrix and t represents the translation vector.

[0054] Based on the above embodiments, optionally, the method further includes: if the wildfire identification result is that no wildfire has occurred, obtaining other monitoring areas in the power transmission corridor and returning to execute S110.

[0055] S130. Based on the external parameter calibration matrix and two-dimensional image data, determine the flame height of the wildfire, and based on the flame height, determine the wildfire alarm level of the monitoring area.

[0056] Specifically, flame height represents the vertical distance between the highest and lowest points of a wildfire.

[0057] In one optional embodiment, determining the flame height of a wildfire based on an extrinsic calibration matrix and two-dimensional image data includes: performing wildfire contour recognition on the two-dimensional image data to obtain the top and bottom pixels of the wildfire; using the extrinsic calibration matrix to convert the top and bottom pixels into the three-dimensional point cloud system corresponding to the monitoring area, respectively, to obtain the top point cloud and the bottom point cloud; and using the difference in z-coordinates between the top point cloud and the bottom point cloud as the flame height of the wildfire.

[0058] In another optional embodiment, determining the flame height of a wildfire based on an extrinsic calibration matrix and two-dimensional image data includes: using an extrinsic calibration matrix to project the two-dimensional image data onto a three-dimensional point cloud system corresponding to the monitoring area to obtain three-dimensional projection data; and determining the flame height of the wildfire based on the three-dimensional projection data.

[0059] Specifically, when the extrinsic parameter calibration matrix represents the spatial transformation relationship from a visual image system to a 3D point cloud system, for example, the pixel P in the 2D image data... C In 3D projection data, it can be represented as a point cloud P. L , where P L =RP C +t. When the extrinsic parameter calibration matrix represents the spatial transformation relationship from a 3D point cloud system to a visual image system, P L =R - 1 P C -t.

[0060] In one optional embodiment, determining the flame height of a wildfire based on three-dimensional projection data includes: performing flame contour recognition on the three-dimensional projection data to obtain the top z-coordinate and bottom z-coordinate of the wildfire, and using the coordinate difference between the top z-coordinate and the bottom z-coordinate as the flame height of the wildfire.

[0061] In one optional embodiment, determining the wildfire alarm level of the monitoring area based on the flame height includes: obtaining the conductor distance from the ground, the distance between conductors, and the maximum canopy height of the vegetation under the transmission lines within the monitoring area; and determining the wildfire alarm level of the monitoring area based on the flame height, conductor distance from the ground, distance between conductors, and maximum canopy height.

[0062] In one specific embodiment, the wildfire alarm level of the monitoring area is determined based on the flame height, the distance between the conductors and the ground, the distance between the conductors and the maximum tree crown height, including: determining the risk of line-to-ground breakdown and the risk of phase-to-phase breakdown based on the flame height, the distance between the conductors and the ground, and the distance between the conductors and the maximum tree crown height; and determining the wildfire alarm level of the monitoring area based on the risk of line-to-ground breakdown and the risk of phase-to-phase breakdown.

[0063] In one embodiment, determining the risk of line-to-ground breakdown and the risk of phase-to-phase breakdown based on flame height, conductor distance from ground, conductor-to-conductor distance, and maximum tree canopy height includes: when the transmission lines in the monitoring area are AC line systems, determining the peak line-to-ground voltage and the peak phase-to-phase voltage based on the line voltage of the transmission lines; determining the line-to-ground insulation strength and the phase-to-phase insulation strength based on flame height, conductor distance from ground, conductor-to-conductor distance, and maximum tree canopy height; using the ratio between the peak line-to-ground voltage and the line-to-ground insulation strength as the risk of line-to-ground breakdown, and using the ratio between the peak phase-to-phase voltage and the phase-to-phase insulation strength as the risk of phase-to-phase breakdown.

[0064] In an AC line system, for example, the peak line-to-ground voltage U a1 Satisfy the following formula:

[0065]

[0066] Interphase voltage peak U a2 Satisfy the following formula:

[0067]

[0068] Among them, U n This indicates the line voltage of the transmission line.

[0069] For example, the risk of line-to-ground breakdown P a1 Satisfy the following formula:

[0070]

[0071] Alternating breakdown risk P a2 Satisfy the following formula:

[0072]

[0073] Among them, U 1acU represents the wire-to-ground insulation strength of an AC line system. 2ac This indicates the phase-to-phase insulation strength of an AC line system.

[0074] Specifically, the line-to-ground breakdown voltage and phase-to-phase breakdown voltage are determined based on the flame height, the distance between conductors and the ground, the distance between conductors and the maximum tree canopy height. Based on the line-to-ground breakdown voltage and phase-to-phase breakdown voltage, the line-to-ground insulation strength and phase-to-phase insulation strength are determined.

[0075] In AC line systems, the line-to-ground insulation strength U 1ac Depends on the line-to-ground gap breakdown voltage U g Breakdown voltage U between ground wire and ground wire s The smaller value in, i.e., U 1ac =min(U g U s Interphase insulation strength and interphase breakdown voltage U p Same, i.e., U 2ac =U p Among them, the line-to-ground gap breakdown voltage U g The grounding gap breakdown voltage U represents the voltage value at which an electrical breakdown occurs between the conductor and the ground. s The phase-to-phase breakdown voltage U represents the voltage value at which an electrical breakdown occurs between the conductor and the ground wire. p This indicates the voltage value at which an electrical breakdown occurs between two conductors.

[0076] In another embodiment, the risk of line-to-ground breakdown and the risk of phase-to-phase breakdown are determined based on the flame height, conductor distance from the ground, conductor-to-conductor distance, and maximum tree canopy height. This includes: when the transmission line in the monitoring area is a DC transmission line system, determining the line-to-ground voltage and phase-to-phase voltage based on the rated voltage of the transmission line; determining the line-to-ground insulation strength and phase-to-phase insulation strength based on the flame height, conductor distance from the ground, conductor-to-conductor distance, and maximum tree canopy height; using the ratio between the line-to-ground voltage and the line-to-ground insulation strength as the risk of line-to-ground breakdown, and using the ratio between the phase-to-phase voltage and the phase-to-phase insulation strength as the risk of phase-to-phase breakdown.

[0077] In a DC line system, the line-to-ground voltage is a single-stage operating voltage. For example, the line-to-ground voltage U... d1 Satisfy the following formula:

[0078] U d1 =U L

[0079] The phase-to-phase voltage is the voltage between two poles; for example, the phase-to-phase voltage U... d2 Satisfy the following formula:

[0080] U d2 =2×U L

[0081] Among them, U L This indicates the rated voltage of the transmission line.

[0082] For example, the risk of line-to-ground breakdown P d1 Satisfy the following formula:

[0083]

[0084] Alternating breakdown risk P d2 Satisfy the following formula:

[0085]

[0086] Among them, U 1dc U represents the line-to-ground insulation strength of a DC line system. 2dc This indicates the phase-to-phase insulation strength of a DC line system.

[0087] Specifically, the line-to-ground breakdown voltage and phase-to-phase breakdown voltage are determined based on the flame height, the distance between conductors and the ground, the distance between conductors and the maximum tree canopy height. Based on the line-to-ground breakdown voltage and phase-to-phase breakdown voltage, the line-to-ground insulation strength and phase-to-phase insulation strength are determined.

[0088] In a DC line system, the line-to-ground insulation strength U 1dc Depends on the line-to-ground gap breakdown voltage U g Breakdown voltage U between ground wire and ground wire s The smaller value in, i.e., U 1ac =C e ×min(U g U s Interphase insulation strength and interphase breakdown voltage U p Related, i.e., U 2ac =C e ×U p Among them, C e This represents the voltage waveform correction factor, for example, C. e It can be 1.19, but is not limited to the example case.

[0089] Based on the above embodiments, optionally, the line-to-ground gap breakdown voltage and phase-to-phase breakdown voltage are determined according to the flame height, the distance between conductors, the distance between conductors, and the maximum canopy height. This includes: taking the sum of the flame height and the maximum canopy height as the flame height above the ground; if the flame height above the ground is greater than or equal to the distance between conductors above the ground, then determining the line-to-ground gap breakdown voltage and phase-to-phase breakdown voltage according to the distance between conductors above the ground, the maximum canopy height, and the average withstand field strength of the flame; if the flame height above the ground is less than the distance between conductors above the ground, then determining the line-to-ground gap breakdown voltage and phase-to-phase breakdown voltage according to the flame height, the distance between conductors above the ground, the maximum canopy height, the average withstand field strength of the flame, and the average withstand field strength of the flue gas.

[0090] In one specific embodiment, the flame height above ground is greater than or equal to the conductor height above ground, indicating complete flame bridging. Since the flame penetrates the entire gap when completely bridging occurs, the line-to-ground gap breakdown voltage U... g and phase-to-phase breakdown voltage U p It is related to the average field strength that the flame can withstand.

[0091] For example, the line-to-ground gap breakdown voltage U g Satisfy the following formula:

[0092] U g =(H l -H t E f

[0093] Interphase breakdown voltage U p Satisfy the following formula:

[0094] U p =d p E f

[0095] Among them, H l H represents the distance of the conductor from the ground. t E represents the maximum canopy height. f d represents the average field strength that the flame can withstand. p Indicates the distance between conductors.

[0096] In another specific embodiment, the flame height above ground is less than the conductor height above ground, indicating incomplete flame bridging. Because the flame is not fully bridged, it only occupies part of the gap; therefore, the line-to-ground gap breakdown voltage U... g and phase-to-phase breakdown voltage U p It is related not only to the average withstand field strength of the flame, but also to the average withstand field strength of the flue gas outside the flame.

[0097] For example, the line-to-ground gap breakdown voltage U g Satisfy the following formula:

[0098] U g =H f E f +(H l -H t -H f E s

[0099] In one specific embodiment, the phase-to-phase breakdown voltage U p Satisfy the following formula:

[0100] U p =d p E s

[0101] Among them, H f E represents the flame height. s This indicates the average field strength that the flue gas can withstand.

[0102] Among them, the average tolerance field strength of flue gas E s Satisfy the following formula:

[0103]

[0104] Where, d s d represents the gap distance. s =H l -H t -H f .

[0105] Specifically, when d s When ≥4, the average withstand field strength E of the flue gas s Depends on d s This indicates that a larger gap distance requires a higher electric field strength. When d s When <4, the average withstand field strength E of the flue gas s The value is a fixed value, indicating that the average withstand field strength E of the flue gas is within a small gap distance. s It tends to a stable value.

[0106] In another specific embodiment, when the flame is not fully bridged, the phase-to-phase breakdown voltage U p Satisfy the following formula:

[0107]

[0108] Among them, E a This represents the standard air average withstand field strength, taken as 359.2 kV / m, C a C represents the altitude correction factor. g C represents the particle density correction factor for the smoke region. σ C represents air density. hThis indicates the air humidity correction factor.

[0109] Based on the above embodiments, optionally, the wildfire alarm level of the monitoring area can be determined according to the line-to-ground breakdown risk and the phase-to-phase breakdown risk, including: obtaining the statistical breakdown risk corresponding to the line-to-ground breakdown risk and the phase-to-phase breakdown risk, and determining the wildfire alarm level of the monitoring area according to the statistical breakdown risk.

[0110] Specifically, when the risk of line-to-ground breakdown is greater than 1, it indicates that the peak voltage to ground or operating voltage of the conductor has exceeded its insulation strength, and the risk of the transmission line tripping is high. When the risk of phase-to-phase breakdown is greater than 1, it indicates that the peak voltage between conductors or operating voltage has exceeded its insulation strength, and the risk of the transmission line tripping is high.

[0111] In this embodiment, the statistical breakdown risk is the maximum or average value corresponding to the line-to-ground breakdown risk and the phase-to-phase breakdown risk. For example, the wildfire alarm level can be extremely high risk, high risk, medium risk, and low risk. The number of wildfire alarm levels and the statistical breakdown risk range corresponding to each wildfire alarm level are not limited here, and can be customized according to actual needs.

[0112] The technical solution of this embodiment determines the wildfire identification result in the monitoring area of ​​the power transmission corridor based on the two-dimensional image data collected by the visual acquisition device. When the wildfire identification result indicates that a wildfire has occurred, the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area is obtained. Based on the extrinsic parameter calibration matrix and the two-dimensional image data, the flame height of the wildfire is determined. Based on the flame height, the wildfire alarm level of the monitoring area is determined. The use of visual acquisition device combined with three-dimensional point cloud system realizes the daily monitoring of wildfire alarms in the power transmission corridor, which solves the problem of high equipment cost of lidar for monitoring wildfires. While ensuring the accuracy of wildfire alarms, it also improves the response speed of wildfire alarms, thereby ensuring the safety of the power transmission corridor.

[0113] Figure 2 This is a flowchart of another wildfire alarm method for a multimodal power transmission corridor provided by an embodiment of the present invention. This embodiment further refines the "obtaining the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area" in the above embodiment. In this embodiment, obtaining the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area includes: obtaining historical image data of the monitoring area collected by the visual acquisition device, and obtaining historical point cloud data of the monitoring area collected by the lidar; wherein, the lidar and the visual acquisition device are set on the same monitoring device; and performing extrinsic parameter calibration based on the historical image data and the historical point cloud data to obtain the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0114] like Figure 2 As shown, the method includes:

[0115] S210. Collect two-dimensional image data of the monitoring area in the power transmission corridor using visual acquisition equipment, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data.

[0116] S210 in this embodiment is the same as that in the above embodiment. Figure 1 The S110 shown is the same or similar, and will not be described again in this embodiment.

[0117] S220. If the wildfire identification result indicates that a wildfire has occurred, acquire historical image data of the monitoring area collected by the visual acquisition device, and acquire historical point cloud data of the monitoring area collected by the lidar.

[0118] In this embodiment, the lidar and visual acquisition device are mounted on the same monitoring device. This monitoring device can be a mobile device or a fixed device within the monitoring area.

[0119] The lidar system consists of a transmitting unit that emits laser beams with specific wavelengths, frequencies, and energies, enabling them to propagate through the air. When these laser beams come into contact with an object, they are reflected, scattered, or absorbed. The receiving unit within the lidar system collects the reflected or scattered laser signals. By measuring the distance and angle of these signals, distance and angle information are obtained. Combining this information allows the location of the three-dimensional coordinates of a surface point on the object. Multiple surface points then constitute a three-dimensional point cloud.

[0120] Specifically, the acquisition times of historical image data and historical point cloud data are within the time error range to ensure the time consistency between historical image data and historical point cloud data.

[0121] In an optional embodiment, the method further includes: performing image preprocessing on the historical image data to obtain preprocessed historical image data. For example, image preprocessing includes, but is not limited to, image enhancement and image denoising. Image enhancement can be implemented using histogram equalization and / or sharpening algorithms. Histogram equalization can improve the image contrast of the historical image data, making details in the historical image data more prominent, thereby improving the visibility of objects in the historical image data. Sharpening algorithms can be implemented using a Laplacian filter to enhance the edges and details of objects in the historical image data. For example, image denoising can use a Gaussian filter, median filter, or mean filter, where a Gaussian filter is a linear smoothing filter that can remove noise from the historical image data while preserving the edge and detail information of objects.

[0122] In an optional embodiment, the method further includes: performing point cloud preprocessing on the historical point cloud data to obtain preprocessed historical point cloud data. Exemplarily, point cloud preprocessing includes, but is not limited to, point cloud denoising, point cloud reconstruction, and point cloud enhancement. Point cloud denoising can be implemented using statistical filters (such as mean filters or median filters) or voxel grid algorithms. Specifically, the historical point cloud data is divided into voxel grids, and the point clouds in each voxel grid are downsampled to achieve point cloud denoising. Point cloud reconstruction can be implemented using interpolation algorithms such as Laplacian smoothing to fill in missing point clouds in the historical point cloud data, thereby improving the completeness and accuracy of the historical point cloud data. Point cloud enhancement can be implemented using normal vector estimation algorithms and point cloud segmentation algorithms.

[0123] There are no restrictions on image preprocessing and point cloud preprocessing here; you can customize the settings according to your actual needs.

[0124] S230. Based on historical image data and historical point cloud data, perform extrinsic parameter calibration to obtain the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0125] In one optional embodiment, extrinsic parameter calibration is performed based on historical image data and historical point cloud data to obtain an extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area. This includes: using a preset calibration algorithm to perform extrinsic parameter calibration based on historical image data and historical point cloud data to obtain an extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0126] For example, the preset calibration algorithms include, but are not limited to, AprilTag calibration method, Iterative Closest Point (ICP) algorithm and Normal Distributions Transform (NDT) algorithm.

[0127] In another optional embodiment, extrinsic parameter calibration is performed based on historical image data and historical point cloud data to obtain an extrinsic parameter calibration matrix between the visual acquisition device and the 3D point cloud system corresponding to the monitoring area. This includes: using a preset calibration algorithm to perform initial extrinsic parameter calibration based on historical image data and historical point cloud data to obtain a reference calibration matrix; inputting the reference calibration matrix, the visual intrinsic parameter matrix of the visual acquisition device, historical image data, and historical point cloud data into an untrained classification and recognition model to obtain the predicted classification result corresponding to the output historical point cloud data; iteratively adjusting the reference calibration matrix in the classification and recognition model based on the predicted classification result; and obtaining the extrinsic parameter calibration matrix between the visual acquisition device and the 3D point cloud system corresponding to the monitoring area in the trained classification and recognition model when the iteration termination condition is met.

[0128] In this embodiment, the predicted classification result includes the projection classification result of each point cloud in the historical point cloud data. The projection classification result includes the probability of whether the projection point of the point cloud in the imaging plane of the visual acquisition device is within the visual cone and / or belongs to a certain object category. The object category can be wire, ground, sky, pillar, or vegetation, etc.

[0129] In one specific embodiment, the classification and recognition model projects historical point cloud data onto the imaging plane of the visual acquisition device based on a reference calibration matrix and the visual intrinsic parameter matrix of the visual acquisition device, obtaining historical projection data. For each projection point in the historical projection data, it determines whether the point cloud corresponding to the projection point is within the visual frustum based on the projection coordinates of the projection point; it then acquires the pixels corresponding to the projection point in the historical image data, and determines the probability that the point cloud corresponding to the projection point belongs to the object category based on the distance between the projection point and the pixel and the object category corresponding to the pixel. Specifically, the greater the distance, the lower the probability that the point cloud belongs to the object category corresponding to the pixel, and vice versa.

[0130] For example, the visual intrinsic parameter matrix K is represented as:

[0131]

[0132] Among them, f x and f y c represents the focal length of the visual acquisition device. x and c y This refers to the optical center of the visual acquisition device.

[0133] For example, when the reference calibration matrix represents the spatial transformation relationship from a 3D point cloud system to a visual image system, for example, point cloud P in historical point cloud data L When converted to a visual image system, it can be represented as a pixel P. C , where P C =RP L +t, then use the visual intrinsic parameter matrix to set the pixel P C The image is projected onto the imaging plane of the vision acquisition device to obtain pixel P. C The projected coordinates (u, v).

[0134] For example, the projected coordinates (u, v) satisfy the following formula:

[0135]

[0136] Among them, (X) C Y C Z C ) represents pixel PC Coordinates in a visual image system.

[0137] In one optional embodiment, the reference calibration matrix in the classification and recognition model is iteratively adjusted according to the predicted classification result, including: determining the classification loss function value based on the predicted classification result and the true classification result corresponding to the historical point cloud data; iteratively adjusting the reference calibration matrix in the classification and recognition model according to the classification loss function value; wherein, the iteration termination condition includes the convergence of the classification loss function value and / or the number of training times reaching a threshold.

[0138] For example, the loss functions corresponding to the classification loss function values ​​include, but are not limited to, the cross loss function, the Hinge loss function, the log loss function, the relative entropy loss function, or the contrastive loss function.

[0139] In another optional embodiment, the reference calibration matrix in the classification and recognition model is iteratively adjusted based on the predicted classification result, including: obtaining historical projection data corresponding to historical point cloud data in the classification and recognition model, and determining the distance cost function value based on the historical projection data and historical image data; wherein, the historical projection data represents the projection result of historical point cloud data on the imaging plane of the visual acquisition device; determining the classification loss function value based on the predicted classification result and the true classification result corresponding to the historical point cloud data; and iteratively adjusting the reference calibration matrix in the classification and recognition model based on the distance cost function value and the classification loss function value; wherein, the iteration termination condition includes at least one of the following: convergence of the distance cost function value, convergence of the classification loss function value, and the number of iterations reaching a threshold.

[0140] Specifically, the distance cost function value represents the distance between a projected point in the historical projection data and its corresponding real pixel in the historical image data. For example, the distance cost function values ​​can be derived from, but are not limited to, the Euclidean distance function, the Manhattan distance function, the Chebyshev distance function, or the Minkowski distance function, etc.

[0141] Based on the above embodiments, optionally, the Gauss-Newton algorithm is used to iteratively adjust the reference calibration matrix in the classification and recognition model according to the classification loss function value and / or cost loss function value.

[0142] The advantage of setting up a classification and recognition model is that, compared with traditional feature point calibration algorithms, the classification and recognition model utilizes the deep three-dimensional geometric features of point cloud data and the two-dimensional visual features of image data during the iterative solution of the extrinsic parameter calibration matrix, providing a richer feature representation for the extrinsic parameter calibration matrix, thereby improving the accuracy of the extrinsic parameter calibration matrix.

[0143] S240. Based on the external parameter calibration matrix and two-dimensional image data, determine the flame height of the wildfire, and based on the flame height, determine the wildfire alarm level of the monitoring area.

[0144] S240 in this embodiment is the same as that in the above embodiment. Figure 1 The S130 shown is the same or similar, and will not be described again in this embodiment.

[0145] The technical solution of this embodiment calibrates the historical point cloud data and historical image data of the monitoring area collected by the lidar and visual acquisition devices installed on the same monitoring device, respectively. This results in the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area. This solves the problem of obtaining the external parameter calibration matrix, improves the accuracy of the external parameter calibration matrix, and thus further improves the accuracy of wildfire alarms in the power transmission corridor.

[0146] Figure 3 This is a flowchart of another wildfire alarm method for a multimodal power transmission corridor provided by an embodiment of the present invention. This embodiment further refines the step of "obtaining the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of vegetation under the power transmission lines in the monitoring area" in the above embodiment. In this embodiment, obtaining the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of vegetation under the power transmission lines in the monitoring area includes: acquiring historical image data of the monitoring area collected by a visual acquisition device and acquiring historical point cloud data of the monitoring area collected by a lidar; wherein the lidar and the visual acquisition device are set on the same monitoring device; inputting the historical image data and the historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs, and the object categories in the point cloud category data include conductors, ground, and vegetation; and determining the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of vegetation under the power transmission lines in the monitoring area based on the point cloud category data.

[0147] like Figure 3 As shown, the method includes:

[0148] S310. Collect two-dimensional image data of the monitoring area in the power transmission corridor through visual acquisition equipment, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data.

[0149] S320. If the wildfire identification result indicates that a wildfire has occurred, obtain the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0150] S330. Determine the flame height of the wildfire based on the external parameter calibration matrix and two-dimensional image data.

[0151] S310-S330 in this embodiment are the same as those in the above embodiments. Figure 1 The S110-S130 shown are the same as or similar to those in the above embodiments. Figure 2 The S210-S240 shown are the same or similar, and will not be described again in this embodiment.

[0152] S340. Acquire historical image data of the monitoring area collected by the visual acquisition device, and acquire historical point cloud data of the monitoring area collected by the lidar.

[0153] In this embodiment, the lidar and visual acquisition device are mounted on the same monitoring device. This monitoring device can be a mobile device or a fixed device within the monitoring area.

[0154] Specifically, the acquisition times of historical image data and historical point cloud data are within the time error range to ensure the time consistency between historical image data and historical point cloud data.

[0155] S350. Input historical image data and historical point cloud data into a pre-trained point cloud classification model to obtain the output point cloud category data.

[0156] In this embodiment, the point cloud category data includes the object category to which each point cloud belongs in the historical point cloud data. The object categories in the point cloud category data include conductors, ground, and vegetation.

[0157] In one optional embodiment, the point cloud classification model includes a visual feature network, a point cloud feature network, and a fusion output network. Accordingly, historical image data and historical point cloud data are input into the pre-trained point cloud classification model to obtain output point cloud category data. Through the visual feature network, features are extracted from the historical image data to obtain visual feature vectors. Through the point cloud feature network, features are extracted from the historical point cloud data to obtain point cloud feature vectors. Through the fusion output network, the visual feature vectors and point cloud feature vectors are fused to obtain fused feature vectors, and point cloud category data is output based on the fused feature vectors.

[0158] In one alternative embodiment, the visual feature network is a convolutional neural network, and the fused output network is an attention network. For example, the point cloud feature network is a RandLaNet network or a VoxelNet network.

[0159] Specifically, visual feature vectors are used to describe visually meaningful information in historical image data. For example, visual feature vectors include, but are not limited to, color feature vectors, texture feature vectors, and spatial feature vectors. Spatial feature vectors are used to describe the relative positional information between multiple objects in historical image data, and can also be used to describe the shape information of each object in historical image data.

[0160] Specifically, point cloud feature vectors are used to describe the geometric and local structural features of historical point cloud data. For example, geometric features include, but are not limited to, location information, normal vectors, and curvature, while local structural features include, but are not limited to, local neighborhood information and local geometric relationships.

[0161] S360. Based on the point cloud category data, determine the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of the vegetation under the transmission lines within the monitoring area.

[0162] In one specific embodiment, the distance between the conductor and the ground of the transmission line within the monitoring area is determined based on the point cloud data corresponding to the conductor and the ground in the point cloud category data.

[0163] Specifically, based on the point cloud data corresponding to the ground in the point cloud category data, a ground plane equation is constructed. Based on the point cloud coordinates of each point cloud corresponding to the conductor in the historical point cloud data and the point-to-plane distance between them and the ground plane equation, the statistical values ​​corresponding to the point-to-plane distances of all point clouds corresponding to the conductor are used as the conductor-to-ground distance of the transmission line within the monitoring area.

[0164] For example, the statistical value is the minimum or average value; there is no limitation on the statistical value used here.

[0165] In one specific embodiment, the distance between conductors of a transmission line is determined based on the point cloud data corresponding to the conductors in the point cloud category data.

[0166] Specifically, based on the point cloud data corresponding to the conductors in the point cloud category data, at least two conductor curve equations are constructed, and the minimum curve distance between the conductor curve equations corresponding to the two adjacent conductors is taken as the distance between the conductors of the transmission line.

[0167] In one specific embodiment, the maximum canopy height of vegetation within the monitoring area is determined based on the point cloud data corresponding to vegetation and ground in the point cloud category data.

[0168] Specifically, based on the point cloud data corresponding to the ground in the point cloud category data, a ground plane equation is constructed. Based on the point cloud coordinates of each point cloud corresponding to vegetation in the historical point cloud data and the point-to-plane distance between them and the ground plane equation, the maximum value of the point-to-plane distance of all point clouds corresponding to vegetation is taken as the maximum canopy height of vegetation in the monitoring area.

[0169] For example, the statistical value is the minimum or average value; there is no limitation on the statistical value used here.

[0170] S370. Determine the wildfire alarm level for the monitoring area based on flame height, distance of the conductor from the ground, distance between conductors, and maximum tree canopy height.

[0171] The technical features of S370 in this embodiment are the same or similar to those in the above embodiments, and will not be described again in this embodiment.

[0172] The technical solution of this embodiment inputs historical image data and historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data. Based on the point cloud category data, the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of the vegetation under the transmission lines in the monitoring area are determined. By utilizing the deep three-dimensional geometric features of the point cloud data and the two-dimensional visual features of the image data, the accuracy of the length measurement parameters used in the wildfire alarm process of the transmission corridor is improved, thereby further improving the accuracy of wildfire alarms in the transmission corridor.

[0173] Based on the above embodiments, optionally, the method further includes: acquiring historical image data of the inspection area collected by a visual acquisition device and historical point cloud data of the inspection area collected by a lidar for each inspection area in the power transmission corridor; inputting the historical image data and historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data; wherein, the point cloud category data includes the vegetation category to which each point cloud in the historical point cloud data belongs; determining the vegetation distribution data of the inspection area based on the point cloud category data; if the vegetation distribution data meets the preset distribution conditions, then the inspection area is designated as a monitoring area; if the vegetation distribution data does not meet the preset distribution conditions, then the inspection area is designated as a non-monitoring area.

[0174] For example, vegetation distribution data includes, but is not limited to, at least one of the following: the number of vegetation categories belonging to the flammable category, vegetation coverage, vegetation height, and vegetation density. Correspondingly, preset distribution conditions include, but are not limited to, at least one of the following: the number of vegetation categories belonging to the flammable category is greater than a preset number, vegetation coverage is greater than a coverage threshold, vegetation height is greater than a height threshold, and vegetation density is greater than a density threshold.

[0175] Specifically, vegetation coverage represents the percentage of the vertical projection area of ​​vegetation on the ground to the total area of ​​the inspection area, while vegetation density represents the amount of vegetation per unit area.

[0176] The advantage of this setup is that by integrating the data features of point cloud data and image data, the number of monitoring areas in the wildfire alarm process of the power transmission corridor is reduced, thereby improving the overall efficiency of wildfire monitoring in the power transmission corridor.

[0177] Figure 4 This is a flowchart illustrating a specific example of a wildfire warning method for a multimodal power transmission corridor provided by an embodiment of the present invention. Specifically, a camera and lidar are mounted on a drone to inspect the power transmission corridor, collecting point cloud data and image data for each inspected area. For each inspected area, the collected point cloud data and image data are preprocessed, and a point cloud classification model is used to determine the point cloud category data based on the preprocessed point cloud data and image data. Based on the point cloud category data, the vegetation distribution data of the inspected area is determined. Based on the vegetation distribution data of each inspected area in the power transmission corridor, at least one monitoring area that meets the preset distribution conditions is selected from multiple inspection areas. External parameter calibration is performed on the preprocessed point cloud data and image data corresponding to each monitoring area to obtain an external parameter calibration matrix.

[0178] In the daily monitoring of wildfire alarms along the power transmission corridor, cameras mounted on drones collect image data of the monitored area. Based on the image data and temperature sensors, the wildfire identification result for the monitored area is determined. The system then determines whether a wildfire has occurred. If not, the drone, equipped with cameras, collects image data of other monitored areas. If a wildfire has occurred, the wildfire alarm level for the monitored area is determined based on the corresponding extrinsic parameter calibration matrix and the image data.

[0179] The following are embodiments of the wildfire alarm device for multimode power transmission corridors provided by the present invention. This device and the wildfire alarm method for multimode power transmission corridors described above belong to the same inventive concept. For details not described in detail in the embodiments of the wildfire alarm device for multimode power transmission corridors, please refer to the content of the wildfire alarm method for multimode power transmission corridors described above.

[0180] Figure 5 This is a schematic diagram of a wildfire alarm device for a multimodal power transmission corridor, provided as an embodiment of the present invention. Figure 5 As shown, the device includes: a wildfire identification result determination module 410, an external parameter calibration matrix acquisition module 420, and a wildfire alarm level determination module 430.

[0181] Among them, the wildfire identification result determination module 410 is used to collect two-dimensional image data of the monitoring area in the power transmission corridor through a visual acquisition device, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data;

[0182] The extrinsic parameter calibration matrix acquisition module 420 is used to acquire the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area when the wildfire identification result indicates that a wildfire has occurred.

[0183] The wildfire alarm level determination module 430 is used to determine the flame height of a wildfire based on the external parameter calibration matrix and two-dimensional image data, and to determine the wildfire alarm level of the monitoring area based on the flame height.

[0184] The technical solution of this embodiment determines the wildfire identification result in the monitoring area of ​​the power transmission corridor based on the two-dimensional image data collected by the visual acquisition device. When the wildfire identification result indicates that a wildfire has occurred, the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area is obtained. Based on the extrinsic parameter calibration matrix and the two-dimensional image data, the flame height of the wildfire is determined. Based on the flame height, the wildfire alarm level of the monitoring area is determined. The use of visual acquisition device combined with three-dimensional point cloud system realizes the daily monitoring of wildfire alarms in the power transmission corridor, which solves the problem of high equipment cost of lidar for monitoring wildfires. While ensuring the accuracy of wildfire alarms, it also improves the response speed of wildfire alarms, thereby ensuring the safety of the power transmission corridor.

[0185] In an optional embodiment, the wildfire alarm level determination module 430 includes:

[0186] The flame height determination unit is used to project two-dimensional image data onto the three-dimensional point cloud system corresponding to the monitoring area using an external parameter calibration matrix to obtain three-dimensional projection data.

[0187] The height of the wildfire flames was determined based on the three-dimensional projection data.

[0188] In an optional embodiment, the extrinsic parameter calibration matrix acquisition module 420 includes:

[0189] The historical point cloud data acquisition unit is used to acquire historical image data of the monitoring area collected by the visual acquisition device and historical point cloud data of the monitoring area collected by the lidar; wherein, the lidar and the visual acquisition device are set on the same monitoring device.

[0190] The extrinsic parameter calibration matrix determination unit is used to perform extrinsic parameter calibration based on historical image data and historical point cloud data, and obtain the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0191] In an optional embodiment, the extrinsic parameter calibration matrix determination unit is specifically used for:

[0192] The reference calibration matrix determines the sub-unit, which is used to obtain the reference calibration matrix by performing initial extrinsic parameter calibration based on historical image data and historical point cloud data using a preset calibration algorithm;

[0193] The prediction classification result output subunit is used to input the reference calibration matrix, the visual intrinsic parameter matrix of the visual acquisition device, historical image data, and historical point cloud data into the untrained classification and recognition model to obtain the prediction classification result corresponding to the output historical point cloud data; wherein, the prediction classification result includes the projection classification result of each point cloud in the historical point cloud data;

[0194] The reference calibration matrix iterative adjustment subunit is used to iteratively adjust the reference calibration matrix in the classification and recognition model based on the predicted classification results.

[0195] The extrinsic calibration matrix determines the sub-unit, which is used to obtain the extrinsic calibration matrix between the visual acquisition device and the corresponding 3D point cloud system of the monitoring area in the trained classification and recognition model until the iteration termination condition is met.

[0196] In one optional embodiment, the reference calibration matrix iteratively adjusts the sub-unit, specifically for:

[0197] Obtain historical projection data corresponding to historical point cloud data in the classification and recognition model, and determine the distance cost function value based on historical projection data and historical image data; where historical projection data represents the projection result of historical point cloud data on the imaging plane of the visual acquisition device;

[0198] The classification loss function value is determined based on the predicted classification results and the actual classification results corresponding to the historical point cloud data.

[0199] The reference calibration matrix in the classification and recognition model is iteratively adjusted based on the distance cost function value and the classification loss function value.

[0200] The iteration termination conditions include convergence of the distance cost function value and / or convergence of the classification loss function value.

[0201] In an optional embodiment, the wildfire alarm level determination module 430 includes:

[0202] The maximum canopy height acquisition unit is used to acquire the distance between the conductors of the transmission lines and the ground, the distance between the conductors, and the maximum canopy height of the vegetation under the transmission lines within the monitoring area.

[0203] The wildfire alarm level determination unit is used to determine the wildfire alarm level of the monitoring area based on the flame height, the distance of the wires from the ground, the distance between the wires, and the maximum tree canopy height.

[0204] In one optional embodiment, the maximum canopy height acquisition unit includes:

[0205] The historical point cloud data acquisition subunit is used to acquire historical image data of the monitoring area collected by the visual acquisition device and historical point cloud data of the monitoring area collected by the lidar; wherein, the lidar and the visual acquisition device are set on the same monitoring device.

[0206] The point cloud category data output subunit is used to input historical image data and historical point cloud data into a pre-trained point cloud classification model to obtain the output point cloud category data. The point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs. The object categories in the point cloud category data include guide lines, ground, and vegetation.

[0207] The maximum canopy height determination subunit is used to determine the distance between the conductors and the ground, the distance between conductors, and the maximum canopy height of the vegetation under the transmission lines within the monitoring area, based on point cloud category data.

[0208] In one alternative embodiment, the point cloud classification model includes a visual feature network, a point cloud feature network, and a fusion output network;

[0209] Correspondingly, the point cloud category data output sub-unit is specifically used for:

[0210] Visual feature vectors are obtained by extracting features from historical image data using a visual feature network.

[0211] Point cloud feature vectors are obtained by extracting features from historical point cloud data using a point cloud feature network.

[0212] By using a fusion output network, visual feature vectors and point cloud feature vectors are fused to obtain a fused feature vector, and point cloud category data is output based on the fused feature vector.

[0213] In an optional embodiment, the wildfire identification result determination module 410 includes:

[0214] A two-dimensional image data output unit is used to input two-dimensional image data into a pre-trained wildfire recognition model; wherein, the wildfire recognition model includes a feature extraction network and a wildfire recognition network;

[0215] The motion feature vector extraction unit is used to extract visual feature vectors and motion feature vectors from two-dimensional image data through a feature extraction network.

[0216] The wildfire prediction output unit is used to output the wildfire prediction results based on visual feature vectors and motion feature vectors through the wildfire recognition network.

[0217] The wildfire identification result output unit is used to determine the wildfire identification result of the monitoring area based on the predicted wildfire results.

[0218] In one optional embodiment, the wildfire identification result output unit is specifically used for:

[0219] Acquire temperature data collected by temperature sensors within the monitoring area;

[0220] If the temperature data exceeds the temperature threshold and the predicted wildfire result is that a wildfire has occurred, then the wildfire identification result of the monitored area will be set to indicate that a wildfire has occurred.

[0221] The wildfire alarm device for the multimode power transmission corridor provided in this embodiment of the invention can execute the wildfire alarm method for the multimode power transmission corridor provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0222] Figure 6 This is a schematic diagram of a wildfire alarm system for a multimodal power transmission corridor provided in one embodiment of the present invention. The wildfire alarm system for the multimodal power transmission corridor in this embodiment provides services for the wildfire alarm method for the multimodal power transmission corridor in the above embodiment.

[0223] like Figure 6 As shown, the wildfire alarm system 500 for the multimodal power transmission corridor includes a visual acquisition device 510 and a terminal device 520 connected by communication; wherein, the visual acquisition device 510 is used to acquire two-dimensional image data of the monitoring area in the power transmission corridor.

[0224] In an optional embodiment, the wildfire alarm system 500 for the multimodal power transmission corridor also includes a lidar 530 that is communicatively connected to a terminal device 520 for collecting point cloud data of the monitoring area in the power transmission corridor.

[0225] In an optional embodiment, the wildfire alarm system 500 for the multimodal transmission corridor also includes monitoring equipment ( Figure 6 (Not shown in the image) is used to set up the visual acquisition device 510 and the lidar 530.

[0226] Figure 7 This is a schematic diagram of a terminal device provided according to one embodiment of the present invention. Terminal device 520 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. Terminal device 520 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0227] like Figure 7As shown, the terminal device 520 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor 11. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the terminal device 520. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0228] Multiple components in terminal device 520 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows terminal device 520 to exchange information or data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0229] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the wildfire alarm method for multimodal transmission corridors provided in the above embodiments.

[0230] In some embodiments, the wildfire alarm method for multimodal transmission corridors provided in the above embodiments can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on terminal device 520 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the wildfire alarm method for multimodal transmission corridors described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the wildfire alarm method for multimodal transmission corridors by any other suitable means (e.g., by means of firmware).

[0231] Various embodiments of the systems and techniques described above herein can be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0232] Computer programs for implementing the wildfire alarm method for multimodal power transmission corridors of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0233] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable storage medium. Examples of machine-readable storage media include, based on an electrical connection of at least one wire, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a terminal device having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the terminal device. Other types of devices can also provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0235] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0236] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0237] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0238] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for wildfire alerting of a multi-modal power transmission corridor, the method comprising: The method comprises the following steps: acquiring two-dimensional image data of a monitoring area in a power transmission corridor by a visual acquisition device, and determining a forest fire identification result of the monitoring area according to the two-dimensional image data; in a case where the forest fire identification result indicates that a forest fire occurs, acquiring an extrinsic calibration matrix between the visual acquisition device and a three-dimensional point cloud system corresponding to the monitoring area; determining a flame height of the forest fire according to the extrinsic calibration matrix and the two-dimensional image data, and determining a forest fire alarm level of the monitoring area according to the flame height; the acquiring of the extrinsic calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area comprises: acquiring historical image data of the monitoring area collected by the visual acquisition device, and acquiring historical point cloud data of the monitoring area collected by a laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device; performing extrinsic calibration according to the historical image data and the historical point cloud data to obtain the extrinsic calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area; the performing of the extrinsic calibration according to the historical image data and the historical point cloud data to obtain the extrinsic calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area comprises: performing initial extrinsic calibration according to the historical image data and the historical point cloud data by using a preset calibration algorithm to obtain a reference calibration matrix; inputting the reference calibration matrix, a visual intrinsic parameter matrix of the visual acquisition device, the historical image data and the historical point cloud data into an untrained classification recognition model to obtain an output predicted classification result corresponding to the historical point cloud data; wherein the predicted classification result comprises a projection classification result of each point cloud in the historical point cloud data; iteratively adjusting the reference calibration matrix in the classification recognition model according to the predicted classification result; until an iteration end condition is met, obtaining the extrinsic calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area in the trained classification recognition model.

2. The method of claim 1, wherein, the determining of the flame height of the forest fire according to the extrinsic calibration matrix and the two-dimensional image data comprises: projecting the two-dimensional image data into the three-dimensional point cloud system corresponding to the monitoring area by using the extrinsic calibration matrix to obtain three-dimensional projection data; determining the flame height of the forest fire according to the three-dimensional projection data.

3. The method of claim 1, wherein, the iteratively adjusting of the reference calibration matrix in the classification recognition model according to the predicted classification result comprises: acquiring historical projection data corresponding to the historical point cloud data in the classification recognition model, and determining a distance cost function value according to the historical projection data and the historical image data; wherein the historical projection data represents a projection result of the historical point cloud data in an imaging plane of the visual acquisition device; determining a classification loss function value according to the predicted classification result and a true classification result corresponding to the historical point cloud data; According to the distance cost function value and the classification loss function value, the reference calibration matrix in the classification recognition model is iteratively adjusted; The iteration end condition includes convergence of the distance cost function value and / or convergence of the classification loss function value.

4. The method of claim 1, wherein, The determination of the forest fire warning level of the monitoring area according to the flame height comprises: Obtaining the conductor-to-ground distance, the conductor-to-conductor distance of the power transmission line in the monitoring area, and the maximum tree crown height of the vegetation under the power transmission line; According to the flame height, the conductor-to-ground distance, the conductor-to-conductor distance and the maximum tree crown height, the forest fire warning level of the monitoring area is determined.

5. The method of claim 4, wherein, The obtaining of the conductor-to-ground distance, the conductor-to-conductor distance of the power transmission line in the monitoring area, and the maximum tree crown height of the vegetation under the power transmission line comprises: Obtaining historical image data of the monitoring area collected by the visual acquisition device and obtaining historical point cloud data of the monitoring area collected by the laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device; The historical image data and the historical point cloud data are input into a pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data contains the object category to which each point cloud in the historical point cloud data belongs, and the object category in the point cloud category data includes conductor, ground and vegetation; According to the point cloud category data, the conductor-to-ground distance, the conductor-to-conductor distance of the power transmission line in the monitoring area, and the maximum tree crown height of the vegetation under the power transmission line are determined.

6. The method of claim 5, wherein, The point cloud classification model comprises a visual feature network, a point cloud feature network and a fusion output network; Correspondingly, the historical image data and the historical point cloud data are input into a pre-trained point cloud classification model to obtain output point cloud category data; The visual feature vector is obtained by feature extraction of the historical image data through the visual feature network; The point cloud feature vector is obtained by feature extraction of the historical point cloud data through the point cloud feature network; The fusion feature vector is obtained by feature fusion of the visual feature vector and the point cloud feature vector through the fusion output network, and the point cloud category data is output according to the fusion feature vector.

7. The method of claim 1, wherein, The determination of the forest fire recognition result of the monitoring area according to the two-dimensional image data comprises: The two-dimensional image data is input into a pre-trained forest fire recognition model; wherein the forest fire recognition model comprises a feature extraction network and a forest fire recognition network; The visual feature vector and the motion feature vector are obtained by feature extraction of the two-dimensional image data through the feature extraction network; The predicted forest fire result is output according to the visual feature vector and the motion feature vector through the forest fire recognition network; The forest fire recognition result of the monitoring area is determined according to the predicted forest fire result.

8. The method of claim 7, wherein, The determination of the forest fire recognition result of the monitoring area according to the predicted forest fire result comprises: Obtaining temperature data collected by a temperature sensor in the monitoring area; If the temperature data exceeds the temperature threshold value and the predicted wildfire result is that a wildfire occurs, a wildfire identification result of the monitoring area is set as that a wildfire occurs.

9. A wildfire alarm device for a multi-mode power transmission corridor, characterized in that, Comprise: A wildfire identification result determination module is configured to collect two-dimensional image data of a monitoring area in a power transmission corridor by a visual collection device, and determine a wildfire identification result of the monitoring area according to the two-dimensional image data; An extrinsic calibration matrix acquisition module is configured to acquire an extrinsic calibration matrix between the visual collection device and a three-dimensional point cloud system corresponding to the monitoring area in a case that the wildfire identification result is that a wildfire occurs; A wildfire alarm level determination module is configured to determine a flame height of a wildfire according to the extrinsic calibration matrix and the two-dimensional image data, and determine a wildfire alarm level of the monitoring area according to the flame height; The extrinsic calibration matrix acquisition module comprises: A historical point cloud data acquisition unit is configured to acquire historical image data of the monitoring area collected by the visual collection device, and acquire historical point cloud data of the monitoring area collected by a laser radar; wherein the laser radar and the visual collection device are arranged on the same monitoring device; An extrinsic calibration matrix determination unit is configured to perform extrinsic calibration according to the historical image data and the historical point cloud data to obtain the extrinsic calibration matrix between the visual collection device and the three-dimensional point cloud system corresponding to the monitoring area; The extrinsic calibration matrix determination unit is specifically configured to: A reference calibration matrix determination subunit is configured to perform initial extrinsic calibration according to the historical image data and the historical point cloud data to obtain a reference calibration matrix by using a preset calibration algorithm; A predicted classification result output subunit is configured to input the reference calibration matrix, a visual intrinsic parameter matrix of the visual collection device, the historical image data and the historical point cloud data into an untrained classification recognition model to obtain an output predicted classification result corresponding to the historical point cloud data; wherein the predicted classification result comprises a projection classification result of each point cloud in the historical point cloud data; A reference calibration matrix iterative adjustment subunit is configured to perform iterative adjustment on the reference calibration matrix in the classification recognition model according to the predicted classification result; An extrinsic calibration matrix determination subunit is configured to obtain the extrinsic calibration matrix between the visual collection device and the three-dimensional point cloud system corresponding to the monitoring area in the trained classification recognition model until an iteration end condition is met.

10. A wildfire alert system for a multi-modal power transmission corridor, the system comprising: The wildfire alarm system of the multi-modal power transmission corridor comprises a visual collection device and a terminal device in communication connection; The visual collection device is configured to collect two-dimensional image data of a monitoring area in a power transmission corridor; The terminal device comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the wildfire alarm method of the multi-modal power transmission corridor according to any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the method for wildfire warning of multi-modal power transmission corridor according to any one of claims 1-8 when executed.

12. A computer program product comprising a computer program which, when executed by a processor, implements the method for wildfire warning of multi-modal power transmission corridor according to any one of claims 1-8.

Images