Mountain fire alarm method, device and system for multi-mode power transmission corridor and medium

Abstract

The invention discloses a forest fire alarm method, device and system for a multi-mode power transmission corridor and a medium. The method comprises the following steps: acquiring two-dimensional image data of a monitoring area in a power transmission corridor through visual acquisition equipment, and determining a forest fire identification result of the monitoring area according to the two-dimensional image data; acquiring an external parameter calibration matrix between the visual acquisition equipment and a three-dimensional point cloud system corresponding to the monitoring area under the condition that the forest fire recognition result is that the forest fire occurs; and determining the flame height of the mountain fire according to the external parameter calibration matrix and the two-dimensional image data, and determining the mountain fire alarm level of the monitoring area according to the flame height, thereby solving the problem of high equipment cost of monitoring the mountain fire by the laser radar, and improving the response speed of the mountain fire alarm while ensuring the accuracy of the mountain fire alarm.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power inspection, and in particular to a mountain fire alarm method, device, system and medium for a multi-modal power transmission corridor. Background Art

[0002] With the continuous development of the power system, the coverage of transmission corridors is getting wider and wider, and a large number of transmission lines pass through mountainous areas, forests and other areas prone to wildfires. Wildfires pose a serious threat to the safe operation of transmission corridors, which may cause accidents such as tripping and damage of transmission lines, thus affecting the stability and reliability of power supply.

[0003] At present, laser radar is mainly used to collect three-dimensional point cloud data for wildfire monitoring. However, the amount of three-dimensional point cloud data is huge, and high-performance hardware facilities are required in multiple technical links such as data storage, data transmission and data analysis. As a result, the equipment cost of wildfire monitoring is high and the real-time monitoring is poor. Summary of the invention

[0004] The embodiments of the present invention provide a wildfire alarm method, device, system and medium for a multi-modal power transmission corridor to solve the problem of high equipment cost of laser radar for monitoring wildfires and improve the response speed of wildfire alarms.

[0005] According to an embodiment of the present invention, a wildfire alarm method for a multi-modal power transmission corridor is provided, the method comprising:

[0006] Collecting two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determining a wildfire identification result of the monitoring area based on the two-dimensional image data;

[0007] When the wildfire identification result is that a wildfire has occurred, obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area;

[0008] The flame height of the wildfire is determined according to the external parameter calibration matrix and the two-dimensional image data, and the wildfire alarm level of the monitoring area is determined according to the flame height.

[0009] According to another embodiment of the present invention, a wildfire warning device for a multi-modal power transmission corridor is provided, the device comprising:

[0010] A wildfire identification result determination module is used to collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data;

[0011] An external parameter calibration matrix acquisition module is used to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area when the wildfire identification result is a wildfire occurrence;

[0012] The wildfire warning level determination module is used to determine the flame height of the wildfire according to the external parameter calibration matrix and the two-dimensional image data, and determine the wildfire warning level of the monitoring area according to the flame height.

[0013] According to another embodiment of the present invention, a wildfire warning system for a multimodal power transmission corridor is provided, the wildfire warning system for a multimodal power transmission corridor includes a visual acquisition device and a terminal device in communication connection;

[0014] Wherein, the visual acquisition device is used to collect two-dimensional image data of the monitoring area in the transmission corridor;

[0015] The terminal device includes: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the wildfire alarm method for the multimodal transmission corridor described in any embodiment of the present invention.

[0016] According to another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the wildfire warning method for a multimodal transmission corridor described in any embodiment of the present invention when executed.

[0017] According to another embodiment of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the wildfire warning method for a multimodal transmission corridor described in any embodiment of the present invention.

[0018] An embodiment of the above invention has at least the following advantages or beneficial effects:

[0019] The wildfire recognition result of the monitoring area in the transmission corridor is determined based on the two-dimensional image data collected by the visual acquisition device. When the wildfire recognition result is a wildfire, the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area is obtained. The flame height of the wildfire is determined based on the external parameter calibration matrix and the two-dimensional image data. According to the flame height, the wildfire alarm level of the monitoring area is determined. The visual acquisition device is combined with the three-dimensional point cloud system to realize the daily monitoring of the wildfire alarm in the transmission corridor, which solves the problem of high cost of laser radar equipment for monitoring wildfires. While ensuring the accuracy of the wildfire alarm, the response speed of the wildfire alarm is improved, thereby ensuring the safety of the transmission corridor.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of a wildfire warning method for a multi-modal power transmission corridor provided by an embodiment of the present invention;

[0023] Figure 2 A flow chart of another multi-modal transmission corridor wildfire warning method provided by one embodiment of the present invention;

[0024] Figure 3 A flow chart of another multi-modal transmission corridor wildfire warning method provided by one embodiment of the present invention;

[0025] Figure 4 A flowchart of a specific example of a wildfire warning method for a multi-modal power transmission corridor provided by an embodiment of the present invention;

[0026] Figure 5 A flow chart of a wildfire warning device for a multi-modal power transmission corridor provided by one embodiment of the present invention;

[0027] Figure 6 A schematic structural diagram of a wildfire warning system for a multi-modal power transmission corridor provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0031] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0032] Figure 1 This is a flowchart of a multi-modal transmission corridor wildfire alarm method provided by an embodiment of the present invention. This embodiment is applicable to the situation of monitoring and alarming wildfires in transmission corridors. The method can be executed by a multi-modal transmission corridor wildfire alarm device. The multi-modal transmission corridor wildfire alarm device can be implemented in the form of hardware and / or software. The multi-modal transmission corridor wildfire alarm device can be configured in a terminal device. Figure 1 As shown, the method includes:

[0033] S110. Collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine a wildfire identification result of the monitoring area based on the two-dimensional image data.

[0034] Specifically, the visual acquisition device refers to a device for acquiring two-dimensional image data, wherein the two-dimensional image data includes one two-dimensional image or multiple two-dimensional images in a time series. Exemplarily, the visual acquisition device may be a camera or a video camera, but is not limited to the exemplary case.

[0035] Exemplarily, the visual acquisition equipment can be fixed on a fixed device in the monitoring area, or it can be mounted on a movable device. For example, the fixed device can be a pole tower, and the movable device can be a drone, a vehicle-mounted monitoring device, or a mobile meteorological control device, but it is not limited to the example scenario.

[0036] In an optional embodiment, a wildfire identification result in a monitoring area is determined based on two-dimensional image data, including: using a histogram equalization algorithm to determine pixel value distribution data of the two-dimensional image data; if the pixel distribution data meets a preset distribution condition, the wildfire identification result in the monitoring area is set to a wildfire occurrence; if the pixel distribution data does not meet the preset distribution condition, the wildfire identification result in the monitoring area is set to a wildfire non-occurrence.

[0037] Specifically, the preset distribution condition is that the pixel ratio corresponding to the preset pixel range reaches a ratio threshold, wherein the pixel ratio represents the ratio of the number of pixels within the preset pixel range to the total number of pixels in the two-dimensional image data.

[0038] The preset pixel range is related to the adopted color system, such as the color system including but not limited to grayscale system, RGB system, CMYK system, HSV system and HSL system, etc. Exemplarily, when the adopted color system is a grayscale system, the preset pixel range may be 200-255.

[0039] In another optional embodiment, determining a wildfire identification result in a monitoring area based on two-dimensional image data includes: inputting the two-dimensional image data into a pre-trained wildfire identification model; wherein the wildfire identification model includes a feature extraction network and a wildfire identification network; performing feature extraction on the two-dimensional image data through the feature extraction network to obtain a visual feature vector and a motion feature vector; outputting a predicted wildfire result based on the visual feature vector and the motion feature vector through the wildfire identification network; and determining a wildfire identification result in the monitoring area based on the predicted wildfire result.

[0040] Specifically, the visual feature vector is used to describe information with visual significance in the two-dimensional image data. Exemplarily, the visual feature vector includes but is not limited to a color feature vector, a texture feature vector, and a spatial feature vector, among which the spatial feature vector is used to describe the relative position information between multiple objects in the two-dimensional image data, and can also be used to describe the shape information of each object in the two-dimensional image data.

[0041] Specifically, the motion feature vector is used to describe the motion feature information of the object in the two-dimensional image data. Exemplarily, the motion feature vector includes but is not limited to a displacement feature vector, a speed feature vector, a rotation feature vector, a trajectory feature vector, and the like.

[0042] In an optional embodiment, the feature extraction network includes a visual feature module for extracting a visual feature vector and a motion feature module for extracting a motion feature vector. Exemplarily, the visual feature module may be a convolutional neural network, a deep residual shrinkage network, a generative adversarial network, or a capsule network, and the motion feature module may be a recursive neural network, an optical flow algorithm, or a motion detection algorithm, but is not limited to the above example scenarios.

[0043] In another optional embodiment, the wildfire recognition model is a two-stream convolutional neural network or a three-dimensional convolutional neural network.

[0044] The advantage of extracting visual feature vectors and motion feature vectors from two-dimensional image data is that since flames have unique visual and motion features, such as the color of flames consisting of orange, red, and yellow, and the irregular shaking and jumping of flames, the accuracy of wildfire identification results can be further improved.

[0045] In an optional embodiment, determining the wildfire identification result of the monitoring area according to the wildfire prediction result includes: using the wildfire prediction result as the wildfire identification result of the monitoring area.

[0046] In another optional embodiment, based on the predicted wildfire result, the wildfire identification result of the monitoring area is determined, including: obtaining temperature data collected by the temperature sensor in the monitoring area; if the temperature data exceeds the temperature threshold and the predicted wildfire result is the occurrence of a wildfire, then the wildfire identification result of the monitoring area is set to the occurrence of a wildfire.

[0047] Based on the above embodiment, optionally, determining the wildfire identification result of the monitoring area according to the predicted wildfire result also includes: if the temperature data does not exceed the temperature threshold, or the predicted wildfire result is that no wildfire has occurred, the wildfire identification result of the monitoring area is set to no wildfire has occurred.

[0048] The advantage of setting up temperature sensor joint identification is that the accuracy and reliability of wildfire identification results are further improved by multimodally fusing two-dimensional image data with temperature data.

[0049] S120. When the wildfire identification result is that a wildfire has occurred, obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0050] Specifically, the three-dimensional point cloud system corresponding to the monitoring area represents the three-dimensional space system to which the discrete points on the surface of objects in the monitoring area belong, and the external parameter calibration matrix represents the spatial transformation relationship between the visual image system and the three-dimensional point cloud system in which the visual acquisition device is located when acquiring in the monitoring area, specifically including the rotation transformation relationship and the translation transformation relationship.

[0051] Specifically, the external parameter calibration matrix T is expressed as:

[0052]

[0053] Among them, R represents the rotation transformation matrix and t represents the translation vector.

[0054] Based on the above embodiment, optionally, the method further includes: when the wildfire identification result is that no wildfire has occurred, acquiring other monitoring areas in the transmission corridor, and returning to execute S110.

[0055] S130, determining the height of the wildfire flame according to the external parameter calibration matrix and the two-dimensional image data, and determining the wildfire alarm level in the monitoring area according to the flame height.

[0056] Specifically, flame height refers to the vertical distance between the highest and lowest points of a wildfire.

[0057] In an optional embodiment, the flame height of the wildfire is determined based on an external parameter calibration matrix and two-dimensional image data, including: performing wildfire contour recognition on the two-dimensional image data to obtain the top pixel points and bottom pixel points of the wildfire; using the external parameter calibration matrix to convert the top pixel points and the bottom pixel points into a three-dimensional point cloud system corresponding to the monitoring area to obtain a top point cloud and a bottom point cloud; and using the z-coordinate difference between the top point cloud and the bottom point cloud as the flame height of the wildfire.

[0058] In another optional embodiment, the flame height of the wildfire is determined based on an external parameter calibration matrix and two-dimensional image data, including: using an external parameter calibration matrix to project the two-dimensional image data into a three-dimensional point cloud system corresponding to the monitoring area to obtain three-dimensional projection data; and determining the flame height of the wildfire based on the three-dimensional projection data.

[0059] Specifically, when the extrinsic calibration matrix represents the spatial transformation relationship from the visual image system to the three-dimensional point cloud system, for example, the pixel point P in the two-dimensional image data C In the three-dimensional projection data, it can be represented as a point cloud P L , where P L =RP C +t. When the external calibration matrix represents the spatial transformation relationship from the three-dimensional point cloud system to the visual image system, P L =R - 1 P C -t.

[0060] In an optional embodiment, the flame height of the wildfire is determined based on the three-dimensional projection data, including: performing flame contour recognition on the three-dimensional projection data to obtain the top z coordinate and the bottom z coordinate of the wildfire, and using the coordinate difference between the top z coordinate and the bottom z coordinate as the flame height of the wildfire.

[0061] In an optional embodiment, the wildfire alarm level of the monitoring area is determined according to the flame height, including: obtaining the distance between the conductors of the transmission lines in the monitoring area and the ground, the distance between the conductors, and the maximum crown height of the vegetation under the transmission lines; determining the wildfire alarm level of the monitoring area according to the flame height, the distance between the conductors and the ground, the distance between the conductors, and the maximum crown height.

[0062] In a specific embodiment, the wildfire alarm level of the monitoring area is determined based on the flame height, the distance of the wire from the ground, the distance between the wires and the maximum tree crown height, including: determining the line-to-ground breakdown risk and the phase-to-phase breakdown risk based on the flame height, the distance of the wire from the ground, the distance between the wires and the maximum tree crown height; determining the wildfire alarm level of the monitoring area based on the line-to-ground breakdown risk and the phase-to-phase breakdown risk.

[0063] In one embodiment, the line-to-ground breakdown risk and the phase-to-phase breakdown risk are determined based on the flame height, the distance of the conductor from the ground, the distance between the conductors and the maximum tree crown height, including: when the transmission line in the monitoring area is an AC line system, the line-to-ground voltage peak and the phase-to-phase voltage peak are determined based on the line voltage of the transmission line; the line-to-ground insulation strength and the phase-to-phase insulation strength are determined based on the flame height, the distance of the conductor from the ground, the distance between the conductors and the maximum tree crown height; the ratio between the line-to-ground voltage peak and the line-to-ground insulation strength is used as the line-to-ground breakdown risk, and the ratio between the phase-to-phase voltage peak and the phase-to-phase insulation strength is used as the phase-to-phase breakdown risk.

[0064] In an AC line system, for example, the line-to-ground voltage peak value U a1 Satisfies the following formula:

[0065]

[0066] Phase-to-phase voltage peak value U a2 Satisfies the following formula:

[0067]

[0068] Among them, U n Indicates the line voltage of the transmission line.

[0069] For example, the line-to-ground breakdown risk P a1 Satisfies the following formula:

[0070]

[0071] Phase breakdown risk P a2 Satisfies the following formula:

[0072]

[0073] Among them, U 1acIndicates the line-to-ground insulation strength of the AC line system, U 2ac Indicates the phase-to-phase insulation strength of the AC line system.

[0074] Specifically, the line-to-ground breakdown voltage and the phase-to-phase breakdown voltage are determined based on the flame height, the distance between the conductors and the ground, the distance between the conductors and the maximum tree crown height, and the line-to-ground insulation strength and the phase-to-phase insulation strength are determined based on the line-to-ground breakdown voltage and the phase-to-phase breakdown voltage.

[0075] In the AC line system, the line-to-ground insulation strength U 1ac Depends on the line-to-ground gap breakdown voltage U g Breakdown voltage U between the ground wire and the s The smaller value of U 1ac =min(U g , U s ). Phase-to-phase insulation strength and phase-to-phase breakdown voltage U p Same, that is, U 2ac =U p Among them, the line-to-ground gap breakdown voltage U g Indicates the voltage value when electrical breakdown occurs between the conductor and the ground, the ground wire gap breakdown voltage U s Indicates the voltage value when electrical breakdown occurs between the conductor and the ground wire, the phase-to-phase breakdown voltage U p Indicates the voltage value when electrical breakdown occurs between two conductors.

[0076] In another embodiment, the line-to-ground breakdown risk and the phase-to-phase breakdown risk are determined based on the flame height, the distance of the conductor from the ground, the distance between the conductors, and the maximum tree crown height, including: when the transmission line in the monitoring area is a DC line system, the line-to-ground voltage and the phase-to-phase voltage are determined based on the rated voltage of the transmission line; the line-to-ground insulation strength and the phase-to-phase insulation strength are determined based on the flame height, the distance of the conductor from the ground, the distance between the conductors, and the maximum tree crown height; the ratio between the line-to-ground voltage and the line-to-ground insulation strength is used as the line-to-ground breakdown risk, and the ratio between the phase-to-phase voltage and the phase-to-phase insulation strength is used as the phase-to-phase breakdown risk.

[0077] In a DC line system, the line-to-ground voltage is a single-stage operating voltage. For example, the line-to-ground voltage U d1 Satisfies the following formula:

[0078] U d1 =U L

[0079] The phase-to-phase voltage is the voltage between two poles. For example, the phase-to-phase voltage U d2 Satisfies the following formula:

[0080] U d2 =2×U L

[0081] Among them, U L Indicates the rated voltage of the transmission line.

[0082] For example, the line-to-ground breakdown risk P d1 Satisfies the following formula:

[0083]

[0084] Phase breakdown risk P d2 Satisfies the following formula:

[0085]

[0086] Among them, U 1dc Indicates the line-to-ground insulation strength of the DC line system, U 2dc Indicates the phase-to-phase insulation strength of the DC line system.

[0087] Specifically, the line-to-ground breakdown voltage and the phase-to-phase breakdown voltage are determined based on the flame height, the distance between the conductors and the ground, the distance between the conductors and the maximum tree crown height, and the line-to-ground insulation strength and the phase-to-phase insulation strength are determined based on the line-to-ground breakdown voltage and the phase-to-phase breakdown voltage.

[0088] In the DC line system, the line-to-ground insulation strength U 1dc Depends on the line-to-ground gap breakdown voltage U g Breakdown voltage U between the ground wire and the s The smaller value of U 1ac =C e ×min(U g ,U s ). Phase-to-phase insulation strength and phase-to-phase breakdown voltage U p Related, that is, U 2ac =C e ×U p Among them, C e represents the voltage waveform correction factor, for example, C e It can be 1.19, but is not limited to the example case.

[0089] On the basis of the above embodiment, optionally, the line-to-ground gap breakdown voltage and the phase-to-phase breakdown voltage are determined according to the flame height, the distance of the conductor from the ground, the distance between the conductors and the maximum crown height, including: taking the sum of the flame height and the maximum crown height as the flame height from the ground; if the flame height from the ground is greater than or equal to the distance of the conductor from the ground, then determining the line-to-ground gap breakdown voltage and the phase-to-phase breakdown voltage according to the distance of the conductor from the ground, the maximum crown height and the average flame tolerance field strength; if the flame height from the ground is less than the distance of the conductor from the ground, then determining the line-to-ground gap breakdown voltage and the phase-to-phase breakdown voltage according to the flame height, the distance of the conductor from the ground, the maximum crown height, the average flame tolerance field strength and the average smoke tolerance field strength.

[0090] In a specific embodiment, the height of the flame from the ground is greater than or equal to the distance between the wire and the ground, indicating that the flame is completely bridged. Since the flame runs through the entire gap when the flame is completely bridged, the line-to-ground gap breakdown voltage U g And the phase-to-phase breakdown voltage U p Related to the average flame tolerance field strength.

[0091] For example, the line-to-ground gap breakdown voltage U g Satisfies the following formula:

[0092] U g =(H l -H t )E f

[0093] Phase-to-phase breakdown voltage U p Satisfies the following formula:

[0094] U p =d p E f

[0095] Among them, H l Indicates the distance between the conductor and the ground, H t represents the maximum crown height, E f Indicates the average flame tolerance field strength, d p Indicates the distance between conductors.

[0096] In another specific embodiment, the height of the flame from the ground is less than the distance between the wire and the ground, indicating that the flame is not completely bridged. Since the flame is not completely bridged, the flame only occupies part of the gap, so the line-to-ground gap breakdown voltage U g And the phase-to-phase breakdown voltage U p It is not only related to the average tolerance field strength of the flame, but also to the average tolerance field strength of the smoke outside the flame.

[0097] For example, the line-to-ground gap breakdown voltage U g Satisfies the following formula:

[0098] U g =H f E f +(H l -H t -H f )E s

[0099] In a specific embodiment, the phase-to-phase breakdown voltage U p Satisfies the following formula:

[0100] U p =d p E s

[0101] Among them, H f Indicates the flame height, E s Indicates the average smoke tolerance field strength.

[0102] Among them, the average smoke tolerance field strength E s Satisfies the following formula:

[0103]

[0104] Among them, d s Indicates the gap distance, d s =H l -H t -H f .

[0105] Specifically, when d s ≥4, the average smoke tolerance field strength E s Depends on s , indicating that a larger gap distance requires a higher electric field strength. s <4, the average smoke tolerance field strength E s is a fixed value, indicating that at a smaller gap distance, the average smoke tolerance field strength E s tends to a stable value.

[0106] In another specific embodiment, when the flame is not completely bridged, the phase-to-phase breakdown voltage U p Satisfies the following formula:

[0107]

[0108] Among them, E a Indicates the average withstand field strength of standard air, the value is 359.2kV / m, C a Indicates the altitude correction factor, C g Indicates the particle density correction factor in the smoke area, C σ Represents the air density, C hIndicates the air humidity correction factor.

[0109] On the basis of the above embodiment, optionally, the wildfire alarm level of the monitoring area is determined according to the line-to-ground breakdown risk and the phase-to-phase breakdown risk, including: obtaining the statistical breakdown risk corresponding to the line-to-ground breakdown risk and the phase-to-phase breakdown risk, and determining the wildfire alarm level of the monitoring area according to the statistical breakdown risk.

[0110] Specifically, when the line-to-ground breakdown risk is greater than 1, it means that the peak voltage or operating voltage of the conductor to the ground has exceeded its insulation strength, and the risk of the transmission line tripping is high. When the phase-to-phase breakdown risk is greater than 1, it means that the peak voltage or operating voltage between the conductors has exceeded its insulation strength, and the risk of the transmission line tripping is high.

[0111] In this embodiment, the statistical breakdown risk is the maximum or average value corresponding to the line-to-ground breakdown risk and the phase-to-phase breakdown risk. Exemplarily, the wildfire warning level can be extremely high risk, high risk, medium risk and low risk. The number of wildfire warning levels and the statistical breakdown risk range corresponding to each wildfire warning level are not limited here, and can be customized according to actual needs.

[0112] The technical solution of this embodiment determines the wildfire recognition result of the monitoring area in the transmission corridor based on the two-dimensional image data collected by the visual acquisition device. When the wildfire recognition result is the occurrence of a wildfire, an external parameter calibration matrix is ​​obtained between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area. The flame height of the wildfire is determined based on the external parameter calibration matrix and the two-dimensional image data. The wildfire alarm level of the monitoring area is determined based on the flame height. The visual acquisition device is combined with the three-dimensional point cloud system to realize daily monitoring of wildfire alarms in the transmission corridor, which solves the problem of high equipment cost of laser radar for monitoring wildfires. While ensuring the accuracy of wildfire alarms, the response speed of wildfire alarms is improved, thereby ensuring the safety of the transmission corridor.

[0113] Figure 2 A flowchart of another multi-modal transmission corridor wildfire alarm method provided by an embodiment of the present invention, this embodiment further refines the "obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area" in the above embodiment. In this embodiment, obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area includes: obtaining historical image data of the monitoring area collected by the visual acquisition device, and obtaining historical point cloud data of the monitoring area collected by the laser radar; wherein the laser radar and the visual acquisition device are set on the same monitoring device; performing external parameter calibration according to the historical image data and the historical point cloud data, and obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0114] like Figure 2 As shown, the method includes:

[0115] S210. Collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine a wildfire identification result of the monitoring area based on the two-dimensional image data.

[0116] S210 in this embodiment is similar to that in the above embodiment. Figure 1 The S110 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0117] S220. When the wildfire identification result is that a wildfire has occurred, obtain historical image data of the monitoring area collected by the visual acquisition device, and obtain historical point cloud data of the monitoring area collected by the laser radar.

[0118] In this embodiment, the laser radar and the visual acquisition device are arranged on the same monitoring device, wherein the monitoring device can be a movable device or a fixed device in the monitoring area.

[0119] The transmitting unit in the laser radar emits laser beams with specific wavelengths, frequencies and energies, which can propagate in the air. When the laser beam contacts an object, it will be reflected, scattered or absorbed. The receiving unit in the laser radar collects the reflected or scattered laser signals, and obtains the distance and angle information by measuring the distance and angle of the laser signals. The distance and angle information can be combined to locate the three-dimensional coordinates of a surface point of the object, and multiple surface points constitute three-dimensional point cloud data.

[0120] Specifically, the collection time corresponding to the historical image data and the historical point cloud data is within a time error range to ensure the time consistency of the historical image data and the historical point cloud data.

[0121] In an optional embodiment, the method further includes: performing image preprocessing on the historical image data to obtain preprocessed historical image data. Exemplarily, image preprocessing includes but is not limited to image enhancement and image noise reduction. Among them, image enhancement can be implemented using a histogram equalization algorithm and / or a sharpening algorithm. The histogram equalization algorithm can improve the image contrast of the historical image data, making the details in the historical image data more prominent, thereby improving the visibility of the object objects in the historical image data. The sharpening algorithm can be implemented using a Laplace filter to enhance the edges and details of the object objects in the historical image data. Exemplarily, image noise reduction can use a Gaussian filter, a median filter or a mean filter, among others, wherein a Gaussian filter is a linear smoothing filter that can remove noise in the historical image data and retain the edge and detail information of the object objects.

[0122] In an optional embodiment, the method further includes: performing point cloud preprocessing on the historical point cloud data to obtain preprocessed historical point cloud data. Exemplarily, point cloud preprocessing includes but is not limited to point cloud denoising, point cloud reconstruction and point cloud enhancement. Point cloud denoising can be implemented using a statistical filter (such as a mean filter or a median filter), or a voxel grid algorithm. Specifically, the historical point cloud data is divided into voxel grids, and the point clouds in each voxel grid are downsampled to achieve a point cloud denoising effect. Point cloud reconstruction can be implemented using interpolation algorithms such as Laplace smoothing to fill in missing point clouds in the historical point cloud data, thereby improving the integrity and accuracy of the historical point cloud data. When performing point cloud enhancement, normal vector estimation algorithms and point cloud segmentation algorithms can be used.

[0123] There is no limitation on image preprocessing and point cloud preprocessing here, and the specific settings can be customized according to actual needs.

[0124] S230. Perform external parameter calibration based on historical image data and historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0125] In an optional embodiment, external parameter calibration is performed based on historical image data and historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area, including: using a preset calibration algorithm, performing external parameter calibration based on historical image data and historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0126] Exemplarily, the preset calibration algorithms include but are not limited to AprilTag calibration method, iterative closest point algorithm (Iterative Closest Point, ICP) and normal distribution transformation algorithm (Normal Distributions Transform, NDT) and the like.

[0127] In another optional embodiment, external parameter calibration is performed based on historical image data and historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area, including: using a preset calibration algorithm to perform initial external parameter calibration based on historical image data and historical point cloud data to obtain a reference calibration matrix; inputting the reference calibration matrix, the visual intrinsic parameter matrix of the visual acquisition device, the historical image data and the historical point cloud data into an untrained classification and recognition model to obtain a predicted classification result corresponding to the output historical point cloud data; iteratively adjusting the reference calibration matrix in the classification and recognition model based on the predicted classification result; until the iteration end condition is met, the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area in the trained classification and recognition model is obtained.

[0128] In this embodiment, the predicted classification result includes the projection classification result of each point cloud in the historical point cloud data, and the projection classification result includes the probability of whether the projection point of the point cloud in the imaging plane of the visual acquisition device is within the visual cone and / or belongs to a certain object category. The object category is a wire, ground, sky, column or vegetation category, etc.

[0129] In a specific embodiment, the classification recognition model projects the historical point cloud data onto the imaging plane of the visual acquisition device according to the reference calibration matrix and the visual intrinsic parameter matrix of the visual acquisition device to obtain the historical projection data. For each projection point in the historical projection data, determine whether the point cloud corresponding to the projection point is within the visual cone according to the projection coordinates of the projection point; obtain the pixel point corresponding to the projection point in the historical image data, and determine the probability that the point cloud corresponding to the projection point belongs to the object category according to the distance between the projection point and the pixel point and the object category corresponding to the pixel point. Specifically, the greater the distance, the smaller the probability that the point cloud belongs to the object category corresponding to the pixel point, and conversely, the smaller the distance, the greater the probability that the point cloud belongs to the object category corresponding to the pixel point.

[0130] Exemplarily, the visual intrinsic parameter matrix K is expressed as:

[0131]

[0132] Among them, f x and f y represents the focal length of the visual acquisition device, c x and c y Represents the optical center of the visual acquisition device.

[0133] Exemplarily, when the reference calibration matrix represents the spatial transformation relationship from the three-dimensional point cloud system to the visual image system, exemplary, the point cloud P in the historical point cloud data L Transformed into a visual image system, it can be represented as a pixel point P C , where P C =RP L +t, and then use the visual internal parameter matrix to transform the pixel point P C Projected onto the imaging plane of the visual acquisition device, the pixel point P is obtained C The projection coordinates (u, v) of .

[0134] Exemplarily, the projection coordinates (u, v) satisfy the following formula:

[0135]

[0136] Among them, (X C , Y C , Z C ) represents the pixel point PC Coordinates in the visual image system.

[0137] In an optional embodiment, the reference calibration matrix in the classification recognition model is iteratively adjusted according to the predicted classification results, including: determining the classification loss function value according to the predicted classification results and the actual classification results corresponding to the historical point cloud data; iteratively adjusting the reference calibration matrix in the classification recognition model according to the classification loss function value; wherein the iteration end conditions include the convergence of the classification loss function value and / or the number of training times reaching a threshold number.

[0138] Exemplarily, the loss function corresponding to the classification loss function value includes but is not limited to a cross loss function, a Hinge loss function, a logarithmic loss function, a relative entropy loss function, or a contrast loss function.

[0139] In another optional embodiment, the reference calibration matrix in the classification recognition model is iteratively adjusted according to the predicted classification result, including: obtaining historical projection data corresponding to the historical point cloud data in the classification recognition model, and determining the distance cost function value according to the historical projection data and the historical image data; wherein the historical projection data represents the projection result of the historical point cloud data in the imaging plane of the visual acquisition device; determining the classification loss function value according to the predicted classification result and the actual classification result corresponding to the historical point cloud data; iteratively adjusting the reference calibration matrix in the classification recognition model according to the distance cost function value and the classification loss function value; wherein the iteration end condition includes at least one of the convergence of the distance cost function value, the convergence of the classification loss function value and the number of iterations reaching a threshold.

[0140] Specifically, the distance cost function value represents the distance between the projection point in the historical projection data and the corresponding real pixel point in the historical image data. Exemplarily, the distance cost function corresponding to the distance cost function value includes but is not limited to the Euclidean distance function, the Manhattan distance function, the Chebyshev distance function or the Minkowski distance function, etc.

[0141] On the basis of the above embodiment, optionally, a Gauss-Newton algorithm is used to iteratively adjust the reference calibration matrix in the classification recognition model according to the classification loss function value and / or the cost loss function value.

[0142] The advantage of setting up a classification recognition model is that, compared with the traditional feature point calibration algorithm, the classification recognition model utilizes the deep three-dimensional geometric features of the point cloud data and the two-dimensional visual features of the image data in the process of iteratively solving the extrinsic parameter calibration matrix, providing a richer feature representation for the extrinsic parameter calibration matrix, thereby improving the accuracy of the extrinsic parameter calibration matrix.

[0143] S240: Determine the height of the wildfire flame according to the external parameter calibration matrix and the two-dimensional image data, and determine the wildfire alarm level in the monitoring area according to the flame height.

[0144] S240 in this embodiment is similar to that in the above embodiment. Figure 1 The S130 shown corresponds to the same or similar steps, and will not be described in detail in this embodiment.

[0145] The technical solution of this embodiment performs external parameter calibration on the historical point cloud data and historical image data of the monitoring area respectively collected by the laser radar and the visual acquisition device set on the same monitoring device, thereby obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area. This solves the problem of obtaining the external parameter calibration matrix, improves the accuracy of the external parameter calibration matrix, and further improves the accuracy of wildfire alarms in the transmission corridor.

[0146] Figure 3 A flowchart of another multi-modal transmission corridor wildfire alarm method provided by an embodiment of the present invention, this embodiment further refines the "obtaining the distance from the ground to the conductor of the transmission line in the monitoring area, the distance between the conductors, and the maximum crown height of the vegetation under the transmission line" in the above embodiment. In this embodiment, obtaining the distance from the ground to the conductor of the transmission line in the monitoring area, the distance between the conductors, and the maximum crown height of the vegetation under the transmission line includes: obtaining the historical image data of the monitoring area collected by the visual acquisition device, and obtaining the historical point cloud data of the monitoring area collected by the laser radar; wherein the laser radar and the visual acquisition device are set on the same monitoring device; inputting the historical image data and the historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs, and the object category in the point cloud category data includes conductors, ground, and vegetation; according to the point cloud category data, determining the distance from the ground to the conductor of the transmission line in the monitoring area, the distance between the conductors, and the maximum crown height of the vegetation under the transmission line.

[0147] like Figure 3 As shown, the method includes:

[0148] S310. Collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine a wildfire identification result of the monitoring area based on the two-dimensional image data.

[0149] S320. When the wildfire identification result is that a wildfire has occurred, obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0150] S330: Determine the flame height of the wildfire according to the external parameter calibration matrix and the two-dimensional image data.

[0151] S310-S330 in this embodiment are similar to those in the above embodiment. Figure 1 The S110-S130 shown in the figure correspond to the same or similar ones, or to the above-mentioned embodiments. Figure 2 The illustrated S210 - S240 correspond to the same or similar ones, and are not described in detail in this embodiment.

[0152] S340, obtaining historical image data of the monitoring area collected by the visual acquisition device, and obtaining historical point cloud data of the monitoring area collected by the laser radar.

[0153] In this embodiment, the laser radar and the visual acquisition device are arranged on the same monitoring device, wherein the monitoring device can be a movable device or a fixed device in the monitoring area.

[0154] Specifically, the collection time corresponding to the historical image data and the historical point cloud data is within a time error range to ensure the time consistency of the historical image data and the historical point cloud data.

[0155] S350, inputting historical image data and historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data.

[0156] In this embodiment, the point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs, and the object categories in the point cloud category data include wires, ground, and vegetation.

[0157] In an optional embodiment, the point cloud classification model includes a visual feature network, a point cloud feature network and a fusion output network; accordingly, historical image data and historical point cloud data are input into a pre-trained point cloud classification model to obtain output point cloud category data; through the visual feature network, feature extraction is performed on the historical image data to obtain a visual feature vector; through the point cloud feature network, feature extraction is performed on the historical point cloud data to obtain a point cloud feature vector; through the fusion output network, feature fusion is performed on the visual feature vector and the point cloud feature vector to obtain a fused feature vector, and point cloud category data is output based on the fused feature vector.

[0158] In an optional embodiment, the visual feature network is a convolutional neural network, and the fusion output network is an attention network. Exemplarily, the point cloud feature network is a RandLaNet network or a VoxelNet network.

[0159] Specifically, the visual feature vector is used to describe information with visual significance in the historical image data. Exemplarily, the visual feature vector includes but is not limited to a color feature vector, a texture feature vector, and a spatial feature vector, among which the spatial feature vector is used to describe the relative position information between multiple objects in the historical image data, and can also be used to describe the shape information of each object in the historical image data.

[0160] Specifically, the point cloud feature vector is used to describe the geometric features and local structural features of the historical point cloud data. For example, the geometric features include but are not limited to position information, normal vectors and curvature, and the local structural features include but are not limited to local neighborhood information and local geometric relationships.

[0161] S360. Determine the distance between the conductors of the power transmission lines in the monitoring area and the ground, the distance between the conductors, and the maximum crown height of the vegetation under the power transmission lines according to the point cloud category data.

[0162] In a specific embodiment, the distance between the conductor of the power transmission line in the monitoring area and the ground is determined based on the point cloud data corresponding to the conductor and the ground in the point cloud category data.

[0163] Specifically, a ground plane equation is constructed based on the point cloud data corresponding to the ground in the point cloud category data, and based on the point-to-plane distance between the point cloud coordinates of each point cloud corresponding to the conductor in the historical point cloud data and the ground plane equation, the statistical value corresponding to the point-to-plane distance of all point clouds corresponding to the conductor is used as the distance from the ground of the transmission line in the monitoring area.

[0164] Exemplarily, the statistical value is a minimum value or an average value, and the statistical value used is not limited here.

[0165] In a specific embodiment, the distance between the conductors of the transmission line is determined based on the point cloud data corresponding to the conductors in the point cloud category data.

[0166] Specifically, at least two conductor curve equations are constructed according to the point cloud data corresponding to the conductor in the point cloud category data, and the minimum curve distance of the conductor curve equations corresponding to two adjacent conductors is used as the distance between the conductors of the transmission line.

[0167] In a specific embodiment, the maximum crown height of vegetation in the monitoring area is determined based on the point cloud data corresponding to vegetation and the ground in the point cloud category data.

[0168] Specifically, according to the point cloud data corresponding to the ground in the point cloud category data, the ground plane equation is constructed, and according to the point-to-plane distance between the point cloud coordinates of each point cloud corresponding to the vegetation in the historical point cloud data and the ground plane equation, the maximum value corresponding to the point-to-plane distance of all point clouds corresponding to the vegetation is taken as the maximum crown height of the vegetation in the monitoring area.

[0169] Exemplarily, the statistical value is a minimum value or an average value, and the statistical value used is not limited here.

[0170] S370. Determine the forest fire warning level in the monitoring area based on the flame height, the distance of the wire from the ground, the distance between the wires and the maximum tree crown height.

[0171] S370 in this embodiment corresponds to the same or similar technical features as those in the above embodiments, and will not be described in detail in this embodiment.

[0172] The technical solution of this embodiment obtains output point cloud category data by inputting historical image data and historical point cloud data into a pre-trained point cloud classification model, and determines the distance between the conductors of the transmission lines in the monitoring area and the maximum crown height of the vegetation under the transmission lines based on the point cloud category data. It utilizes the deep three-dimensional geometric features of the point cloud data and the two-dimensional visual features of the image data, improves the accuracy of the length measurement parameters used in the wildfire alarm process of the transmission corridor, and thus further improves the accuracy of the wildfire alarm in the transmission corridor.

[0173] On the basis of the above embodiments, optionally, the method further includes: for each inspection area in the transmission corridor, obtaining historical image data of the inspection area collected by the visual acquisition device, and obtaining historical point cloud data of the inspection area collected by the lidar; inputting the historical image data and the historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data includes the vegetation category to which each point cloud in the historical point cloud data belongs; determining the vegetation distribution data of the inspection area according to the point cloud category data; if the vegetation distribution data meets the preset distribution conditions, the inspection area is used as a monitoring area; if the vegetation distribution data does not meet the preset distribution conditions, the inspection area is used as a non-monitoring area.

[0174] Exemplarily, the vegetation distribution data includes but is not limited to at least one of the number of vegetation categories belonging to the flammable category, vegetation coverage, vegetation height and vegetation density, etc. Correspondingly, the preset distribution conditions include but are not limited to at least one of the number of vegetation categories belonging to the flammable category being greater than a preset number, the vegetation coverage being greater than a coverage threshold, the vegetation height being greater than a height threshold, and the vegetation density being greater than a density threshold, etc.

[0175] Specifically, vegetation coverage indicates the percentage of the vertical projection area of ​​vegetation on the ground to the total area of ​​the inspection area, and vegetation density indicates the amount of vegetation per unit area.

[0176] The advantage of this setting is that by integrating the data features of point cloud data and image data, the number of monitoring areas in the wildfire alarm process of the transmission corridor is reduced, thereby improving the overall efficiency of wildfire monitoring in the transmission corridor.

[0177] Figure 4 A flowchart of a specific example of a multi-modal transmission corridor wildfire warning method provided by an embodiment of the present invention. Specifically, a camera and a laser radar are mounted on a drone to inspect the transmission corridor, and point cloud data and image data of each inspection area in the transmission corridor are collected. For each inspection area, the collected point cloud data and image data of the inspection area are preprocessed respectively, and a point cloud classification model is used to determine the point cloud category data of the point cloud data based on the preprocessed point cloud data and image data, and the vegetation distribution data of the inspection area is determined based on the point cloud category data. Based on the vegetation distribution data of each inspection area in the transmission corridor, at least one monitoring area that meets the preset distribution conditions is screened from multiple inspection areas, and the preprocessed point cloud data and image data corresponding to each monitoring area are calibrated with external parameters to obtain an external parameter calibration matrix.

[0178] In the daily monitoring of wildfire alarms in the transmission corridor, the camera is mounted on the drone to collect image data of the monitoring area, and the wildfire identification result of the monitoring area is determined based on the image data combined with the temperature sensor. It is judged whether the wildfire identification result indicates that a wildfire has occurred. If not, the drone is equipped with a camera to collect image data of other monitoring areas. If yes, the wildfire alarm level of the monitoring area is determined based on the external parameter calibration matrix and image data corresponding to the monitoring area.

[0179] The following is an embodiment of a wildfire alarm device for a multimodal power transmission corridor provided in an embodiment of the present invention. The device and the wildfire alarm method for a multimodal power transmission corridor of the above-mentioned embodiment belong to the same inventive concept. For details not fully described in the embodiment of the wildfire alarm device for a multimodal power transmission corridor, reference may be made to the contents of the wildfire alarm method for a multimodal power transmission corridor in the above-mentioned embodiment.

[0180] Figure 5 A schematic diagram of the structure of a multi-mode power transmission corridor wildfire warning device provided by an embodiment of the present invention. Figure 5 As shown, the device includes: a wildfire identification result determination module 410, an external parameter calibration matrix acquisition module 420 and a wildfire alarm level determination module 430.

[0181] The wildfire identification result determination module 410 is used to collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data;

[0182] The external parameter calibration matrix acquisition module 420 is used to obtain the external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area when the wildfire identification result is that a wildfire has occurred;

[0183] The wildfire warning level determination module 430 is used to determine the flame height of the wildfire according to the external parameter calibration matrix and the two-dimensional image data, and determine the wildfire warning level of the monitoring area according to the flame height.

[0184] The technical solution of this embodiment determines the wildfire recognition result of the monitoring area in the transmission corridor based on the two-dimensional image data collected by the visual acquisition device. When the wildfire recognition result is the occurrence of a wildfire, an external parameter calibration matrix is ​​obtained between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area. The flame height of the wildfire is determined based on the external parameter calibration matrix and the two-dimensional image data. The wildfire alarm level of the monitoring area is determined based on the flame height. The visual acquisition device is combined with the three-dimensional point cloud system to realize daily monitoring of wildfire alarms in the transmission corridor, which solves the problem of high equipment cost of laser radar for monitoring wildfires. While ensuring the accuracy of wildfire alarms, the response speed of wildfire alarms is improved, thereby ensuring the safety of the transmission corridor.

[0185] In an optional embodiment, the wildfire warning level determination module 430 includes:

[0186] The flame height determination unit is used to project the two-dimensional image data into the three-dimensional point cloud system corresponding to the monitoring area by using an external parameter calibration matrix to obtain three-dimensional projection data;

[0187] Determine the flame height of the wildfire based on the three-dimensional projection data.

[0188] In an optional embodiment, the extrinsic calibration matrix acquisition module 420 includes:

[0189] A historical point cloud data acquisition unit is used to acquire historical image data of the monitoring area acquired by the visual acquisition device, and to acquire historical point cloud data of the monitoring area acquired by the laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device;

[0190] The external parameter calibration matrix determination unit is used to perform external parameter calibration based on historical image data and historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

[0191] In an optional embodiment, the extrinsic calibration matrix determination unit is specifically used to:

[0192] A reference calibration matrix determination subunit is used to obtain a reference calibration matrix by performing initial external parameter calibration based on historical image data and historical point cloud data using a preset calibration algorithm;

[0193] The prediction classification result output subunit is used to input the reference calibration matrix, the visual intrinsic parameter matrix of the visual acquisition device, the historical image data and the historical point cloud data into the untrained classification recognition model to obtain the prediction classification result corresponding to the output historical point cloud data; wherein the prediction classification result includes the projection classification result of each point cloud in the historical point cloud data;

[0194] A reference calibration matrix iterative adjustment subunit is used to iteratively adjust the reference calibration matrix in the classification recognition model according to the predicted classification result;

[0195] The extrinsic parameter calibration matrix determination subunit is used to obtain the extrinsic parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area in the trained classification recognition model until the iteration end condition is met.

[0196] In an optional embodiment, the reference calibration matrix iterative adjustment subunit is specifically configured to:

[0197] Obtaining historical projection data corresponding to the historical point cloud data in the classification recognition model, and determining the distance cost function value according to the historical projection data and the historical image data; wherein the historical projection data represents the projection result of the historical point cloud data in the imaging plane of the visual acquisition device;

[0198] Determine the classification loss function value based on the predicted classification result and the actual classification result corresponding to the historical point cloud data;

[0199] Iteratively adjust the reference calibration matrix in the classification recognition model according to the distance cost function value and the classification loss function value;

[0200] The iteration termination condition includes the convergence of the distance cost function value and / or the convergence of the classification loss function value.

[0201] In an optional embodiment, the wildfire warning level determination module 430 includes:

[0202] A maximum tree crown height acquisition unit is used to acquire the distance between the conductors of the power transmission lines in the monitoring area and the distance between the conductors, as well as the maximum tree crown height of the vegetation under the power transmission lines;

[0203] The wildfire warning level determination unit is used to determine the wildfire warning level in the monitoring area based on the flame height, the distance of the wire from the ground, the distance between the wires and the maximum tree crown height.

[0204] In an optional embodiment, the maximum tree crown height acquisition unit includes:

[0205] The historical point cloud data acquisition subunit is used to acquire the historical image data of the monitoring area acquired by the visual acquisition device, and acquire the historical point cloud data of the monitoring area acquired by the laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device;

[0206] The point cloud category data output subunit is used to input the historical image data and the historical point cloud data into the pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs, and the object categories in the point cloud category data include wires, ground and vegetation;

[0207] The maximum crown height determination subunit is used to determine the distance between the conductors of the transmission lines in the monitoring area and the ground, the distance between the conductors, and the maximum crown height of the vegetation under the transmission lines based on the point cloud category data.

[0208] In an optional embodiment, the point cloud classification model includes a visual feature network, a point cloud feature network, and a fusion output network;

[0209] Correspondingly, the point cloud category data output subunit is specifically used for:

[0210] Through the visual feature network, the historical image data is extracted to obtain the visual feature vector;

[0211] Through the point cloud feature network, feature extraction is performed on the historical point cloud data to obtain the point cloud feature vector;

[0212] Through the fusion output network, the visual feature vector and the point cloud feature vector are fused to obtain a fused feature vector, and the point cloud category data is output based on the fused feature vector.

[0213] In an optional embodiment, the wildfire identification result determination module 410 includes:

[0214] A two-dimensional image data output unit, used to input the two-dimensional image data into a pre-trained wildfire recognition model; wherein the wildfire recognition model includes a feature extraction network and a wildfire recognition network;

[0215] A motion feature vector extraction unit, used for extracting features from two-dimensional image data through a feature extraction network to obtain a visual feature vector and a motion feature vector;

[0216] A wildfire prediction result output unit, used to output a wildfire prediction result according to a visual feature vector and a motion feature vector through a wildfire recognition network;

[0217] The wildfire identification result output unit is used to determine the wildfire identification result of the monitoring area according to the predicted wildfire result.

[0218] In an optional embodiment, the wildfire identification result output unit is specifically used to:

[0219] Acquire temperature data collected by temperature sensors in the monitoring area;

[0220] If the temperature data exceeds the temperature threshold and the predicted wildfire result is a wildfire occurrence, the wildfire identification result in the monitored area is set to a wildfire occurrence.

[0221] The wildfire warning device for a multimodal power transmission corridor provided in an embodiment of the present invention can execute the wildfire warning method for a multimodal power transmission corridor provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0222] Figure 6 A structural schematic diagram of a wildfire alarm system for a multimodal power transmission corridor provided in one embodiment of the present invention. The wildfire alarm system for a multimodal power transmission corridor in this embodiment provides services for the wildfire alarm method for a multimodal power transmission corridor in the above-mentioned embodiment.

[0223] like Figure 6 As shown, the wildfire warning system 500 for the multimodal power transmission corridor includes a visual acquisition device 510 and a terminal device 520 that are communicatively connected; wherein the visual acquisition device 510 is used to collect two-dimensional image data of the monitoring area in the power transmission corridor.

[0224] In an optional embodiment, the wildfire warning system 500 for the multimodal power transmission corridor further includes a laser radar 530 communicatively connected to the terminal device 520 for collecting point cloud data of the monitoring area in the power transmission corridor.

[0225] In an optional embodiment, the wildfire warning system 500 for the multi-modal transmission corridor further includes a monitoring device ( Figure 6 ), which is not shown in the figure, is used to set up the visual acquisition device 510 and the laser radar 530.

[0226] Figure 7 A schematic diagram of the structure of a terminal device provided for one embodiment of the present invention. The terminal device 520 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The terminal device 520 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0227] like Figure 7As shown, the terminal device 520 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 to the RAM 13. Various programs and data required for the operation of the terminal device 520 can also be stored in the RAM 13. The processor 11, the ROM 12, and the RAM 13 are connected to each other through the bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0228] A number of components in the terminal device 520 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the terminal device 520 to exchange information or data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0229] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the wildfire alarm method for the multimodal transmission corridor provided in the above embodiments.

[0230] In some embodiments, the wildfire warning method for a multimodal power transmission corridor provided in the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the terminal device 520 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the wildfire warning method for a multimodal power transmission corridor described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the wildfire warning method for a multimodal power transmission corridor in any other appropriate manner (e.g., by means of firmware).

[0231] Various embodiments of the systems and techniques described above herein may be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0232] The computer program for implementing the wildfire warning method of the multimodal transmission corridor of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0233] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable storage medium. Examples of machine readable storage media may include electrical connections based on at least one line, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0234] To provide interaction with a user, the systems and techniques described herein may be implemented on a terminal device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the terminal device. Other types of devices may also provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0235] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.

[0236] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0237] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0238] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A wildfire warning method for a multi-modal power transmission corridor, characterized in that: include: Collecting two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determining a wildfire identification result of the monitoring area based on the two-dimensional image data; When the wildfire identification result is that a wildfire has occurred, obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area; The flame height of the wildfire is determined according to the external parameter calibration matrix and the two-dimensional image data, and the wildfire alarm level of the monitoring area is determined according to the flame height.

2. The method according to claim 1, characterized in that Determining the flame height of the wildfire according to the external parameter calibration matrix and the two-dimensional image data includes: Using the external parameter calibration matrix, projecting the two-dimensional image data into a three-dimensional point cloud system corresponding to the monitoring area to obtain three-dimensional projection data; The flame height of the wildfire is determined based on the three-dimensional projection data.

3. The method according to claim 1, characterized in that The step of obtaining an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area includes: Acquire the historical image data of the monitoring area collected by the visual acquisition device, and acquire the historical point cloud data of the monitoring area collected by the laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device; External parameter calibration is performed based on the historical image data and the historical point cloud data to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area.

4. The method according to claim 3, characterized in that: The extrinsic calibration is performed according to the historical image data and the historical point cloud data to obtain an extrinsic calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area, including: Using a preset calibration algorithm, an initial external parameter calibration is performed according to the historical image data and the historical point cloud data to obtain a reference calibration matrix; Input the reference calibration matrix, the visual intrinsic parameter matrix of the visual acquisition device, the historical image data and the historical point cloud data into an untrained classification recognition model to obtain a predicted classification result corresponding to the outputted historical point cloud data; wherein the predicted classification result includes a projection classification result of each point cloud in the historical point cloud data; Iteratively adjusting the reference calibration matrix in the classification recognition model according to the predicted classification result; When the iteration end condition is met, an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area in the trained classification recognition model is obtained.

5. The method according to claim 4, characterized in that The iterative adjustment of the reference calibration matrix in the classification recognition model according to the predicted classification result includes: Acquire historical projection data corresponding to the historical point cloud data in the classification and recognition model, and determine a distance cost function value according to the historical projection data and the historical image data; wherein the historical projection data represents a projection result of the historical point cloud data in an imaging plane of the visual acquisition device; Determining a classification loss function value according to the predicted classification result and the actual classification result corresponding to the historical point cloud data; Iteratively adjusting a reference calibration matrix in the classification recognition model according to the distance cost function value and the classification loss function value; The iteration end condition includes the convergence of the distance cost function value and / or the convergence of the classification loss function value.

6. The method according to claim 1, characterized in that Determining the wildfire warning level of the monitoring area according to the flame height includes: Obtaining the distance between the conductors of the power transmission lines in the monitoring area and the distance between the conductors, and the maximum crown height of the vegetation under the power transmission lines; The wildfire warning level of the monitoring area is determined according to the flame height, the distance of the wire from the ground, the distance between the wires and the maximum tree crown height.

7. The method according to claim 6, characterized in that The obtaining of the distance between the conductors of the power transmission lines in the monitoring area and the distance between the conductors and the maximum crown height of the vegetation under the power transmission lines includes: Acquire the historical image data of the monitoring area collected by the visual acquisition device, and acquire the historical point cloud data of the monitoring area collected by the laser radar; wherein the laser radar and the visual acquisition device are arranged on the same monitoring device; Inputting the historical image data and the historical point cloud data into a pre-trained point cloud classification model to obtain output point cloud category data; wherein the point cloud category data includes the object category to which each point cloud in the historical point cloud data belongs, and the object categories in the point cloud category data include wires, ground and vegetation; According to the point cloud category data, the distance between the conductors of the transmission lines in the monitoring area and the distance between the conductors and the maximum crown height of the vegetation under the transmission lines are determined.

8. The method according to claim 7, characterized in that The point cloud classification model includes a visual feature network, a point cloud feature network and a fusion output network; Correspondingly, the historical image data and the historical point cloud data are input into a pre-trained point cloud classification model to obtain output point cloud category data; Extracting features from the historical image data using the visual feature network to obtain a visual feature vector; Through the point cloud feature network, feature extraction is performed on the historical point cloud data to obtain a point cloud feature vector; The visual feature vector and the point cloud feature vector are subjected to feature fusion through the fusion output network to obtain a fused feature vector, and point cloud category data is output according to the fused feature vector.

9. The method according to claim 1, characterized in that: Determining the wildfire identification result of the monitoring area according to the two-dimensional image data includes: Inputting the two-dimensional image data into a pre-trained wildfire recognition model; wherein the wildfire recognition model includes a feature extraction network and a wildfire recognition network; Through the feature extraction network, feature extraction is performed on the two-dimensional image data to obtain a visual feature vector and a motion feature vector; Outputting a predicted wildfire result according to the visual feature vector and the motion feature vector through the wildfire recognition network; The wildfire identification result of the monitoring area is determined according to the predicted wildfire result.

10. The method according to claim 9, characterized in that Determining the wildfire identification result in the monitoring area according to the predicted wildfire result includes: Acquire temperature data collected by a temperature sensor in the monitoring area; If the temperature data exceeds the temperature threshold, and the predicted wildfire result is the occurrence of a wildfire, the wildfire identification result of the monitoring area is set as the occurrence of a wildfire.

11. A wildfire warning device for a multi-modal power transmission corridor, characterized in that: include: A wildfire identification result determination module is used to collect two-dimensional image data of the monitoring area in the transmission corridor through a visual acquisition device, and determine the wildfire identification result of the monitoring area based on the two-dimensional image data; An external parameter calibration matrix acquisition module is used to obtain an external parameter calibration matrix between the visual acquisition device and the three-dimensional point cloud system corresponding to the monitoring area when the wildfire identification result is a wildfire occurrence; The wildfire warning level determination module is used to determine the flame height of the wildfire according to the external parameter calibration matrix and the two-dimensional image data, and determine the wildfire warning level of the monitoring area according to the flame height.

12. A forest fire warning system for a multi-modal power transmission corridor, characterized in that: The mountain fire warning system of the multimodal power transmission corridor includes a visual acquisition device and a terminal device connected in communication; Wherein, the visual acquisition device is used to collect two-dimensional image data of the monitoring area in the transmission corridor; The terminal device comprises: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the wildfire alarm method for the multimodal transmission corridor described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the wildfire warning method for a multimodal transmission corridor as described in any one of claims 1-10 when executed.

14. A computer program product, comprising a computer program, which, when executed by a processor, implements the wildfire warning method for a multimodal power transmission corridor according to any one of claims 1-10.

Images