Intelligent smoke and fire detection method and equipment applied to unmanned aerial vehicle inspection and medium
By using technical means such as real-time video stream detection, GPS coordinate countercalculating and circumvention in drone inspection, the problem that traditional drone inspection methods are difficult to fully cover risk areas in complex environments and lack of independent decision-making, and efficient and accurate firework detection and independent inspection capabilities are achieved.
Patent Information
- Application Number
- CN202411893330.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-05-13
AI Technical Summary
Traditional drone inspection methods are difficult to fully cover all potential risk areas when facing complex and changing environments, and lack independent decision-making capabilities, resulting in an increase in the risk of missed inspections and false alarms, and the complex and changeable types of fireworks lead to errors in detection results.
By obtaining the real-time video stream of the drone on the preset patrol route, performing initial detection based on the preset firework detection model, counter-calculating the GPS coordinates of the suspicious area, performing proximity flight detection, and inputting the detected image to the firework prediction model to generate firework risk information.
It significantly improves the accuracy of firework detection, enhances the autonomous inspection capabilities of drones, achieves comprehensive, timely and accurate monitoring of fire risks, reduces the probability of missed inspections and false alarms, and increases the response speed of emergency rescue.
Smart Images
Figure CN119992372A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of drone intelligent inspection, and in particular to an intelligent fireworks detection method, device and medium applied to drone inspection. Background Art
[0002] Currently, drone inspection technology is showing unprecedented application potential in many fields with its unique advantages of high efficiency, flexibility and wide coverage. The core of this technology is to combine advanced image recognition and processing methods, make full use of the maneuverability and high-altitude vision of drones, and quickly capture the smoke and flames at the beginning of a fire. This combination not only greatly improves the detection efficiency, but also wins precious response time for emergency rescue operations, which is of great significance for reducing disaster losses and protecting people's lives and property.
[0003] However, although the traditional drone inspection method has achieved fire prevention capabilities to a certain extent, its inherent limitations cannot be ignored. Specifically, traditional drone inspections usually rely on preset fixed inspection routes, which are inadequate in the face of complex and changing environments. Due to the complexity and uncertainty of the environment, fixed inspection routes often fail to fully cover all potential risk areas, resulting in an increased risk of missed inspections.
[0004] In addition, traditional drone inspection methods lack autonomous decision-making capabilities during the inspection process and cannot flexibly adjust inspection strategies according to on-site conditions. This means that when drones encounter emergencies or abnormal phenomena, they cannot respond effectively in a timely manner, and may miss critical inspection opportunities.
[0005] What’s more serious is that the complexity and variety of fireworks also poses a great challenge to drone fireworks detection. Factors such as changes in smoke concentration and the various forms of flames may lead to errors in detection results, which in turn may cause false alarms or missed alarms. These defects not only reduce the accuracy and reliability of drone inspections, but may also cause unnecessary interference and delays to emergency rescue operations. Summary of the invention
[0006] The embodiments of the present application provide an intelligent fire and smoke detection method, device and medium for drone inspection, which are used to provide an efficient, accurate drone inspection method that can adapt to complex environments, so as to achieve comprehensive, timely and accurate monitoring of fire risks.
[0007] In the first aspect, an embodiment of the present application provides an intelligent fireworks detection method for drone inspection, characterized in that the method includes: obtaining a real-time video stream of the drone on a preset inspection route, and performing an initial detection on the real-time video stream based on a preset fireworks detection model to determine whether there is a suspicious area and corresponding suspicious information; if so, based on the suspicious information, the GPS coordinates of the suspicious area are reversed, and an approach and circumvention detection is performed based on the GPS coordinates to determine whether risk targets can be detected continuously; if so, a number of detection images corresponding to the continuous detection of risk targets are input into a preset fireworks prediction model for prediction to generate fireworks risk information, and the fireworks risk information is transmitted back for alarm.
[0008] In one implementation of the present application, before obtaining the real-time video stream of the drone on the preset inspection route, the method also includes: connecting the pan-tilt camera and the edge computing module to the drone as mounts; configuring the drone with a preset inspection route based on preset inspection requirements; and setting the RTSP address of the pan-tilt camera and the alarm information receiving address of the video surveillance center in the edge computing module.
[0009] In one implementation of the present application, before performing an initial detection on the real-time video stream based on a preset smoke and fire detection model, the method further includes:
[0010] In one implementation of the present application, a fireworks detection model is trained using the YOLOV8s model as a benchmark model; an initial detection is performed on the real-time video stream based on a preset fireworks detection model to determine whether there is a suspicious area and corresponding suspicious information, specifically including: inputting the image to be initially inspected into the fireworks detection model to output a detection result; wherein the detection result includes at least one of the following: whether there is a suspicious area, a suspicious area report; the suspicious area report includes: suspicious information corresponding to the suspicious area; the suspicious information includes: the number of suspicious areas in the image to be initially inspected, image pixel coordinate information of a preset image reference position corresponding to each suspicious area, image size information in the image corresponding to each suspicious area, the detection category of each suspicious area and the corresponding category probability.
[0011] In one implementation of the present application, based on the suspicious information, the GPS coordinates of the suspicious area are inversely calculated, specifically including: converting the image pixel coordinate information into image physical coordinate information by transforming the image pixel coordinate system into the image physical coordinate system; converting the image physical coordinate information into camera coordinate information by transforming the image physical coordinate system into the camera coordinate system; converting the camera coordinate information into world coordinate information relative to the drone by transforming the camera coordinate system into the drone world coordinate system; and converting the world coordinate information into GPS coordinate information by transforming the world coordinate system into the WGS84 coordinate system.
[0012] In one implementation of the present application, an approach and circling detection is performed based on GPS coordinates to determine whether risky targets can be detected continuously, specifically including: flying an unmanned aerial vehicle (UAV) close to the area based on the GPS coordinates, and performing circling detection with a preset radius with the center of the suspicious area as the origin; determining whether the number of risky targets detected within the preset circling time is greater than a preset threshold, and if so, determining that risky targets can be detected continuously.
[0013] In one implementation of the present application, the number of detected images is a preset threshold; the fireworks prediction model is obtained based on ResNet50 network training; the detected images corresponding to the continuous detection of risk targets are input into the preset fireworks prediction model for prediction to generate fireworks risk information, specifically including: cropping the detected images based on GPS coordinates, and inputting the cropped detected images into the fireworks prediction model in sequence to obtain three-dimensional panoramic view data of the suspicious area and the corresponding fireworks risk information; wherein the fireworks risk information includes at least: fireworks risk category.
[0014] In one implementation of the present application, several detected images are cropped based on GPS coordinates and suspicious information, specifically including: calibrating corresponding suspicious areas in several detected images based on GPS coordinates and suspicious information; and cropping the calibrated suspicious areas in several detected images.
[0015] In a second aspect, an embodiment of the present application also provides an intelligent fireworks detection device for use in drone inspections, characterized in that the device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an intelligent fireworks detection method for use in drone inspections such as any one of the above.
[0016] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium for intelligent fireworks detection for drone inspection, which stores computer executable instructions, and is characterized in that when the computer executable instructions are executed, an intelligent fireworks detection method for drone inspection as described above is implemented.
[0017] The embodiments of the present application provide an intelligent fireworks detection method, device and medium for drone inspection, which have the following beneficial effects:
[0018] 1. Significantly improve the accuracy of fireworks detection: By introducing autonomous decision-making and multiple screening mechanisms, this application enables drones to conduct more detailed and in-depth inspections of suspected fireworks areas, ensuring accurate detection even in the face of complex and changeable types of fireworks, such as changes in smoke concentration and diverse flame shapes, greatly reducing the probability of missed detection and false alarms.
[0019] 2. Enhance the autonomous inspection capability of drones: Traditional drone inspection methods often rely on preset inspection routes and lack autonomous decision-making capabilities. However, this application integrates advanced image recognition and processing technology and intelligent algorithms to enable drones to flexibly adjust inspection strategies according to on-site conditions, thus achieving truly intelligent inspections, improving inspection efficiency, and enhancing the adaptability and flexibility of drones.
[0020] 3. Achieve accurate position back-calculation and fly-around detection: This application uses an accurate position back-calculation module to convert the image area into actual GPS coordinates, thereby achieving accurate positioning of suspicious areas. At the same time, through the fly-around detection mechanism, the drone can conduct multi-angle inspections of suspicious areas, further improving the accuracy and reliability of detection.
[0021] 4. Improve the real-time performance and response speed of fire and smoke detection: This application uses edge computing modules to process and detect real-time video streams, which can identify suspicious areas in a short time and conduct follow-up investigations immediately. This efficient detection process wins more valuable response time for emergency rescue operations and helps reduce losses caused by fires. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0023] Figure 1 A flow chart of an intelligent fireworks detection method for drone inspection provided in an embodiment of the present application;
[0024] Figure 2 A schematic diagram of the internal structure of an intelligent fireworks detection device used for drone inspections provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0026] The embodiments of the present application provide an intelligent fire and smoke detection method, device and medium for drone inspection, which are used to provide an efficient, accurate drone inspection method that can adapt to complex environments, so as to achieve comprehensive, timely and accurate monitoring of fire risks.
[0027] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.
[0028] Figure 1 This is a flow chart of an intelligent fireworks detection method for drone inspection provided in an embodiment of the present application. Figure 1 As shown, an intelligent fireworks detection method for drone inspection provided in an embodiment of the present application specifically includes the following steps:
[0029] Step 101: obtain a real-time video stream of the drone on a preset inspection route, and perform an initial detection on the real-time video stream based on a preset fireworks detection model to determine whether there is a suspicious area and corresponding suspicious information.
[0030] In one embodiment of the present application, in order to realize intelligent fireworks detection applied to drone inspection, it is first necessary to deploy drone equipment and set inspection routes.
[0031] Specifically, the PTZ camera and edge computing module are mounted on the drone; based on the preset inspection requirements, the drone is configured with a preset inspection route; the RTSP address of the PTZ camera and the alarm information receiving address of the video surveillance center are set in the edge computing module.
[0032] Furthermore, real-time video stream of the drone on the preset inspection route is obtained.
[0033] Specifically, the edge computing module is responsible for pulling and processing the real-time video stream from the PTZ camera. Optionally, the edge computing module uses GStreamer, a powerful multimedia framework, to connect to the PTZ camera through the RTSP address to obtain real-time video data.
[0034] Furthermore, after acquiring the real-time video stream, it is also necessary to pre-process the real-time video stream.
[0035] Specifically, the real-time video stream is parsed to determine the video encoding format corresponding to the pan-tilt camera; based on the video encoding format, the edge computing module matches the corresponding preset GStreamer decoder to decode the real-time video stream to convert the frame image in the real-time video stream into an image to be initially inspected in RGB image format.
[0036] In one embodiment, according to the video encoding format provided by the PTZ camera (such as H.264 or H.265, etc.), the edge computing module will select the corresponding GStreamer decoder to decode these video streams. Specifically, when processing H.264-encoded video streams, the module will select the omxh264dec decoder designed for NVIDIA hardware acceleration. Through this decoding process, the video stream is finally converted into RGB image format.
[0037] Furthermore, an initial detection is performed on the real-time video stream based on a preset fireworks detection model to determine whether there are suspicious areas and corresponding suspicious information.
[0038] In one embodiment of the present application, the fireworks detection model is trained with the YOLOV8s model as the benchmark model; the real-time video stream is initially detected based on the preset fireworks detection model to determine whether there is a suspicious area and corresponding suspicious information, specifically including: inputting the image to be initially inspected into the fireworks detection model to output the detection result; wherein the detection result includes at least one of the following: whether there is a suspicious area, a suspicious area report; the suspicious area report includes: suspicious information corresponding to the suspicious area; the suspicious information includes: the number of suspicious areas in the image to be initially inspected, the image pixel coordinate information of the preset image reference position corresponding to each suspicious area, the image size information in the image corresponding to each suspicious area, the detection category of each suspicious area and the corresponding category probability. It should be noted that the preset image reference position is preferably: the center position or the upper left corner or the upper right corner or the lower left corner or the lower right corner of the corresponding suspicious area in the image to be initially inspected.
[0039] For example, in the initial detection stage, an input RGB image may contain multiple suspicious areas at the same time, denoted by R = {r1, r2, ..., r n}, where r i It represents the suspicious area of the i-th detected image, and any suspicious area contains information r i =[x i ,y i ,w i ,h i ,c i ,p i ], where x i ,y iThe image pixel coordinate information representing the center position of the i-th detected suspicious area, w i ,h i Indicates the width and height of the detected suspicious area, c i represents the detected category, p i Represents the probability of detected category.
[0040] Step 102: If so, based on the suspicious information, reverse calculate the GPS coordinates of the suspicious area, and perform approach and fly-around detection based on the GPS coordinates to determine whether risky targets can be continuously detected.
[0041] It should be noted that when the existence of suspicious areas and corresponding suspicious information is determined, the drone will conduct a secondary inspection of each suspicious area detected. i Taking the suspicious area as an example, based on the coordinate information of the suspicious area and the GPS information of its own location, the drone uses the position inversion module to inversely calculate the image area to the actual GPS coordinates, conducts close inspection based on the inversely calculated GPS coordinates, and flies around the suspicious area to conduct inspection based on the inversely calculated actual center position.
[0042] In one embodiment of the present application, based on the suspicious information, the GPS coordinates of the suspicious area are inversely calculated, specifically including: converting the image pixel coordinate information into image physical coordinate information by transforming the image pixel coordinate system into the image physical coordinate system; converting the image physical coordinate information into camera coordinate information by transforming the image physical coordinate system into the camera coordinate system; converting the camera coordinate information into world coordinate information relative to the drone by transforming the camera coordinate system into the drone world coordinate system; converting the world coordinate information into GPS coordinate information by transforming the world coordinate system into the WGS84 coordinate system.
[0043] The following is a detailed description of the back calculation steps:
[0044] 1) Since the image pixel coordinates and image physical coordinates have the following transformation formula:
[0045]
[0046] In this way, the actual physical coordinates of each pixel in the image can be obtained as {x_pixel, y_pixel}.
[0047] 2) According to the actual physical coordinates obtained in 1), the actual distance actual_length between the camera and the imaging origin and the focal length f of the camera are measured by laser, and then the transformation formula of the camera coordinate system is obtained:
[0048] Z_c=actual_length
[0049] X_c=x_pixel*Z_c / f
[0050] Y_c=y_pixel*Z_c / f
[0051] 3) Based on the camera coordinates {X_c, Y_c, Z_c} generated in 2), the camera coordinates are further converted to the world coordinate system in combination with the attitude information {pitch, roll, yaw} of the drone. Since rigid body transformation only involves the spatial position translation and orientation rotation of the object, but does not change its shape, it can be described by two variables: the rotation matrix R and the three-dimensional translation vector t. The three-dimensional translation vector t contains the translation amounts {t_x, t_y, t_z} in three dimensions respectively. On the other hand, rotation also contains three degrees of freedom, namely rotation around x, y, and z. According to the rotation angle, the rotation matrix R in the three directions can be obtained respectively. x , R y , R z , and then we get the rotation matrix R = R x ×R y ×R z . Where R x , R y , R z They are:
[0052]
[0053] The conversion equation can be further simplified as follows:
[0054]
[0055] 4) Convert the world coordinates relative to the drone into the actual geographic longitude and latitude GPS coordinates. According to the standard value in the WGS84 coordinate system, the earth's major axis a is about 6378137.0 meters, and the earth's flattening f is about 1 / 298.257223563. The radius of curvature N of the yoke is obtained and the formula is recorded as:
[0056]
[0057] Where N is the radius of curvature of the earth ellipsoid at a given latitude, and lat_rad is the arc representation of the current latitude of the current drone. Given the current longitude and latitude information, the number of meters corresponding to each degree of change in longitude and latitude is calculated as follows:
[0058]
[0059] Calculate the change in latitude and longitude of the world coordinates from the center of the image. The calculation formula for the change is as follows:
[0060]
[0061] Finally, calculate the actual GPS coordinates of the target center point in the suspicious area:
[0062] object_lat=drone_lat+delta_lat
[0063] object_lon=drone_lon+delta_lon
[0064] Among them, drone_lat and drone_lon are the longitude and latitude information of the current drone, and object_lat and object_lon are the longitude and latitude information of the corresponding pixels in the obtained image. Through the above steps, the conversion from the drone's world coordinates to GPS coordinates is realized, and the curvature change of the earth's ellipsoid shape at different latitudes is taken into account.
[0065] In one embodiment of the present application, after the GPS coordinates of the suspicious area are inversely calculated, an approach and fly-around detection is performed based on the GPS coordinates to determine whether risky targets can be continuously detected.
[0066] Specifically, based on the GPS coordinates, the drone flies close and performs circling detection with a preset radius with the center of the suspicious area as the origin; it is determined whether the number of risky targets detected during the preset circling time is greater than a preset threshold. If so, it is determined that risky targets can be detected continuously.
[0067] It is understandable that if it is not greater than the preset threshold, then the suspicious area is jumped out and a secondary inspection of the next suspicious area is carried out.
[0068] Step 103: If yes, a plurality of detected images corresponding to the continuous detection of risk targets are input into a preset fire and smoke prediction model for prediction to generate fire and smoke risk information, and the fire and smoke risk information is transmitted back for alarm.
[0069] It should be noted that, when it is determined that risk targets can be detected continuously, a third target confirmation is required. The main purpose of this stage is to check for multiple similar false targets that mislead the detection model and produce false fireworks alarms. The third confirmation stage mainly relies on several detected images obtained during the second flyby continuous detection, which are further input into the neural network for classification tasks.
[0070] In one embodiment of the present application, the number of detected images is a preset threshold; the fireworks prediction model is obtained based on ResNet50 network training; the detected images corresponding to the continuous detection of risk targets are input into the preset fireworks prediction model for prediction to generate fireworks risk information, specifically including: cropping the detected images based on GPS coordinates, and inputting the cropped detected images into the fireworks prediction model in sequence to obtain three-dimensional panoramic view data of the suspicious area and the corresponding fireworks risk information; wherein the fireworks risk information includes at least: fireworks risk category.
[0071] The method of cropping a plurality of detected images based on the GPS coordinates and the suspicious information specifically includes: calibrating corresponding suspicious areas in the plurality of detected images based on the GPS coordinates and the suspicious information; and cropping the calibrated suspicious areas in the plurality of detected images.
[0072] It should be noted that the present application crops several detected images based on GPS coordinates, and uses the cropped N detected images as the viewing angle sequence of the detection target, denoted as x = {x1, x2, ..., x N}, input into the ResNet50 network for fireworks prediction, the model pools multiple views, aggregates the feature information of multiple views, generates an aggregate descriptor representing the 3D model, denoted as X, and then inputs into the ResNet50 model to achieve classification of multi-view images. This method can solve the problem of insufficient feature description of a single view, and at the same time combine the panoramic view data of the 3D model to further improve the classification accuracy.
[0073] Furthermore, the fireworks risk information is transmitted back for alarm.
[0074] In one embodiment of the present application, after all suspicious areas have been checked, the drone will return to the normally set inspection route, the drone will return to the GPS point where it first moved, and continue with subsequent inspection tasks.
[0075] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides an intelligent fireworks detection device for drone inspection, whose structure is as follows: Figure 2 shown.
[0076] Figure 2 This is a schematic diagram of the internal structure of an intelligent fireworks detection device used for drone inspections provided in an embodiment of the present application. Figure 2 As shown, the device includes:
[0077] at least one processor 201;
[0078] and, a memory 202 communicatively connected to the at least one processor;
[0079] The memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 to enable at least one processor 201 to:
[0080] Obtain the real-time video stream of the drone on the preset inspection route, and perform an initial detection on the real-time video stream based on the preset fireworks detection model to determine whether there are suspicious areas and corresponding suspicious information;
[0081] If there is, based on the suspicious information, the GPS coordinates of the suspicious area are calculated, and an approach and fly-around detection is performed based on the GPS coordinates to determine whether the risk target can be detected continuously;
[0082] If so, a preset fire and fireworks prediction model will be input for prediction based on a number of detected images corresponding to the continuous detection of risk targets to generate fire and fireworks risk information, and the fire and fireworks risk information will be transmitted back for alarm.
[0083] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for intelligent fireworks detection for unmanned aerial vehicle inspection, storing computer executable instructions, wherein the computer executable instructions are set as follows:
[0084] Obtain the real-time video stream of the drone on the preset inspection route, and perform an initial detection on the real-time video stream based on the preset fireworks detection model to determine whether there are suspicious areas and corresponding suspicious information;
[0085] If there is, based on the suspicious information, the GPS coordinates of the suspicious area are calculated, and an approach and fly-around detection is performed based on the GPS coordinates to determine whether the risk target can be detected continuously;
[0086] If so, a preset fire and fireworks prediction model will be input for prediction based on a number of detected images corresponding to the continuous detection of risk targets to generate fire and fireworks risk information, and the fire and fireworks risk information will be transmitted back for alarm.
[0087] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0088] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.
[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0094] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0095] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0097] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. An intelligent fireworks detection method for drone inspection, characterized in that: The method comprises: Obtaining a real-time video stream of the drone on a preset inspection route, and performing an initial detection on the real-time video stream based on a preset fireworks detection model to determine whether there is a suspicious area and corresponding suspicious information; If so, based on the suspicious information, the GPS coordinates of the suspicious area are calculated inversely, and an approach and fly-around detection is performed based on the GPS coordinates to determine whether risky targets can be detected continuously; If so, a preset fire and smoke prediction model is input for prediction based on a number of detected images corresponding to the continuous detection of risk targets to generate fire and smoke risk information, and the fire and smoke risk information is transmitted back for alarm.
2. The intelligent fireworks detection method for drone inspection according to claim 1 is characterized in that: Before obtaining the real-time video stream of the drone on the preset inspection route, the method further includes: Connect the gimbal camera and edge computing module to the drone as external mounts; Based on the preset inspection requirements, configuring the preset inspection route for the drone; The RTSP address of the PTZ camera and the alarm information receiving address of the video surveillance center are set in the edge computing module.
3. The intelligent fireworks detection method for drone inspection according to claim 2 is characterized in that: Before performing an initial detection on the real-time video stream based on a preset smoke and fire detection model, the method further includes: Parsing the real-time video stream to determine the video encoding format corresponding to the PTZ camera; Based on the video encoding format, the edge computing module matches the corresponding GStreamer decoder preset therein to decode the real-time video stream to convert the frame image in the real-time video stream into an image to be initially inspected in RGB image format.
4. The intelligent fireworks detection method for drone inspection according to claim 3 is characterized in that: The fireworks detection model is trained using the YOLOV8s model as a benchmark model; The real-time video stream is initially detected based on a preset smoke and fire detection model to determine whether there is a suspicious area and corresponding suspicious information, specifically including: Inputting the image to be initially inspected into the fireworks detection model to output a detection result; Among them, the detection result includes at least one of the following: whether there is a suspicious area, a suspicious area report; the suspicious area report includes: suspicious information corresponding to the suspicious area; the suspicious information includes: the number of suspicious areas in the image to be initially inspected, the image pixel coordinate information of the preset image reference position corresponding to each suspicious area, the image size information in the image corresponding to each suspicious area, the detection category of each suspicious area and the corresponding category probability.
5. The intelligent fireworks detection method for drone inspection according to claim 4 is characterized in that: Based on the suspicious information, the GPS coordinates of the suspicious area are calculated, specifically including: The image pixel coordinate information is converted into image physical coordinate coordinate information by transforming the image pixel coordinate system into the image physical coordinate system; The image physical coordinate information is converted into camera coordinate information by transforming the image physical coordinate system into the camera coordinate system; The camera coordinate information is converted into world coordinate information relative to the drone by transforming the camera coordinate system into the drone world coordinate system; The world coordinate information is converted into GPS coordinate information by transforming the world coordinate system into the WGS84 coordinate system.
6. The intelligent fireworks detection method for drone inspection according to claim 5 is characterized in that: Based on the GPS coordinates, an approach and circumvention detection is performed to determine whether risk targets can be continuously detected, specifically including: The UAV flies close to the suspicious area based on the GPS coordinates, and performs a fly-around detection with a preset radius taking the center of the suspicious area as the origin; Determine whether the number of times risk targets are detected during the preset circling time is greater than a preset threshold. If so, determine that risk targets can be detected continuously.
7. The intelligent fireworks detection method for drone inspection according to claim 6 is characterized in that: The number of the detected images is the preset threshold; the fireworks prediction model is obtained based on ResNet50 network training; The corresponding images detected based on the continuous detection of risk targets are input into the preset fireworks prediction model for prediction to generate fireworks risk information, including: The plurality of detected images are cropped based on the GPS coordinates and the suspicious information, and the cropped plurality of detected images are sequentially input into the fireworks prediction model to obtain three-dimensional panoramic view data of the suspicious area and corresponding fireworks risk information; wherein the fireworks risk information at least includes: fireworks risk category.
8. The intelligent fireworks detection method for drone inspection according to claim 7 is characterized in that: The plurality of detected images are cropped based on the GPS coordinates and the suspicious information, specifically comprising: Based on the GPS coordinates and the suspicious information, calibrate the corresponding suspicious areas in the plurality of detected images; The suspicious areas marked in the plurality of detected images are cropped.
9. An intelligent fireworks detection device used for drone inspection, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent fireworks detection method applied to drone inspection as described in any one of claims 1-8.
10. A non-volatile computer storage medium for intelligent fireworks detection for drone inspection, storing computer executable instructions, characterized in that: When the computer executable instructions are executed, an intelligent fireworks detection method for drone inspection as described in any one of claims 1 to 8 is implemented.