A method, apparatus, device and medium for detecting a pyrotechnic

Abstract

Embodiments of the present application provide a kind of fireworks detection method, device, equipment and medium, belong to fireworks detection technical field.The method determines first target position information by the first image collected by unmanned aerial vehicle, controls unmanned aerial vehicle to fly to the fireworks detection position corresponding to first target position information in the inspection area and rephotographs second image and third image, and according to the second pixel coordinate of second image, the second pixel coordinate of the target firework to be confirmed is mapped in third image, to confirm target firework by the pixel area corresponding to second pixel coordinate, can improve the speed and accuracy of fireworks detection, can solve the problem of limitation in detection reaction speed and accuracy in traditional fireworks detection method.

Description

Technical Field

[0001] This invention relates to the field of fireworks detection technology, and in particular to a method, apparatus, equipment and medium for fireworks detection. Background Technology

[0002] With the continuous development of intelligent monitoring technology, smoke and fire detection algorithms have been widely used in the security field. Traditional fire detection methods mostly rely on temperature sensors and smoke detectors, which have certain limitations in terms of response speed and accuracy. Especially in open outdoor environments or complex indoor spaces, traditional methods often fail to detect fires in time, easily delaying firefighting efforts and causing serious losses. Summary of the Invention

[0003] This invention provides a method, apparatus, equipment, and medium for detecting smoke and fire, which can identify smoke and fire in a timely and accurate manner.

[0004] In a first aspect, this embodiment provides a method for detecting fireworks, the method comprising: controlling a drone to acquire a first image of an inspection area; determining first target location information corresponding to a target fireworks to be confirmed identified from the first image; controlling the drone to fly to the fireworks detection location corresponding to the first target location information in the inspection area according to the first target location information to acquire a second image and a third image; wherein the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; mapping the second pixel coordinates of the target fireworks to be confirmed in the third image based on the first pixel coordinates of the target fireworks to be confirmed in the second image; detecting the pixel area corresponding to the second pixel coordinates in the third image to confirm the target fireworks to be confirmed.

[0005] Secondly, this embodiment provides a fireworks detection device, which includes: a first image acquisition module for controlling a drone to acquire a first image of an inspection area; a first target location information determination module for determining first target location information corresponding to a target fireworks to be confirmed identified from the first image; a second and third image acquisition module for controlling the drone to fly to the fireworks detection location corresponding to the first target location information in the inspection area based on the first target location information to acquire a second image and a third image; wherein the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; a second pixel coordinate determination module for mapping the second pixel coordinate of the target fireworks to be confirmed in the third image based on the first pixel coordinate of the target fireworks to be confirmed in the second image; and a target fireworks confirmation module for detecting the pixel area corresponding to the second pixel coordinate in the third image and confirming the target fireworks to be confirmed.

[0006] Thirdly, this embodiment provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the fireworks detection method according to any embodiment of the present invention.

[0007] Fourthly, this embodiment provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the smoke detection method as described in any embodiment of the present invention.

[0008] Fifthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the smoke detection method as described in any embodiment of the present invention.

[0009] The technical solution of this embodiment involves controlling a drone to acquire a first image of an inspection area; determining the first target location information corresponding to a target firework identified from the first image; controlling the drone to fly to the firework detection location corresponding to the first target location information in the inspection area to acquire a second image and a third image based on the first target location information; wherein the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; mapping the second pixel coordinates of the target firework to be confirmed in the third image based on the first pixel coordinates of the target firework to be confirmed; detecting the pixel area corresponding to the second pixel coordinates in the third image to confirm the target firework. This embodiment, by determining the first target location information through the first image acquired by the drone, controlling the drone to fly to the firework detection location corresponding to the first target location information in the inspection area to retake the second and third images, and mapping the second pixel coordinates of the target firework to be confirmed in the third image based on the first pixel coordinates of the second image, can confirm the target firework through the pixel area corresponding to the second pixel coordinates. This improves the speed and accuracy of firework detection and solves the limitations in detection reaction speed and accuracy in traditional firework detection methods.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a smoke detection method provided in an embodiment of the present invention; Figure 2 This is a schematic flowchart of another smoke detection method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a smoke detection device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a schematic flowchart of a fireworks detection method provided in an embodiment of this application. This disclosure applies to fireworks detection scenarios. The method can be executed by a fireworks detection device, which can be implemented in software and / or hardware, optionally through an electronic device, such as a mobile terminal, PC, or server. Figure 1 As shown, the method includes: S110: Control the drone to collect the first image of the inspection area.

[0016] The drone is equipped with an edge device, which deploys a smoke detection network model. This model can infer in real time whether the captured images contain suspected smoke targets. In this embodiment, the smoke detection network model can be quantized to improve inference speed. The smoke detection network model can be any inference model, such as the YOLOv11-n network model. The smoke detection dataset used to train the model can be an open-source dataset or images captured by drones, cameras, or other hardware. This embodiment can also use generative adversarial networks to generate new smoke images on background images that originally did not contain smoke. Before training the model, the smoke detection dataset can be augmented. Augmentation methods can include random cropping, mosaic enhancement, mixup enhancement, and copy-paste enhancement.

[0017] In this embodiment, there are no restrictions on the inspection area; for example, it can be an inspection area such as a factory, forest, power grid, or industrial park.

[0018] In this embodiment, the type of camera used to acquire the first image is not limited; for example, it can be a visible light camera, an infrared thermal imaging camera, etc. For instance, the camera type used to acquire the first image is a visible light camera, and the first image can be a visible light image. The first image is used to initially screen out suspected smoke targets, i.e., smoke targets to be confirmed.

[0019] In this embodiment, the drone can be controlled to fly along a pre-set route. During the flight, the drone is controlled to vertically downward to collect images of the inspection area, i.e., the first image, so as to achieve full coverage inspection of the inspection area.

[0020] Optionally, controlling the drone to collect the first image of the inspection area includes: setting multiple waypoints for the inspection area; controlling the drone to fly according to the flight path formed by the multiple waypoints; and controlling the drone to collect the first image vertically downwards.

[0021] In this embodiment, several spatial waypoints can be set on the electronic map of the inspection area according to the geographical range of the inspection area, the field of view of the visible light camera on the UAV, and the set inspection overlap rate. The distance between adjacent waypoints can ensure that the images captured by the visible light camera have an overlap greater than the set inspection overlap rate, so as to achieve blind-spot-free coverage of the inspection area.

[0022] In this embodiment, the drone can be controlled to fly sequentially according to preset waypoints to form a route covering the entire inspection area. During the flight, the drone is controlled to maintain a vertical attitude towards the ground (e.g., a pitch angle of -90 degrees) and collect the first image downwards at fixed intervals.

[0023] In this embodiment, by controlling a drone to fly along a preset route and capture first images downwards, fully automated blind-spot-free inspection can be achieved, reducing missed and false detections caused by viewing angle or obstruction, and improving the accuracy of smoke and fire detection. Furthermore, the drone can be controlled to monitor the inspection area in real time, enabling timely detection of smoke and fire.

[0024] In this embodiment, the method of acquiring the first image vertically downwards reduces perspective distortion caused by tilted shooting, providing a high-precision imaging basis for subsequently determining the location information of the first target.

[0025] S120. Determine the location information of the first target corresponding to the target fireworks identified from the first image.

[0026] Among them, the target fireworks to be confirmed can be understood as suspected fireworks targets (candidate fireworks targets) that have not yet been verified.

[0027] The first target location information can be understood as the geographical latitude and longitude coordinates of the target fireworks to be confirmed in the first image.

[0028] In this embodiment, the target fireworks to be confirmed can be identified from the first image through a fireworks detection network model. The camera intrinsic parameter matrix and distortion coefficients corresponding to the first image are used to correct the distortion of the pixel coordinates of the target fireworks to be confirmed. Then, combined with the UAV flight parameters when the first image was acquired, the pixel coordinates of the target fireworks to be confirmed are converted into the first target position information.

[0029] In this embodiment, the conversion from pixels to geographic coordinates (i.e., the first target location information) is achieved through the pinhole camera imaging model, so as to obtain the precise location of the target fireworks to be confirmed, and provide accurate positioning basis for subsequent re-shooting confirmation.

[0030] S130. Based on the first target location information, control the drone to fly to the smoke and fire detection location corresponding to the first target location information in the inspection area and collect the second and third images.

[0031] In this embodiment, after the drone flies to the location corresponding to the first target location information, it can take a second picture. A second image and a third image can be acquired simultaneously. The second image is a visible light verification image taken by the drone after reaching the location corresponding to the first target location information, used to extract the geographical latitude and longitude coordinates of the target smoke / fire to be confirmed. The third image is an infrared thermal imaging verification image acquired synchronously by the drone after reaching the location corresponding to the first target location information, used to extract temperature values ​​to confirm the target smoke / fire.

[0032] In this embodiment, the type of camera used to acquire the second and third images is not limited; for example, it can be a visible light camera or an infrared thermal imaging camera. For instance, the second image can be captured by a visible light camera, and the third image can be captured by an infrared thermal imaging camera.

[0033] Optionally, according to the first target location information, the drone is controlled to fly to the smoke detection location corresponding to the first target location information in the inspection area to collect a second image and a third image, including: controlling the drone to fly to the smoke detection location corresponding to the first target location information; and simultaneously collecting a second image and a third image.

[0034] The cameras used to capture the second and third images are different, and the positional deviation of the two cameras meets the set positional deviation condition; the positional information of the target fireworks in the second image is the same as the positional information of the target fireworks in the third image.

[0035] The positional deviation can be a deviation relative to the installation position or relative attitude. In this embodiment, the camera acquiring the second image and the camera acquiring the third image are two independent imaging devices, and their deviations in relative installation position and relative attitude meet the set positional deviation conditions. In this embodiment, there are no restrictions on the set positional deviation conditions, such as the relative installation position deviation not exceeding 0.5mm and the relative attitude deviation not exceeding 0.05 degrees.

[0036] In this embodiment, the location information corresponding to the target fireworks in the second image and the target fireworks in the third image can both refer to the geographical latitude and longitude coordinates of the target fireworks in the real world. The second image and the third image are taken of the same real target fireworks, and their corresponding geographical latitude and longitude coordinates are consistent.

[0037] In this embodiment, after the drone flies to the smoke detection location (which can be understood as the suspected smoke location) corresponding to the first target location information obtained from the initial screening, it simultaneously acquires the second and third images. This provides homogeneous and synchronous dual-modal basic data for the subsequent confirmation of the target smoke, which can improve the accuracy of subsequent verification or confirmation.

[0038] S140. Based on the first pixel coordinates of the target fireworks to be confirmed in the second image, map the second pixel coordinates of the target fireworks to be confirmed into the third image.

[0039] The first pixel coordinates can be the distorted corner pixel coordinates of the bounding rectangle corresponding to the target fireworks in the second image. The second pixel coordinates can be the distorted corner pixel coordinates of the bounding rectangle corresponding to the target fireworks in the third image.

[0040] In this embodiment, the first pixel coordinates of the target fireworks in the second image can be converted into the geographic latitude and longitude coordinates of the target fireworks in the second image by combining the UAV flight parameters and camera calibration parameters when the second image was acquired. Then, using the geographic latitude and longitude coordinates of the target fireworks in the second image as a unified intermediate reference, and combining the UAV flight parameters and camera calibration parameters when the third image was acquired, the geographic latitude and longitude coordinates of the target fireworks in the second image can be converted into the second pixel coordinates of the target fireworks in the third image.

[0041] S150. Detect the pixel region corresponding to the second pixel coordinate in the third image, and confirm the target fireworks to be confirmed.

[0042] In this embodiment, the outer rectangle corresponding to the target fireworks in the third image can be determined by detecting the pixel area corresponding to the second pixel coordinates. The target fireworks are then confirmed by the temperature values ​​of each pixel within the outer rectangle and the smoke concentration monitored by the smoke sensor.

[0043] Optionally, the step of detecting the pixel region corresponding to the second pixel coordinate in the third image to confirm the target fireworks includes: determining the second circumscribed rectangle corresponding to the target fireworks in the third image based on the pixel region corresponding to the second pixel coordinate in the third image; extracting the temperature value of each pixel within the second circumscribed rectangle; confirming the existence of a fire point if the number of pixels with a temperature value greater than a preset temperature threshold meets a preset quantity condition; and confirming the existence of smoke if the detected smoke concentration is greater than a set concentration threshold.

[0044] The second circumscribed rectangle can be the pixel region (detection region) in the third image that is generated with the second pixel coordinates as the corner points and encloses the target fireworks to be confirmed.

[0045] In this embodiment, no specific limitations are placed on the preset temperature threshold, preset quantity condition, and set concentration threshold. These can all be flexibly adjusted according to the inspection area to adapt to different detection accuracy and false detection rate requirements. For example, the preset temperature threshold is 100 degrees Celsius. The preset quantity condition can be the proportion of high-temperature pixels to the total pixels within the second outer rectangle, such as 60%. The set concentration threshold can be 0.3 mg / m³. 3 .

[0046] In this embodiment, the temperature values ​​of all pixels within the second outer rectangle are extracted. The number of pixels within the second outer rectangle whose temperature values ​​exceed a preset temperature threshold is counted. It is then determined whether a preset quantity condition is met. If the preset quantity condition is met, a fire point is confirmed to exist, i.e., there is real smoke or fire, triggering a fire alarm. If the preset quantity condition is not met, it can be assumed that there is no fire point, ruling out false fire detection.

[0047] In this embodiment, the method for obtaining smoke concentration is not limited; for example, it can be an airborne smoke sensor, a smoke recognition algorithm, etc. In this embodiment, if the detected smoke concentration is greater than a set concentration threshold, the presence of smoke can be confirmed.

[0048] In this embodiment, by using dual verification of pixel temperature and smoke concentration, abnormal interference can be eliminated, greatly reducing the false detection rate and ensuring the accuracy of smoke detection results, facilitating timely notification to on-site personnel. The technical solution of this embodiment involves controlling a drone to acquire a first image of an inspection area; determining the first target location information corresponding to the target smoke identified from the first image; controlling the drone to fly to the smoke detection location corresponding to the first target location information in the inspection area to acquire a second image and a third image based on the first target location information; wherein the second image is a visible light verification image, and the third image is an infrared thermal imaging verification image; based on the first pixel coordinates of the target smoke in the second image, mapping the second pixel coordinates of the target smoke in the third image; detecting the pixel area corresponding to the second pixel coordinates in the third image, and confirming the target smoke. In this embodiment, the location information of a first target is determined by a first image captured by a drone. The drone is then controlled to fly to the smoke detection location corresponding to the first target location information in the inspection area to retake the second and third images. The second pixel coordinates of the target smoke to be confirmed are mapped in the third image based on the first pixel coordinates of the second image. The target smoke is then confirmed by the pixel area corresponding to the second pixel coordinates. This method can improve the speed and accuracy of smoke detection and solve the problem of limitations in detection response speed and accuracy in traditional smoke detection methods.

[0049] In this embodiment, compared to traditional methods, drones can be flexibly deployed at high altitudes or in hard-to-reach areas. Equipped with visual sensors, they can significantly expand the monitoring field of view, compensating for blind spots in fixed ground-based monitoring. Combined with the technical solution provided by this embodiment, drones can not only achieve early fire detection but also reduce missed and false detections caused by viewing angles or obstructions through multi-angle, dynamic cruising. By processing the dual-modal data of the second and third images, more reliable auxiliary verification information is provided for smoke and fire detection, further improving the detection accuracy and robustness in complex environments, and protecting personal and property safety.

[0050] Figure 2 This is a flowchart illustrating another fireworks detection method provided by an embodiment of the present invention. This embodiment is a further optimization of the above embodiment. In this embodiment, the definition of "determining the first target location information corresponding to the target fireworks identified from the first image" and the definition of "determining the second pixel coordinates of the target fireworks in the third image based on the first pixel coordinates of the target fireworks in the second image" are further optimized. Figure 2 As shown, this embodiment provides a smoke detection method, which specifically includes the following steps: S201, Control the drone to collect the first image of the inspection area.

[0051] S202, Obtain the first camera calibration parameters and the first UAV flight parameters corresponding to the first image.

[0052] The first camera calibration parameters include the camera intrinsic parameter matrix and distortion coefficients used to acquire the first image. The first UAV flight parameters can be the UAV flight parameters used to acquire the first image, such as flight altitude and yaw angle.

[0053] S203. Using the calibration parameters of the first camera, the pixel coordinates of the first distorted corner point corresponding to the target fireworks in the first image are corrected to obtain the pixel coordinates of the first standard corner point.

[0054] It should be noted that the first distorted corner pixel coordinates, the first pixel coordinates, and the second pixel coordinates can all include two corner coordinates, such as the lower left corner coordinates and the upper right corner coordinates.

[0055] For example, taking any corner coordinate (x1, y1) as an example, the first distorted corner pixel coordinate is corrected using the camera intrinsic parameter matrix camera_matrix and distortion coefficients dist_coeffs of the first camera calibration parameters. The corrected coordinates, i.e., the first standard corner pixel coordinates, are (x1_distorted, y1_distorted).

[0056] S204. Based on the first pixel offset of the first standard corner point pixel coordinates relative to the center of the first image, and the first ground distance corresponding to a single pixel, determine the first planar ground offset of the target smoke in the first image relative to the ground point below the drone.

[0057] The first ground distance is obtained based on the calibration parameters of the first camera and the flight parameters of the first UAV.

[0058] For example, the camera intrinsic parameter matrix of the first camera calibration parameters, `camera_matrix`, is [[fx,0,cx],[0,fy,cy],[0,0,1], and the distortion coefficients, `dist_coeffs`, are [k1,k2,p1,p2,k3]. At an altitude of H for the first UAV's flight parameters, the first ground distance corresponding to a single pixel includes the distance represented by the pixel in the x-direction and the distance represented by the pixel in the y-direction of the first image, respectively represented as: `ground_per_pixel_x` is H / fx, and `ground_per_pixel_y` is height / fy.

[0059] The first pixel offset of the first standard corner point pixel coordinates relative to the center of the first image includes the offset in the x direction and the offset in the y direction, which are respectively represented as: dx_pix is ​​cx-x1_distorted, and dy_pix is ​​cy-y1_distorted.

[0060] The first plane ground offset includes the ground x-direction offset and y-direction offset, which are respectively represented as: dx_ground is dx_pix × ground_per_pixel_x, and dy_ground is dy_pix × ground_per_pixel_y.

[0061] S205. Based on the flight parameters of the first UAV, convert the first plane offset into the first eastward offset and the first northward offset in the geodetic coordinate system.

[0062] For example, the first UAV flight parameters also include yaw angle. The yaw angle and roll angle are converted into an equivalent yaw angle, which is equal to the sum of the yaw angle and roll angle. The first plane offset is converted to the east-west and north-south directions: the first eastward offset dx_geo is dx_ground×cos_yaw (cosine of yaw) + dy_ground×sin_yaw (sine of yaw), and the first northward offset dy_geo is -dx_ground×sin_yaw + dy_ground×cos_yaw. A positive value for dx_geo indicates that the target firework is east of the center of the photo, and vice versa; a positive value for dy_geo indicates that the target firework is north of the center of the photo, and vice versa.

[0063] S206. Based on the first eastward offset, the first northward offset, and the flight parameters of the first UAV, determine the position information corresponding to the single corner point of the target smoke in the first image.

[0064] For example, the first eastward and first northward offsets are then converted into latitude and longitude offsets. One latitude offset, dist_per_lat, is 111319.55 meters. The longitude offset is related to the longitude of the UAV itself in the flight parameters of the first UAV, and one longitude offset, dist_per_lon, is dist_per_lat × cos(lat_rad), where lat_rad is the longitude in radians. Therefore, the longitude offset dlon of the target firework to be confirmed is dx_geo / dist_per_lon, and the latitude offset dlat is dy_geo / dist_per_lat. Adding the latitude and longitude offsets to the UAV's own latitude and longitude in the flight parameters of the first UAV yields the geographic latitude and longitude coordinates (res_lon, res_lat) of a corner point, which is the location information corresponding to the single corner point of the target firework to be confirmed in the first image.

[0065] S207. Determine the first target location information corresponding to the target fireworks to be confirmed in the first image based on the location information of multiple corner points.

[0066] Among them, multiple corner points are the two opposite corner points of the first circumscribed rectangle of the target fireworks to be confirmed in the first image.

[0067] For example, calculating the coordinates of another corner point (x2, y2) can also yield the corresponding position information, and averaging the two can give the position information of the first target.

[0068] Optionally, the first target location information corresponding to the target fireworks to be confirmed can be determined based on the location information of multiple corner points, including: taking the average of the location information of two opposite corner points to obtain the first target location information corresponding to the target fireworks to be confirmed.

[0069] The location information is the geographic latitude and longitude coordinates.

[0070] In this embodiment, by averaging the position information of two corner points to obtain the center of the outer rectangle corresponding to the target firework to be confirmed, and using it as the first target position information, the positioning accuracy of the target firework to be confirmed can be improved.

[0071] S208. Based on the first target location information, control the drone to fly to the smoke and fire detection location corresponding to the first target location information in the inspection area and collect the second and third images.

[0072] For example, the camera parameters of the second image are: width1, height1, camera focal length fx1, fy1, camera optical center cx1, cy1, distortion coefficient dist_coeffs1, and the drone flight parameters corresponding to the second image can be flight altitude H1, latitude and longitude lon1 and lat1 captured by the drone's visible light camera, yaw1, pitch1, and roll1.

[0073] For example, the width and height of the third image are width2 and height2, the camera focal length is fx2 and fy2, the camera optical center is cx2 and cy2, the distortion coefficient is dist_coeffs2, and the flight parameters of the second UAV can be flight altitude H2, latitude and longitude lon2 and lat2 captured by the infrared camera, yaw angle yaw2, pitch angle pitch2, and roll angle roll2.

[0074] S209. Determine the second target position information corresponding to the target fireworks in the second image based on the first pixel coordinates, and use it as the third target position information corresponding to the target fireworks in the third image.

[0075] In this embodiment, the specific process of determining the location information of the second target is similar to that of determining the location information of the first target, and will not be described again. In this embodiment, the location information of the second target firework in the second image can be used as the location information of the third target firework in the third image, that is, the geographical latitude and longitude coordinates of the target firework in the second image can be used as the geographical latitude and longitude coordinates of the target firework in the third image.

[0076] For example, the second target location information corresponding to the two corner point coordinates of the first pixel coordinates are respectively: (x11, y11) geographical longitude res_lon1, latitude res_lat1; (x22, y22) geographical longitude res_lon2, latitude res_lat2.

[0077] S210. Determine the position offset based on the third target's position information and the second UAV's flight parameters corresponding to the third image.

[0078] For example, based on the infrared camera capturing latitude and longitude coordinates lon2 and lat2 according to the flight parameters of the second UAV, the positional offset of the latitude and longitude of one corner point corresponding to the third image from the center of the third image (latitude and longitude coordinates lon2 and lat2) is dlon1 = res_lon1 – lon2, dlat1 = res_lat1 – lat2. The positional offset of the latitude and longitude of the other corner point from the center of the third image is dlon2 = res_lon2 – lon2, dlat2 = res_lat2 – lat2.

[0079] S211. Convert the position offset into the second eastward offset and the second northward offset in the geodetic coordinate system.

[0080] For example, dist_per_lat and dist_per_lon can be derived from the coordinates lon2 and lat2 of the infrared camera's shooting point. For the geodetic offset of one corner point, the second eastward offset dx1 in the geodetic coordinate system is dlon1×dist_per_lon, and the second northward offset d1y is dloat1×dist_per_lat; for the geodetic offset of another corner point, the second eastward offset dx2 in the geodetic coordinate system is dlon2×dist_per_lon, and the second northward offset dy2 is dlat2×dist_per_lat.

[0081] S212. Based on the flight parameters of the second UAV, the eastward offset and the northward offset are converted into the second plane ground offset of the smoke of the target to be confirmed in the third image relative to the ground point below the UAV.

[0082] For example, since the infrared camera also has a yaw angle yaw2 when taking pictures, and the pitch angle pitch2 is -90°, yaw2 and roll2 are converted into an equivalent yaw angle yaw_visible, which is equal to yaw2 plus roll2. With the equivalent yaw angle, the ground offset can be projected onto the displacement in the x and y directions of the third image. The second-plane ground offset of one corner point of the bounding rectangle of the target fireworks to be confirmed includes the x-direction displacement dx1_ground_visible and the y-direction displacement dy1_ground_visible, where dx1_ground_visible is dx1×cos_yaw_visible – dy1×sin_yaw_visible and dy1_ground_visible is dx1×sin_yaw_visible + dy1×cos_yaw; the second-plane ground offset of the other corner point includes the x-direction displacement dx2_ground_visible and the y-direction displacement dy2_ground_visible, where dx2_ground_visible is dx2×cos_yaw_visible – dy2×sin_yaw_visible and dy2_ground_visible is dx2×sin_yaw_visible + dy2×cos_yaw_visible.

[0083] S213. Based on the second ground distance corresponding to a single pixel, convert the second plane ground offset into a second pixel offset.

[0084] The second ground distance is obtained based on the second camera calibration parameters corresponding to the third image and the second UAV flight parameters.

[0085] For example, at an altitude of H2, the second ground distance corresponding to a single pixel in the third image includes the distance represented by one pixel in the x-direction: ground_per_pixel_x_visible, and the distance in the y-direction: ground_per_pixel_y_visible. ground_per_pixel_x_visible is H2 / fx2, and ground_per_pixel_y_visible is H2 / fy2. Therefore, the second plane ground offset can be converted into the pixel distance between the target firework and the center point of the third image. The second pixel offset includes the pixel distance in the x-direction: dx_pix_visible, and the pixel distance in the y-direction: dy_pix_visible. dx_pix_visible = dx_ground_visible / ground_per_pixel_x_visible; dy_pix_visible = dy_ground_visible / ground_per_pixel_y_visible.

[0086] S214. Convert the second pixel offset into the second standard corner pixel coordinates.

[0087] For example, the theoretical undistorted corner pixel coordinates, i.e., the second standard corner pixel coordinates, can be calculated using the optical centers cx2 and cy2 of a visible light camera. The second standard corner pixel coordinates (undistorted_x, undistorted_y) are: undistorted_x = dx_pix_visible + cx2, undistorted_y = cy2 - dy_pix_visible.

[0088] S215. Using the calibration parameters of the second camera, the pixel coordinates of the second standard corner point are transformed into the pixel coordinates of the second distorted corner point, which are then used as the second pixel coordinates.

[0089] For example, since infrared light also exhibits distortion, the theoretically undistorted second standard corner pixel coordinates need to be distorted. This process can be accomplished using the infrared camera distortion coefficients dist_coeffs2 and the visible camera intrinsic parameters camera_matrix_visible matrix [[fx2,0,cx2],[0,fy2,cy2],[0,0,1]], thereby obtaining the second pixel coordinates.

[0090] S216. Confirm the target fireworks based on the second pixel coordinates.

[0091] This embodiment provides a complete coordinate transformation link by clearly defining the corner pixel coordinates of the second or first image into geographic latitude and longitude coordinates, and then determining the pixel coordinates of the third image using the geographic latitude and longitude coordinates of the second image. This enables pixel mapping from visible light images to infrared images without the need for dual-camera joint extrinsic parameter calibration, resulting in stronger adaptability, stable mapping accuracy, significantly reduced maintenance costs, and stronger robustness. It effectively improves the accuracy and convenience of secondary confirmation of the authenticity of fireworks.

[0092] Figure 3 This is a schematic diagram of a smoke detection device provided in an embodiment of the present invention. This device is applicable to smoke detection and can be implemented in hardware and / or software, and is generally integrated into an electronic device. For example... Figure 3 As shown, the device includes: a first image acquisition module 31, a first target location information determination module 32, a second and third image acquisition module 33, a second pixel coordinate determination module 34, and a target smoke and fire confirmation module 35, wherein, The first image acquisition module 31 is used to control the drone to acquire the first image of the inspection area; The first target location information determination module 32 is used to determine the first target location information corresponding to the target fireworks to be confirmed identified from the first image; The second and third image acquisition modules 33 are used to control the UAV to fly to the smoke and fire detection position corresponding to the first target location information in the inspection area according to the first target location information to acquire the second and third images; wherein, the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; The second pixel coordinate determination module 34 is used to map the second pixel coordinate of the target fireworks to be confirmed in the third image based on the first pixel coordinate of the target fireworks to be confirmed in the second image. The target fireworks confirmation module 35 is used to detect the pixel region corresponding to the second pixel coordinate in the third image and to confirm the target fireworks to be confirmed.

[0093] The technical solution of this embodiment involves controlling a drone to acquire a first image of an inspection area via a first image acquisition module; determining the first target location information corresponding to the target fireworks identified from the first image via a first target location information determination module; controlling the drone to fly to the fireworks detection location corresponding to the first target location information in the inspection area via a second and third image acquisition module based on the first target location information to acquire a second image and a third image; mapping the second pixel coordinates of the target fireworks to be confirmed in the third image based on the first pixel coordinates of the target fireworks to be confirmed in the second image via a second pixel coordinate determination module; and detecting the pixel area corresponding to the second pixel coordinates in the third image via a target fireworks confirmation module to confirm the target fireworks to be confirmed. In this embodiment, the location information of a first target is determined by a first image collected by a drone. The drone is then controlled to fly to the smoke detection location corresponding to the first target location information in the inspection area to retake the second and third images. The second pixel coordinates of the target smoke to be confirmed are mapped in the third image based on the first pixel coordinates of the second image. The target smoke is then confirmed by the pixel area corresponding to the second pixel coordinates. This method can improve the speed and accuracy of smoke detection and solve the problem of limitations in reaction speed and accuracy in traditional smoke detection methods.

[0094] Optionally, the first target location information determination module is specifically used for: acquiring the first camera calibration parameters and the first UAV flight parameters corresponding to the first image; correcting the first distorted corner pixel coordinates corresponding to the target fireworks to be confirmed in the first image using the first camera calibration parameters to obtain the first standard corner pixel coordinates; determining the first planar ground offset of the target fireworks to be confirmed in the first image relative to the ground point below the UAV based on the first pixel offset of the first standard corner pixel coordinates relative to the center of the first image and the first ground distance corresponding to a single pixel; wherein, the first ground distance is obtained based on the first camera calibration parameters and the first UAV flight parameters; converting the first planar offset into a first eastward offset and a first northward offset in the geodetic coordinate system based on the first UAV flight parameters; determining the location information corresponding to a single corner of the target fireworks to be confirmed in the first image based on the first eastward offset, the first northward offset, and the first UAV flight parameters; and determining the first target location information corresponding to the target fireworks to be confirmed in the first image based on the location information of multiple corner points.

[0095] Wherein, the plurality of corner points are two opposite corner points of the first circumscribed rectangle of the target fireworks to be confirmed in the first image. Optionally, the first target location information determination module is further configured to: average the location information of the two opposite corner points to obtain the first target location information corresponding to the target fireworks to be confirmed. Wherein, the location information is geographic latitude and longitude coordinates.

[0096] Optionally, the second pixel coordinate determination module is specifically used for: determining the second target position information corresponding to the target fireworks to be confirmed in the second image based on the first pixel coordinates, as the third target position information corresponding to the target fireworks to be confirmed in the third image; determining the position offset based on the third target position information and the second UAV flight parameters corresponding to the third image; converting the position offset into a second eastward offset and a second northward offset in the geodetic coordinate system; converting the eastward offset and the northward offset into a second plane ground offset of the target fireworks to be confirmed relative to the ground point below the UAV in the third image based on the second UAV flight parameters; converting the second plane ground offset into a second pixel offset according to the second ground distance corresponding to a single pixel; wherein the second ground distance is obtained according to the second camera calibration parameters corresponding to the third image and the second UAV flight parameters; converting the second pixel offset into second standard corner pixel coordinates; and using the second camera calibration parameters, converting the second standard corner pixel coordinates into second distorted corner pixel coordinates, as the second pixel coordinates.

[0097] Optionally, the target smoke confirmation module is specifically used to: determine the second circumscribed rectangle corresponding to the target smoke in the third image based on the pixel region corresponding to the second pixel coordinate in the third image; extract the temperature value of each pixel within the second circumscribed rectangle; if the number of pixels with a temperature value greater than a preset temperature threshold meets a preset quantity condition, then the existence of a fire point is confirmed; if the detected smoke concentration is greater than a set concentration threshold, then the existence of smoke is confirmed.

[0098] Optionally, the second and third image acquisition modules are specifically used to: control the UAV to fly to the smoke detection position corresponding to the first target position information; and simultaneously acquire the second and third images; wherein the cameras acquiring the second and third images are different, and the positional deviation of the two cameras meets the set positional deviation condition; the positional information corresponding to the target smoke in the second image is the same as the positional information corresponding to the target smoke in the third image.

[0099] Optionally, the first image acquisition module is specifically used to: set multiple waypoints for the inspection area; control the UAV to fly according to the flight path formed by the multiple waypoints; and control the UAV to acquire a first image vertically downwards.

[0100] The smoke detection device provided in the embodiments of the present invention can execute the smoke detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0101] Figure 4 This is a schematic diagram of an electronic device provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0102] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 and a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0103] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as smoke detection methods.

[0105] In some embodiments, the fireworks detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the fireworks detection method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the fireworks detection method by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0111] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0112] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the smoke detection method provided in any embodiment of this invention.

[0113] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0115] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting smoke and fire, characterized in that, include: Control the drone to collect the first image of the inspection area; Determine the location information of the first target corresponding to the target fireworks identified from the first image; Based on the first target location information, the UAV is controlled to fly to the smoke and fire detection location corresponding to the first target location information in the inspection area to collect a second image and a third image; wherein, the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; Based on the first pixel coordinates of the target fireworks to be confirmed in the second image, the second pixel coordinates of the target fireworks to be confirmed are mapped in the third image; The pixel region corresponding to the second pixel coordinate in the third image is detected to confirm the target fireworks.

2. The method according to claim 1, characterized in that, The step of determining the location information of the first target corresponding to the target fireworks identified from the first image includes: Obtain the first camera calibration parameters and the first UAV flight parameters corresponding to the first image; The first standard corner pixel coordinates are obtained by correcting the first distorted corner pixel coordinates corresponding to the target fireworks in the first image using the first camera calibration parameters. Based on the first pixel offset of the first standard corner point pixel coordinates relative to the center of the first image, and the first ground distance corresponding to a single pixel, the first planar ground offset of the target smoke in the first image relative to the ground point below the drone is determined; wherein, the first ground distance is obtained based on the first camera calibration parameters and the first drone flight parameters; Based on the flight parameters of the first UAV, the first planar offset is converted into the first eastward offset and the first northward offset in the geodetic coordinate system; Based on the first eastward offset, the first northward offset, and the first UAV flight parameters, the position information corresponding to the single corner point of the target firework to be confirmed in the first image is determined; The first target location information corresponding to the target fireworks to be confirmed in the first image is determined based on the position information of multiple corner points.

3. The method according to claim 2, characterized in that, in, The multiple corner points are the two opposite corner points of the first circumscribed rectangle of the target fireworks to be confirmed in the first image; Determining the first target location information corresponding to the target fireworks to be confirmed based on the location information of multiple corner points includes: The average value of the position information of the two diagonal points is used to obtain the first target position information corresponding to the target fireworks to be confirmed; wherein, the position information is geographical latitude and longitude coordinates.

4. The method according to claim 1, characterized in that, The step of mapping the second pixel coordinates of the target fireworks to be confirmed in the third image based on the first pixel coordinates of the target fireworks in the second image includes: Based on the first pixel coordinates, the second target location information corresponding to the target fireworks to be confirmed in the second image is determined, and used as the third target location information corresponding to the target fireworks to be confirmed in the third image. The position offset is determined based on the third target location information and the second UAV flight parameters corresponding to the third image; The position offset is converted into a second eastward offset and a second northward offset in the geodetic coordinate system; Based on the flight parameters of the second UAV, the eastward offset and the northward offset are converted into the second plane ground offset of the smoke of the target to be confirmed relative to the ground point below the UAV in the third image; Based on the second ground distance corresponding to a single pixel, the second planar ground offset is converted into a second pixel offset; wherein, the second ground distance is obtained based on the second camera calibration parameters corresponding to the third image and the second UAV flight parameters; Convert the second pixel offset into the second standard corner pixel coordinates; Using the second camera calibration parameters, the second standard corner point pixel coordinates are transformed into the second distorted corner point pixel coordinates, which are then used as the second pixel coordinates.

5. The method according to claim 1, characterized in that, The step of detecting the pixel region corresponding to the second pixel coordinates in the third image and confirming the target fireworks includes: Based on the pixel region corresponding to the second pixel coordinates in the third image, determine the second outer rectangle corresponding to the target fireworks to be confirmed in the third image; Extract the temperature value of each pixel within the second bounding rectangle; If the number of pixels with a temperature value greater than the preset temperature threshold meets the preset quantity condition, then a fire point is confirmed to exist. If the detected smoke concentration is greater than the set concentration threshold, the presence of smoke is confirmed.

6. The method according to claim 1, characterized in that, The step of controlling the drone to fly to the smoke detection position corresponding to the first target location information in the inspection area based on the first target location information to collect a second image and a third image includes: Control the drone to fly to the smoke detection location corresponding to the first target location information; Simultaneously, the second image and the third image are acquired; wherein the cameras used to acquire the second image and the third image are different, and the positional deviation of the two cameras meets a set positional deviation condition; the positional information corresponding to the target fireworks to be confirmed in the second image is the same as the positional information corresponding to the target fireworks to be confirmed in the third image.

7. The method according to claim 1, characterized in that, The first image of the inspection area collected by the controlled drone includes: Multiple waypoints are set for the inspection area; The drone is controlled to fly according to the flight path formed by the multiple waypoints, and the drone is controlled to capture the first image vertically downwards.

8. A smoke and fire detection device, characterized in that, include: The first image acquisition module is used to control the drone to acquire the first image of the inspection area; The first target location information determination module is used to determine the first target location information corresponding to the target fireworks identified from the first image. The second and third image acquisition modules are used to control the UAV to fly to the smoke and fire detection position corresponding to the first target location information in the inspection area according to the first target location information to acquire the second and third images; wherein, the second image is a visible light verification image and the third image is an infrared thermal imaging verification image; The second pixel coordinate determination module is used to map the second pixel coordinate of the target fireworks to be confirmed in the third image based on the first pixel coordinate of the target fireworks to be confirmed in the second image. The target fireworks confirmation module is used to detect the pixel region corresponding to the second pixel coordinate in the third image and to confirm the target fireworks to be confirmed.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the fireworks detection method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the fireworks detection method as described in any one of claims 1-7.

Images