Fire detection method, device and equipment based on tunnel robot and medium

Abstract

The invention relates to a fire detection method, device and equipment based on a tunnel robot and a medium. The method comprises the following steps: acquiring an infrared image and a visible light image which are acquired by the tunnel robot and aim at a fire scene in a tunnel; performing image fusion on the infrared image and the visible light image to obtain a fused flame image; and detecting a flame target in the fused flame image to obtain position information of an actual flame area of the fire scene. According to the method of the invention, the flame characterizing features in the infrared image and the visible light image are fused together, and then the position information of the fire area is identified, so that the image features of the two different types of images can be comprehensively considered, and the obtained position information of the actual flame area is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a fire detection method, device, equipment and medium based on a tunnel robot. Background Art

[0002] In the prior art, fire detection in tunnels is usually performed based on one type of image to detect the fire area. In one type of image, the characteristics representing the flame are easily affected by other features in the image, making the final detection result inaccurate. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a fire detection method, device, equipment and medium based on a tunnel robot, aiming to solve at least one of the above technical problems.

[0004] In the first aspect, the technical solution of the present invention to solve the above technical problem is as follows: a fire detection method based on a tunnel robot, the method comprising:

[0005] Acquire infrared images and visible light images of the fire scene in the tunnel through the tunnel robot;

[0006] Performing image fusion on the infrared image and the visible light image to obtain a fused flame image;

[0007] The flame target in the fused flame image is detected to obtain the position information of the actual flame area at the fire scene.

[0008] The beneficial effect of the present invention is that: for two different types of images of the fire scene in the tunnel, namely infrared images and visible light images, image fusion is first performed to fuse the features representing the flames in the infrared images and the visible light images, and then the location information of the fire area is identified, thereby comprehensively considering the image features of the two different types of images, so that the location information of the actual flame area obtained is more accurate.

[0009] Based on the above technical solution, the present invention can also be improved as follows.

[0010] Further, the performing image fusion on the infrared image and the visible light image to obtain a fused flame image includes:

[0011] The infrared image and the visible light image are fused by means of a pre-trained target detection model to obtain a fused flame image, wherein the target detection model is trained based on a deep learning network of Yolov5s, the training data set of the target detection model includes a self-annotated fused image flame data set, and the verification data set of the target detection model includes a self-annotated infrared flame data set, a self-annotated visible light flame data set, and an open source FLAME infrared flame data set.

[0012] Furthermore, the target detection model is trained based on the following method:

[0013] Acquire the training data set, wherein each sample image in the training data set corresponds to an actual result, and for each sample image, the actual result represents a real target frame corresponding to the flame in the corresponding sample image;

[0014] The deep learning network of Yolov5s is trained according to the training data set to obtain a prediction result corresponding to each sample image, wherein the prediction result represents a predicted target frame of the flame in the corresponding sample image;

[0015] According to the prediction results and actual results corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function;

[0016] The target detection model is trained according to the loss value.

[0017] Further, for each of the sample images, there is a common area between the predicted target box and the real target box corresponding to the sample image, and the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function according to the predicted results and actual results corresponding to all the sample images, including:

[0018] For each of the sample images, determining the respective areas of the predicted target box and the real target box corresponding to the sample image, the intersection-over-union ratio between the predicted target box and the real target box corresponding to the sample image, and the area of ​​the common area corresponding to the sample image;

[0019] For each of the sample images, based on the predicted target frame corresponding to the sample image and the area of ​​the target frame with the smallest area in the real target frame, the intersection-over-union ratio corresponding to the sample image, and the area of ​​the common area corresponding to the sample image, determine the sub-loss value corresponding to the sample image;

[0020] According to the sub-loss values ​​corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined.

[0021] Furthermore, the method further comprises:

[0022] According to the position information of the actual flame area at the fire scene and the current position of the tunnel robot, the tunnel robot is controlled to move to the actual flame area to extinguish the fire.

[0023] Further, the current position of the tunnel robot is determined based on a laser radar provided on the tunnel robot;

[0024] The controlling the tunnel robot to move to the actual flame area to extinguish the fire according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot comprises:

[0025] Acquire the angle between the laser radar and the forward direction of the tunnel robot;

[0026] Determine the straight-line distance between the laser radar and the actual flame area according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot;

[0027] The moving distance of the tunnel robot along the track to the actual flame area is determined according to the straight-line distance and the angle, so as to control the tunnel robot to move to the actual flame area for fire extinguishing based on the moving distance.

[0028] Furthermore, the method further comprises:

[0029] The distance from the tunnel robot traveling along the track to the nearest point of the actual flame area is determined according to the angle and the straight-line distance.

[0030] In a second aspect, in order to solve the above technical problems, the present invention further provides a fire detection device based on a tunnel robot, the device comprising:

[0031] An image acquisition module, used to acquire infrared images and visible light images of the fire scene in the tunnel acquired by the tunnel robot;

[0032] An image fusion module, used for fusing the infrared image and the visible light image to obtain a fused flame image;

[0033] The fire detection module is used to detect the flame target in the fused flame image to obtain the position information of the actual flame area at the fire scene.

[0034] In the third aspect, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the fire detection method based on the tunnel robot of the present application is implemented.

[0035] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the fire detection method based on a tunnel robot of the present application is implemented.

[0036] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0038] Figure 1 A schematic flow chart of a fire detection method based on a tunnel robot provided by an embodiment of the present invention;

[0039] Figure 2 A schematic flow chart of another fire detection method based on a tunnel robot provided by an embodiment of the present invention;

[0040] Figure 3 (a) is a schematic diagram of a verification data set provided by an embodiment of the present invention;

[0041] Figure 3 (b) is a schematic diagram of another verification data set provided by an embodiment of the present invention;

[0042] Figure 3 (c) is a schematic diagram of another verification data set provided by an embodiment of the present invention;

[0043] Figure 3 (d) is a schematic diagram of a training data set provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a verification result provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a test result based on an infrared image provided by an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of a test result based on a visible light image provided by an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of a test result based on a fused flame image provided by an embodiment of the present invention;

[0048] Figure 8 A schematic diagram of shooting positions of a tunnel robot provided by an embodiment of the present invention;

[0049] Fig. 9 A schematic structural diagram of a fire detection device based on a tunnel robot provided by one embodiment of the present invention;

[0050] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0052] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0053] The solution provided by the embodiment of the present invention can be applied to any application scenario that requires identification of the location information of the actual flame area at the fire scene in the tunnel. The solution provided by the embodiment of the present invention can be executed by any electronic device, for example, it can be a user's terminal device, including at least one of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device. The user's terminal device can communicate with the tunnel robot to control the tunnel robot.

[0054] The embodiment of the present invention provides a possible implementation method, such as Figure 1 As shown, a flow chart of a fire detection method based on a tunnel robot is provided. The solution can be executed by any electronic device. For the convenience of description, the method provided by the embodiment of the present invention will be described below by taking the terminal device as the execution subject as an example. Figure 1 The method may include the following steps:

[0055] S10, acquiring an infrared image and a visible light image of the fire scene in the tunnel acquired by the tunnel robot;

[0056] S20, performing image fusion on the infrared image and the visible light image to obtain a fused flame image;

[0057] S30, detecting the flame target in the fused flame image to obtain position information of the actual flame area at the fire scene.

[0058] By the method of the present invention, two different types of images of the fire scene in the tunnel, namely, an infrared image and a visible light image, are firstly fused to fuse the features representing the flames in the infrared image and the visible light image, and then the position information of the fire area is identified, thereby comprehensively considering the image features of the two different types of images, so that the position information of the actual flame area obtained is more accurate.

[0059] The scheme of the present invention is further described below in conjunction with the following specific embodiments. In this embodiment, see Figure 2 , a fire detection method based on a tunnel robot is provided, which may include the following steps:

[0060] S10, acquiring an infrared image and a visible light image of the fire scene in the tunnel acquired by the tunnel robot;

[0061] The tunnel robot may be equipped with a dual-spectrum camera, based on which an infrared image and a visible light image may be acquired, both of which include flames. The tunnel robot may be arranged on the top of the tunnel via a track, and travel in the tunnel via the track.

[0062] S20, performing image fusion on the infrared image and the visible light image to obtain a fused flame image;

[0063] Optionally, one possible implementation of S20 is to first perform image registration according to the infrared image and the visible light image, and then perform image fusion based on the image registration result to obtain a fused flame image.

[0064] S30, detecting the flame target in the fused flame image to obtain position information of the actual flame area at the fire scene.

[0065] The above-mentioned location information may be coordinate information.

[0066] Optionally, another implementation of S20 is: by using a pre-trained target detection model, the infrared image and the visible light image are fused to obtain a fused flame image, wherein the target detection model is trained based on the deep learning network of Yolov5s, the training data set of the target detection model includes a self-annotated fused image flame data set, and the verification data set of the target detection model includes a self-annotated infrared flame data set, a self-annotated visible light flame data set, and an open source FLAME infrared flame data set. The self-annotated fused image flame data set can be found in Figure 3 (d) as shown.

[0067] Among them, the validation dataset is used to verify the accuracy of the target detection model, and the specific verification process will be described below. The self-annotated infrared flame dataset refers to a dataset formed by multiple infrared images containing flames, and the corresponding flame area is annotated on each infrared image. Similarly, the self-annotated visible light flame dataset refers to a dataset formed by multiple visible light images containing flames, and the corresponding flame area is annotated on each visible light image. The open source FLAME infrared flame dataset refers to a dataset formed by multiple infrared images containing flames, and the corresponding flame area is annotated on each infrared image. Each annotation described above can be a annotation box, which is used to annotate the flame area in the corresponding image. For details, please refer to Figure 3 (a) to Figure 3 (c) Schematic diagram of different types of verification data shown.

[0068] Optionally, the self-annotated infrared image dataset and the fused flame image dataset are determined by infrared images collected from the perspective of the tunnel robot, thereby ensuring the reliability of the fire scene dataset and enhancing the robustness of the flame recognition algorithm.

[0069] Optionally, the target detection model is trained based on the following method:

[0070] S1, obtaining the training data set, wherein each sample image in the training data set corresponds to an actual result, and for each sample image, the actual result represents a real target frame corresponding to the flame in the corresponding sample image;

[0071] Optionally, if the number of sample images in the training data set is less than the set number, the data enhancement method can be used to enhance each sample image in the training data set to obtain the enhanced training data set for subsequent training. Through data enhancement, the number of samples in the training data set can be supplemented.

[0072] Optionally, the enhancement processing includes but is not limited to rotating at different angles, flipping up and down, left and right, cropping the picture, and splicing the cropped images.

[0073] S2, training the deep learning network of Yolov5s according to the training data set to obtain a prediction result corresponding to each sample image, wherein the prediction result represents a predicted target frame of the flame in the corresponding sample image;

[0074] S3, determining the loss value of the deep learning network of Yolov5s by using the GIoU loss function according to the prediction results and actual results corresponding to all sample images;

[0075] S4: According to the loss value, train and obtain the target detection model.

[0076] In the present application, the advantage of the GIoU loss function is that it not only considers the overlapping area, that is, the common area, but also the non-overlapping area, which can better reflect the relative position relationship between the two bounding boxes (the real target box and the predicted target box). In addition, the value range of GIoU is [-1,1], where GIoU is 1 when the two bounding boxes completely overlap, and -1 when they do not overlap at all and are infinitely far apart, which makes GIoU a better distance metric.

[0077] Optionally, for each of the sample images, there is a common area between the predicted target box and the real target box corresponding to the sample image. In the above S3, according to the predicted results and actual results corresponding to all the sample images, the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function, including:

[0078] S31, for each of the sample images, determining the respective areas of the predicted target box and the real target box corresponding to the sample image, the intersection-over-union ratio between the predicted target box and the real target box corresponding to the sample image, and the area of ​​the common area corresponding to the sample image;

[0079] S32, for each of the sample images, based on the predicted target frame corresponding to the sample image and the area of ​​the target frame with the smallest area in the real target frame, the intersection-over-union ratio corresponding to the sample image, and the area of ​​the common area corresponding to the sample image, determine the sub-loss value corresponding to the sample image;

[0080] S33, determining the loss value of the Yolov5s deep learning network according to the sub-loss values ​​corresponding to all sample images.

[0081] Among them, in S32, for each of the sample images, based on the predicted target box corresponding to the sample image and the area of ​​the target box with the smallest area in the real target box, the intersection-over-union ratio corresponding to the sample image, and the area of ​​the common area corresponding to the sample image, one implementable manner of determining the sub-loss value corresponding to the sample image is:

[0082] For each of the sample images, based on the predicted target box corresponding to the sample image and the area of ​​the target box with the smallest area in the real target box, the intersection-over-union ratio corresponding to the sample image and the area of ​​the common area corresponding to the sample image, the sub-loss value corresponding to the sample image is determined by the following formula:

[0083]

[0084] G loss =1-GIoU

[0085] Among them, IoU represents the intersection over union ratio of the sample image, Ac represents the area of ​​the smallest target box between the predicted target box and the real target box corresponding to the sample image, U represents the area of ​​the common area corresponding to the sample image, and G loss Represents the sub-loss value corresponding to the sample image.

[0086] It can be understood that the loss value of the deep learning network of Yolov5s described above can be the average of the sub-loss values ​​corresponding to the sample image.

[0087] In this application, the GIoU loss function is used to more accurately evaluate the gap between the predicted results and the actual results in fire detection. Compared with the traditional IoU method, GIoU has fewer defects in the calculation of bounding box loss, while retaining the advantages of IoU.

[0088] Optionally, in the above S30, a possible implementation method of detecting the flame target in the fused flame image to obtain the position information of the actual flame area at the fire scene is:

[0089] The fused flame image is input into the fusion detection module, and the flame target is detected by the detection algorithm in the fusion detection module, and finally the predicted frame coordinate information of the actual flame area, that is, the position information of the actual flame area, is output.

[0090] After training the target detection model, the accuracy of the target detection model can be verified based on the validation data set. The solution is as follows:

[0091] The infrared image set in the verification data set (self-annotated infrared flame data set and open source FLAME infrared flame data set) and the verification image in the self-annotated visible light flame data set are respectively input into the target detection model to obtain the detection box output by the target detection model;

[0092] Based on the detection box corresponding to the infrared image set and the detection box corresponding to the visible light data set, determine the verification result of the target detection model, which includes precision, recall and detection accuracy;

[0093] The detection accuracy of the target detection model is verified based on the verification results, so that when the detection accuracy of the target detection model is less than the set value, the model parameters are adjusted and the target detection model is retrained.

[0094] As an example, see Figure 4 The accuracy verification result diagram is shown in Figure 4In this paper, the target detection model trained by YOLOv5s model is used to detect the verification images in the infrared image set, the verification images in the self-annotated visible light flame data set, and the fused flame images in the self-annotated fused image flame data set. The experimental results show that in terms of recognition accuracy, compared with the verification image input in the infrared image set, the fused flame image is improved by 15.36%; compared with the input of the visible light image, the fused flame image is improved by 5.35%. In terms of recall rate, compared with the verification image input in the infrared image set, the fused flame image is improved by 18.80%; compared with the input of the visible light image, the fused flame image is improved by 2.74%. In terms of detection accuracy, compared with the verification image input in the infrared image set, the fused flame image is improved by up to 24.44%; compared with the input of the visible light image, the fused image is improved by 8.85%. The experimental results show that the fused flame image contains the feature information of infrared and visible light images, which can improve the detection effect of the fire area.

[0095] In the present application, in addition to outputting the location information of the actual flame area, the target detection model can also output the probability value of the detection box corresponding to the actual flame area. The size of the probability value represents the probability that the target in the area corresponding to the corresponding detection box is a flame. The larger the probability value, the greater the probability that it is a flame.

[0096] The specific effects of the above verification process can be found in Figures 5 to 7 As shown in Figure 2, an infrared image containing flames is input into the target detection model, and the position information (detection box) of the actual flame area is obtained as follows: Figure 5 As shown, in Figure 5 In the figure, the rectangular box represents the location information of the actual flame area output by the target detection model. By recognizing the infrared image, Figure 5 The infrared characteristics of the flame in the middle area are clearly identified as flames due to their obvious temperature characteristics. Figure 5 The infrared signature of the small flame on the right side is accurately detected as a flame. Figure 5 The infrared image features generated by the high temperature of the vehicle tire on the left side of the center are mistakenly detected as flames. Since the high temperature area in the fire scene will affect the infrared image flame recognition, other high-temperature objects will be misidentified.

[0097] A visible light image containing flames is input into the target detection model, and the position information (detection box) of the actual flame area is obtained as follows: Figure 6 As shown, in Figure 6 In the figure, the rectangular box represents the location information of the actual flame area output by the target detection model. By recognizing the visible light image, Figure 6In the visible light images, there are cases where flame-like objects are misidentified as flames, such as printed flame paper, flames reflected on the water surface, etc. The printed paper between two real brazier flames is misidentified. Figure 6 The printed flame posted on the right rear was also misdetected. Figure 6 The flame reflected by the water under the left brazier was mistakenly identified as a flame. Figure 6 The small flame in the brazier on the right was not detected, resulting in missed detection.

[0098] A fused flame image containing flames is input into the target detection model, and the position information (detection box) of the actual flame area is obtained as follows: Figure 7 As shown, in Figure 7 In the figure, the rectangular box represents the location information of the actual flame area output by the target detection model. By recognizing the fused flame image, Figure 7 The fused flame image in can accurately detect the two burning areas. Figure 7 The large flame on the left can be better identified. Figure 7 The smaller flame area on the right side of can also be identified by fusion of flame images. Figures 5 to 7 The comparison of fire area recognition effects under infrared images, visible light images and fused flame images is shown respectively. Compared with the recognition by a single visual sensor, the recognition accuracy of flame recognition using the fused flame image method has been greatly improved and the misrecognition rate has been reduced.

[0099] In summary, the image fusion method based on YOLOv5s shows high accuracy, recall and precision in flame recognition, which shows that this method is feasible in flame recognition tasks and can effectively reduce the problems of missed detection and false detection of fire areas in complex fire environments. It verifies the reliability of the fused flame image obtained by fusing infrared images and visible light images for flame detection in complex backgrounds.

[0100] Based on the above methods S10 to S30, the fire scene can be efficiently detected and the location information of the actual flame area can be obtained. On this basis, the tunnel robot can be controlled to move to the actual flame area for fire extinguishing according to the location information of the actual flame area at the fire scene and the current position of the tunnel robot.

[0101] Another implementation method is to install a laser radar in the tunnel robot, which can calculate the exact distance from the tunnel robot to the actual flame area based on the TOF flight time.

[0102] The specific implementation is as follows:

[0103] Obtaining the angle between the laser radar and the forward direction of the tunnel robot, that is, the angle between the straight line formed between the position point of the laser radar and the position point represented by the position information of the actual flame area and the forward direction of the tunnel robot, denoted as θ;

[0104] According to the position information of the actual flame area at the fire scene and the current position of the tunnel robot, determine the straight-line distance between the laser radar and the actual flame area, that is, the straight-line distance from the laser radar to the actual flame area, recorded as D;

[0105] According to the straight-line distance D and the angle θ, the moving distance X of the tunnel robot along the track to the actual flame area is determined, so as to control the tunnel robot to move to the actual flame area for fire extinguishing based on the moving distance.

[0106] The current position of the tunnel robot is determined based on a laser radar installed on the tunnel robot. Figure 8 As shown in the schematic diagram, the track where the tunnel robot is located is set parallel to the ground. In the right triangle formed by the current position of the tunnel robot and the position information of the actual flame area, the moving distance X can be calculated based on the following formula, given that the hypotenuse D and the angle θ of the triangle are known:

[0107]

[0108] As an example, assume that the straight-line distance from the laser radar to the actual flame area is D=10 meters, and the angle θ between the laser radar and the forward direction of the tunnel robot is 30°.

[0109] Then, the moving distance X of the tunnel robot along the track can be calculated as follows:

[0110] X=10·cos(30°)

[0111] X≈10·0.866

[0112] X≈8.66 m

[0113] This means that in order to reach the closest point to the fire, the tunnel robot needs to travel approximately 8.66 meters along the track.

[0114] Furthermore, the method further comprises:

[0115] According to the angle and the straight-line distance, the distance from the tunnel robot traveling along the track to the nearest point of the actual flame area is determined, which is recorded as Y. The nearest point refers to a point on the ground.

[0116] The specific implementation method is:

[0117] Based on the angle θ and the straight-line distance D, the distance Y from the tunnel robot to the nearest point of the actual flame area along the track is determined based on the following formula:

[0118]

[0119] Based on Figure 1 The present invention also provides a fire detection device 20 based on a tunnel robot, based on the same principle as the method shown in Fig. 9 As shown in , the fire detection device 20 based on the tunnel robot may include an image acquisition module 210, an image fusion module 220 and an image fusion module 230, wherein:

[0120] An image acquisition module 210 is used to acquire infrared images and visible light images of the fire scene in the tunnel acquired by the tunnel robot;

[0121] An image fusion module 220 is used to fuse the infrared image and the visible light image to obtain a fused flame image;

[0122] The image fusion module 230 is used to detect the flame target in the fused flame image to obtain the position information of the actual flame area at the fire scene.

[0123] Optionally, when the image fusion module 220 fuses the infrared image and the visible light image to obtain a fused flame image, it is specifically used to:

[0124] The infrared image and the visible light image are fused by means of a pre-trained target detection model to obtain a fused flame image, wherein the target detection model is trained based on a deep learning network of Yolov5s, the training data set of the target detection model includes a self-annotated fused image flame data set, and the verification data set of the target detection model includes a self-annotated infrared flame data set, a self-annotated visible light flame data set, and an open source FLAME infrared flame data set.

[0125] Optionally, the target detection model is obtained by training based on the following training modules, wherein the training modules are used to:

[0126] Acquire the training data set, wherein each sample image in the training data set corresponds to an actual result, and for each sample image, the actual result represents a real target frame corresponding to the flame in the corresponding sample image;

[0127] The deep learning network of Yolov5s is trained according to the training data set to obtain a prediction result corresponding to each sample image, wherein the prediction result represents a predicted target frame of the flame in the corresponding sample image;

[0128] According to the prediction results and actual results corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function;

[0129] The target detection model is trained according to the loss value.

[0130] Optionally, for each of the sample images, there is a common area between the predicted target box and the real target box corresponding to the sample image, and the training module determines the loss value of the deep learning network of Yolov5s through the GIoU loss function according to the predicted results and actual results corresponding to all the sample images, specifically for:

[0131] For each of the sample images, determining the respective areas of the predicted target box and the real target box corresponding to the sample image, the intersection-over-union ratio between the predicted target box and the real target box corresponding to the sample image, and the area of ​​the common area corresponding to the sample image;

[0132] For each of the sample images, based on the predicted target frame corresponding to the sample image and the area of ​​the target frame with the smallest area in the real target frame, the intersection-over-union ratio corresponding to the sample image, and the area of ​​the common area corresponding to the sample image, determine the sub-loss value corresponding to the sample image;

[0133] According to the sub-loss values ​​corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined.

[0134] Optionally, the device further comprises:

[0135] The fire extinguishing module is used to control the tunnel robot to move to the actual flame area to extinguish the fire according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot.

[0136] Optionally, the current position of the tunnel robot is determined based on a laser radar provided on the tunnel robot;

[0137] The fire extinguishing module controls the tunnel robot to move to the actual flame area to extinguish the fire according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot, and is specifically used to:

[0138] Acquire the angle between the laser radar and the forward direction of the tunnel robot;

[0139] Determine the straight-line distance between the laser radar and the actual flame area according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot;

[0140] The moving distance of the tunnel robot along the track to the actual flame area is determined according to the straight-line distance and the angle, so as to control the tunnel robot to move to the actual flame area for fire extinguishing based on the moving distance.

[0141] Optionally, the device further comprises:

[0142] The distance determination module is used to determine the distance from the tunnel robot traveling along the track to the nearest point of the actual flame area according to the angle and the straight-line distance.

[0143] The tunnel robot-based fire detection device of the embodiment of the present invention can execute the tunnel robot-based fire detection method provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the tunnel robot-based fire detection device in each embodiment of the present invention correspond to the steps in the tunnel robot-based fire detection method in each embodiment of the present invention. For the detailed functional description of each module of the tunnel robot-based fire detection device, please refer to the description of the corresponding tunnel robot-based fire detection method shown in the previous text, which will not be repeated here.

[0144] Among them, the above-mentioned fire detection device based on the tunnel robot can be a computer program (including program code) running in a computer device. For example, the fire detection device based on the tunnel robot is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiment of the present invention.

[0145] In some embodiments, the fire detection device based on the tunnel robot provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the fire detection device based on the tunnel robot provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the fire detection method based on the tunnel robot provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0146] In other embodiments, the fire detection device based on the tunnel robot provided by the embodiments of the present invention can be implemented in a software manner. Fig. 9 A fire detection device based on a tunnel robot stored in a memory is shown, which may be software in the form of a program and a plug-in, and includes a series of modules, including an image acquisition module 210, an image fusion module 220 and an image fusion module 230, for implementing the fire detection method based on a tunnel robot provided in an embodiment of the present invention.

[0147] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0148] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0149] In an alternative embodiment, an electronic device is provided, such as Fig.10 As shown, Fig.10The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0150] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0151] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0152] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0153] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0154] The electronic device may also be a terminal device. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0155] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0156] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0157] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0158] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0160] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0161] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A fire detection method based on a tunnel robot, characterized in that: The following steps are involved: Acquire infrared images and visible light images of the fire scene in the tunnel through the tunnel robot; Performing image fusion on the infrared image and the visible light image to obtain a fused flame image; The flame target in the fused flame image is detected to obtain the position information of the actual flame area at the fire scene.

2. The method according to claim 1, characterized in that The step of fusing the infrared image and the visible light image to obtain a fused flame image includes: The infrared image and the visible light image are fused by means of a pre-trained target detection model to obtain a fused flame image, wherein the target detection model is trained based on a deep learning network of Yolov5s, the training data set of the target detection model includes a self-annotated fused image flame data set, and the verification data set of the target detection model includes a self-annotated infrared flame data set, a self-annotated visible light flame data set, and an open source FLAME infrared flame data set.

3. The method according to claim 2, characterized in that The target detection model is trained based on the following method: Acquire the training data set, wherein each sample image in the training data set corresponds to an actual result, and for each sample image, the actual result represents a real target frame corresponding to the flame in the corresponding sample image; The deep learning network of Yolov5s is trained according to the training data set to obtain a prediction result corresponding to each sample image, wherein the prediction result represents a predicted target frame of the flame in the corresponding sample image; According to the prediction results and actual results corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function; The target detection model is trained according to the loss value.

4. The method according to claim 3, characterized in that: For each of the sample images, there is a common area between the predicted target box and the real target box corresponding to the sample image, and the loss value of the deep learning network of Yolov5s is determined by the GIoU loss function according to the predicted results and actual results corresponding to all the sample images, including: For each of the sample images, determining the respective areas of the predicted target box and the real target box corresponding to the sample image, the intersection-over-union ratio between the predicted target box and the real target box corresponding to the sample image, and the area of ​​the common area corresponding to the sample image; For each of the sample images, based on the predicted target frame corresponding to the sample image and the area of ​​the target frame with the smallest area in the real target frame, the intersection-over-union ratio corresponding to the sample image, and the area of ​​the common area corresponding to the sample image, determine the sub-loss value corresponding to the sample image; According to the sub-loss values ​​corresponding to all sample images, the loss value of the deep learning network of Yolov5s is determined.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: According to the position information of the actual flame area at the fire scene and the current position of the tunnel robot, the tunnel robot is controlled to move to the actual flame area to extinguish the fire.

6. The method according to claim 5, characterized in that The current position of the tunnel robot is determined based on a laser radar disposed on the tunnel robot; The controlling the tunnel robot to move to the actual flame area to extinguish the fire according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot comprises: Acquire the angle between the laser radar and the forward direction of the tunnel robot; Determine the straight-line distance between the laser radar and the actual flame area according to the position information of the actual flame area at the fire scene and the current position of the tunnel robot; The moving distance of the tunnel robot along the track to the actual flame area is determined according to the straight-line distance and the angle, so as to control the tunnel robot to move to the actual flame area to extinguish the fire based on the moving distance.

7. The method according to claim 6, characterized in that The method further comprises: The distance from the tunnel robot traveling along the track to the nearest point of the actual flame area is determined according to the angle and the straight-line distance.

8. A fire detection device based on a tunnel robot, characterized in that: include: An image acquisition module, used to acquire infrared images and visible light images of the fire scene in the tunnel acquired by the tunnel robot; An image fusion module, used for fusing the infrared image and the visible light image to obtain a fused flame image; The fire detection module is used to detect the flame target in the fused flame image to obtain the position information of the actual flame area at the fire scene.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images