Binocular holder fire source automatic positioning method and device based on deep learning

Abstract

The invention relates to the technical field of emergency fire fighting, in particular to a binocular holder fire source automatic positioning method and device based on deep learning. According to the invention, a target fire environment is shot through a cradle head carrying a binocular camera, the shot video stream is processed to obtain aligned video frames and a parallax map, and then the video frames are analyzed in real time by using a trained flame detection model to detect the pixel position of the flame so as to obtain the parallax. And finally, the absolute position of the fire source is calculated by combining parameters of the binocular camera and the holder. Accurate flame detection and positioning are achieved in the whole process, manual intervention is reduced, and the safety coefficient of personnel is increased. According to the method, the problems that application of deep learning to fire source positioning in the prior art is difficult to realize and the effect has improvement space are solved.

Description

Technical Field

[0001] The present invention relates to the field of emergency fire fighting technology, and more specifically, to: 1. A binocular pan-tilt automatic fire source positioning method based on deep learning; 2. A binocular pan-tilt automatic fire source positioning device based on deep learning. Background Art

[0002] Traditional fire detection usually relies on manual inspection or simple sensor detection - these methods have problems such as slow response, inaccurate positioning, low safety, etc. when facing complex and changing fire environments, and it is difficult to cope with the rapid spread of fire.

[0003] In addition, traditional monocular cameras cannot accurately measure the distance to the fire source due to the lack of depth information, resulting in inaccurate fire-fighting strategy planning. Even if some systems are equipped with flame and smoke detection functions, they are unable to quickly and accurately determine the specific location of the fire source. This makes fire-fighting operations inefficient when responding to sudden fires, prolongs rescue time, and affects the fire-fighting effect.

[0004] In recent years, the application of deep learning in the field of target detection has made great progress. The inventor considered applying deep learning to fire source location, but found that how to achieve it is still a technical problem, and there are still problems such as difficulty in detecting tiny flame smoke targets, and easy false detection or missed detection in complex and changeable fire environments. Summary of the invention

[0005] Based on this, it is necessary to provide a binocular pan-tilt automatic fire source positioning method and device based on deep learning in order to address the problems that the existing application of deep learning to fire source positioning is difficult to implement and the effect has room for improvement.

[0006] The present invention is implemented by the following technical solutions:

[0007] In a first aspect, the present invention discloses a binocular pan-tilt automatic fire source positioning method based on deep learning, comprising:

[0008] Step 1: Use a pan / tilt equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams;

[0009] Among them, the binocular camera has completed binocular positioning in advance;

[0010] Step 2: decompose the left and right video streams into a plurality of left and right video frames in time sequence, and take the left and right video frames at the same time as a group;

[0011] Step 3: stereo correction and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated;

[0012] Step 4: Detect several left-channel video frames in chronological order through the trained flame detection model, until a fire source is detected in a left-channel video frame, and output the pixel position (x_l, y_l) of the fire source in the left-channel video frame, and then obtain the corresponding disparity value d0 from the left disparity map based on (x_l, y_l); wherein x_l represents the pixel horizontal coordinate of the fire source in the left-channel video frame, and y_l represents the pixel vertical coordinate of the fire source in the left-channel video frame;

[0013] Or, several right-channel video frames are detected in chronological order through the trained flame detection model, until a fire source is detected in a right-channel video frame, and the pixel position (x_r, y_r) of the fire source in the right-channel video frame is output, and then the corresponding disparity value d is obtained from the right disparity map based on (x_r, y_r); wherein x_r represents the pixel horizontal coordinate of the fire source in the right-channel video frame, and y_r represents the pixel vertical coordinate of the fire source in the right-channel video frame;

[0014] Step 5: Based on the disparity value d0 and combined with the parameters of the binocular camera and the gimbal, the absolute position (x, y, z) of the fire source is calculated;

[0015] Among them, x represents the absolute X-axis coordinate of the fire source, y represents the absolute Y-axis coordinate of the fire source, and z represents the absolute Z-axis coordinate of the fire source.

[0016] This deep learning-based binocular pan-tilt automatic fire source positioning method implements the method or process according to the embodiment of the present disclosure.

[0017] In a second aspect, the present invention discloses a binocular pan-tilt automatic fire source positioning device based on deep learning, which uses the binocular pan-tilt automatic fire source positioning method based on deep learning disclosed in the first aspect.

[0018] The binocular pan-tilt fire source automatic positioning device based on deep learning includes: a pan-tilt equipped with a binocular camera and a host computer.

[0019] The gimbal equipped with a binocular camera is used to perform step one of the binocular gimbal fire source automatic positioning method based on deep learning as disclosed in the first aspect.

[0020] The host computer is used to perform steps 2 to 5 of the binocular pan-tilt automatic fire source positioning method based on deep learning as disclosed in the first aspect.

[0021] This deep learning-based binocular pan-tilt automatic fire source positioning device implements the method or process according to the embodiment of the present disclosure.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention uses a pan-tilt head equipped with a binocular camera to shoot the target fire environment, then processes the shot video stream to obtain aligned video frames and disparity maps, then uses a trained flame detection model to analyze these video frames in real time to detect the pixel position of the flame and then obtain the disparity, and finally calculates the absolute position of the fire source by combining the parameters of the binocular camera and the pan-tilt head; the whole process realizes accurate flame detection and positioning, reduces manual intervention, and increases the safety factor of personnel.

[0024] 2. The present invention realizes the application of deep learning to fire source location detection, and designs a fire source detection model to enhance the adaptability and detection accuracy to complex fire scenes, which can respond to fires quickly and make firefighting more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A flow chart of a binocular pan-tilt automatic fire source positioning method based on deep learning provided in Example 1 of the present invention;

[0027] Figure 2 for Figure 1 Structural diagram of the flame detection model;

[0028] Figure 3 for Figure 2 The structural diagram of the CBS layer;

[0029] Figure 4 for Figure 2 Structural diagram of the GELAN layer. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component at the same time. When a component is considered to be "fixed to" another component, it may be directly fixed on the other component or there may be a central component at the same time.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" used herein includes any and all combinations of one or more related listed items.

[0033] Example 1

[0034] See also Figure 1 , Figure 1 The flow chart of a binocular PTZ fire source automatic positioning method based on deep learning is presented, which includes:

[0035] Step 1: Use a pan / tilt equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams;

[0036] Among them, the binocular camera has completed binocular calibration in advance (it is recommended to use Zhang's calibration method).

[0037] The gimbal can pitch and rotate horizontally at a set speed so that the binocular camera can cover a wide detection area and avoid monitoring blind spots.

[0038] Since the camera equipped with the PTZ is dual-purpose, one of them captures the left video stream and the other captures the right video stream.

[0039] Step 2: Decompose the left and right video streams into a number of left and right video frames in time sequence, and take the left and right video frames at the same time as a group.

[0040] Since the left and right video streams can be regarded as a connection of video frames that increase over time, they can be decomposed into a number of left and right video frames in chronological order.

[0041] In order to facilitate subsequent processing, the left and right video frames at the same time are taken as a group, that is, several groups of video frames are obtained.

[0042] Step three, stereo rectification and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated.

[0043] Among them, stereo correction aims to adjust the left and right video frames to the same horizontal plane; stereo matching aims to correct the left and right video frames to the same horizontal line so that the corresponding pixels are in the same row.

[0044] Then, after stereo rectification and stereo matching, the left and right video frames of the same group are pixel-aligned.

[0045] The calculation method of the left disparity map includes:

[0046] For pixel a in the left video frame, calculate the cost function C(a,d)=|I L (a)-I R (ad)|——The difference between corresponding pixels.

[0047] Among them, I L (a) represents the pixel position of pixel a in the left video frame; R (ad) represents the pixel position of pixel point a in the right video frame.

[0048] For pixel a in the left video frame, find the disparity value d that minimizes the cost function C(a,d) min ——It represents the depth information of pixel a in the left video frame. Then, by traversing all the pixels of the left video frame, a left disparity map can be generated, which contains the depth information of all the pixels of the left video frame.

[0049] Then, we only need to obtain the pixel position (x_l, y_l) of the fire source in the left video frame, and then we can obtain the corresponding disparity value d0 from the left disparity map.

[0050] Similarly, the calculation method of the right disparity map includes:

[0051] For pixel b in the right video frame, calculate the cost function C(b,d)=|I R (b)-I L (bd)|——The difference between corresponding pixels;

[0052] Among them, I R (b) represents the pixel position of pixel point a in the right video frame; R (ad) represents the pixel position of pixel point b in the left video frame.

[0053] For pixel b in the right video frame, find the disparity value d that minimizes the cost function C(b,d) min——It represents the depth information of pixel b in the right video frame. Then, by traversing all the pixels of the right video frame, a right disparity map can be generated, which contains the depth information of all the pixels of the right video frame.

[0054] Then, we only need to obtain the pixel position (x_r, y_r) of the fire source in the right video frame, and then we can obtain the corresponding disparity value d0 from the right disparity map.

[0055] Step 4: Detect several left-channel video frames in chronological order through the trained flame detection model until a fire source is detected in a left-channel video frame, and output the pixel position (x_l, y_l) of the fire source in the left-channel video frame, and then obtain the corresponding disparity value d0 from the left disparity map based on (x_l, y_l);

[0056] Among them, x_l represents the pixel horizontal coordinate of the fire source in the left video frame, and y_l represents the pixel vertical coordinate of the fire source in the left video frame.

[0057] Of course, step ④ can also be replaced by:

[0058] Several right-channel video frames are detected in chronological order through the trained flame detection model until a fire source is detected in a right-channel video frame, and the pixel position (x_r, y_r) of the fire source in the right-channel video frame is output, and then the corresponding disparity value d is obtained from the disparity map based on (x_r, y_r);

[0059] Among them, x_r represents the pixel horizontal coordinate of the fire source in the right video frame, and y_r represents the pixel vertical coordinate of the fire source in the right video frame.

[0060] It should be noted that the flame detection model structures used in the above two steps 4 are the same, and are introduced uniformly here:

[0061] See also Figure 2 ,The flame detection model includes: feature extraction part, attention part, feature fusion part, and output part.

[0062] The following are the results one by one:

[0063] ①The feature extraction unit is used to perform multi-level feature extraction on the input image.

[0064] See above, the input image is a left video frame or a right video frame.

[0065] like Figure 2 As shown, the feature extraction unit includes: 5 CBS layers CBS1~CBS5, and 4 GELAN layers GELAN1~GELAN5.

[0066] See also Figure 3For the CBS layer, it is composed of the Conv layer, BN (batch normalization) layer and SiLU (SiLU activation function) layer connected in sequence: the Conv layer is responsible for performing convolution operations on the input of the CBS layer to extract features from the image; the BN layer normalizes the output of the convolution layer so that the data meets a certain distribution law (usually the mean is 0 and the variance is 1), thereby accelerating the convergence speed of the model and improving the stability and generalization ability of the model; the SiLU layer introduces nonlinear factors so that the model can learn more complex mapping relationships. In general, the CBS layer realizes feature extraction and nonlinear transformation of its input.

[0067] For the GELAN layer, it combines CSPNet (Cross-stage Local Network) and ELAN (Efficient Layer Aggregation Network): its input is divided into two paths; one for dense feature extraction, and the other uses cross-stage connections to cross the stage before merging. This can enrich information processing and reduce redundancy, which not only significantly reduces the number of network parameters and makes the model more lightweight, but also enhances the network's learning ability and improves the accuracy of the model.

[0068] For details, see Figure 4 , the GELAN layer includes: 4 Conv layers, 1 Chunk layer, 2 RepNCSP layers, and 1 CONCAT layer. In a certain GELAN layer: the input of the GELAN layer is used as the input of the first Conv layer; the output of the first Conv layer is used as the input of the Chunk layer. The output of the Chunk layer is divided into two paths: Split1 and Split2; Split2 is used as the input of the first RepNCSP layer, and the output of the first RepNCSP layer is used as the input of the second Conv layer; the output of the second Conv layer is used as the input of the second RepNCSP layer, and the output of the second RepNCSP layer is used as the input of the third Conv layer; Split1, Split2, the output of the second Conv layer, and the output of the third Conv layer are used as the input of the CONCAT layer; the output of the CONCAT layer is used as the input of the fourth Conv layer, and the output of the fourth Conv layer is used as the output of the GELAN layer.

[0069] Then, in the feature extraction unit: the input of the feature extraction unit is used as the input of CBS1; the output of CBS1 is used as the input of CBS2, the output of CBS2 is used as the input of GELAN1, the output of GELAN1 is used as the input of CBS3, the output of CBS3 is used as the input of GELAN2, the output of GELAN2 is used as the input of CBS4, the output of CBS4 is used as the input of GELAN3, the output of GELAN3 is used as the input of CBS5, and the output of CBS5 is used as the input of GELAN4; the outputs of GELAN2, GELAN3 and GELAN4 constitute the output of the feature extraction unit.

[0070] ② The attention unit is used to process part of the output of the feature extraction unit through the attention mechanism to focus on the salient features.

[0071] like Figure 2 As shown, the attention unit includes: 1 CBAM layer CBAM1 and 1 SPPF layer SPPF1.

[0072] The CBAM layer consists of two parts: 1. Channel Attention: Emphasize the features of important channels and suppress irrelevant channel information; 2. Spatial Attention: Focus on the salient areas in the image and enhance the information of important spatial areas. The CBAM layer enables the model to reduce noise and irrelevant details while retaining basic information, thereby enhancing its ability to detect fire sources.

[0073] For the SPPF layer, it uses the spatial pyramid pooling technology to perform multiple pooling operations of different sizes on the input features, and then splices these pooling results to finally form an output feature map that integrates multi-scale information. In this way, more information can be captured from multiple scales, adapt to targets of different sizes, enhance the model's perception of targets of different sizes, and reduce the amount of calculation.

[0074] Then, in the attention unit: the output of GELAN4 serves as the input of the attention unit; the input of the attention unit serves as the input of CBAM1, the output of CBAM1 serves as the input of SPPF1, and the output of SPPF1 serves as the output of the attention unit.

[0075] ③ The feature fusion unit is used to perform multi-level fusion on the remaining outputs of the feature extraction unit and the output of the attention unit to obtain multi-scale features.

[0076] like Figure 2As shown, the feature fusion unit includes: 4 CBS layers CBS6~CBS9, 4 CSP2 layers CSP21~CSP24, 2 Upsample layers Upsample1~Upsample2, and 4 BiFPN layers BiFPN1~BiFPN4.

[0077] For the CSP2 layer, it adopts a branching structure, which divides the input feature map into two parts, processes these parts through different convolution operations, and then merges them. In this way, it can enhance the feature expression ability while ensuring computational efficiency. This method helps the model learn richer features and improves the expressiveness of the model, especially for complex or diverse targets.

[0078] For the Upsample layer, it is used for upsampling operations, which can make more spatial information available by increasing the low-resolution feature map to a higher resolution. This is very helpful for small targets that require fine positioning, because high-resolution feature maps can provide more detailed information, helping the model to better identify and locate targets.

[0079] For the BiFPN layer, it performs multiple rounds of upsampling and downsampling on high-level and low-level feature maps through its bidirectional feature propagation structure, which not only improves the integration capability of multi-scale features, but also can effectively fuse feature maps from different resolutions and enhance the detection capability of targets of different scales. In addition, the BiFPN layer optimizes feature selection through a weighted mechanism, improves the accuracy and robustness of the model, and helps to better identify and locate targets in complex environments.

[0080] Then, in the feature fusion unit, the output of the attention unit is used as the input of the feature fusion unit; the input of the feature fusion unit is used as the input of CBS6; the output of CBS6 is used as the input of Upsample1, the outputs of Upsample1 and GELAN3 are used as the dual inputs of BiFPN1, the output of BiFPN1 is used as the input of CSP21, the output of CSP21 is used as the input of CBS7, the output of CBS7 is used as the input of Upsample2, the output of Upsample2 and GELAN3 is used as the dual input of BiFPN2, the output of BiFPN2 is used as the input of CSP22, the output of CSP22 is used as the input of CBS8, the outputs of CSP21 and CBS8 are used as the dual inputs of BiFPN3, the output of BiFPN3 is used as the input of CSP23, the output of CSP23 is used as the input of CBS9, the outputs of CBS6 and CBS9 are used as the dual inputs of BiFPN4, and the output of BiFPN4 is used as the input of CSP24.

[0081] ④ The output part is used to convert the multi-scale features into several candidate boxes to mark the fire source on the input image.

[0082] like Figure 2 As shown, the output part includes: 3 Conv layers Conv1~Conv3.

[0083] For the Conv layer, it converts the multi-scale features after feature extraction and fusion into the final prediction output.

[0084] Then, in the output part: the output of CSP22 is used as the input of Conv1, and the output of Conv1 is the small-sized candidate box Prediction1; the output of CSP23 is used as the input of Conv2, and the output of Conv2 is the medium-sized candidate box Prediction2; the output of CSP4 is used as the input of Conv3, and the output of Conv3 is the large-sized candidate box Prediction3.

[0085] The three candidate frames of different sizes will be subsequently merged into a detection frame, and the fire source will be marked on the input image - generally the pixel coordinates of the center point of the detection frame are taken as the pixel coordinates of the fire source.

[0086] It should be noted that the present invention requires the use of a well-trained fire source detection model with optimal model parameters to ensure its effectiveness.

[0087] The principle of training the fire source detection model is similar to that of existing deep learning, including:

[0088] S1, obtain a set of fire source sample images with true labels;

[0089] Among them, the fire source sample image set contains a large number of flame images, non-flame images, flame-like images, etc., ensuring that the dataset contains a variety of fire scenes, including images of different scales, different lighting and different environments, and the real labels are marked using the LabelImg tool.

[0090] S2, divide the fire source sample image set with real labels into training set, validation set and test set;

[0091] Among them, the generally recommended division ratio of training set, validation set, and test set is 7:2:1.

[0092] S3, perform N rounds of training on the fire source detection model based on the training set; verify the fire source detection model after each round of training based on the validation set until the fire source detection model with the best performance is selected; evaluate the fire source detection model with the best performance based on the test set, if it meets the evaluation requirements, use it as the trained fire source detection model, otherwise, repeat S3.

[0093] Step 5: Based on the disparity value d0 and combined with the parameters of the binocular camera and the gimbal, the absolute position (x, y, z) of the fire source is calculated.

[0094] Specifically, the calculation formula for the absolute position (x, y, z) of the fire source is:

[0095]

[0096] Where x represents the absolute X-axis coordinate of the fire source, y represents the absolute Y-axis coordinate of the fire source, and z represents the absolute Z-axis coordinate of the fire source; f represents the focal length of the binocular camera, B represents the baseline of the binocular camera; θ represents the pitch angle of the gimbal; φ represents the horizontal rotation angle of the gimbal.

[0097] At this point, the detection and location of the fire source have been completed.

[0098] Simulation Verification

[0099] This Example 1 verifies the effectiveness of the above-mentioned method:

[0100] The other five groups of models (including: YOLOv5s, MobileNet-YOLO, FCDM, CCAB, YOLOv7-Tiny) are introduced to compare the fire source detection performance in the complex fire environment of forest fire with the fire source detection model (referred to as LRENet) proposed in this embodiment 1. The performance comparison results of the performance indicators are shown in Table 1.

[0101] Table 1 Performance index comparison

[0102]

[0103] As shown in Table 1, the mAP@0.5 (average accuracy, the larger the value, the better) of LRENet is 89.4%, which is better than the other five groups of models. In addition, LRENet is also at the forefront of the six groups of models in terms of Precision (accuracy, the larger the value, the better), Recall (recall rate, the larger the value, the better), and FPS (processing speed, the larger the value, the better). Although the Parameters (the number of model parameters, the smaller the value, the better) of LRENet reaches 6.46M, it is still a lightweight model and has low requirements for the platform on which it is installed.

[0104] In summary, the fire source detection model proposed in this embodiment 1 has certain advantages in use, is suitable for fire source detection in complex fire environments, and can ensure the accuracy of fire source positioning.

[0105] Example 2

[0106] This embodiment 2 provides a binocular pan-tilt fire source automatic positioning device based on deep learning, including: a pan-tilt equipped with a binocular camera and a host computer.

[0107] The gimbal equipped with a binocular camera is used to perform step 1 of the binocular gimbal fire source automatic positioning method based on deep learning as disclosed in Example 1.

[0108] The host computer is used to perform steps 2 to 5 of the binocular pan-tilt automatic fire source positioning method based on deep learning as disclosed in Example 1.

[0109] Specifically, the pan / tilt head equipped with a binocular camera may be the commercially available LD-PTZ20-2IP-W or other purchasable models, which may be connected to a host computer via a data cable to achieve transmission of a video stream.

[0110] The host computer can adopt NVIDIA Orin machine or other edge computing device, which has built-in video stream processing unit, flame detection unit (loaded with trained flame detection model), and coordinate calculation unit; wherein, the video stream processing unit is used to perform steps 2 to 3 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1; the flame detection unit is used to perform step 4 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1; the coordinate calculation unit is used to perform step 5 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1.

[0111] Since this device uses the binocular pan-tilt automatic fire source positioning method based on deep learning in Example 1, it also has the same effect and will not be repeated here.

[0112] Example 3

[0113] This embodiment 3 discloses a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the binocular pan-tilt automatic fire source positioning method based on deep learning disclosed in embodiment 1 are implemented.

[0114] This embodiment 3 also discloses a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of the binocular pan-tilt automatic fire source positioning method based on deep learning disclosed in embodiment 1 are executed.

[0115] This embodiment 3 also discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the binocular pan-tilt automatic fire source positioning method based on deep learning disclosed in embodiment 1 are implemented.

[0116] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A binocular PTZ fire source automatic positioning method based on deep learning, characterized in that: include: Step 1: Use a pan / tilt equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams; Among them, the binocular camera has completed binocular positioning in advance; Step 2: decompose the left and right video streams into a plurality of left and right video frames in time sequence, and take the left and right video frames at the same time as a group; Step 3: stereo correction and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated; Step 4: Detect several left-channel video frames in chronological order through the trained flame detection model, until a fire source is detected in a left-channel video frame, and output the pixel position (x_l, y_l) of the fire source in the left-channel video frame, and then obtain the corresponding disparity value d0 from the left disparity map based on (x_l, y_l); wherein x_l represents the pixel horizontal coordinate of the fire source in the left-channel video frame, and y_l represents the pixel vertical coordinate of the fire source in the left-channel video frame; Or, several right-channel video frames are detected in chronological order through the trained flame detection model, until a fire source is detected in a right-channel video frame, and the pixel position (x_r, y_r) of the fire source in the right-channel video frame is output, and then the corresponding disparity value d is obtained from the right disparity map based on (x_r, y_r); wherein x_r represents the pixel horizontal coordinate of the fire source in the right-channel video frame, and y_r represents the pixel vertical coordinate of the fire source in the right-channel video frame; Step 5: Based on the disparity value d0 and combined with the parameters of the binocular camera and the gimbal, the absolute position (x, y, z) of the fire source is calculated; Among them, x represents the absolute X-axis coordinate of the fire source, y represents the absolute Y-axis coordinate of the fire source, and z represents the absolute Z-axis coordinate of the fire source.

2. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: In step 4, the fire source detection model includes: A feature extraction unit, which is used to perform multi-level feature extraction on the input image; The attention unit is used to process part of the output of the feature extraction unit through the attention mechanism to focus on the salient features; A feature fusion unit, which is used to perform multi-level fusion on the remaining outputs of the feature extraction unit and the output of the attention unit to obtain multi-scale features; as well as An output unit, which is used to convert the multi-scale features into a number of candidate boxes to mark the fire source on the input image; The input image is a left video frame or a right video frame.

3. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 2 is characterized in that: The feature extraction unit includes: 5 CBS layers CBS1-CBS5, 4 GELAN layers GELAN1-GELAN5; The input of the feature extraction unit is used as the input of CBS1; the output of CBS1 is used as the input of CBS2, the output of CBS2 is used as the input of GELAN1, the output of GELAN1 is used as the input of CBS3, the output of CBS3 is used as the input of GELAN2, the output of GELAN2 is used as the input of CBS4, the output of CBS4 is used as the input of GELAN3, the output of GELAN3 is used as the input of CBS5, and the output of CBS5 is used as the input of GELAN4; the outputs of GELAN2, GELAN3 and GELAN4 constitute the output of the feature extraction unit.

4. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 3 is characterized in that: The attention unit includes: a CBAM layer CBAM1, a SPPF layer SPPF1; The output of GELAN4 is used as the input of the attention unit; the input of the attention unit is used as the input of CBAM1, the output of CBAM1 is used as the input of SPPF1, and the output of SPPF1 is used as the output of the attention unit.

5. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 4 is characterized in that: The feature fusion unit includes: 4 CBS layers CBS6-CBS9, 4 CSP2 layers CSP21-CSP24, 2 Upsample layers Upsample1-Upsample2, and 4 BiFPN layers BiFPN1-BiFPN4; The output of the attention unit is used as the input of the feature fusion unit; the input of the feature fusion unit is used as the input of CBS6; the output of CBS6 is used as the input of Upsample1, the outputs of Upsample1 and GELAN3 are used as the dual input of BiFPN1, the output of BiFPN1 is used as the input of CSP21, the output of CSP21 is used as the input of CBS7, the output of CBS7 is used as the input of Upsample2, the outputs of Upsample2 and GELAN3 are used as the dual input of BiFPN2, the output of BiFPN2 is used as the input of CSP22, the output of CSP22 is used as the input of CBS8, the outputs of CSP21 and CBS8 are used as the dual input of BiFPN3, the output of BiFPN3 is used as the input of CSP23, the output of CSP23 is used as the input of CBS9, the outputs of CBS6 and CBS9 are used as the dual input of BiFPN4, and the output of BiFPN4 is used as the input of CSP24.

6. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 5 is characterized in that: The output unit includes: 3 Conv layers Conv1 to Conv3; The output of CSP22 is used as the input of Conv1, and the output of Conv1 is the small-sized candidate box Prediction1; the output of CSP23 is used as the input of Conv2, and the output of Conv2 is the medium-sized candidate box Prediction2; the output of CSP4 is used as the input of Conv3, and the output of Conv3 is the large-sized candidate box Prediction3.

7. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: In step four, Where f represents the focal length of the binocular camera, B represents the baseline of the binocular camera, θ represents the pitch angle of the gimbal, and φ represents the horizontal rotation angle of the gimbal.

8. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: The training methods of the fire source detection model include: S1, obtain a set of fire source sample images with true labels; S2, divide the fire source sample image set with real labels into training set, validation set and test set; S3, perform N rounds of training on the fire source detection model based on the training set; verify the fire source detection model after each round of training based on the validation set until the fire source detection model with the best performance is selected; evaluate the fire source detection model with the best performance based on the test set, if it meets the evaluation requirements, use it as the trained fire source detection model, otherwise, repeat S3.

9. A binocular PTZ fire source automatic positioning device based on deep learning, characterized in that: It includes: A pan / tilt equipped with a binocular camera, which is used to perform step 1 of the binocular pan / tilt fire source automatic positioning method based on deep learning as described in any one of claims 1 to 8; as well as A host computer is used to perform steps 2 to 5 of the binocular pan-tilt automatic fire source positioning method based on deep learning as described in claims 1-8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the binocular pan-tilt automatic fire source positioning method based on deep learning as described in any one of claims 1 to 8 are implemented.

Images