A binocular PTZ fire source automatic positioning method and device based on deep learning

Abstract

The present invention relates to the field of emergency firefighting technology, and specifically to a binocular pan-tilt head fire source automatic positioning method and device based on deep learning. The present invention uses a pan-tilt head equipped with a binocular camera to shoot the target fire environment, then processes the shot video stream to obtain aligned video frames and disparity maps, then uses a trained flame detection model to analyze these video frames in real time to detect the pixel position of the flame and thus obtain the disparity, and finally calculates the absolute position of the fire source by combining the parameters of the binocular camera and the pan-tilt head; the entire process achieves accurate flame detection and positioning, reduces manual intervention, and increases the safety factor of personnel. The present invention solves the problem that the existing application of deep learning to fire source positioning is difficult to implement and there is room for improvement in the effect.

Description

Technical Field

[0001] The present invention relates to the field of emergency fire fighting technology, and more specifically, to: 1. A binocular pan-tilt automatic fire source positioning method based on deep learning; 2. A binocular pan-tilt automatic fire source positioning device based on deep learning. Background Art

[0002] Traditional fire detection typically relies on manual inspections or simple sensor detection. These methods suffer from slow response, inaccurate positioning, and low safety when faced with complex and changing fire environments, making them unable to cope with the rapid spread of fire.

[0003] Furthermore, traditional monocular cameras lack depth information and are unable to accurately measure the distance to a fire source, leading to inaccurate firefighting strategy planning. Even systems equipped with flame and smoke detection capabilities struggle to quickly and accurately pinpoint the fire source. This results in inefficient firefighting operations, prolonged rescue efforts, and compromised firefighting effectiveness.

[0004] In recent years, the application of deep learning in target detection has made significant progress. The inventors considered applying deep learning to fire source location, but found that specific implementation remained a technical challenge. Furthermore, there were issues such as difficulty detecting tiny flames and smoke targets, and the tendency to misdetect or miss detections in complex and changing fire environments. Summary of the Invention

[0005] Based on this, it is necessary to provide a binocular pan-tilt automatic fire source positioning method and device based on deep learning to address the problems that the existing application of deep learning to fire source positioning is difficult to implement and the effect has room for improvement.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] In a first aspect, the present invention discloses a binocular pan-tilt automatic fire source positioning method based on deep learning, comprising:

[0008] Step 1: Use a pan / tilt camera equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams;

[0009] Among them, the binocular camera has completed binocular positioning in advance;

[0010] Step 2: Decompose the left and right video streams into a number of left and right video frames in chronological order, and take the left and right video frames at the same time as a group;

[0011] Step 3: Stereo correction and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated;

[0012] Step 4: Detect several left-channel video frames in chronological order using the trained flame detection model until a fire source is detected in a left-channel video frame and the pixel position (x_l, y_l) of the fire source in the left-channel video frame is output. Then, based on (x_l, y_l), the corresponding disparity value d0 is obtained from the left disparity map; where x_l represents the horizontal coordinate of the pixel of the fire source in the left-channel video frame, and y_l represents the vertical coordinate of the pixel of the fire source in the left-channel video frame;

[0013] Alternatively, several right-channel video frames are sequentially detected using the trained flame detection model in chronological order until a fire source is detected in a certain right-channel video frame, and the pixel position (x_r, y_r) of the fire source in the right-channel video frame is output. Then, based on (x_r, y_r), the corresponding disparity value d is obtained from the right disparity map; where x_r represents the horizontal coordinate of the pixel of the fire source in the right-channel video frame, and y_r represents the vertical coordinate of the pixel of the fire source in the right-channel video frame;

[0014] Step 5: Based on the disparity value d0 and combined with the parameters of the binocular camera and the gimbal, the absolute position (x, y, z) of the fire source is calculated;

[0015] Where x represents the absolute X-axis coordinate of the fire source, y represents the absolute Y-axis coordinate of the fire source, and z represents the absolute Z-axis coordinate of the fire source.

[0016] This deep learning-based binocular pan-tilt automatic fire source positioning method implements the method or process according to the embodiment of the present disclosure.

[0017] In the second aspect, the present invention discloses a binocular pan-tilt automatic fire source positioning device based on deep learning, which uses the binocular pan-tilt automatic fire source positioning method based on deep learning disclosed in the first aspect.

[0018] The binocular pan-tilt head fire source automatic positioning device based on deep learning includes: a pan-tilt head equipped with a binocular camera and a host computer.

[0019] The pan-tilt head equipped with a binocular camera is used to perform step one of the binocular pan-tilt head fire source automatic positioning method based on deep learning as disclosed in the first aspect.

[0020] The host computer is used to perform steps 2 to 5 of the binocular pan-tilt head fire source automatic positioning method based on deep learning as disclosed in the first aspect.

[0021] This deep learning-based binocular pan-tilt automatic fire source positioning device implements the method or process according to the embodiment of the present disclosure.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention uses a pan-tilt head equipped with a binocular camera to capture the target fire environment, then processes the captured video stream to obtain aligned video frames and disparity maps. A trained flame detection model is then used to analyze these video frames in real time to detect the pixel position of the flame and thereby obtain the disparity. Finally, the absolute position of the fire source is calculated by combining the parameters of the binocular camera and the pan-tilt head. The entire process achieves accurate flame detection and positioning, reduces manual intervention, and increases personnel safety.

[0024] 2. The present invention applies deep learning to fire source location detection and designs a fire source detection model to enhance the adaptability and detection accuracy to complex fire scenes, which can quickly respond to fires and make firefighting more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 Flowchart of the binocular pan-tilt automatic fire source positioning method based on deep learning provided in Example 1 of the present invention;

[0027] Figure 2 for Figure 1 Structural diagram of the flame detection model;

[0028] Figure 3 for Figure 2 The structural diagram of the CBS layer;

[0029] Figure 4 for Figure 2 Structural diagram of the GELAN layer. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] It should be noted that when a component is referred to as being "mounted on" another component, it may be directly on the other component or there may be a central component. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be a central component. When a component is considered to be "fixed to" another component, it may be directly fixed to the other component or there may be a central component.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0033] Example 1

[0034] See Figure 1 , Figure 1 A flowchart of a deep learning-based binocular pan-tilt automatic fire source location method is presented, which includes:

[0035] Step 1: Use a pan / tilt camera equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams;

[0036] Among them, the binocular camera has completed binocular calibration in advance (it is recommended to use Zhang's calibration method).

[0037] The gimbal can pitch and rotate horizontally at a set speed, allowing the binocular camera to cover a wide detection area and avoid monitoring blind spots.

[0038] Since the camera equipped with the PTZ is dual-purpose, one of the cameras captures the left video stream and the other captures the right video stream.

[0039] Step 2: Decompose the left and right video streams into a number of left and right video frames in chronological order, and take the left and right video frames at the same time as a group.

[0040] Since the left and right video streams can be viewed as a connection of video frames that increase over time, they can be decomposed into a number of left and right video frames in chronological order.

[0041] In order to facilitate subsequent processing, the left and right video frames at the same time are taken as a group, that is, several groups of video frames are obtained.

[0042] Step 3: Stereo rectification and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated.

[0043] Among them, stereo rectification aims to adjust the left and right video frames to the same horizontal plane; stereo matching aims to correct the left and right video frames to the same horizontal line so that the corresponding pixels are in the same row.

[0044] Then, after stereo rectification and stereo matching, the left and right video frames of the same group are pixel-aligned.

[0045] The calculation method of the left disparity map includes:

[0046] For pixel a in the left video frame, calculate the cost function C(a,d)=|I L (a)-I R (ad)|——The difference between corresponding pixels.

[0047] Among them, I L (a) represents the pixel position of pixel a in the left video frame; I R (ad) represents the pixel position of pixel point a in the right video frame.

[0048] For pixel a in the left video frame, find the disparity value d that minimizes the cost function C(a,d) min ——It represents the depth information of pixel a in the left video frame. Then, by traversing all the pixels of the left video frame, a left disparity map can be generated, which contains the depth information of all the pixels of the left video frame.

[0049] Then, we only need to obtain the pixel position (x_l, y_l) of the fire source in the left video frame, and then we can obtain the corresponding disparity value d0 from the left disparity map.

[0050] Similarly, the calculation method of the right disparity map includes:

[0051] For the pixel b in the right video frame, calculate the cost function C(b,d)=|I R (b)-I L (bd)|——The difference between corresponding pixels;

[0052] Among them, I R (b) represents the pixel position of pixel a in the right video frame; R (ad) represents the pixel position of pixel point b in the left video frame.

[0053] For pixel b in the right video frame, find the disparity value d that minimizes the cost function C(b,d) min——It represents the depth information of pixel b in the right video frame. Then, by traversing all the pixels of the right video frame, a right disparity map can be generated, which contains the depth information of all the pixels of the right video frame.

[0054] Then, we only need to obtain the pixel position (x_r, y_r) of the fire source in the right video frame, and then we can obtain the corresponding disparity value d0 from the right disparity map.

[0055] Step 4: Detect several left-channel video frames in chronological order using the trained flame detection model until a fire source is detected in a left-channel video frame. The pixel position (x_l, y_l) of the fire source in the left-channel video frame is output, and then the corresponding disparity value d0 is obtained from the left disparity map based on (x_l, y_l).

[0056] Among them, x_l represents the pixel horizontal coordinate of the fire source in the left video frame, and y_l represents the pixel vertical coordinate of the fire source in the left video frame.

[0057] Of course, step ④ can also be replaced by:

[0058] The trained flame detection model is used to detect several right-channel video frames in chronological order until a fire source is detected in a right-channel video frame. The pixel position (x_r, y_r) of the fire source in the right-channel video frame is output, and the corresponding disparity value d is obtained from the disparity map based on (x_r, y_r).

[0059] Among them, x_r represents the pixel horizontal coordinate of the fire source in the right video frame, and y_r represents the pixel vertical coordinate of the fire source in the right video frame.

[0060] It should be noted that the flame detection model structure used in the above two steps 4 is the same, and is introduced here uniformly:

[0061] See Figure 2 ,The flame detection model includes: feature extraction part, attention part, feature fusion part, and output part.

[0062] The following are the results one by one:

[0063] ①The feature extraction unit is used to perform multi-level feature extraction on the input image.

[0064] Referring to the above, the input image is a left-channel video frame or a right-channel video frame.

[0065] like Figure 2 As shown, the feature extraction unit includes: 5 CBS layers CBS1~CBS5, and 4 GELAN layers GELAN1~GELAN5.

[0066] See Figure 3The CBS layer consists of a sequentially connected Conv layer, BN (batch normalization) layer, and SiLU (SiLU activation function) layer: The Conv layer is responsible for performing convolution operations on the CBS layer input to extract features from the image; the BN layer normalizes the output of the convolution layer so that the data conforms to a certain distribution pattern (usually a mean of 0 and a variance of 1), thereby accelerating the convergence of the model and improving its stability and generalization ability; the SiLU layer introduces nonlinear factors, enabling the model to learn more complex mapping relationships. In short, the CBS layer implements feature extraction and nonlinear transformation of its input.

[0067] The GELAN layer combines CSPNet (Cross-Stage Partial Network) and ELAN (Efficient Layer Aggregation Network): its input is split into two paths; one for dense feature extraction, and the other using cross-stage connections to span the pre-merger stages. This enriches information processing and reduces redundancy, significantly reducing the number of network parameters and making the model more lightweight. It also enhances the network's learning ability and improves model accuracy.

[0068] For details, see Figure 4 The GELAN layer consists of four Conv layers, one Chunk layer, two RepNCSP layers, and one Concat layer. In a given GELAN layer, the input of the GELAN layer serves as the input of the first Conv layer; the output of the first Conv layer serves as the input of the Chunk layer. The output of the Chunk layer is divided into two paths: Split1 and Split2. Split2 serves as the input of the first RepNCSP layer, and the output of the first RepNCSP layer serves as the input of the second Conv layer. The output of the second Conv layer serves as the input of the second RepNCSP layer, and the output of the second RepNCSP layer serves as the input of the third Conv layer. Split1, Split2, the output of the second Conv layer, and the output of the third Conv layer serve as the input of the Concat layer. The output of the Concat layer serves as the input of the fourth Conv layer, and the output of the fourth Conv layer serves as the output of the GELAN layer.

[0069] Then, in the feature extraction unit: the input of the feature extraction unit is used as the input of CBS1; the output of CBS1 is used as the input of CBS2, the output of CBS2 is used as the input of GELAN1, the output of GELAN1 is used as the input of CBS3, the output of CBS3 is used as the input of GELAN2, the output of GELAN2 is used as the input of CBS4, the output of CBS4 is used as the input of GELAN3, the output of GELAN3 is used as the input of CBS5, and the output of CBS5 is used as the input of GELAN4; the outputs of GELAN2, GELAN3, and GELAN4 constitute the output of the feature extraction unit.

[0070] ②The attention unit is used to process part of the output of the feature extraction unit through the attention mechanism to focus on salient features.

[0071] like Figure 2 As shown, the attention unit includes: 1 CBAM layer CBAM1 and 1 SPPF layer SPPF1.

[0072] The CBAM layer consists of two components: 1. Channel Attention, which emphasizes the features of important channels and suppresses irrelevant channel information; and 2. Spatial Attention, which focuses on salient regions in the image and enhances information in these important spatial regions. The CBAM layer enables the model to retain essential information while reducing noise and irrelevant details, thereby enhancing its ability to detect fire sources.

[0073] The SPPF layer uses spatial pyramid pooling technology to perform multiple pooling operations of different sizes on the input features, then concatenates these pooling results to form an output feature map that integrates multi-scale information. This method captures more information from multiple scales, adapts to objects of different sizes, enhances the model's perception of objects of different sizes, and reduces the amount of computation.

[0074] Then, in the attention unit: the output of GELAN4 serves as the input of the attention unit; the input of the attention unit serves as the input of CBAM1, the output of CBAM1 serves as the input of SPPF1, and the output of SPPF1 serves as the output of the attention unit.

[0075] ③ The feature fusion unit is used to perform multi-level fusion on the remaining outputs of the feature extraction unit and the output of the attention unit to obtain multi-scale features.

[0076] like Figure 2As shown, the feature fusion unit includes: 4 CBS layers CBS6~CBS9, 4 CSP2 layers CSP21~CSP24, 2 Upsample layers Upsample1~Upsample2, and 4 BiFPN layers BiFPN1~BiFPN4.

[0077] The CSP2 layer uses a branching structure, splitting the input feature map into two parts, processing these parts through different convolution operations, and then merging them. This approach enhances feature expression while maintaining computational efficiency. This approach helps the model learn richer features and improves its expressiveness, especially for complex or diverse targets.

[0078] The Upsample layer performs upsampling, which increases the resolution of low-resolution feature maps to a higher resolution, thereby making more spatial information available. This is particularly helpful for small targets that require precise positioning, as high-resolution feature maps provide more detailed information, helping the model better identify and locate the target.

[0079] The BiFPN layer, through its bidirectional feature propagation structure, performs multiple rounds of upsampling and downsampling on both high- and low-level feature maps. This not only improves the ability to integrate multi-scale features, but also effectively fuses feature maps from different resolutions, enhancing the ability to detect objects of different scales. Furthermore, the BiFPN layer optimizes feature selection through a weighted mechanism, improving the accuracy and robustness of the model and helping to better identify and locate objects in complex environments.

[0080] Then, in the feature fusion unit, the output of the attention unit serves as the input of the feature fusion unit; the input of the feature fusion unit serves as the input of CBS6; the output of CBS6 serves as the input of Upsample1, the output of Upsample1 and GELAN3 serves as the dual input of BiFPN1, the output of BiFPN1 serves as the input of CSP21, the output of CSP21 serves as the input of CBS7, the output of CBS7 serves as the input of Upsample2, the output of Upsample2 and GELAN3 serves as the dual input of BiFPN2, the output of BiFPN2 serves as the input of CSP22, the output of CSP22 serves as the input of CBS8, the output of CSP21 and CBS8 serves as the dual input of BiFPN3, the output of BiFPN3 serves as the input of CSP23, the output of CSP23 serves as the input of CBS9, the output of CBS6 and CBS9 serves as the dual input of BiFPN4, and the output of BiFPN4 serves as the input of CSP24.

[0081] ④ Output part, which is used to convert multi-scale features into several candidate boxes to mark the fire source on the input image.

[0082] like Figure 2 As shown, the output part includes: 3 Conv layers Conv1~Conv3.

[0083] For the Conv layer, it converts the multi-scale features after feature extraction and fusion into the final prediction output.

[0084] Then, in the output part: the output of CSP22 is used as the input of Conv1, and the output of Conv1 is the small-sized candidate box Prediction1; the output of CSP23 is used as the input of Conv2, and the output of Conv2 is the medium-sized candidate box Prediction2; the output of CSP4 is used as the input of Conv3, and the output of Conv3 is the large-sized candidate box Prediction3.

[0085] The three candidate boxes of different sizes are subsequently merged into a detection box, and the fire source is marked on the input image. Generally, the pixel coordinates of the center point of the detection box are taken as the pixel coordinates of the fire source.

[0086] It should be noted that the present invention requires the use of a well-trained fire source detection model with optimal model parameters to ensure its effectiveness.

[0087] The principles of training the fire source detection model are similar to those of existing deep learning methods, including:

[0088] S1, obtain a set of fire source sample images with true labels;

[0089] Among them, the fire source sample image set contains a large number of flame images, non-flame images, flame-like images, etc., ensuring that the dataset contains a variety of fire scenes, including images of different scales, different lighting and different environments, and uses the LabelImg tool to mark the real labels.

[0090] S2, divide the fire source sample image set with real labels into training set, validation set and test set;

[0091] Among them, the generally recommended division ratio of training set, validation set, and test set is 7:2:1.

[0092] S3, perform N rounds of training on the fire source detection model based on the training set; verify the fire source detection model after each round of training based on the validation set until the fire source detection model with the best performance is selected; evaluate the fire source detection model with the best performance based on the test set. If it meets the evaluation requirements, it will be used as the trained fire source detection model. Otherwise, repeat S3.

[0093] Step 5: Based on the disparity value d0 and combined with the parameters of the binocular camera and the gimbal, the absolute position (x, y, z) of the fire source is calculated.

[0094] Specifically, the calculation formula for the absolute position (x, y, z) of the fire source is:

[0095]

[0096] Where x represents the absolute X-axis coordinate of the fire source, y represents the absolute Y-axis coordinate of the fire source, and z represents the absolute Z-axis coordinate of the fire source; f represents the focal length of the binocular camera, B represents the baseline of the binocular camera; θ represents the pitch angle of the gimbal; and φ represents the horizontal rotation angle of the gimbal.

[0097] At this point, the detection and location of the fire source are completed.

[0098] Simulation Verification

[0099] This Example 1 verifies the effectiveness of the above-mentioned method:

[0100] Five other models (including YOLOv5s, MobileNet-YOLO, FCDM, CCAB, and YOLOv7-Tiny) were introduced to compare their fire source detection performance in a complex fire environment such as forest fires. The performance comparison results are shown in Table 1.

[0101] Table 1 Performance index comparison

[0102]

[0103] As shown in Table 1, LRENet achieves a mAP@0.5 (average accuracy, higher values ​​are better) of 89.4%, outperforming the other five models. Furthermore, LRENet leads the six models in terms of Precision (higher values ​​are better), Recall (higher values ​​are better), and FPS (higher values ​​are better). Although LRENet's parameters (lower values ​​are better) reach 6.46M, it is still a lightweight model and has low platform requirements.

[0104] In summary, the fire source detection model proposed in this embodiment 1 has certain advantages in use, is suitable for fire source detection in complex fire environments, and can ensure the accuracy of fire source positioning.

[0105] Example 2

[0106] This embodiment 2 provides a binocular pan-tilt head fire source automatic positioning device based on deep learning, including: a pan-tilt head equipped with a binocular camera and a host computer.

[0107] The pan-tilt head equipped with a binocular camera is used to perform step 1 of the binocular pan-tilt head fire source automatic positioning method based on deep learning as disclosed in Example 1.

[0108] The host computer is used to perform steps 2 to 5 of the binocular pan-tilt head fire source automatic positioning method based on deep learning as disclosed in Example 1.

[0109] Specifically, the pan / tilt head equipped with a binocular camera can be the commercially available LD-PTZ20-2IP-W or other purchasable models, which can be connected to a host computer via a data cable to achieve video stream transmission.

[0110] The host computer can be an NVIDIA Orin machine or other edge computing device, which has a built-in video stream processing unit, a flame detection unit (loaded with a trained flame detection model), and a coordinate calculation unit; wherein the video stream processing unit is used to perform steps 2 to 3 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1; the flame detection unit is used to perform step 4 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1; and the coordinate calculation unit is used to perform step 5 of the binocular pan-tilt fire source automatic positioning method based on deep learning as disclosed in Example 1.

[0111] Since this device uses the binocular pan-tilt head fire source automatic positioning method based on deep learning in Example 1, it also has the same effect and will not be repeated here.

[0112] Example 3

[0113] This embodiment 3 discloses a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the binocular pan-tilt head fire source automatic positioning method based on deep learning disclosed in embodiment 1.

[0114] This embodiment 3 also discloses a readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps of the binocular pan-tilt head fire source automatic positioning method based on deep learning disclosed in embodiment 1 are executed.

[0115] This embodiment 3 further discloses a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the binocular pan-tilt head fire source automatic positioning method based on deep learning disclosed in embodiment 1 are implemented.

[0116] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A binocular PTZ fire source automatic positioning method based on deep learning, characterized in that: include: Step 1: Use a pan / tilt camera equipped with a binocular camera to shoot the target fire environment to obtain left and right video streams; wherein, the binocular camera has completed binocular positioning in advance; Step 2: Decompose the left and right video streams into a number of left and right video frames in chronological order, and take the left and right video frames at the same time as a group; Step 3: Stereo correction and stereo matching are performed on the left and right video frames of the same group to ensure image alignment, and a left disparity map based on the left video frame or a right disparity map based on the right video frame is calculated; Step 4: Detect several left-channel video frames in chronological order through the trained flame detection model until a fire source is detected in a left-channel video frame, and output the pixel position of the fire source in the left-channel video frame ( x _ l , y _ l ), then based on ( x _ l , y _ l ) Get the corresponding disparity value from the left disparity map d 0; among them, x _ l Indicates the pixel horizontal coordinate of the fire source in the left video frame, y _ l Indicates the pixel vertical coordinate of the fire source in the left video frame; Alternatively, several right-channel video frames are sequentially detected by the trained flame detection model in chronological order until a fire source is detected in a right-channel video frame, and the pixel position of the fire source in the right-channel video frame is output ( x _ r , y _ r ), then based on ( x _ r , y _ r ) Get the corresponding disparity value from the right disparity map d ;in, x _ r Indicates the pixel horizontal coordinate of the fire source in the right video frame, y _ r Indicates the pixel vertical coordinate of the fire source in the right video frame; Step 5: Based on the disparity value d 0, and combined with the parameters of the binocular camera and the pan / tilt, calculate the absolute position of the fire source ( x , y , z );in, x Indicates the absolute source of fire X Axis coordinates, y Indicates the absolute source of fire Y Axis coordinates, z Indicates the absolute source of fire z axis coordinates; In step 4, the fire source detection model includes: A feature extraction unit, which is used to perform multi-level feature extraction on the input image; The attention unit is used to process part of the output of the feature extraction unit through the attention mechanism to focus on salient features; A feature fusion unit, configured to perform multi-level fusion on the remaining outputs of the feature extraction unit and the output of the attention unit to obtain multi-scale features; and The output part is used to convert the multi-scale features into a number of candidate frames to mark the fire source on the input image; wherein the input image is a left video frame or a right video frame.

2. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: The feature extraction unit includes: 5 CBS layers CBS1~CBS5, 4 GELAN layers GELAN1~GELAN5; The input of the feature extraction unit is used as the input of CBS1; the output of CBS1 is used as the input of CBS2, the output of CBS2 is used as the input of GELAN1, the output of GELAN1 is used as the input of CBS3, the output of CBS3 is used as the input of GELAN2, the output of GELAN2 is used as the input of CBS4, the output of CBS4 is used as the input of GELAN3, the output of GELAN3 is used as the input of CBS5, and the output of CBS5 is used as the input of GELAN4; the outputs of GELAN2, GELAN3, and GELAN4 constitute the output of the feature extraction unit.

3. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 2 is characterized in that: The attention unit includes: a CBAM layer CBAM1, a SPPF layer SPPF1; The output of GELAN4 serves as the input of the attention unit; the input of the attention unit serves as the input of CBAM1, the output of CBAM1 serves as the input of SPPF1, and the output of SPPF1 serves as the output of the attention unit.

4. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 3 is characterized in that: The feature fusion unit includes: 4 CBS layers CBS6~CBS9, 4 CSP2 layers CSP21~CSP24, 2 Upsample layers Upsample1~Upsample2, and 4 BiFPN layers BiFPN1~BiFPN4; The output of the attention unit serves as the input of the feature fusion unit; the input of the feature fusion unit serves as the input of CBS6; the output of CBS6 serves as the input of Upsample1, the outputs of Upsample1 and GELAN3 serve as the dual input of BiFPN1, the output of BiFPN1 serves as the input of CSP21, the output of CSP21 serves as the input of CBS7, the output of CBS7 serves as the input of Upsample2, the output of Upsample2 and GELAN3 serves as the dual input of BiFPN2, the output of BiFPN2 serves as the input of CSP22, the output of CSP22 serves as the input of CBS8, the outputs of CSP21 and CBS8 serve as the dual input of BiFPN3, the output of BiFPN3 serves as the input of CSP23, the output of CSP23 serves as the input of CBS9, the outputs of CBS6 and CBS9 serve as the dual input of BiFPN4, and the output of BiFPN4 serves as the input of CSP24.

5. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 4 is characterized in that: The output unit includes: 3 Conv layers Conv1~Conv3; The output of CSP22 is used as the input of Conv1, and the output of Conv1 is the small-sized candidate box Prediction1; the output of CSP23 is used as the input of Conv2, and the output of Conv2 is the medium-sized candidate box Prediction2; the output of CSP4 is used as the input of Conv3, and the output of Conv3 is the large-sized candidate box Prediction3.

6. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: In step four, ; Where, f Indicates the focal length of the binocular camera, B Indicates the baseline of the binocular camera; θ Indicates the pitch angle of the gimbal; Indicates the horizontal rotation angle of the gimbal.

7. The binocular PTZ fire source automatic positioning method based on deep learning according to claim 1 is characterized in that: The training methods for the fire source detection model include: S1, obtain a set of fire source sample images with true labels; S2, divide the fire source sample image set with real labels into training set, validation set and test set; S3, fire source detection model based on the training set N rounds of training; verify the fire source detection model after each round of training based on the validation set until the fire source detection model with the best performance is selected; evaluate the fire source detection model with the best performance based on the test set, if it meets the evaluation requirements, it will be used as the trained fire source detection model, otherwise, repeat S3.

8. A binocular PTZ fire source automatic positioning device based on deep learning, characterized in that: It includes: A pan / tilt platform equipped with a binocular camera, which is used to perform step 1 of the binocular pan / tilt platform fire source automatic positioning method based on deep learning as described in any one of claims 1 to 7; as well as The host computer is used to perform steps 2 to 5 of the binocular pan-tilt head fire source automatic positioning method based on deep learning as described in claims 1-7.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the binocular pan-tilt head fire source automatic positioning method based on deep learning as described in any one of claims 1 to 7 are implemented.

Images