Fire source distance measuring method based on binocular vision

Abstract

The invention discloses a fire source distance measuring method based on binocular vision in the technical field of computer vision and image processing. The fire source distance measuring method comprises the following steps: S1, improving a YOLOv5 algorithm; s2, sending the fire source data set into an improved YOLO algorithm for training, and obtaining a model weight; s3, performing accelerated reasoning on the model by adopting an ONNX method; s4, calibrating the binocular camera, and obtaining internal parameters of the camera; s5, carrying out distortion correction and stereo correction on the pictures collected by the binocular camera; s6, calculating and generating a disparity map by adopting an SGBM algorithm; s7, the distance is calculated according to the geometric principle of binocular vision, rapid detection of a fire source is achieved through the improved YOLOv5 algorithm and the ONNX accelerated reasoning technology, and the fire monitoring requirement in the complex environment is met; three-dimensional depth information is obtained by using a binocular vision technology, and accurate measurement of the distance of a fire source in various scenes is ensured by combining accurate camera calibration and image distortion correction.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and in particular to a fire source distance measurement method. Background Art

[0002] Fire source detection and ranging technology has received extensive attention in the field of fire safety in recent years. Traditional fire detection methods, such as smoke detectors and temperature sensors, can detect the occurrence of fires, but cannot accurately determine the specific location and distance of the fire source, and usually require a higher concentration or temperature threshold to trigger an alarm. This limits its application in complex environments. In addition, the existing fire monitoring system has problems with detection blind spots and false alarms when facing outdoor scenes or large areas. Therefore, how to improve the accuracy and real-time performance of fire source detection has become a key issue in fire prevention and control research. Against this background, fire source ranging technology based on binocular vision has gradually emerged. Binocular vision technology imitates the working principle of the human eye and estimates the depth of the scene through the difference in perspective captured by two cameras. Compared with monocular vision, binocular vision can directly obtain the three-dimensional position information of the target object, thereby achieving more accurate distance measurement. The detection of fire sources uses the YOLO algorithm in deep learning to quickly identify flames through a trained neural network model. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a fire source ranging method based on binocular vision, which solves the shortcomings of traditional fire detection methods in real-time, detection accuracy and adaptability, thereby improving the intelligence level and response speed of fire prevention and control. Through the improved YOLOv5 algorithm and ONNX accelerated reasoning technology, rapid detection of fire sources can be achieved to meet the needs of fire monitoring in complex environments; binocular vision technology is used to obtain three-dimensional depth information, combined with precise camera calibration and image distortion correction, to ensure that the distance to the fire source can be accurately measured in various scenarios.

[0004] The object of the present invention is achieved in this way: a fire source distance measurement method based on binocular vision, characterized in that it includes the following parts:

[0005] Step S1, improving the YOLOv5 algorithm;

[0006] Step S2, sending the fire source data set to the improved YOLO algorithm for training to obtain model weights;

[0007] Step S3, using ONNX method to accelerate the reasoning of the model;

[0008] Step S4, calibrate the binocular camera to obtain the camera internal parameters;

[0009] Step S5: performing distortion correction and stereo correction on the photos collected by the binocular camera;

[0010] Step S6, using the SGBM algorithm to calculate and generate a disparity map;

[0011] Step S7: Calculate the distance using the geometric principle of binocular vision.

[0012] Furthermore, in step S1, improving the YOLO algorithm specifically includes the following steps:

[0013] Step S1.1, replace the Focus module in the original YOLOv5 algorithm with the convolution module;

[0014] First, replace the Focus module in the original YOLOv5 algorithm with the convolution module. For the input feature map 𝑋∈R 𝐶×𝐻×𝑊 , where C is the number of channels, H and W are the height and width of the image respectively. After being processed by the convolution module, the output feature map Y is obtained. Y is the feature map obtained by weighted summation of the convolution kernel under the sliding window. The specific formula is:

[0015] (1)

[0016] Where k is the size of the convolution kernel, X C，i+m,j+n Represents the pixel value of the input image X at channel C and spatial position (i+m, j+n), W C,m,n Represents the weight parameters of the convolution kernel;

[0017] Secondly, the remaining layers in the backbone network are replaced by Shuffle_Block modules; specifically, the first layer is a convolutional layer, and the remaining layers are Shuffle_Block modules.

[0018] Finally, the Concat layer in Neck is improved with the WConcat structure. Before fusion, the WConcat structure first defines a learnable weight parameter w, which is a tensor of length 2, representing the initial weights of the input feature maps x0 and x1, and is continuously updated through back propagation and gradient descent during training.

[0019] Through the normalization operation, the sum of the two weights is ensured to be 1, so as to avoid a certain layer of feature maps occupying too much weight in the fusion process. The calculation formula is:

[0020] (2)

[0021] (3)

[0022] Where ε is a small constant used to avoid the situation where the denominator is 0. weight0 and weight1 correspond to the weights of x0 and x1 respectively.

[0023] Finally, the weights and feature maps are correspondingly multiplied and concatenated, and the output feature map x is:

[0024] (4)

[0025] Among them, Concat is a concatenation operation;

[0026] Furthermore, in the step S2, step S2 specifically includes: obtaining 3,000 photos of fire sources, and sending them into the network for training according to the ratio of verification set to training set 8:2, setting the batch number to 16 and the round number to 50, and finally completing the training to obtain the model weight.

[0027] Furthermore, step S3 includes the following specific steps: exporting the model generated in step S2 into ONNX format, and using ONNX Runtime to perform accelerated reasoning of the model.

[0028] Furthermore, the step S4 of calibrating the binocular camera specifically includes:

[0029] First, use a binocular camera to shoot chessboard patterns in at least two different directions. The photos must be clear and the chessboard content must be complete. After shooting, the left and right images are cropped out according to the pixels of the image. The left and right images are sent to the calibration toolbox of Matlab respectively, and the images with large errors are deleted. Finally, the distortion parameters and rotation matrix of the camera are generated and the data is saved.

[0030] Furthermore, the step S5 specifically includes the following steps:

[0031] S5.1 calculate camera parameters;

[0032] Use the cv2.stereoRectify() function in opencv to calculate the rotation matrix and projection matrix of the left camera and the right camera. The input parameters are the intrinsic parameter matrix of the left camera, the distortion coefficient of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the right camera, the size of the picture, the rotation matrix and the translation vector. The output parameters are the rotation matrix R1 of the left camera, the rotation matrix R2 of the right camera, the projection matrix P1 of the left camera, the projection matrix P2 of the right camera, the reprojection matrix Q and the effective area of ​​the left and right images after correction.

[0033] S5.2 performs distortion correction;

[0034] The iterative method is used to solve the problem, and the ideal pixel coordinates without distortion (x, y) are deduced from the distorted pixel coordinates (x', y'). The correction formula is:

[0035] (12)

[0036] (13)

[0037] Among them, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients, and r ’2 =x ’2 +y' 2 is the square of the radial distance of the distorted pixel.

[0038] S5.3 performs stereo correction;

[0039] In the stereo correction stage, the pixels in the image are rotated and projected through the rotation matrices R1 and R2, and the projection matrices P1 and P2 to ensure that the images of the left and right cameras are coplanar and the optical axes are parallel. The specific formula is:

[0040] (14)

[0041] (15)

[0042] Where (x l ,y l ) ,(x r ,y r ) are the pixel coordinates after correction for the left camera and the right camera, and the transformation relationship between the left and right images is saved using mapping matrices map1 and map2 respectively.

[0043] S5.4 remapping;

[0044] Redistribute the pixels in the original image to the corrected image according to the mapping matrix. For each pixel (u, v) in the corrected image, find its corresponding pixel position (x, y) in the original image from map1 (u, v) and map2 (u, v), and assign the pixel at that position to the pixel of the corrected image, as follows:

[0045] (16)

[0046] The two sides of the equation represent the pixel values ​​of the original image and the corrected image respectively.

[0047] Furthermore, in step S6, the SGBM algorithm is used to calculate and generate the disparity map, including the following steps:

[0048] S6.1 Image preprocessing;

[0049] The SGBM algorithm first preprocesses the image using the Sobel operator, and its formula is:

[0050] (17)

[0051] Where x, y represent the horizontal and vertical coordinates of the pixel respectively, and P(x, y) represents the value of the pixel;

[0052] After being processed by the Sobel operator, each pixel processed by the Sobel operator is mapped into a new image through a mapping function. The specific formula is as follows:

[0053] (18)

[0054] Where P represents the pixel value at each position after being processed by the Sobel operator, P NEW Represents the pixel value after being processed by the mapping function. preFilterCap is a constant parameter, which is generally 15 in opencv.

[0055] S6.2 solves the optimal matching disparity by semi-global path aggregation;

[0056] By comparing the pixel value with the interpolated value of its neighboring pixel values, the formula is:

[0057] (19)

[0058] Among them I L (x,y) and I R (x, y) represents the value of the pixel (x, y) in the left image and the right image respectively, d is the disparity value, I R (xd-0.5,y) and I R (x-d+0.5,y) represent the interpolated pixels corresponding to the pixels of the right image and the work image respectively;

[0059] By propagating the cost in different directions, a globally optimized cost is finally formed; the path aggregation formula is as follows (20):

[0060] (20)

[0061] Where S(x,y,z) represents the total cost at pixel (x,y) with disparity d, C(x,y,d) is the matching cost function, P1 is the penalty term when the disparity difference |d-d'|=1, P2 is the penalty term when the disparity difference |d-d'|>1, δ(|d-d'|>1) is the indicator function, δ=1 when the disparity difference is greater than 1, otherwise δ=0;

[0062] SGBM calculates the cumulative cost along multiple paths and selects multiple directions for aggregation; for each pixel p = (x, y), the total aggregation cost S (p, d) is the sum of the cumulative costs along all paths, and its formula is as follows (21):

[0063] (twenty one)

[0064] Where r represents the direction;

[0065] For each pixel p=(x,y), select the disparity value d that minimizes the cumulative cost S(p,d) opt As the optimal disparity; its formula is as follows (22):

[0066] (twenty two)

[0067] S6.3 Post-processing;

[0068] The disparity map generated in the above steps is post-processed; it is mainly divided into confidence detection, left-right consistency detection, and sub-pixel interpolation;

[0069] Confidence detection is usually calculated by matching the minimum and second minimum values ​​of the cost; for a certain pixel, the disparity cost function is C(d), where d is the disparity, and the confidence detection formula is (23):

[0070] (twenty three)

[0071] Where C(d1) is the minimum value of the disparity cost, corresponding to the optimal disparity d1, and C(d2) is the second minimum value, corresponding to the suboptimal disparity d2; and the confidence threshold is set to Tconf; if the confidence is lower than the threshold, the disparity estimate may be a mismatch and can be marked as invalid;

[0072] Left-right consistency detection is used to determine the disparity d calculated from the left view left and the parallax d from the right view right Are they consistent? The parallax of a pixel x in the left image is d left (x), then the corresponding pixel position in the right figure is x'=xd left (x); then the difference between the left and right parallax is:

[0073] (twenty four)

[0074] Set the consistency threshold to T lr ,If the error is larger than this threshold, the inspection estimate is inconsistent and is considered invalid;

[0075] Under high confidence and left-right consistent disparity estimation, sub-pixel interpolation can be applied to improve the accuracy of disparity; at integer disparity d and its surroundings d-1 and d+1, there are matching costs C(d-1), C(d), and C(d+1); then the interpolation formula is:

[0076] (25)

[0077] where d is the initial integer disparity estimate, d refined It is the sub-pixel disparity value after parabolic interpolation to improve accuracy.

[0078] Furthermore, in step S7, the distance is calculated using the geometric principle of binocular vision, including the following steps:

[0079] The normalized target center point obtained in target recognition is c x , c y , the length and width of the picture are h and w respectively, then the parallax d of the target object can be obtained as:

[0080] (26)

[0081] The distance Z is calculated according to the geometric principle of binocular vision. The specific process is:

[0082] (27)

[0083] Where f is the focal length of the camera and B is the baseline distance between the two lenses.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] The present invention adopts the improved YOLOv5 algorithm for fire source identification, and introduces ONNX technology to accelerate the reasoning process, which significantly improves the real-time performance of fire source detection. Traditional fire source detection methods (such as temperature sensors or smoke sensors) often require a long response time, while the present invention can achieve rapid positioning of the fire source and adapt to the emergency treatment needs of sudden fires. The three-dimensional position of the fire source is obtained by binocular vision technology. Compared with the monocular ranging method, the present invention has higher accuracy in obtaining depth information. The camera calibration and image distortion correction are combined with Zhang Zhengyou's calibration method, which makes the distance measurement of the fire source more accurate and effectively reduces the ranging error; through this technology, the fire source can be quickly detected and its distance can be measured in complex fire scenes, providing more accurate real-time data support for the automated fire fighting system. This technology can be applied to drone fire monitoring, intelligent fire fighting robots and fixed monitoring equipment to improve the efficiency and accuracy of fire response. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0087] Figure 1 It is a flow chart of the present invention.

[0088] Figure 2 It is a structural diagram of the improved YOLO algorithm in the present invention.

[0089] Figure 3 It is a schematic diagram of the Shuffle_Block structure in the present invention.

[0090] Figure 4 This is a comparison chart of the effects of the improved algorithm in the present invention and the original YOLOv5 algorithm.

[0091] Figure 5 It is the chessboard diagram calibrated in the present invention. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] A fire source ranging method based on binocular vision can efficiently realize accurate identification and distance measurement of fire sources. First, the flame is detected using the improved YOLOv5 algorithm, and the ONNX technology is used to accelerate model reasoning to improve the real-time performance of fire source identification. Secondly, the binocular camera is accurately calibrated by the Zhang Zhengyou calibration method to obtain internal reference information to ensure the accuracy of ranging. Then, the collected image is distorted to eliminate the geometric distortion caused by the lens. Then, the SGBM algorithm is used to generate a disparity map to obtain the depth information of the scene. Finally, the distance of the fire source is calculated based on the disparity map and camera parameters. This method significantly improves the accuracy and response speed of fire source detection, and is suitable for fire prevention and rescue application scenarios such as drone fire monitoring and intelligent firefighting robots.

[0094] Example 1

[0095] Step S1, improve the YOLOv5 algorithm

[0096] First, replace the Focus module in the original YOLOv5 algorithm with the convolution module. For the input feature map 𝑋∈R 𝐶×𝐻×𝑊 , where C is the number of channels, H and W are the height and width of the image respectively. After being processed by the convolution module, the output feature map Y is obtained. Y is the feature map obtained by weighted summation of the convolution kernel under the sliding window. The specific formula is:

[0097] (twenty two)

[0098] Where k is the size of the convolution kernel, X C，i+m,j+n Represents the pixel value of the input image X at channel C and spatial position (i+m, j+n), W C,m,n Represents the weight parameters of the convolution kernel.

[0099] Secondly, the remaining layers in the backbone network are replaced with Shuffle_Block modules. Specifically, the first layer is a convolutional layer, and the remaining layers are Shuffle_Block modules.

[0100] Finally, the Concat layer in Neck is improved with the WConcat structure. Before fusion, the WConcat structure first defines a learnable weight parameter w, which is a tensor of length 2, representing the initial weights of the input feature maps x0 and x1, and is continuously updated through back propagation and gradient descent during training.

[0101] Through the normalization operation, the sum of the two weights is ensured to be 1, so as to avoid a certain layer of feature maps occupying too much weight in the fusion process. The calculation formula is:

[0102] (twenty three)

[0103] (twenty four)

[0104] Where ε is a small constant used to avoid the situation where the denominator is 0. weight0 and weight1 correspond to the weights of x0 and x1 respectively.

[0105] Finally, the weights and feature maps are correspondingly multiplied and concatenated, and the output feature map x is:

[0106] (25)

[0107] Concat is a concatenation operation.

[0108] Step S2: Send the data set to the improved YOLOv5 model for training.

[0109] 3,000 fire source photos were obtained and sent to the network for training with a validation set and training set ratio of 8:2. The batch size was set to 16 and the number of rounds was set to 100. Finally, the training was completed and the model file was obtained.

[0110] Step S3: Use ONNX to accelerate the reasoning process of the model.

[0111] Export the model generated in step S2 to ONNX format, and use ONNX Runtime to perform accelerated inference of the model;

[0112] Finally, the model ran at a speed of 15FPS on the Raspberry Pi 4B.

[0113] Step S4, calibrating the binocular camera, includes the following steps:

[0114] First, use a binocular camera to take 30 photos of the chessboard pattern in at least two different directions. The photos must be clear and the chessboard content must be complete.

[0115] After shooting, the left and right images are cropped out according to the pixels of the picture. Since the size of the image shot by this camera is 640×320, the left and right images are:

[0116] (26)

[0117] Among them, img is the original image, left and right are the cropped images.

[0118] The left and right images are sent to the calibration toolbox of Matlab respectively, and the images with large errors are deleted. Finally, the distortion parameters, rotation matrix, etc. of the camera are generated and the data is saved.

[0119] The parameters of the left and right cameras after calibration are:

[0120] (27)

[0121] (28)

[0122] The rotation matrix R1 is:

[0123] (29)

[0124] Step S5, performing distortion correction and stereo correction on the image generated by the binocular camera, includes the following steps:

[0125] S5.1 Calculation of camera parameters

[0126] The cv2.stereoRectify() function in opencv is used to calculate the rotation matrix and projection matrix of the left and right cameras. The input parameters are the intrinsic parameter matrix of the left camera, the distortion coefficient of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the right camera, the size of the picture, the rotation matrix and the translation vector. The output parameters are the rotation matrix R1 of the left camera, the rotation matrix R2 of the right camera, the projection matrix P1 of the left camera, the projection matrix P2 of the right camera, the reprojection matrix Q and the effective area of ​​the left and right images after correction.

[0127] S5.2 for distortion correction

[0128] The purpose of distortion correction is to infer the ideal coordinates (x, y) without distortion from the distorted coordinates (x', y'). The correction formula is solved by iterative method:

[0129] (30)

[0130] (31)

[0131] Among them, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients, and r 2 =x 2 +y 2 is the distance from the ideal pixel without distortion to the center of the image.

[0132] S5.3 Stereo Correction

[0133] In the stereo correction stage, the pixels in the image are rotated and projected through the rotation matrices R1 and R2, and the projection matrices P1 and P2 to ensure that the images of the left and right cameras are coplanar and the optical axes are parallel. The specific formula is:

[0134] (32)

[0135] (33)

[0136] Where (x l ,y l ) ,(x r ,y r ) are the pixel coordinates after correction for the left camera and the right camera, and the transformation relationship between the left and right images is saved using mapping matrices map1 and map2 respectively.

[0137] S5.4 Remapping

[0138] Remapping is to redistribute the pixels in the original image to the corrected image according to the mapping matrix. For each pixel (u, v) in the corrected image, find its corresponding pixel position (x, y) in the original image from map1 (u, v) and map2 (u, v), and assign the pixel at that position to the pixel of the corrected image. The specific principle is as follows:

[0139] (34)

[0140] The two sides of the equation represent the pixel values ​​of the original image and the corrected image respectively.

[0141] In step S6, the SGBM algorithm is used to calculate and generate a disparity map.

[0142] S6.1 Image preprocessing

[0143] The SGBM algorithm first preprocesses the image using the Sobel operator. It can provide important edge information for structural analysis and shape recognition in the image, so as to better identify the shape or contour of the object in subsequent computer vision or image analysis tasks. Its formula is:

[0144] (35)

[0145] Where x and y represent the horizontal and vertical coordinates of the pixel respectively, and P(x,y) represents the value of the pixel. The pixel value of each pixel is finally obtained after the Sobel operator is processed. After the Sobel operator is processed, the edge contour of the image becomes very obvious, the contour and structure are highlighted, and the image is denoised.

[0146] After being processed by the Sobel operator, each pixel processed by the Sobel operator is mapped into a new image through a mapping function. Its formula is as follows (36):

[0147] (36)

[0148] Where P represents the pixel value at each position after being processed by the Sobel operator, P NEW Represents the pixel value after processing by the mapping function. preFilterCap is a constant parameter, which is generally 15 in opencv.

[0149] S6.2 Solving the Optimal Matching Disparity via Semi-Global Path Aggregation

[0150] In traditional matching costs (such as the SAD algorithm), only the difference between corresponding pixels in the image is usually considered, which is easily affected by sampling errors and quantization noise. The SGBM algorithm calculates the BT cost, which can better handle sub-pixel errors and improve matching accuracy by comparing the pixel value with the interpolation of its adjacent pixel values. Its formula is shown in (37).

[0151] (37)

[0152] Among them I L (x,y) and I R (x, y) represents the value of the pixel (x, y) in the left image and the right image respectively, d is the disparity value, I R (xd-0.5,y) and I R (x-d+0.5, y) represent the interpolated pixels corresponding to the pixels of the right image and the work image, respectively, which are used to process sub-pixel level matching.

[0153] SGBM optimizes the disparity estimation by summing the costs on multiple paths. This approach combines the efficiency of local matching and the accuracy of global optimization. By propagating the costs along different directions, a globally optimized cost is finally formed, as shown in formula (38):

[0154] (38)

[0155] Where S(x,y,z) represents the total cost at pixel (x,y) with disparity d, C(x,y,d) is the matching cost function, P1 is the penalty term when the disparity difference |d-d'|=1, P2 is the penalty term when the disparity difference |d-d'|>1, δ(|d-d'|>1) is the indicator function, δ=1 when the disparity difference is greater than 1, otherwise δ=0.

[0156] SGBM calculates the cumulative cost along multiple paths and selects 8 directions for aggregation. The common 8 directions are horizontal, vertical, diagonal, and anti-diagonal. For each pixel p=(x,y), the total aggregation cost p=(x,y) is the sum of the cumulative costs along all paths, and its formula is as follows:

[0157] (39)

[0158] Where r represents the direction.

[0159] For each pixel p=(x,y), select the disparity value d that minimizes the cumulative cost S(p,d) opt As the optimal disparity. Its formula is as follows:

[0160] (40)

[0161] S6.3 Post-processing

[0162] To improve the disparity map, the disparity map is post-processed, which is mainly divided into confidence detection, left-right consistency detection, and sub-pixel interpolation.

[0163] The goal of confidence detection is to measure the reliability of disparity estimation, which is usually calculated by matching the minimum and second minimum values ​​of the cost. Assuming that for a certain pixel, the disparity cost function is C(d), where d is the disparity, the formula for confidence detection is:

[0164] (41)

[0165] Where C(d1) is the minimum value of the disparity cost, corresponding to the optimal disparity d1, and C(d2) is the second minimum value, corresponding to the suboptimal disparity d2. And set the confidence threshold to Tconf. If the confidence is lower than the threshold, the disparity estimate may be a mismatch and can be marked as invalid.

[0166] Left-right consistency detection is used to determine the disparity d calculated from the left view left and the parallax d from the right view right Are they consistent? Assume that the disparity of a pixel x in the left image is d left (x), then the corresponding pixel position in the right figure is x'=xd left (x). Then the difference between the left and right parallax is:

[0167] (42)

[0168] Set the consistency threshold to T lr ,If the error is greater than the threshold, the inspection is inconsistent and is considered invalid.

[0169] Under high confidence and left-right consistent disparity estimation, sub-pixel interpolation can be applied to improve the accuracy of disparity. Assume that there are matching costs C(d-1), C(d), and C(d+1) at integer disparity d and its surroundings d-1 and d+1. Then the interpolation formula is:

[0170] (43)

[0171] where d is the initial integer disparity estimate, d refined It is the sub-pixel disparity value after parabolic interpolation to improve accuracy.

[0172] In step S7, the distance is calculated using the geometric principle of binocular vision, including the following steps:

[0173] Assume that the normalized target center point obtained in target recognition is c x , c y, the length and width of the picture are h and w respectively, then the parallax d of the target object can be obtained as:

[0174] (44)

[0175] The distance Z is calculated according to the geometric principle of binocular vision. The specific process is as follows:

[0176] (45)

[0177] Where f is the focal length of the camera and B is the baseline distance between the two lenses.

[0178] The actual test showed excellent performance and wide applicability. The improved YOLOv5 algorithm combined with ONNX accelerated inference technology greatly improved the real-time and accuracy of fire source detection; experiments showed that mAP_0.5 increased by about 9%, mAP_0.5:0.95 increased by about 4%, precision increased by about 13%, and recall increased by about 2%. In addition, the binocular camera was accurately calibrated by Zhang Zhengyou calibration method, and the distortion correction and stereo correction methods were used to effectively eliminate the geometric distortion of the image and ensure the reliability of parallax calculation. The experimental results show that this method can accurately detect fire sources and measure distance in a variety of complex scenarios, and the ranging error is significantly lower than that of traditional methods, which fully reflects its potential and application value in intelligent fire protection systems.

[0179] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A fire source distance measurement method based on binocular vision, characterized in that: Includes the following sections: Step S1, improving the YOLOv5 algorithm; Step S2, sending the fire source data set to the improved YOLO algorithm for training to obtain model weights; Step S3, using ONNX method to accelerate the reasoning of the model; Step S4, calibrate the binocular camera to obtain the camera internal parameters; Step S5: performing distortion correction and stereo correction on the photos collected by the binocular camera; Step S6, using the SGBM algorithm to calculate and generate a disparity map; Step S7: Calculate the distance using the geometric principle of binocular vision.

2. The method for measuring fire source distance based on binocular vision according to claim 1, characterized in that: In step S1, improving the YOLO algorithm specifically includes the following steps: Step S1.1, replace the Focus module in the original YOLOv5 algorithm with the convolution module; First, replace the Focus module in the original YOLOv5 algorithm with the convolution module. For the input feature map X∈R C×H×W , where C is the number of channels, H and W are the height and width of the image respectively. After being processed by the convolution module, the output feature map Y is obtained. Y is the feature map obtained by weighted summation of the convolution kernel under the sliding window. The specific formula is: (1) Where k is the size of the convolution kernel, X C，i+m,j+n Represents the pixel value of the input image X at channel C and spatial position (i+m, j+n), W C,m,n Represents the weight parameter of the convolution kernel; Secondly, the remaining layers in the backbone network are replaced with Shuffle_Block modules; specifically, the first layer is a convolutional layer, and the remaining layers are Shuffle_Block modules; Finally, the Concat layer in Neck is improved with the WConcat structure. Before fusion, the WConcat structure first defines a learnable weight parameter w, which is a tensor of length 2, representing the initial weights of the input feature maps x0 and x1, and is continuously updated through back propagation and gradient descent during training. Through the normalization operation, the sum of the two weights is ensured to be 1, so as to avoid a certain layer of feature maps occupying too much weight in the fusion process. The calculation formula is: (2) (3) Where ε is a small constant to avoid the situation where the denominator is 0; weight0 and weight1 correspond to the weights of x0 and x1 respectively; Finally, the weights and feature maps are correspondingly multiplied and concatenated, and the output feature map x is: (4)； Concat is a concatenation operation.

3. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: In the step S2, step S2 specifically includes: obtaining 3,000 photos of fire sources, and sending them into the network for training according to the ratio of verification set to training set 8:2, setting the batch number to 16 and the round number to 50, and finally completing the training to obtain the model weight.

4. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: The step S3 includes the following specific steps: exporting the model generated in step S2 into ONNX format, and using ONNX Runtime to perform accelerated reasoning of the model.

5. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: The step S4 of calibrating the binocular camera specifically includes: First, use a binocular camera to shoot chessboard patterns in at least two different directions. The photos must be clear and the chessboard content must be complete. After shooting, the left and right images are cropped out according to the pixels of the image. The left and right images are sent to the calibration toolbox of Matlab respectively, and the images with large errors are deleted. Finally, the distortion parameters and rotation matrix of the camera are generated and the data is saved.

6. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: The step S5 specifically comprises the following steps: S5.1 calculate camera parameters; Use the cv2.stereoRectify() function in opencv to calculate the rotation matrix and projection matrix of the left camera and the right camera. The input parameters are the intrinsic parameter matrix of the left camera, the distortion coefficient of the left camera, the intrinsic parameter matrix of the right camera, the distortion coefficient of the right camera, the size of the picture, the rotation matrix and the translation vector. The output parameters are the rotation matrix R1 of the left camera, the rotation matrix R2 of the right camera, the projection matrix P1 of the left camera, the projection matrix P2 of the right camera, the reprojection matrix Q and the effective area of ​​the left and right images after correction. S5.2 performs distortion correction; The iterative method is used to solve the problem, and the ideal pixel coordinates without distortion (x, y) are deduced from the distorted pixel coordinates (x', y'). The correction formula is: (12)； (13)； Among them, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients, and r ’2 =x ’2 +y' 2 is the square of the radial distance of the distorted pixel; S5.3 performs stereo correction; In the stereo correction stage, the pixels in the image are rotated and projected through the rotation matrices R1 and R2, and the projection matrices P1 and P2 to ensure that the images of the left and right cameras are coplanar and the optical axes are parallel. The specific formula is: (14)； (15)； Where (x l ,y l ) ,(x r ,y r ) are the pixel coordinates of the left camera and the right camera after correction, and the transformation relationship between the left and right images is saved using mapping matrices map1 and map2 respectively; S5.4 remapping; Redistribute the pixels in the original image to the corrected image according to the mapping matrix. For each pixel (u, v) in the corrected image, find its corresponding pixel position (x, y) in the original image from map1 (u, v) and map2 (u, v), and assign the pixel at that position to the pixel of the corrected image, as follows: (16)； The two sides of the equation represent the pixel values ​​of the original image and the corrected image respectively.

7. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: In step S6, the SGBM algorithm is used to calculate and generate a disparity map, including the following steps: S6.1 Image preprocessing; The SGBM algorithm first preprocesses the image using the Sobel operator, and its formula is: (17)； Where x, y represent the horizontal and vertical coordinates of the pixel respectively, and P(x, y) represents the value of the pixel; After being processed by the Sobel operator, each pixel processed by the Sobel operator is mapped into a new image through a mapping function. The specific formula is as follows: (18)； Where P represents the pixel value at each position after being processed by the Sobel operator, P NEW Represents the pixel value after being processed by the mapping function. preFilterCap is a constant parameter, which is generally 15 in opencv. S6.2 solves the optimal matching disparity by semi-global path aggregation; By comparing the pixel value with the interpolated value of its neighboring pixel values, the formula is: (19)； Among them, I L (x,y) and I R (x, y) represents the value of the pixel (x, y) in the left image and the right image respectively, d is the disparity value, I R (xd-0.5,y) and I R (x-d+0.5,y) represent the interpolated pixels corresponding to the pixels of the right image and the work image respectively; By propagating the cost in different directions, a globally optimized cost is finally formed; the formula is as follows (20): (20) Where S(x,y,z) represents the total cost at pixel (x,y) with disparity d, C(x,y,d) is the matching cost function, P1 is the penalty term when the disparity difference |d-d'|=1, P2 is the penalty term when the disparity difference |d-d'|>1, δ(|d-d'|>1) is the indicator function, δ=1 when the disparity difference is greater than 1, otherwise δ=0; SGBM calculates the cumulative cost along multiple paths and selects multiple directions for aggregation; for each pixel p = (x, y), the total aggregation cost S (p, d) is the sum of the cumulative costs along all paths, and its formula is as follows (21): (21)； Where r represents the direction; For each pixel p=(x,y), select the disparity value d that minimizes the cumulative cost S(p,d) opt As the optimal disparity; its formula is as follows (22): (22)； S6.3 Post-processing; The disparity map generated in the above steps is post-processed; it is mainly divided into confidence detection, left-right consistency detection, and sub-pixel interpolation; Confidence detection is usually calculated by matching the minimum and second minimum values ​​of the cost; for a certain pixel, the disparity cost function is C(d), where d is the disparity, and the confidence detection formula is (23): (23)； Where C(d1) is the minimum value of the disparity cost, corresponding to the optimal disparity d1, and C(d2) is the second minimum value, corresponding to the suboptimal disparity d2; and the confidence threshold is set to Tconf; if the confidence is lower than the threshold, the disparity estimate may be a mismatch and can be marked as invalid; Left-right consistency detection is used to determine the disparity d calculated from the left view left and the parallax d from the right view right Are they consistent? The parallax of a pixel x in the left image is d left (x), then the corresponding pixel position in the right figure is x'=xd left (x); then the difference between the left and right parallax is: (24)； Set the consistency threshold to T lr ,If the error is larger than this threshold, the inspection estimate is inconsistent and is considered invalid; Under high confidence and left-right consistent disparity estimation, sub-pixel interpolation can be applied to improve the accuracy of disparity; at integer disparity d and its surroundings d-1 and d+1, there are matching costs C(d-1), C(d), and C(d+1); then the interpolation formula is: (25)； where d is the initial integer disparity estimate, d refined It is the sub-pixel disparity value after parabolic interpolation to improve accuracy.

8. The fire source ranging algorithm based on binocular vision according to claim 1 is characterized in that: In step S7, the distance is calculated using the geometric principle of binocular vision, including the following steps: The normalized target center point obtained in target recognition is c x , c y , the length and width of the picture are h and w respectively, then the parallax d of the target object can be obtained as: (26)； The distance Z is calculated according to the geometric principle of binocular vision. The specific process is: (27)； Where f is the focal length of the camera and B is the baseline distance between the two lenses.

Images