Fire behavior parameter measurement method and system based on flame binocular image self-correction

Abstract

The invention provides a fire behavior parameter measurement method and system based on flame binocular image self-correction, and belongs to the field of fire monitoring. The objective of the invention is to solve the problem that the accuracy of fire behavior parameter calculation is affected due to low precision of acquired flame depth information caused by limitation and difference between images shot by left and right eyes when the existing binocular vision technology is used for monitoring rapidly changing flames. According to the method, a self-made binocular camera is adopted to shoot a single flame, and a flame contour line is extracted through binarization operation, edge detection and a filtering algorithm; dividing a flame contour area by using the extracted flame contour line, and extracting and matching flame response feature points from the flame contour area; a B spline grid is constructed, and an optimal deformation displacement field is obtained through iteration to realize correction of left and right eye flame images, so that the calculation precision of flame key point information is improved, and finally accurate measurement of fire behavior parameters is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of fire monitoring, and in particular to a method and system for measuring fire behavior parameters based on flame binocular image self-correction. Background Art

[0002] In fire management, fire behavior parameters are important indicators of flame research and directly affect the scope of fire and the difficulty of prevention and control. Therefore, researching and developing new fire behavior parameter measurement methods is crucial to timely control the scale of fire spread.

[0003] Traditional fire behavior parameter measurement methods mainly rely on manual observation or reference object comparison detection technology, but these methods have certain limitations, such as limited observation range, low accuracy and poor real-time performance. With the rapid development of technologies such as computers, thermal imaging, laser scanning and binocular vision, new ideas are provided for measuring fire behavior parameters. Although image processing and deep learning technologies can extract fire behavior parameter information, they require a large amount of measured data as data sets, and the accuracy needs to be improved; thermal imaging technology has advantages in flame temperature extraction, but it is more difficult to extract flame depth information; laser scanning technology can provide relatively accurate three-dimensional flame coordinates, but it requires the use of specific equipment, such as altimeters or laser rangefinders, which are complex and expensive and require professional operation and maintenance, limiting the popularization and application scope of the technology; compared with the above technologies, binocular vision technology has obvious advantages. Binocular vision detection technology is a height measurement method based on the principle of stereo vision. The binocular vision detection system is usually deployed on the ground. The technology uses two cameras to simultaneously shoot the same flame from different angles and calculate its parallax, thereby obtaining relatively accurate three-dimensional information of the object. At the same time, the binocular vision system is relatively low in price and is not limited by sensors.

[0004] At present, binocular vision flame measurement technology is mostly focused on optimizing the camera baseline length and improving the calculation quality of flame depth information, and improving the accuracy of flame three-dimensional coordinate calculation through methods such as long baseline and depth error optimization. However, these methods are difficult to deal with flame image distortion caused by complex fire environments and insufficient reliability of the camera's own hardware and algorithms, and the types of fire behavior parameters measured are also relatively small. For example, the existing patent is a calibration and correction method for a flame space positioning system based on binocular vision (application number: CN202211412692.1), in which the flame space positioning system based on binocular vision includes: an image acquisition unit, used to synchronously acquire flame images and / or videos from different perspectives, and the image acquisition unit is two binocular cameras with different perspectives; and a flame three-dimensional information acquisition unit, which solves the flame three-dimensional information based on the flame image and / or video; the calibration and correction method includes: establishing a camera model, including a small hole forming model of the camera and a camera imaging transformation relationship model; calibrating and correcting a monocular camera based on the camera model and a 2D plane target, and the calibration and correction of the monocular camera includes camera parameter solution and monocular camera calibration implementation; stereo calibration and correction of the binocular system composed of binocular cameras is performed based on the calibration and correction results of the monocular camera and the principle of epipolar geometry. Binocular vision technology has limitations when monitoring rapidly changing flames. There are differences between the images captured by the left and right eyes. These differences are caused by image delay, jitter and viewing angle difference, resulting in low accuracy of flame depth information obtained, which in turn affects the accuracy of fire behavior parameter calculation. Therefore, it is of great significance to study the fire behavior parameter measurement method based on flame binocular image self-correction. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to solve the problem that when the existing binocular vision technology monitors rapidly changing flames, the images collected by the left and right eyes are different due to image delay, jitter and viewing angle difference, resulting in low accuracy of the flame depth information obtained, affecting the accuracy of fire behavior parameter calculation.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The present invention provides a method for measuring fire behavior parameters based on flame binocular image self-correction, comprising the following steps:

[0009] S100, collecting flame binocular image data through left and right binocular cameras;

[0010] S200, extracting the flame contour line using the flame binocular image data collected in step S100, calculating the centroid points of the left and right flame contour lines and eliminating parallax information;

[0011] S300, expanding the flame contour line extracted and processed in step S200 to generate a mask, overlapping the mask with the flame image to generate a flame contour area, extracting flame response type feature points from the area, and matching between the left and right flame contour areas;

[0012] S400, constructing a two-dimensional cubic B-spline control point grid on the flame binocular image, and using the matching point pairs obtained in step S300 to initialize the deformation displacement field of the control points; establishing a similarity measurement function to quantify the difference between the left and right eye flame contour area images, and iteratively searching for the optimal deformation displacement field to minimize the difference, thereby achieving correction of the binocular image in the flame contour area;

[0013] S500, restoring the parallax information of the flame binocular image, calculating the depth information of the flame outline area, and calculating the three-dimensional coordinates of the key points of the flame outline in combination with the pixel positions of the flame outline, and calculating the fire behavior parameters through these three-dimensional coordinates.

[0014] Furthermore, in step S200, the flame image is preprocessed, the flame outline is extracted by edge extraction and image filtering algorithm, the centroid of the left and right flame outlines is calculated and the parallax information is eliminated. The specific process is as follows:

[0015] S210, using the differences between the flame color and the surrounding environment color in terms of hue h, saturation s and brightness value v to filter the ambient color; firstly, converting the color space of the flame binocular image into the HSV color space, and defining a color threshold interval H according to the color characteristics of the flame; traversing each pixel in the flame binocular image, if the color value of the pixel is not within the threshold interval, setting the pixel to zero, otherwise setting the pixel to the maximum grayscale value, so as to complete the binarization of the flame image and filter the ambient color;

[0016] S220, using a structure element of size 5×5 to perform a morphological opening operation on the binary image to remove noise and bubbles, thereby achieving preliminary filtering of the image;

[0017] S230, using the Canny algorithm to extract the contour of the binary image after preliminary filtering, and calculating the area of ​​all extracted contours; using the clustering algorithm to cluster the contour area, and setting the clustering types to 3; after the clustering algorithm is completed, find the category with the largest cluster center value, find the minimum value in the category and set it as the flame recognition threshold σ; traverse the extracted contour area, if the contour area is less than σ, then determine that the contour is a flame pore or noise and delete it; if the contour area is greater than or equal to σ, then determine that the contour is a flame and retain it, thereby completing the re-filtration of the binary image and completing the extraction of the flame contour line;

[0018] S240, obtaining the pixel coordinates of the left eye flame contour line, and saving these coordinates to the list point; the list point contains three sublists: point.x stores the contour line horizontal coordinate, point.y stores the contour line vertical coordinate, and point.count stores the number of contour points; calculating the arithmetic mean of point.x and point.y to determine the horizontal coordinate Cx and the vertical coordinate Cy of the centroid point C, thereby obtaining the centroid point coordinate C of the left eye flame contour line l , specifically through formula (1):

[0019]

[0020] Use the same method to get the centroid coordinates C of the right eye flame outline r ;

[0021] S250, calculate the vector difference C=C of the center of mass of the left and right flame contour lines l -C r , sum the coordinates of each pixel point of the flame right eye image and the flame right eye contour line with the centroid point vector difference C in turn, and obtain the flame image and flame contour line adjusted for the right eye that coincide with the centroid of the left eye, thereby eliminating the parallax information between the left eye and the right eye. The adjusted image is called the right eye de-parallax flame image and the right eye de-parallax flame contour line.

[0022] Further, in step S300, the flame contour line is expanded to generate a mask, the mask is overlapped with the flame image to generate a flame contour area, the flame response type feature points are extracted from the area, and matching is performed between the left and right flame contour areas, specifically including:

[0023] S310, expanding a total of n pixels along the extracted left-eye flame outline and the right-eye parallax-free flame outline to the inside and outside of the outline to form a flame outline mask, overlapping the mask with the left-eye flame image and the right-eye parallax-free flame image respectively, if a pixel point in the flame image is located in the mask area, marking the pixel point as a flame outline point, if not located in the mask area, marking the pixel point as a non-outline point; the area formed by the flame outline points is recorded as the flame outline area;

[0024] S320, in the flame outline area, calculating the grayscale score Inorm and the color score Col for each pixel, and using the color threshold interval H set in step S210 as the calculation standard of the color score, the grayscale score and the color score are calculated by formula (2);

[0025]

[0026] Among them, I(x,y) represents the gray value of the current pixel; d h d sd v They represent the distance between the hue h, saturation s and brightness v of the current pixel and the color threshold interval, respectively. The calculation is performed by formula (3):

[0027]

[0028] The parameters h, s, and v represent the hue, saturation, and brightness of the pixel. min 、h max 、s min 、s max 、v min 、v max represents the color threshold interval set in step S210;

[0029] S330, according to the grayscale score I norm (x, y) and the color score Col(x, y), and the final score S(x, y) of each pixel is calculated by weighted fusion. The weighted calculation is performed by formula (4):

[0030] S(x,y)=w i I norm (,y)+w c ·Col(x,y) (4)

[0031] Among them, w i With w c Represent the weighting factors of grayscale score and color score respectively, and satisfy w i +w c =1;

[0032] S340. In the flame contour area, randomly select a pixel point p as a candidate key point. Among the 16 pixel points around p, check whether there are at least 9 consecutive pixel points whose weighted scores are significantly higher or lower than the weighted score of p. If so, p is taken as the key point. Extract these key points and their descriptors in the flame contour areas of the left and right eyes, and match them to obtain flame contour matching point pairs.

[0033] Further, in step S400, a two-dimensional cubic B-spline control point grid is constructed on the flame binocular image, and the deformation displacement field of the control point is initialized using the matching point pairs obtained in step S300; a similarity measurement function is established to quantify the difference between the left and right flame contour area images, and the optimal deformation displacement field is iteratively searched to minimize the difference, thereby achieving correction of the binocular image in the flame contour area, specifically including:

[0034] S410, construct a two-dimensional cubic B-spline control point grid on the flame binocular image, set the grid spacing to δ, and record the matching point pairs obtained in step S300 as in Refers to the pixel coordinate value of the i-th feature point on the left image. Refers to the pixel coordinate value of the i-th feature point on the right image, and calculates the displacement vector difference between the matching point pairs Traverse to find The control point with the closest Euclidean distance to the feature point in the left image control point grid is recorded as Where τ is the index of the control point, X τ ,Y τ are the x-coordinate and y-coordinate of the control point respectively; the right eye control point with the same index Sum the difference d with the displacement vector to get the coordinates of the initialization control point on the right. Repeat the above operation for all matched right eye feature points to initialize the deformation displacement field of the control points of the right eye image;

[0035] S420, applying the initial deformation displacement field of the right eye image to each pixel of the right eye image to complete the initial deformation of the right eye image. The displacement vector of each pixel can be calculated by formula (5);

[0036]

[0037] Where l and m represent the basis function indexes in the x and y directions respectively; φ j,k represents the control point vector; j, k represent the index of the control point in the x direction and y direction respectively, which is related to the grid spacing δ; B l , B m They represent the lth and mth basis functions respectively. The cubic B-spline basis function is shown in formula (6):

[0038]

[0039] Where t is a parameter, and t∈[0,1], which represents the relative position from one control point to the next control point; B0, B1, B2, and B3 represent the four basis functions of the cubic B-spline curve respectively;

[0040] S430, quantifying the difference between the left and right eye flame contour area images by a similarity measurement function, and constructing a mathematical model as shown in formula (7), wherein the measurement function in the mathematical model is composed of a data fidelity term and a regularization term;

[0041]

[0042] in, represents the optimal deformation displacement field found; L and R represent the left image and the right image respectively; represents the right eye image after displacement field correction; D represents the image used to quantify L and The data fidelity term of the difference between them; represents the regularization term used to impose constraints on the displacement field; λ represents the empirical coefficient for adjusting the weight of the regularization term;

[0043] S440, using the Euclidean distance between the flame contour matching point pairs obtained in step S300 and the gray intensity difference between the left and right eye flame contour area images, a weighted operation is performed as a data fidelity item, as shown in formula (8):

[0044]

[0045] Where Ω represents the image domain of the flame contour area; I L (x p ) represents the grayscale intensity of the p-th pixel in the left image;

[0046] represents the grayscale intensity of the pth pixel in the right image after the deformation displacement field; n represents the number of matched point pairs; It represents the i-th feature point in the right image that matches the left image after the deformation displacement field. α and β both represent empirical weight factors, which are used to balance the influence of gray intensity difference and Euclidean distance on the function value.

[0047] S450. Using the constructed mathematical model (7) in combination with formula (5), the deformation displacement field and the pixel coordinates of the right eye flame contour area image are iteratively updated until the minimum value of the image difference is less than the set threshold ε. Using the iterated deformation displacement field in combination with formula (5), the coordinates of each pixel point of the right eye flame contour area after deformation are calculated, thereby achieving correction of the binocular image in the flame contour area.

[0048] Further, in step S500, the parallax information of the flame binocular image is restored, the depth information of the flame outline area is calculated, and the three-dimensional coordinates of the key points of the flame outline are calculated in combination with the pixel point positions of the flame outline, and the fire behavior parameters are calculated through these three-dimensional coordinates, specifically including:

[0049] S510, restoring the right flame image and the right flame outline by using the centroid point difference C in step S200;

[0050] S520, traverse the pixel points of the left flame outline, find the coordinate points with the smallest and largest y coordinate values, and record them as the highest point p of the flame respectively. h (x h ,y h ) and the lowest point of the flame p l (x l ,y l );

[0051] S530, calculating the depth value in the flame contour area of ​​the corrected image using formula (9);

[0052]

[0053] Where, f represents the focal length of the camera; B represents the baseline length of the binocular camera; X L With X R are the imaging coordinates of the flame outline area in the left camera and the right camera;

[0054] S540, combining the obtained flame key point with the depth value calculated by formula (9) to calculate the three-dimensional coordinates of the flame key point, the calculation formula of which is shown in formula (10):

[0055]

[0056] Where P represents the three-dimensional coordinates of the flame key point; K represents the intrinsic parameter of the camera; Z represents the depth value; [uv 1] T Represents the corresponding pixel coordinates; after calculation, the three-dimensional coordinates of the highest point of the flame are obtained P H =[X H Y H Z H ] T The three-dimensional coordinates P of the lowest point of the flame L =[X L Y L Z L ] T ;

[0057] S550, calculate the flame height F according to the known three-dimensional coordinate points H =Y H -Y L , flame length Flame Angle

[0058] A fire behavior parameter measurement system based on flame binocular image self-correction, the system has a program module corresponding to the above steps, and executes the steps in the above fire behavior parameter measurement method based on flame binocular image self-correction when running.

[0059] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of a method for measuring fire behavior parameters based on flame binocular image self-correction when called by a processor.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. Simple equipment composition, wide adaptability and low cost: Existing flame height measurement technologies mostly rely on complex and expensive professional equipment such as altimeters or laser rangefinders, which not only have high operation and maintenance requirements, but also limit the wide application of the technology. The present invention only requires two visible light cameras with the same parameters, and a binocular visual imaging system can be formed with the help of a simple bracket. The system can flexibly adjust the physical baseline between cameras according to specific application scenarios, and arrange the cameras on the ground or mount them on drones according to task requirements, which expands the scope of application of the cameras; in addition, the built-in self-correction mechanism can effectively deal with image distortion problems caused by the limitations of the camera itself or changes in the external environment, ensuring that the image quality meets the measurement requirements, enabling the equipment to cope with complex environments such as forest fires, expanding the scope of application while reducing dependence on high-cost imaging equipment; by only extracting and matching feature points in a limited flame contour area, the demand for computing resources is reduced, which enables the system to collect and process high frame rate data on the one hand to cope with rapidly changing flame environments, and on the other hand further reduces the cost of the overall system.

[0062] 2. High measurement accuracy: The present invention uses the HSV color channel to filter out the binary image of the flame contour, and uses the morphological opening operation and area clustering algorithm to perform double denoising on the image, thereby ensuring the accuracy of the extracted flame contour line; on this basis, the flame contour area is defined, and a specially designed flame responsive feature extraction algorithm is used to extract and match feature points from the flame contour area, thereby improving the accuracy of feature point extraction and reducing the probability of mismatching, thereby ensuring the accuracy of matching point pairs; by constructing a B-spline grid and combining the matched flame contour point pairs, an optimization problem based on the difference in flame contour images is constructed, and an iterative method is used to find the optimal deformation displacement field, thereby correcting the left and right eye images to reduce the calculation error of the depth value in the flame contour area, thereby improving the calculation accuracy of the three-dimensional coordinates of the flame key points and improving the measurement accuracy of the fire behavior parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a method for measuring fire behavior parameters based on flame binocular image self-correction in an embodiment of the present invention;

[0064] Figure 2 This is a physical picture of a long baseline bracket equipped with a binocular camera in an embodiment of the present invention;

[0065] Figure 3 The original flame image captured by the binocular camera in the embodiment of the present invention, wherein (a) is the captured left-eye image, and (b) is the captured right-eye image;

[0066] Figure 4 This is a flame profile diagram obtained after preprocessing in an embodiment of the present invention;

[0067] Figure 5 is a flame contour feature point diagram extracted in an embodiment of the present invention;

[0068] Figure 6 is the optimal deformation displacement field diagram obtained after iteration in the embodiment of the present invention;

[0069] Figure 7 3 is a comparison diagram of the flame image before and after correction in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In the description of the present invention, it should be noted that the terminology nouns in each embodiment, such as "left", "right" and other words indicating directions, are only for simplifying the description of the positional relationship based on the drawings in the specification, and do not mean that the referred elements and devices must be operated according to the specific directions and defined operations and methods and structures in the specification. Such directional nouns do not constitute limitations to the present invention.

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0072] Specific implementation plan 1: Combine Figures 1 to 7 As shown, the present invention provides a method for measuring fire behavior parameters based on flame binocular image self-correction, comprising the following steps:

[0073] S100, collecting binocular image data of a single flame by using left and right binocular cameras arranged on a long baseline support, such as Figure 3 As shown; the long baseline bracket can also be set on a drone to facilitate capturing forest fires or other types of fires that spread rapidly;

[0074] S200, extracting the flame contour line using the flame binocular image data collected in step S100, calculating the centroid points of the left and right flame contour lines and eliminating parallax information;

[0075] S300, expanding the flame contour line extracted and processed in step S200 to generate a mask, overlapping the mask with the flame image to generate a flame contour area, extracting flame response type feature points from the area, and matching between the left and right flame contour areas;

[0076] S400, constructing a two-dimensional cubic B-spline control point grid on the flame binocular image, and using the matching point pairs obtained in step S300 to initialize the deformation displacement field of the control points; establishing a similarity measurement function to quantify the difference between the left and right eye flame contour area images, and iteratively searching for the optimal deformation displacement field to minimize the difference, thereby achieving correction of the binocular image in the flame contour area;

[0077] S500, restoring the parallax information of the flame binocular image, calculating the depth information of the flame outline area, and calculating the three-dimensional coordinates of the key points of the flame outline in combination with the pixel positions of the flame outline, and calculating the fire behavior parameters through these three-dimensional coordinates.

[0078] Use Figure 2 The self-made long baseline left and right binocular camera shown in the figure is placed about 10 meters in front of the flame to collect Figure 3 The visible light image of the flame shown in the figure has a resolution of 640×480 and a sampling frequency of 30 Hz for monocular image acquisition. The calibration of the intrinsic and extrinsic parameters of the left and right binocular cameras has been completed in advance.

[0079] Specific implementation scheme 2: For step S200, if Figure 4 As shown, the flame image is preprocessed, the flame outline is extracted through edge extraction and image filtering algorithm, the centroid points of the left and right flame outlines are calculated and the parallax information is eliminated. The specific process is as follows:

[0080] S210, since there are large differences between the flame color and the surrounding environment color in terms of hue h, saturation s and brightness value v, the difference can be used to filter the ambient color; first, the color space of the flame binocular image is converted into the HSV color space, and a color threshold interval H is defined according to the difference between the flame color and the ambient color in the image actually sampled in this embodiment: h∈[0,35], s∈[8,150], v∈[125,255]; traverse each pixel point in the flame binocular image, if the color value of the pixel point is not within the threshold interval, then the pixel point is set to zero, otherwise the pixel point is set to the maximum grayscale value, so as to complete the binarization of the flame image and filter the ambient color;

[0081] S220, using a 5×5 structure element to perform a morphological opening operation on the binary image to remove noise and bubbles and achieve preliminary filtering of the image, wherein the morphological opening operation is an image processing technique that removes small objects and noise in the image by first corroding and then dilating, while retaining larger structural features, thereby achieving the effect of smoothing boundaries and eliminating small-scale details;

[0082] S230, using the Canny algorithm to extract the contour of the binary image after preliminary filtering, and calculating the area of ​​all extracted contours; using the clustering algorithm to cluster the contour area, and setting the clustering types to 3 types, namely flame area, large area noise and pores, and small area noise; after the clustering algorithm is completed, find the category with the largest cluster center value, find the minimum value in the category and set it as the flame recognition threshold σ; traverse the extracted contour area, if the contour area is less than σ, then determine that the contour is a flame pore or noise and delete it; if the contour area is greater than or equal to σ, then determine that the contour is a flame and retain it, thereby completing the re-filtration of the binary image and completing the extraction of the flame contour line;

[0083] S240, obtaining pixel coordinates of the left eye flame contour line, and saving these coordinates to the list point; the list point contains three sublists: point.x stores the contour line horizontal coordinate, point.y stores the contour line vertical coordinate, and point.count stores the number of contour points; calculating the arithmetic mean of point.x and point.y to determine the horizontal coordinate Cx and the vertical coordinate Cy of the centroid point C, thereby obtaining the centroid point coordinate Cl of the left eye flame contour line, specifically by formula (1):

[0084]

[0085] Use the same method to get the centroid coordinates C of the right eye flame outline r ;

[0086] S250, calculate the vector difference C=C of the center of mass of the left and right flame contour lines l -C r , sum the coordinates of each pixel point of the flame right-eye image and the flame right-eye contour line with the centroid point vector difference C in turn, and obtain the flame image and flame contour line that coincide with the left-eye centroid after right-eye adjustment, thereby eliminating the parallax information between the left and right eyes. The adjusted image is called the right-eye de-parallax flame image and the right-eye de-parallax flame contour line. Since the left-eye image is set as the main image in this method, the above-mentioned adjustment operation and subsequent correction operations are all performed on the right-eye image.

[0087] Although the flame contours are inconsistent due to a certain degree of image distortion in the left and right images of the flame, the centroids are stable, so theoretically, after eliminating the parallax information, the centroids of the left and right flame contours should be aligned. Therefore, the parallax information can be effectively eliminated by aligning the centroids.

[0088] It should be noted that the purpose of eliminating the parallax information between the left eye and the right eye is to reduce the influence of the parallax information on the correction of the flame outline area images of the left and right eyes, thereby improving the correction accuracy.

[0089] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0090] Specific implementation scheme three: For step S300, the flame contour line is expanded to generate a mask, the mask is overlapped with the flame image to generate a flame contour area, the flame response type feature points are extracted from the area, and matching is performed between the left and right flame contour areas, specifically including:

[0091] S310, expanding a total of n pixels along the extracted left-eye flame outline and the right-eye parallax-free flame outline to the inside and outside of the outline to form a flame outline mask; in this embodiment, according to the proportion of the flame in the image, 10 pixels are selected to expand to generate a suitable flame outline mask; the mask is overlapped with the left-eye flame image and the right-eye parallax-free flame image respectively, if the pixel point in the flame image is located in the mask area, the pixel point is marked as a flame outline point, if it is not located in the mask area, it is marked as a non-outline point; the area composed of the flame outline points is recorded as the flame outline area;

[0092] S320, in the flame outline area, calculating the grayscale score Inorm and the color score Col for each pixel, and using the color threshold interval H set in step S210 as the calculation standard of the color score, the grayscale score and the color score are calculated by formula (2);

[0093]

[0094] Among them, I(x,y) is the gray value of the current pixel, d h d s d v They represent the distance between the hue h, saturation s and brightness v of the current pixel and the color threshold interval, respectively. The calculation is performed by formula (3):

[0095]

[0096] Among them, the parameters h, s, and v are the hue, saturation, and brightness values ​​of the pixel. min 、h max 、s min 、s max 、v min 、vx ax is the color threshold interval set in step S210;

[0097] S330, according to the grayscale score I norm (x, y) and the color score Col(x, y), and the final score S(x, y) of each pixel is calculated by weighted fusion. The weighted calculation is performed by formula (4):

[0098] S(x,y)=w i I norm (x,y)+w c ·Col(x,y) (4)

[0099] Among them, w i With w c are the weighting factors of grayscale score and color score respectively, and satisfy w i +w c =1;

[0100] S340, such as Figure 5 As shown in the figure, in the flame contour area, any pixel point p is selected as a key point candidate point. Among the 16 pixel points around p (forming a circle with a radius of 3), it is checked whether there are at least 9 consecutive pixel points whose weighted scores are significantly higher or lower than the weighted score of p. If so, p is taken as the key point; these key points and their descriptors are extracted in the flame contour areas of the left and right eyes, and then matched to obtain flame contour matching point pairs.

[0101] There are many methods for matching feature points, such as brute force matching, nearest neighbor ratio test, fast nearest neighbor search library, etc., which can be selected according to actual needs.

[0102] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0103] Specific implementation scheme 4: For step S400, a two-dimensional cubic B-spline control point grid is constructed on the flame binocular image, and the deformation displacement field of the control points is initialized using the matching point pairs obtained in step S300; a similarity measurement function is established to quantify the difference between the left and right eye flame contour area images, and the optimal deformation displacement field is iteratively searched to minimize the difference, thereby achieving correction of the binocular image in the flame contour area, specifically including:

[0104] S410, construct a two-dimensional cubic B-spline control point grid on the flame binocular image, set the grid spacing to δ, and record the matching point pairs obtained in step S300 as in Refers to the pixel coordinate value of the i-th feature point on the left image. Refers to the pixel coordinate value of the i-th feature point on the right image, and calculates the displacement vector difference between the matching point pairs Traverse to find The control point with the closest Euclidean distance to the feature point in the left image control point grid is recorded as Where τ is the index of the control point, X τ ,Y τare the x-coordinate and y-coordinate of the control point respectively; the right eye control point with the same index Sum the difference d with the displacement vector to get the coordinates of the initialization control point on the right. Repeat the above operation for all matched right eye feature points to initialize the deformation displacement field of the control points of the right eye image;

[0105] S420, applying the initial deformation displacement field of the right eye image to each pixel of the right eye image to complete the initial deformation of the right eye image. The displacement vector of each pixel can be calculated by formula (5);

[0106]

[0107] Among them, l and m are the basis function indices in the x-direction and y-direction respectively; φ j,k is the control point vector, j and k represent the index of the control point in the x and y directions respectively, and the index is related to the grid spacing δ; B l , B m They represent the lth and mth basis functions respectively. The cubic B-spline basis function is shown in formula (6):

[0108]

[0109] Where t represents the relative position from one control point to the next control point (t∈[0,1]), B0, B1, B2, and B3 represent the four basis functions of the cubic B-spline curve respectively;

[0110] S430, quantifying the difference between the left and right eye flame contour area images by a similarity measurement function, and constructing a mathematical model as shown in formula (7), wherein the measurement function in the mathematical model is composed of a data fidelity term and a regularization term;

[0111]

[0112] in, To find the optimal deformation displacement field, L and R are the left and right images, is the right eye image after displacement field correction, and D is used to quantify L and The data fidelity term of the difference between is the regularization term used to impose constraints on the displacement field, and λ is the empirical coefficient for adjusting the weight of the regularization term;

[0113] S440, using the Euclidean distance between the flame contour matching point pairs obtained in step S300 and the gray intensity difference between the left and right eye flame contour area images, a weighted operation is performed as a data fidelity item, as shown in formula (8):

[0114]

[0115] Where Ω represents the image domain of the flame contour area, I L (x p ) represents the grayscale intensity of the pth pixel in the left image, represents the grayscale intensity of the pth pixel in the right image after the deformation displacement field; n is the number of matched point pairs, is the i-th feature point in the left image, is the i-th feature point in the right image that matches the left image after the deformation displacement field; α and β are empirical weight factors used to balance the influence of gray intensity difference and Euclidean distance on the function value;

[0116] S450, using the constructed mathematical model (7) and formula (5), the deformation displacement field and the pixel coordinates of the right eye flame contour area image are iteratively updated until the minimum value of the image difference is less than the set threshold ε. The optimal deformation displacement field obtained by iteration is as follows: Figure 6 As shown in Fig. 1, the optimal deformation displacement field after iteration is combined with formula (5) to calculate the coordinates of each pixel point in the flame contour area of ​​the right eye after image deformation, so as to realize the correction of the binocular image in the flame contour area, as shown in Fig. Figure 7 Shown are comparison pictures before and after correction.

[0117] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0118] Specific implementation scheme 5: For step S500, the parallax information of the flame binocular image is restored, the depth information of the flame outline area is calculated, and the three-dimensional coordinates of the key points of the flame outline are calculated in combination with the pixel point positions of the flame outline, and the fire behavior parameters are calculated through these three-dimensional coordinates, which specifically includes:

[0119] S510, restoring the right flame image and the right flame outline by using the centroid point difference C in step S200;

[0120] S520, traverse the pixel points of the left flame outline, find the coordinate points with the smallest and largest y coordinate values, and record them as the highest point p of the flame respectively. h (x h ,y h ) and the lowest point of the flame p l (x l ,y l ), and is called the flame key point. Since the starting point is usually located in the upper left corner in pixel coordinates, when the y coordinate value is the smallest, the point is generally located at the highest point of the flame;

[0121] In this embodiment, the coordinates of the highest point of the flame are obtained as p h(475,58), the coordinate of the lowest point of the flame is p l (402,386);

[0122] S530, calculating the depth value in the flame contour area of ​​the corrected image using formula (9);

[0123]

[0124] Wherein, f is the focal length of the camera (2.45 mm in this embodiment), B is the baseline length of the binocular camera (120 mm in this embodiment), X L With X R are the imaging coordinates of the flame outline area in the left camera and the right camera;

[0125] S540, combining the obtained flame key point with the depth value calculated by formula (9) to calculate the three-dimensional coordinates of the flame key point, the calculation formula of which is shown in formula (10):

[0126]

[0127] Among them, P is the three-dimensional coordinate, K is the intrinsic parameter of the camera, Z is the depth value, [uv 1] T is the corresponding pixel coordinate; after calculation, the three-dimensional coordinate P of the highest point of the flame is obtained H =[X H Y H Z H ] T The three-dimensional coordinates P of the lowest point of the flame L =[X L Y L Z L ] T ;

[0128] In this embodiment, the calculated three-dimensional coordinates of the flame key points are shown in Table 1;

[0129] Table 1 Three-dimensional coordinates of flame key points (unit: meter)

[0130]

[0131] S550, calculate the flame height F according to the known three-dimensional coordinate points H =Y H -Y L , flame length Flame Angle

[0132] In this embodiment, the calculated fire behavior parameters are shown in Table 2;

[0133] Table 2 Fire behavior parameters

[0134]

[0135] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0136] Specific implementation scheme six: The present invention provides a fire behavior parameter measurement system based on flame binocular image self-correction. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned fire behavior parameter measurement method based on flame binocular image self-correction during operation.

[0137] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0138] Specific implementation scheme seven: The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a fire behavior parameter measurement method based on flame binocular image self-correction when called by a processor.

[0139] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0140] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for measuring fire behavior parameters based on flame binocular image self-correction, characterized in that: The following steps are involved: S100, collecting flame binocular image data through left and right binocular cameras; S200, extracting the flame contour line using the flame binocular image data collected in step S100, calculating the centroid points of the left and right flame contour lines and eliminating parallax information; S300, expanding the flame contour line extracted and processed in step S200 to generate a mask, overlapping the mask with the flame image to generate a flame contour area, extracting flame response type feature points from the area, and matching between the left and right flame contour areas; S400, constructing a two-dimensional cubic B-spline control point grid on the flame binocular image, and using the matching point pairs obtained in step S300 to initialize the deformation displacement field of the control points; A similarity measurement function is established to quantify the difference between the left and right eye flame contour area images. The optimal deformation displacement field is iteratively found to minimize the difference, thereby achieving correction of the binocular image in the flame contour area. S500, restoring the parallax information of the flame binocular image, calculating the depth information of the flame outline area, and calculating the three-dimensional coordinates of the key points of the flame outline in combination with the pixel positions of the flame outline, and calculating the fire behavior parameters through these three-dimensional coordinates.

2. The method for measuring fire behavior parameters based on flame binocular image self-correction according to claim 1, characterized in that: In step S200, the flame image is preprocessed, the flame outline is extracted through edge extraction and image filtering algorithm, the centroid points of the left and right flame outlines are calculated and the parallax information is eliminated. The specific process is as follows: S210, using the differences between the flame color and the surrounding environment color in terms of hue h, saturation s and brightness value v to filter the ambient color; firstly, converting the color space of the flame binocular image into the HSV color space, and defining a color threshold interval H according to the color characteristics of the flame; traversing each pixel in the flame binocular image, if the color value of the pixel is not within the threshold interval, setting the pixel to zero, otherwise setting the pixel to the maximum grayscale value, so as to complete the binarization of the flame image and filter the ambient color; S220, using a structure element of size 5×5 to perform a morphological opening operation on the binary image to remove noise and bubbles, thereby achieving preliminary filtering of the image; S230, using the Canny algorithm to extract the contour of the binary image after preliminary filtering, and calculating the area of ​​all extracted contours; using the clustering algorithm to cluster the contour area, and setting the clustering types to 3; after the clustering algorithm is completed, find the category with the largest cluster center value, find the minimum value in the category and set it as the flame recognition threshold σ; traverse the extracted contour area, if the contour area is less than σ, then determine that the contour is a flame pore or noise and delete it; if the contour area is greater than or equal to σ, then determine that the contour is a flame and retain it, thereby completing the re-filtration of the binary image and completing the extraction of the flame contour line; S240, obtaining the pixel coordinates of the left flame outline, and saving these coordinates to the list point; The list point contains three sublists: point.x stores the horizontal coordinate of the contour line, point.y stores the vertical coordinate of the contour line, and point.count stores the number of contour points; Calculate the arithmetic mean of point.x and point.y to determine the horizontal coordinate Cx and vertical coordinate Cy of the centroid point C, thereby obtaining the centroid point coordinate C of the left eye flame outline l , specifically through formula (1): Use the same method to get the centroid coordinates C of the right eye flame outline r ; S250, calculate the vector difference C=C of the center of mass of the left and right flame contour lines l -C r , sum the coordinates of each pixel point of the flame right eye image and the flame right eye contour line with the centroid point vector difference c in turn, and obtain the flame image and flame contour line adjusted for the right eye that coincide with the centroid of the left eye, thereby eliminating the parallax information between the left eye and the right eye. The adjusted image is called the right eye de-parallax flame image and the right eye de-parallax flame contour line.

3. The method for measuring fire behavior parameters based on flame binocular image self-correction according to claim 2, characterized in that: In step S300, the flame contour line is expanded to generate a mask, the mask is overlapped with the flame image to generate a flame contour area, flame response type feature points are extracted from the area, and matching is performed between the left and right flame contour areas, specifically including: S310, expanding a total of n pixels along the extracted left-eye flame outline and the right-eye parallax-free flame outline to the inside and outside of the outline to form a flame outline mask, overlapping the mask with the left-eye flame image and the right-eye parallax-free flame image respectively, if a pixel point in the flame image is located in the mask area, marking the pixel point as a flame outline point, if not located in the mask area, marking the pixel point as a non-outline point; the area formed by the flame outline points is recorded as the flame outline area; S320, in the flame outline area, calculating the grayscale score Inorm and the color score Col for each pixel, and using the color threshold interval H set in step S210 as the calculation standard of the color score, the grayscale score and the color score are calculated by formula (2); Where I(x,y) represents the gray value of the current pixel; d h ,d s ,d v They represent the distance between the hue h, saturation s and brightness v of the current pixel and the color threshold interval, respectively. The calculation is performed by formula (3): The parameters h, s, and v represent the hue, saturation, and brightness of the pixel. min 、h max 、s min 、s max 、v min 、v max represents the color threshold interval set in step S210; S330, according to the grayscale score I norm (x, y) and the color score Col(x, y), and the final score S(x, y) of each pixel is calculated by weighted fusion. The weighted calculation is performed by formula (4): S(x,y)=w i ·I norm (x,y)+w c ·Col(x,y) (4) Among them, w i With w c Represent the weighting factors of grayscale score and color score respectively, and satisfy w i +w c =1; S340. In the flame contour area, randomly select a pixel point p as a candidate key point. Among the 16 pixel points around p, check whether there are at least 9 consecutive pixel points whose weighted scores are significantly higher or lower than the weighted score of p. If so, p is taken as the key point. Extract these key points and their descriptors in the flame contour areas of the left and right eyes, and match them to obtain flame contour matching point pairs.

4. The method for measuring fire behavior parameters based on flame binocular image self-correction according to claim 3 is characterized in that: In step S400, a two-dimensional cubic B-spline control point grid is constructed on the flame binocular image, and the deformation displacement field of the control points is initialized using the matching point pairs obtained in step S300; A similarity measurement function is established to quantify the difference between the left and right eye flame contour area images. The optimal deformation displacement field is iteratively found to minimize the difference, thereby achieving the correction of the binocular image in the flame contour area. Specifically, it includes: S410, construct a two-dimensional cubic B-spline control point grid on the flame binocular image, set the grid spacing to δ, and record the matching point pairs obtained in step S300 as in Refers to the pixel coordinate value of the i-th feature point on the left image. Refers to the pixel coordinate value of the i-th feature point on the right image, and calculates the displacement vector difference between the matching point pairs Traverse to find The control point with the closest Euclidean distance to the feature point in the left image control point grid is recorded as Where τ is the index of the control point, X τ ,Y τ are the x-coordinate and y-coordinate of the control point respectively; the right eye control point with the same index Sum the difference d with the displacement vector to get the coordinates of the initialization control point on the right. Repeat the above operation for all matched right eye feature points to initialize the deformation displacement field of the control points of the right eye image; S420, applying the initial deformation displacement field of the right eye image to each pixel of the right eye image to complete the initial deformation of the right eye image. The displacement vector of each pixel can be calculated by formula (5); Where l and m represent the basis function indexes in the x and y directions respectively; φ j,k represents the control point vector; j, k represent the index of the control point in the x direction and y direction respectively, which is related to the grid spacing δ; B l , B m They represent the lth and mth basis functions respectively. The cubic B-spline basis function is shown in formula (6): Where t is a parameter, and t∈[0,1], which represents the relative position from one control point to the next control point; B0, B1, B2, and B3 represent the four basis functions of the cubic B-spline curve respectively; S430, quantifying the difference between the left and right eye flame contour area images by a similarity measurement function, and constructing a mathematical model as shown in formula (7), wherein the measurement function in the mathematical model is composed of a data fidelity term and a regularization term; in, represents the optimal deformation displacement field found; L and R represent the left image and the right image respectively; represents the right eye image after displacement field correction; D represents the image used to quantify L and The data fidelity term of the difference between them; represents the regularization term used to impose constraints on the displacement field; λ represents the empirical coefficient for adjusting the weight of the regularization term; S440, using the Euclidean distance between the flame contour matching point pairs obtained in step S300 and the gray intensity difference between the left and right eye flame contour area images, a weighted operation is performed as a data fidelity item, as shown in formula (8): Where Ω represents the image domain of the flame contour area; I L (x p ) represents the grayscale intensity of the p-th pixel in the left image; represents the grayscale intensity of the pth pixel in the right image after the deformation displacement field; n represents the number of matched point pairs; It represents the i-th feature point in the right image that matches the left image after the deformation displacement field. α and β both represent empirical weight factors, which are used to balance the influence of gray intensity difference and Euclidean distance on the function value. S450. Using the constructed mathematical model (7) in combination with formula (5), the deformation displacement field and the pixel coordinates of the right eye flame contour area image are iteratively updated until the minimum value of the image difference is less than the set threshold ε. Using the iterated deformation displacement field in combination with formula (5), the coordinates of each pixel point of the right eye flame contour area after deformation are calculated, thereby achieving correction of the binocular image in the flame contour area.

5. The method for measuring fire behavior parameters based on flame binocular image self-correction according to claim 4, characterized in that: In step S500, the parallax information of the flame binocular image is restored, the depth information of the flame outline area is calculated, and the three-dimensional coordinates of the key points of the flame outline are calculated in combination with the pixel positions of the flame outline. The fire behavior parameters are calculated through these three-dimensional coordinates, which specifically include: S510, restoring the right flame image and the right flame outline by using the centroid point difference C in step S200; S520, traverse the pixel points of the left flame outline, find the coordinate points with the smallest and largest y coordinate values, and record them as the highest point p of the flame respectively. h (x h ,y h ) and the lowest point of the flame p l (x l ,y l ); S530, calculating the depth value in the flame contour area of ​​the corrected image using formula (9); Where, f represents the focal length of the camera; B represents the baseline length of the binocular camera; X L With X R are the imaging coordinates of the flame outline area in the left camera and the right camera; S540, combining the obtained flame key point with the depth value calculated by formula (9) to calculate the three-dimensional coordinates of the flame key point, the calculation formula of which is shown in formula (10): Where P represents the three-dimensional coordinates of the flame key point; K represents the intrinsic parameter of the camera; Z represents the depth value; [uv 1] T Represents the corresponding pixel coordinates; after calculation, the three-dimensional coordinates of the highest point of the flame are obtained P H =[X H Y H Z H ] T The three-dimensional coordinates P of the lowest point of the flame L =[X L Y L Z L ] T ; S550, calculate the flame height F according to the known three-dimensional coordinate points H =Y H -Y L , flame length Flame Angle 6. A fire behavior parameter measurement system based on flame binocular image self-correction, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 5, and executes the steps of the fire behavior parameter measurement method based on flame binocular image self-correction when running.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the fire behavior parameter measurement method based on flame binocular image self-correction according to any one of claims 1 to 5 when called by a processor.

Images