A method for locating the source of tunnel fire based on binocular vision

By installing a binocular camera in the tunnel and building a convolutional neural network model, the problems of poor flexibility of fixed position sensors and high image processing time in the tunnel fire monitoring system are solved, and accurate three-dimensional positioning and real-time positioning of the fire source are achieved. .

CN118037827BActive Publication Date: 2024-11-22CHINA UNIV OF MINING & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410334910.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-22
Publication Date
2024-11-22
Estimated Expiration
2044-03-22

AI Technical Summary

Technical Problem

The fixed position sensor in the existing tunnel fire monitoring system has poor flexibility and cannot calculate the three-dimensional spatial information in the fire source image in real time. The fire source positioning method based on image processing is costly and cannot meet the real-time requirements.

Method used

Using the tunnel fire source positioning method based on binocular vision, by installing a binocular camera, image correction and normalization processing are carried out, convolutional neural network model is constructed, and the fire source feature recognition model is trained using the model distillation method, combined with the binocular camera. The parameters map the characteristic points of the fire source to a three-dimensional coordinate system to achieve accurate positioning of the fire source.

Benefits of technology

This method can more accurately estimate the three-dimensional position of an object, provide more accurate spatial positioning information, improve the expression ability and accuracy of the model, and is suitable for scenarios with high real-time requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118037827B_ABST
    Figure CN118037827B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for locating the source of a tunnel fire based on binocular vision, and the technical field to which it belongs is the field of fire source locating, comprising: installing a binocular camera inside the tunnel, and photographing after visually aligning the binocular camera based on the tunnel environment to obtain tunnel image data; performing image correction and normalization processing on the tunnel image data to obtain a processed data set; constructing a convolutional neural network model to obtain fire source feature points; and mapping the fire source feature points to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points. The binocular vision system of the present invention can obtain the depth information of objects in the scene through parallax calculation of two cameras. The three-dimensional position of an object can be estimated more accurately, and more accurate spatial positioning information can be provided for fire source locating.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of fire source positioning, and in particular relates to a method for positioning a tunnel fire source based on binocular vision. Background Art

[0002] With the continuous and rapid advancement of infrastructure construction across the country, the number and scale of tunnels have grown rapidly, greatly promoting economic and social development. However, there are many types of combustibles on vehicles and the load is large. Once a fire occurs, it spreads rapidly and the temperature in the tunnel rises rapidly. Due to the small space and lack of oxygen in the tunnel, the degree of incomplete combustion of combustibles is high, which easily produces a large amount of toxic and harmful smoke, posing a huge threat to the life safety of the occupants. Rapidly extinguishing the fire can effectively ensure the safe evacuation and rescue of tunnel personnel.

[0003] However, most fire monitoring systems in tunnels are fixed-position sensors, and most of them are smoke and temperature sensors connected to the fire protection system as detection units. They need to be installed during construction, have poor flexibility in fixed-position deployment, and cannot calculate the three-dimensional spatial information in the fire source image; in addition, the fire source positioning method based on image processing has a high time cost and cannot meet the real-time requirements in actual application scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a method for locating the fire source of a tunnel fire based on binocular vision to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, the present invention provides a method for locating the fire source of a tunnel fire based on binocular vision, comprising:

[0006] A binocular camera is installed inside the tunnel, and the binocular camera is visually aligned based on the tunnel environment to take pictures to obtain tunnel image data;

[0007] Performing image correction and normalization processing on the tunnel image data to obtain a processed data set;

[0008] Constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method to obtain a fire source feature recognition model, inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature points;

[0009] The fire source feature points are mapped to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points.

[0010] Preferably, the process of obtaining tunnel image data includes:

[0011] Select two zoom cameras of the same size and determine the relative positions of the two cameras based on the tunnel environment;

[0012] Setting the position and orientation of the zoom camera based on the relative position;

[0013] The variable-focus camera is turned on to photograph the tunnel to obtain the tunnel image data.

[0014] Preferably, the process of obtaining the processed data set includes:

[0015] Calculate a basic matrix between the binocular cameras based on the internal parameters and external parameters of the binocular cameras;

[0016] Aligning the epipolar lines of the feature points of the tunnel image data based on the basic matrix to obtain aligned image data;

[0017] Remapping the aligned image data based on a correction transformation matrix to obtain a correction data set;

[0018] The calibration data set is normalized to obtain the processed data set.

[0019] Preferably, the process of constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method, and obtaining a fire source feature recognition model includes:

[0020] Build convolutional neural network models and long short-term memory networks;

[0021] Obtaining a fire source sample set, forward-propagating the long short-term memory network through the fire source sample set, and obtaining an output of the long short-term memory network;

[0022] The output of the long short-term memory network is used as the input of the convolutional neural network model for training to obtain a student network model;

[0023] Constructing an attention mechanism module and a residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network;

[0024] The improved convolutional neural network model is trained by using the fire source sample set to obtain the fire source feature recognition model.

[0025] Preferably, the process of training the convolutional neural network model by using the output of the long short-term memory network as the input of the convolutional neural network model to obtain the student network model includes:

[0026] Constructing data hard labels based on the fire source sample set, and using the output of the long short-term memory network as a soft label;

[0027] The data hard labels and the soft labels are input into the convolutional neural network model for training, and the accuracy of the convolutional neural network model is optimized based on a multi-scale supervision method to obtain the student network model.

[0028] Preferably, the process of constructing an attention mechanism module and a residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network includes:

[0029] Constructing the attention mechanism module and the residual module;

[0030] Introducing the attention mechanism module into the first three layers of the hidden layer of the student network model to obtain an attention classification model;

[0031] The residual module is introduced into the attention classification model to generate the improved convolutional neural network.

[0032] Preferably, the process of inputting the processed data set into the fire source feature recognition model for calculation to obtain the fire source feature points includes:

[0033] Inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature pixel points;

[0034] The fire source characteristic pixel points are spatially transformed to obtain the fire source characteristic points.

[0035] Preferably, the process of mapping the fire source feature points to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points includes:

[0036] Calculate the corresponding relationship of the fire source feature points based on the SIFT algorithm to obtain the matching relationship of the feature points;

[0037] The matching relationship between the parameters of the binocular camera and the feature points is calculated based on a triangulation method to obtain the fire source coordinate point.

[0038] The technical effects of the present invention are:

[0039] The binocular vision system of the present invention can obtain the depth information of objects in the scene through the parallax calculation of the two cameras. The three-dimensional position of the object can be estimated more accurately, and for the location of the fire source, more accurate spatial positioning information can be provided. The use of convolutional neural networks for fire source location can better model complex nonlinear relationships. Neural networks can learn complex mappings from binocular vision input to the location of the fire source, improving the expressive power and accuracy of the model. At the same time, through the model distillation method, the long short-term memory network is used as the teacher network and the convolutional neural network is used as the student network, which can improve the ability of the convolutional neural network to handle the timing problems of the binocular camera, and perform well in scenes with high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 This is a flow chart of a method for locating a tunnel fire source based on binocular vision in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of using the output of a long-short time neural network and corresponding target labels to train a convolutional network model in an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of a convolutional network in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Embodiment 1

[0047] like Figure 1 As shown, this embodiment provides a method for locating a fire source in a tunnel based on binocular vision, comprising:

[0048] A binocular camera is installed inside the tunnel, and the binocular camera is visually aligned based on the tunnel environment to take pictures to obtain tunnel image data;

[0049] Performing image correction and normalization processing on the tunnel image data to obtain a processed data set;

[0050] Constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method to obtain a fire source feature recognition model, inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature points;

[0051] The fire source feature points are mapped to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points.

[0052] Further optimizing the scheme, the process of obtaining tunnel image data includes:

[0053] Two zoom cameras of the same size are selected, and the relative positions of the two cameras are determined based on the tunnel environment; the distance between the two cameras is called the baseline.

[0054] Setting the position and orientation of the zoom camera based on the relative position; acquiring the relative position and posture relationship between the two cameras through spatial reconstruction or other techniques;

[0055] The variable-focus camera is turned on to photograph the tunnel to obtain the tunnel image data.

[0056] Further optimizing the scheme, the process of obtaining the processed data set includes:

[0057] Calculate a basic matrix between the binocular cameras based on the internal parameters and external parameters of the binocular cameras;

[0058] Intrinsic parameter calibration: Each camera has its own intrinsic parameters, such as focal length, principal point offset, etc. Use camera calibration techniques (such as Zhang's calibration method or checkerboard calibration) to determine the intrinsic parameters of each camera. This step ensures that accurate camera parameters are used during image processing.

[0059] External parameter calibration: External parameters refer to the position and orientation of the camera. The relative position and attitude relationship between two cameras is obtained through spatial reconstruction or other techniques. Usually, it is necessary to use known 3D points or structures in the scene to calculate the external parameters of the camera.

[0060] Aligning the epipolar lines of the feature points of the tunnel image data based on the basic matrix to obtain aligned image data;

[0061] In stereo vision, given two views (e.g., images taken by left and right cameras), the feature points in one image can be projected onto the other image through the fundamental matrix, and the lines connecting the corresponding feature points in the two images are all on the same straight line, which is called the epipolar line.

[0062] The relationship between the basic matrix (F) and the feature points (x) and (x') can be expressed as follows:

[0063] [x'^TFx=0]

[0064] Where (x) and (x') are the homogeneous coordinates of the feature points on the two images respectively. This equation expresses the epipolar geometric relationship between the corresponding point (x') on the other image and the original point (x).

[0065] Substituting (x) into the above formula, we can get the polar equation corresponding to the specific feature point (x):

[0066] [l'=Fx]

[0067] Among them, (l') is the polar line corresponding to (x').

[0068] Remapping the aligned image data based on a correction transformation matrix to obtain a correction data set;

[0069] In this embodiment, the correction transformation matrix is ​​a camera intrinsic parameter matrix, which is usually expressed as (K), and includes the internal parameters of the camera, such as focal length, principal point coordinates, etc. The standard form of the camera intrinsic parameter matrix is ​​as follows:

[0070] [K=\begin{bmatrix}f_x&0&c_x\0&f_y&c_y\0&0&1\end{bmatrix}]

[0071] Where (f_x) and (f_y) are the focal lengths of the camera in the horizontal and vertical directions, and (c_x) and (c_y) are the coordinates of the principal point.

[0072] The calibration data set is normalized to obtain the processed data set.

[0073] The formula for min-max normalization is:

[0074] [X_{\text{norm}}=\frac{X-X_{\text{min}}}{X_{\text{max}}-X_{\text{min}}}]

[0075] in:

[0076] (X_{\text{norm}}) is the normalized data.

[0077] (X) is the original data.

[0078] (X_{\text{min}}) is the minimum value of the original data.

[0079] (X_{\text{max}}) is the maximum value of the original data.

[0080] Further optimizing the scheme, the process of constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method, and obtaining a fire source feature recognition model includes:

[0081] Build convolutional neural network models and long short-term memory networks;

[0082] Obtaining a fire source sample set, forward-propagating the long short-term memory network through the fire source sample set, and obtaining an output of the long short-term memory network;

[0083] The output of the long short-term memory network is used as the input of the convolutional neural network model for training to obtain a student network model;

[0084] Constructing an attention mechanism module and a residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network;

[0085] The improved convolutional neural network model is trained by using the fire source sample set to obtain the fire source feature recognition model.

[0086] In this embodiment, the process of constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method, and obtaining a fire source feature recognition model is model distillation, and the specific process is:

[0087] First, the LSTM network is used as the teacher network for training. LSTM is a recurrent neural network suitable for sequence data. It has the ability to capture long-term dependencies and is suitable for processing time series data or tasks with time correlation.

[0088] A convolutional neural network (CNN) is designed as the student network. CNN is mainly used for image processing and can effectively capture spatial features in images. The student network is usually designed to be lighter than the teacher network and more suitable for running in resource-constrained environments.

[0089] The teacher network is used to make predictions on the training data, and these predicted soft labels (probability distributions) are used as targets instead of the original hard labels. Such soft labels contain the relative confidence information of the teacher network for each category.

[0090] The student network is trained using soft labels. The goal of the student network is to replicate the behavior of the teacher network as much as possible, that is, to be as consistent as possible with the teacher network in terms of prediction. At this point, the loss function of the original hard label and the soft label is usually combined.

[0091] The student network is evaluated to see how well it performs on the test set. Due to the distillation process, the student network is usually able to achieve similar performance to the teacher network with a relatively small model capacity.

[0092] Further optimization scheme, such as Figure 2-3 As shown, the process of training the convolutional neural network model by using the output of the long short-term memory network as the input of the convolutional neural network model to obtain the student network model includes:

[0093] Constructing data hard labels based on the fire source sample set, and using the output of the long short-term memory network as a soft label;

[0094] The data hard labels and the soft labels are input into the convolutional neural network model for training, and the accuracy of the convolutional neural network model is optimized based on a multi-scale supervision method to obtain the student network model.

[0095] The loss function is:

[0096]

[0097] Ldice=1-dice

[0098] Where dice is the DICE coefficient, X represents the first image data of the binocular camera, Y represents the second image data of the binocular camera, and L dice is the loss function.

[0099] Further optimizing the scheme, the process of constructing the attention mechanism module and the residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network includes:

[0100] Constructing the attention mechanism module and the residual module;

[0101] Introducing the attention mechanism module into the first three layers of the hidden layer of the student network model to obtain an attention classification model;

[0102] The residual module is introduced into the attention classification model to generate the improved convolutional neural network.

[0103] Further optimizing the scheme, the processed data set is input into the fire source feature recognition model for calculation, and the process of obtaining the fire source feature points includes:

[0104] Inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature pixel points;

[0105] The fire source characteristic pixel points are spatially transformed to obtain the fire source characteristic points.

[0106] Further optimizing the scheme, mapping the fire source feature points to a three-dimensional coordinate system based on the parameters of the binocular camera, and obtaining the fire source coordinate points includes:

[0107] The correspondence of the fire source feature points is calculated based on the SIFT algorithm to obtain the matching relationship of the feature points; after the fire source feature points are identified in the images of different viewing angles, the correspondence of these feature points in different images is found through the feature point matching algorithm. This can use some computer vision algorithms, such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), etc., and SIFT (Scale Invariant Feature Transform) is used in this embodiment.

[0108] The matching relationship between the parameters of the binocular camera and the feature points is calculated based on the triangulation method to obtain the coordinate point of the fire source. The three-dimensional coordinates of the target point can be calculated using the triangulation method through the known matching relationship between the camera parameters and the feature points. Common methods include direct linear transformation (DirectLinearTransform, DLT) or methods based on nonlinear optimization, such as optimizing the coordinates of three-dimensional points by minimizing the reprojection error. The calculated three-dimensional coordinates are usually relative to the camera coordinate system. If it needs to be mapped to the global world coordinate system, the position and orientation of the camera need to be considered. This can be done by coordinate transformation through camera extrinsics.

[0109] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for locating the source of tunnel fire based on binocular vision. It is characterized in that The following steps are involved: A binocular camera is installed inside the tunnel, and the binocular camera is visually aligned based on the tunnel environment to take pictures to obtain tunnel image data; Performing image correction and normalization processing on the tunnel image data to obtain a processed data set; Constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method to obtain a fire source feature recognition model, inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature points; Mapping the fire source feature points to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points; The process of constructing a convolutional neural network model, improving and training the convolutional neural network model based on a model distillation method, and obtaining a fire source feature recognition model includes: Build convolutional neural network models and long short-term memory networks; Obtaining a fire source sample set, forward-propagating the long short-term memory network through the fire source sample set, and obtaining an output of the long short-term memory network; The output of the long short-term memory network is used as the input of the convolutional neural network model for training to obtain a student network model; Constructing an attention mechanism module and a residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network; The improved convolutional neural network model is trained by using the fire source sample set to obtain the fire source feature recognition model; The process of training the convolutional neural network model by using the output of the long short-term memory network as the input of the convolutional neural network model to obtain the student network model includes: Constructing data hard labels based on the fire source sample set, and using the output of the long short-term memory network as a soft label; Inputting the data hard labels and the soft labels into the convolutional neural network model for training, and optimizing the accuracy of the convolutional neural network model based on a multi-scale supervision method to obtain the student network model; The process of constructing an attention mechanism module and a residual module, and introducing the attention mechanism module and the residual module into the student network model to generate an improved convolutional neural network includes: Constructing the attention mechanism module and the residual module; Introducing the attention mechanism module into the first three layers of the hidden layer of the student network model to obtain an attention classification model; Introducing the residual module into the attention classification model to generate the improved convolutional neural network; The process of inputting the processed data set into the fire source feature recognition model for calculation to obtain the fire source feature points includes: Inputting the processed data set into the fire source feature recognition model for calculation to obtain fire source feature pixel points; Performing spatial transformation on the fire source characteristic pixel points to obtain the fire source characteristic points; The process of mapping the fire source feature points to a three-dimensional coordinate system based on the parameters of the binocular camera to obtain the fire source coordinate points includes: Calculate the corresponding relationship of the fire source feature points based on the SIFT algorithm to obtain the matching relationship of the feature points; The matching relationship between the parameters of the binocular camera and the feature points is calculated based on a triangulation method to obtain the fire source coordinate point.

2. The method for locating the source of a tunnel fire based on binocular vision according to claim 1, It is characterized in that The process of obtaining tunnel image data comprises: Select two zoom cameras of the same size to form a binocular camera, and determine the relative positions of the binocular cameras based on the tunnel environment; Setting the position and orientation of the binocular camera based on the relative position; The binocular camera is turned on to photograph the tunnel to obtain the tunnel image data.

3. The method for locating the source of a tunnel fire based on binocular vision according to claim 1, It is characterized in that The process of obtaining the processed data set includes: Calculate a basic matrix between the binocular cameras based on the internal parameters and external parameters of the binocular cameras; Aligning the epipolar lines of the feature points of the tunnel image data based on the basic matrix to obtain aligned image data; Remapping the aligned image data based on a correction transformation matrix to obtain a correction data set; The calibration data set is normalized to obtain the processed data set.