Fire source identification method, system and equipment based on deep learning and medium

Abstract

The invention relates to a fire source identification method, system and device based on deep learning and a medium. The method comprises the steps of obtaining a current frame image and a historical frame image sequence of a flame scene, extracting multi-scale spatial features based on a weight sharing deep convolutional neural network, and performing time dimension fusion to obtain a fused spatial-temporal feature map; inputting the fused spatial-temporal characteristic graph into a segmentation network based on an encoder-decoder to obtain a flame region segmentation probability graph; constructing a cross-view space-time matching cost body according to the fused space-time feature map, and calculating soft maximization probability distribution along the depth dimension to obtain depth probability distribution; a flame three-dimensional space probability body is obtained through calculation based on the flame region segmentation probability graph and the depth probability distribution, the flame three-dimensional space probability body and the historical estimation state are input into a time sequence inference model based on a recurrent neural network, and three-dimensional coordinates and estimation covariance of a fire source base point are obtained. By adopting the method, accurate and stable three-dimensional positioning of the fire source base point can be realized.

Description

Fire source identification methods, systems, devices, and media based on deep learning Technical Field

[0001] This invention belongs to the field of fire identification technology, and in particular relates to a fire source identification method, system, device and medium based on deep learning. Background Technology

[0002] With the development of fire extinguishing technology, automatic fire extinguishing devices based on computer vision have emerged, which can accurately locate the root of the fire source to carry out fire suppression.

[0003] In traditional technologies, the root of a fire source is mainly based on 3D reconstruction technology using feature points. This relies on stable and repeatable feature point matching between different views or consecutive frames. Depth is calculated through triangulation or motion parallax to complete the 3D localization of the fire source base point. Alternatively, a deep learning fire source recognition model can be used to first output a 2D bounding box or segmentation mask of the image plane, and then combine it with the depth map of LiDAR and static background binocular vision to query the 2D centroid depth.

[0004] However, the aforementioned methods fundamentally hinder accurate 3D localization due to the highly non-rigid and rapidly changing physical phenomena of flames, as well as their irregular flickering and swaying shapes and brightness. For feature-point 3D reconstruction techniques, the constantly changing surface texture of flames and the easy annihilation or deformation of feature points from the previous frame lead to frequent feature matching failures and mismatches, causing drastic jumps or even failures in depth calculations, making it impossible to provide stable 3D coordinates. Deep learning models can only output 2D projection information, and simply combining this with depth map queries amplifies the 2D centroid fluctuations caused by flame jitter into 3D coordinate noise, making it impossible to stably associate the fire source base point with the instantaneous visual appearance. Summary of the Invention

[0005] Therefore, it is necessary to provide a deep learning-based fire source identification method, system, device, and medium that can treat the flame area as a whole dynamic process and use time series information to infer the three-dimensional position of the relatively stable fire source base point enclosed by it.

[0006] Firstly, this application provides a fire source identification method based on deep learning, comprising:

[0007] Obtain the current frame image and historical frame image sequence of the fire scene; the current frame image includes the left-view current frame image and the right-view current frame image;

[0008] Based on a weight-shared deep convolutional neural network, multi-scale spatial features of each frame in the current frame image and the historical frame image sequence are extracted to obtain the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence. The current frame multi-scale spatial features and the historical multi-scale spatial feature sequence are then fused in the time dimension to obtain the fused spatiotemporal feature map of the current frame image.

[0009] The fused spatiotemporal feature map is input into an encoder-decoder-based segmentation network to obtain a flame region segmentation probability map and a segmentation uncertainty map; the segmentation probability map represents the likelihood that each pixel in the current frame belongs to the flame.

[0010] Based on the preset discrete depth assumption range, a cross-view spatiotemporal matching cost body is constructed according to the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map. After regularizing the cross-view spatiotemporal matching cost body through a pre-trained three-dimensional convolutional neural network, the soft maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution.

[0011] The three-dimensional spatial probability volume of the flame is calculated based on the flame region segmentation probability map and depth probability distribution. The three-dimensional spatial probability volume of the flame and the historical estimated state are then input into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance.

[0012] In one embodiment, based on a weight-shared deep convolutional neural network, multi-scale spatial features of each frame in the current frame image and the historical frame image sequence are extracted respectively to obtain the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence. The current frame multi-scale spatial features and the historical multi-scale spatial feature sequence are then fused in the time dimension to obtain the fused spatiotemporal feature map of the current frame image, including:

[0013] Spatial features of the current frame image of the left eye and the current frame image of the right eye at multiple scales are extracted respectively to obtain the spatial feature map of the current frame of the left eye and the spatial feature map of the current frame of the right eye.

[0014] Extract left-eye historical spatial feature map sequences and right-eye historical spatial feature map sequences at the same scale based on historical frame image sequences;

[0015] The spatial feature map of the current frame of the left eye and the sequence of spatial feature maps of the left eye are concatenated in the time dimension to obtain the spatiotemporal feature block of the left eye;

[0016] The right eye's current frame spatial feature map and the right eye's historical spatial feature map sequence are concatenated in the time dimension to obtain the right eye's spatiotemporal feature block;

[0017] Spatiotemporal convolution is performed on the left-eye and right-eye spatiotemporal feature blocks to obtain a fused spatiotemporal feature map. The fused spatiotemporal feature map includes the left-eye fused spatiotemporal feature map and the right-eye fused spatiotemporal feature map. Spatiotemporal convolution corresponds to performing convolution operations in the time dimension to aggregate temporal context information and nonlinear activation function processing.

[0018] In one embodiment, based on a preset discrete depth assumption range, a cross-view spatiotemporal matching cost body is constructed according to the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, including:

[0019] Based on the left-eye fusion spatiotemporal feature map, extract the left-eye feature vector of any pixel in the current frame image of the left eye;

[0020] Based on the epipolar geometry of the binocular system and the preset depth assumption, the pixel points corresponding to the feature vector of the left eye are projected onto the current frame image of the right eye to obtain the right eye projection coordinates corresponding to the pixel points.

[0021] Extract the right eye feature vector at the right eye projection coordinates of the right eye fusion spatiotemporal feature map;

[0022] Calculate the cosine similarity between the left and right feature vectors to obtain the basic matching metric, and obtain the left-eye segmentation uncertainty value at the corresponding pixel of the left-eye feature vector and the right-eye segmentation uncertainty value at the right-eye projection coordinates.

[0023] Based on the adaptive weighted matching cost function, and combining the basic matching metric, the left-eye segmentation uncertainty value, and the right-eye segmentation uncertainty value, the matching cost of a pixel under the depth assumption is calculated; the expression of the adaptive weighted matching cost function is: ,in, For the left eye feature vector, The right eye feature vector, For left eye segmentation uncertainty value, For the uncertainty value of right eye segmentation, To balance the weighting coefficients, These are the pixel coordinates. The coordinates of the right eye projection. This is the assumed value for depth;

[0024] By iterating through all pixels and depth assumptions of the current frame image of the left eye, and arranging each matching cost in the spatial and depth dimensions, a cross-view spatiotemporal matching cost volume is obtained.

[0025] In one embodiment, after regularizing the cross-view spatiotemporal matching cost volume using a pre-trained 3D convolutional neural network, a soft-maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution, including:

[0026] By stacking three-dimensional convolutional layers and three-dimensional pooling layers in a three-dimensional convolutional neural network, the cross-view spatiotemporal matching cost volume is filtered sequentially in the spatial and depth dimensions, and contextual information is aggregated to suppress noise and isolated high-cost points caused by incorrect matching, thus obtaining a regularized cost volume.

[0027] Based on the depth dimension, the softmax function is applied to each pixel of the regularized cost volume to calculate the probability distribution under different depth assumptions, thus obtaining the depth probability distribution; the expression for the softmax function is: ,in, Represents pixels The depth is the The probability of a depth hypothesis value The regularization cost volume at the pixel And the depth is the The value of the depth hypothesis. This represents the total number of depth hypotheses.

[0028] In one embodiment, a three-dimensional spatial probability volume of the flame is calculated based on the flame region segmentation probability map and the depth probability distribution, including:

[0029] For any pixel in the current frame image of the left eye, the pixel is back-projected into a three-dimensional spatial point in the camera coordinate system according to the physical depth value corresponding to the left eye camera intrinsic parameter matrix and the depth assumption value.

[0030] The unnormalized joint probability density of the three-dimensional spatial points is obtained by multiplying the flame probability value of the flame region segmentation probability map at the pixel point with the probability value of the depth probability distribution at the same pixel point with the depth assumption.

[0031] Traverse all pixels in the current frame image of the left eye, and fill the three-dimensional space grid with the unnormalized joint probability density of the three-dimensional space points corresponding to each pixel to obtain the three-dimensional space probability volume of the flame.

[0032] In one embodiment, the three-dimensional spatial probability volume of the flame and the historical estimated state are input into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance, including:

[0033] The three-dimensional spatial probability volume of the flame is compressed into a fixed-dimensional feature vector by using a three-dimensional convolutional network.

[0034] Obtain the hidden state of the previous time step and the three-dimensional coordinates of the predicted fire source base point of the previous time step in the memory layer;

[0035] The feature vector is concatenated with the three-dimensional coordinates of the predicted fire source base point at the previous time step, and then input into the gated loop unit along with the hidden state at the previous time step to obtain the hidden state at the current time step.

[0036] The hidden state at the current moment is input into the multilayer perceptron to predict the displacement increment and covariance matrix of the fire source base point at the current moment relative to the previous moment.

[0037] The displacement increment is added to the predicted three-dimensional coordinates of the fire source base point at the previous moment to obtain the three-dimensional coordinates of the fire source base point at the current moment. The estimated covariance of the corresponding three-dimensional coordinates of the fire source base point is obtained by combining the covariance matrix.

[0038] In one embodiment, the unnormalized joint probability density of the three-dimensional spatial points is obtained by multiplying the flame probability value of the flame region segmentation probability map at a pixel with the probability value of the depth probability distribution having a depth assumption at the same pixel, including:

[0039] Based on the segmentation uncertainty map, obtain the left-eye segmentation uncertainty value corresponding to the current pixel;

[0040] Based on the uncertainty value of left eye segmentation, the confidence weight of each pixel is calculated using a confidence weight based on an exponential function; the expression for the confidence weight is as follows: ,in, The preset positive scaling factor. For left eye segmentation uncertainty value;

[0041] Based on the confidence weight, the flame probability value of a pixel is multiplied by the probability value of the same pixel having a depth hypothesis value to obtain the unnormalized joint probability density.

[0042] Secondly, this application also provides a fire source identification system based on deep learning, comprising:

[0043] The image acquisition module is used to acquire the current frame image and the sequence of historical frame images of the flame scene; the current frame image includes the left-view current frame image and the right-view current frame image;

[0044] The feature extraction module is used to extract multi-scale spatial features of each frame in the current frame image and the historical frame image sequence based on a weight-shared deep convolutional neural network, to obtain the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence, and to fuse the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence in the time dimension to obtain the fused spatiotemporal feature map of the current frame image.

[0045] The flame determination module is used to input the fused spatiotemporal feature map into the encoder-decoder-based segmentation network to obtain the flame region segmentation probability map and the segmentation uncertainty map; the segmentation probability map represents the likelihood that each pixel in the current frame image belongs to the flame.

[0046] The spatial module is used to construct a cross-view spatiotemporal matching cost body based on the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, according to a preset discrete depth assumption range. After regularizing the cross-view spatiotemporal matching cost body through a pre-trained three-dimensional convolutional neural network, the soft-maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution.

[0047] The flame location module is used to calculate the three-dimensional spatial probability volume of the flame based on the flame region segmentation probability map and the depth probability distribution. The three-dimensional spatial probability volume of the flame and the historical estimated state are then input into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance.

[0048] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-described deep learning-based fire source identification methods.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above-described deep learning-based fire source identification methods.

[0050] The aforementioned deep learning-based fire source identification method, system, device, and medium acquire current and historical image sequences from binocular vision, extract multi-scale spatial features using a weight-sharing deep convolutional neural network, and fuse them temporally to generate a fused spatiotemporal feature map containing dynamic flame information. An encoder-decoder segmentation network simultaneously outputs the flame region probability and its uncertainty quantization map. Combining binocular fusion features and uncertainty information, a cross-view spatiotemporal matching cost volume is constructed and regularized within a preset depth range, resulting in a more robust depth probability distribution. The depth distribution representing the spatial existence probability of the flame is combined with the segmentation probability to form a three-dimensional spatial probability volume, which is then input into a temporal inference model. A recurrent neural network is used to learn the flame dynamic prior and filter instantaneous flicker noise, thereby outputting a spatiotemporally consistent, stable, and reliable three-dimensional coordinates of the fire source base point and its covariance estimate. This effectively solves the problem of feature matching failure and depth estimation jumps caused by drastic changes in the non-rigid shape of the flame, achieving accurate and stable three-dimensional positioning of the fire source base point. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 is a flowchart illustrating the deep learning-based fire source identification method of the present invention;

[0053] Figure 2 is a structural diagram of the fire source identification system based on deep learning according to the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In one embodiment, as shown in Figure 1, a deep learning-based fire source identification method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] S101. Obtain the current frame image and historical frame image sequence of the flame scene; the current frame image includes the left-view current frame image and the right-view current frame image.

[0057] In illustrative terms, image acquisition is implemented using a binocular vision acquisition system. This system satisfies the requirement of spatiotemporal synchronization, ensuring that the left and right cameras capture image data of the flame scene simultaneously. The historical frame image sequence is a collection of continuously acquired images preceding the current frame, maintained through a temporal buffer queue. The length of the buffer queue is determined based on the frequency of dynamic changes in the flame and the configuration of computing resources. Preprocessing operations are performed on the acquired image data, including image distortion correction based on camera intrinsic parameter calibration results and pixel value normalization, ensuring that the image data meets the input requirements of subsequent deep convolutional neural networks and eliminating the image quality impact caused by differences in hardware devices.

[0058] S102. Based on a weight-shared deep convolutional neural network, extract the multi-scale spatial features of each frame in the current frame image and the historical frame image sequence to obtain the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence. Then, fuse the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence in the time dimension to obtain the fused spatiotemporal feature map of the current frame image.

[0059] Furthermore, the weight-shared deep convolutional neural network employs a backbone network architecture with strong feature extraction capabilities. ResNet-50 or ConvNeXt can be selected as the core backbone. Its weight-sharing characteristic ensures that the feature extraction process for the left and right eye images follows consistent mapping rules, avoiding feature heterogeneity between the left and right eyes due to differences in network parameters. Multi-scale spatial feature extraction is achieved through multi-level convolution and pooling operations in the backbone network. The outputs of different layers correspond to feature maps of different resolutions. High-resolution feature maps retain detailed information about the flame region, while low-resolution feature maps contain global semantic information about the flame, forming a set covering features at different scales. Historical multi-scale spatial feature sequences are obtained by performing the same feature extraction process on each historical frame image in the cache queue, ensuring that they have the same dimension and feature representation as the current frame's multi-scale spatial features. Furthermore, temporal fusion associates and aggregates the multi-scale spatial features of the current frame with the historical multi-scale spatial feature sequences in the temporal dimension. It performs temporal convolution operations through lightweight 3D convolutional layers or spatiotemporal separation convolution (2DConv+1DTemporalConv) to learn the rules of flame feature changes over time, suppress feature fluctuations caused by instantaneous flame flickering, and generate a fused spatiotemporal feature map that contains both the visual information of the current frame and the motion trend information of the historical frames.

[0060] S103. Input the fused spatiotemporal feature map into the encoder-decoder-based segmentation network to obtain the flame region segmentation probability map and the segmentation uncertainty map; the segmentation probability map represents the likelihood that each pixel in the current frame image belongs to the flame.

[0061] Optionally, the encoder-decoder-based segmentation network adopts a symmetrical U-Net structure. The encoder part reuses the feature extraction module of the aforementioned weight-sharing deep convolutional neural network, and enhances the semantic expressive power of the features through progressive downsampling operations. The decoder part gradually restores the spatial resolution of the feature map through upsampling operations, and fuses the feature maps of different levels of the encoder with the feature maps of the corresponding levels of the decoder through a skip connection mechanism, which ensures both the semantic accuracy of the segmentation results and improves the boundary localization accuracy of the flame region. Furthermore, the output layer of the segmentation network adopts a dual-branch structure. The first branch converts the feature map into a segmentation probability map through the sigmoid activation function. The value of each pixel in this map ranges from [0,1], and the larger the value, the higher the probability that the pixel belongs to the flame region. The second branch generates a segmentation uncertainty map through an uncertainty estimation module composed of Monte Carlo Dropout or learning prediction mechanisms. This map has the same spatial resolution as the segmentation probability map, and its pixel value quantifies the credibility of the segmentation result at the corresponding position. Regions with blurred features or drastic changes in the flame boundary during the segmentation process have higher uncertainty values, while regions with stable combustion and clear contours have lower uncertainty values. The segmentation network is trained under the supervision of segmentation loss, which employs weighted binary cross-entropy (BCE) loss or Dice loss to minimize the difference between the predicted segmentation probability map and the real flame mask. This loss, as an important component of the total loss function, participates in the end-to-end joint optimization of the network.

[0062] S104. Based on the preset discrete depth assumption range, construct a cross-view spatiotemporal matching cost body according to the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map. After regularizing the cross-view spatiotemporal matching cost body through a pre-trained three-dimensional convolutional neural network, calculate the soft-maximization probability distribution along the depth dimension to obtain the depth probability distribution.

[0063] In a schematic manner, the preset discrete depth assumption range is determined according to the actual application requirements of the fire extinguishing scenario, covering the possible fire source depth range. This continuous range is uniformly discretized into several depth assumption values, forming a depth assumption set. The construction of the cross-view spatiotemporal matching cost body is based on the epipolar geometric constraints of the binocular camera. For each spatial position in the left-eye fusion spatiotemporal feature map, its corresponding epipolar position in the right-eye fusion spatiotemporal feature map is calculated according to each depth assumption value. Then, the feature vectors of the corresponding positions in the left and right eyes are extracted. The initial matching cost is obtained by calculating the matching degree of the feature vectors. The initial matching costs of all spatial positions under different depth assumptions are arranged according to the spatial dimension and the depth dimension to form the initial cross-view spatiotemporal matching cost body. The pre-trained 3D convolutional neural network consists of multiple stacked 3D convolutional layers and 3D pooling layers. The 3D convolutional layers use 3×3×3 kernels, and the 3D pooling layers use 2×2×2 kernels. The network filters the initial cost volume in both the spatial and depth dimensions, aggregates contextual information, suppresses noise and isolated high-cost points caused by incorrect matches, and outputs a regularized cost volume. The parameters of this 3D CNN are optimized under supervised depth estimation loss. The depth probability distribution is obtained by soft maximization along the depth dimension of the regularized cost volume, which can comprehensively represent the matching confidence of each pixel under different depth assumptions and is more adaptable to the dynamic characteristics of flames than single depth value regression. The depth estimation loss uses negative log-likelihood loss (NLL) to encourage the network to maximize the probability value at the true depth location. The loss for non-flame regions is removed by masking, and this loss participates in the weighted summation of the total loss function.

[0064] S105. Based on the probability map of flame region segmentation and the probability distribution of depth, the three-dimensional spatial probability volume of the flame is calculated, and the three-dimensional spatial probability volume of the flame and the historical estimated state are input into the time-series inference model based on recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance.

[0065] Specifically, the calculation of the flame 3D spatial probability volume is based on the camera imaging model. Each pixel in the left-eye fusion spatiotemporal feature map is back-projected into a 3D spatial point in the camera coordinate system according to the physical depth corresponding to the camera intrinsic parameter matrix and each depth hypothesis value. The flame likelihood of the pixel in the segmentation probability map is multiplied by the probability of the depth hypothesis value corresponding to the pixel in the depth probability distribution to obtain the unnormalized joint probability density of the 3D spatial point. By traversing all pixels and depth hypothesis values, the unnormalized joint probability density of each 3D spatial point is arranged according to the 3D spatial grid to form the flame 3D spatial probability volume. This probability volume quantitatively represents the joint probability of each location in the 3D space belonging to the fire source region. The time-series inference model based on recurrent neural networks includes a feature compression module, a gated recurrent unit (GRU) or a long short-term memory network (LSTM), and a multilayer perceptron (MLP) output prediction module. The feature compression module compresses the three-dimensional spatial probability volume of the flame into a fixed-dimensional feature vector through a lightweight three-dimensional convolutional network, which is convenient for subsequent recurrent unit processing. The historical estimated state includes the three-dimensional coordinates of the fire source base point output by the time-series inference model at the previous time step and the hidden state of the gated recurrent unit, where the hidden state encodes the flame movement trend information of all previous time steps. The gated recurrent unit receives the compressed current frame feature vector, the three-dimensional coordinates of the fire source base point at the previous time step, and the hidden state, and adaptively updates the hidden state through a gating mechanism, fusing current observation information and historical state information. The output prediction module adopts an MLP structure consisting of at least two fully connected layers, mapping the updated hidden state to the displacement increment and covariance matrix of the fire source base point at the current time step relative to the previous time step. The three-dimensional coordinates of the fire source base point at the current time step are obtained by superimposing the coordinates and displacement increment at the previous time step step, and the covariance matrix quantifies the uncertainty of the current base point coordinate estimation result, providing an accuracy reference for the control decision of downstream fire extinguishing devices.

[0066] Optionally, an end-to-end training strategy is adopted. The training data includes binocular video sequences and the 3D ground truth coordinates of the fire source base point for each frame. The ground truth data is obtained by using a high-precision motion capture system under controlled conditions to track burning material markers or by using synthetic data generated from reconstructing static combustion scenes using precision 3D scanning equipment. The total loss function is a multi-part weighted sum, i.e. ,in, This is the segmentation loss, used to monitor the segmentation accuracy of the flame region; To estimate the loss at depth, supervise the accuracy of the depth probability distribution; The regression loss is used to directly monitor the prediction accuracy of the three-dimensional coordinates of the fire source base point. , To predict coordinates, For true coordinates, The length of the training sequence; To compensate for temporal smoothness loss, L2 loss is used to penalize drastic changes in the coordinates of the base points between adjacent frames. Forced output trajectory smoothing; To compensate for uncertainty in the calibration loss, negative log-likelihood loss is used. , To predict the covariance matrix and ensure that the covariance matrix truly reflects the estimation error; , , , and The optimal values ​​for the balanced weights of each loss term are determined through experimental debugging to achieve collaborative optimization of each training objective.

[0067] In the aforementioned deep learning-based fire source identification method, the current frame image and historical frame image sequence of the flame scene are acquired by binoculars. Multi-scale spatial features of each frame are extracted using a weight-shared deep convolutional neural network. The current frame multi-scale spatial features and historical multi-scale spatial feature sequences are fused temporally to obtain a fused spatiotemporal feature map. This fused spatiotemporal feature map is input into an encoder-decoder-based segmentation network to obtain a flame region segmentation probability map and a segmentation uncertainty map representing the likelihood of pixel flames. Based on a preset discrete depth assumption range, a cross-view spatiotemporal matching cost body is constructed by combining the left and right eye fused spatiotemporal feature maps. After regularization by a pre-trained 3D convolutional neural network, the cost body is then applied along the depth... The depth probability distribution is obtained by soft maximization of the probability distribution in the degree dimension. Based on the flame region segmentation probability map and the depth probability distribution, the three-dimensional spatial probability volume of the flame is calculated. This volume, along with the historical estimated state, is input into a time-series inference model based on a recurrent neural network. The output is the three-dimensional coordinates of the fire source base point and the estimated covariance. By deeply fusing time-series information and multi-view geometric information, the dependence on stable feature points is eliminated. Stable three-dimensional positioning of non-rigid flames is achieved through an end-to-end process. This effectively resists the interference of instantaneous flame flickering and morphological changes, improves the accuracy, stability, and robustness of fire source base point positioning, and provides reliable positioning results information, providing reliable target instructions for relevant fire extinguishing devices.

[0068] In one embodiment, based on a weight-shared deep convolutional neural network, multi-scale spatial features of each frame in the current frame image and the historical frame image sequence are extracted respectively to obtain the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence. The current frame multi-scale spatial features and the historical multi-scale spatial feature sequence are then fused in the time dimension to obtain the fused spatiotemporal feature map of the current frame image, including:

[0069] S11. Extract the spatial features of the current frame image of the left eye and the current frame image of the right eye at multiple scales respectively to obtain the spatial feature map of the current frame of the left eye and the spatial feature map of the current frame of the right eye.

[0070] Indicatively, the spatial feature extraction processes of the left and right current frame images are completely synchronized and follow a consistent network computation logic, both implemented through a backbone network of a weight-sharing deep convolutional neural network. The backbone network is constructed by alternating stacks of multiple convolutional and pooling layers. Each convolutional layer uses a fixed-size kernel for feature extraction, with stride controlling the downsampling ratio of the feature map resolution. Pooling layers are used to retain key feature information and further compress the feature map dimension. Spatial features at multiple scales correspond to the outputs of different layers of the backbone network. From shallow to deep layers, the spatial resolution of the feature maps gradually decreases, while the number of channels gradually increases, forming a multi-scale feature hierarchy. The spatial feature maps of the left and right current frames have the same spatial resolution, number of channels, and feature dimension at each corresponding scale, ensuring compatibility with subsequent temporal concatenation and spatiotemporal convolution processing. The element values ​​of each feature map quantify the semantic and textural features of the corresponding spatial location.

[0071] S12. Extract the left-eye historical spatial feature map sequence and the right-eye historical spatial feature map sequence at the same scale based on the historical frame image sequence.

[0072] Furthermore, the historical frame image sequence is read from the temporal buffer queue. During the extraction process, for the left-eye and right-eye historical frame images, a backbone network and parameter configuration completely consistent with the current frame's spatial feature extraction are used to extract spatial feature maps of each historical frame at multiple scales. The construction of the left-eye and right-eye historical spatial feature map sequences must strictly follow the scale matching principle. That is, for each feature scale of the current frame, the left-eye and right-eye feature maps of all historical frames at that scale are collected to form a historical feature map sequence that corresponds one-to-one with the feature scale of the current frame. The spatial resolution and number of channels of the feature maps in each historical spatial feature map sequence are consistent with the spatial feature map of the corresponding scale of the current frame, ensuring the data dimension uniformity when stitching together the temporal dimension.

[0073] S13. The spatial feature map of the current frame of the left eye and the sequence of spatial feature maps of the history of the left eye are concatenated in the time dimension to obtain the spatiotemporal feature block of the left eye.

[0074] Specifically, the stitching operation is based on temporal sequence. The left-eye historical spatial feature map sequence is arranged in order of acquisition time from earliest to latest. The spatial feature map of the current left-eye frame is used as the last element of the temporal sequence, and is dimensionally expanded and stitched with the historical feature map sequence in the time dimension. The resulting left-eye spatiotemporal feature block is a four-dimensional data structure, whose dimensions include the number of channels, spatial height, spatial width, and time step, where the time step equals the sum of the number of historical frames and the number of current frames. During the stitching process, it is necessary to ensure that all feature maps involved in the stitching are completely consistent in the number of channels, spatial height, and spatial width dimensions to avoid data misalignment caused by dimensional mismatch. The left-eye spatiotemporal feature block completely preserves the spatial feature change trajectory of the flame scene captured by the left-eye camera over a continuous time series.

[0075] S14. The right eye current frame spatial feature map and the right eye historical spatial feature map sequence are concatenated in the time dimension to obtain the right eye spatiotemporal feature block.

[0076] Similarly, the construction process of the right-eye spatiotemporal feature block is completely consistent with that of the left-eye spatiotemporal feature block. It also uses temporal sequence as the stitching criterion, sorting the right-eye historical spatial feature map sequence by acquisition time and then stitching it with the right-eye current frame spatial feature map in the temporal dimension. The stitched right-eye spatiotemporal feature block has the same four-dimensional data structure as the left-eye spatiotemporal feature block, with consistent channel count, spatial height, spatial width, and time step, ensuring the isomorphism of the left and right-eye spatiotemporal feature blocks. This stitching process strictly follows the same dimensional verification rules as the left-eye process, ensuring the integrity and data consistency of the right-eye temporal feature sequence, enabling the right-eye spatiotemporal feature block to accurately reflect the spatial feature evolution of the flame scene captured by the right-eye camera in the continuous temporal dimension.

[0077] S15. Perform spatiotemporal convolution on the left-eye spatiotemporal feature block and the right-eye spatiotemporal feature block to obtain a fused spatiotemporal feature map; the fused spatiotemporal feature map includes the left-eye fused spatiotemporal feature map and the right-eye fused spatiotemporal feature map; spatiotemporal convolution corresponds to performing convolution operations in the time dimension to aggregate temporal context information and nonlinear activation function processing.

[0078] Optionally, the spatiotemporal convolution processes for the left-eye and right-eye spatiotemporal feature blocks are independent, each processed by a structurally identical spatiotemporal convolution module to ensure the symmetry of left-eye feature fusion. The spatiotemporal convolution module consists of temporal and spatial convolutional layers. The temporal convolutional layer uses a one-dimensional kernel, the length of which is designed based on the time-cycle characteristics of the flame's dynamic changes. By sliding convolution along the time step dimension, it aggregates feature information from adjacent time frames, capturing the dependencies of flame features over time and suppressing feature abrupt changes caused by instantaneous flicker. The spatial convolutional layer uses a two-dimensional kernel, performing convolution operations in the spatial dimension of the feature map to further enhance the semantic relevance and local consistency of spatial features. During spatiotemporal convolution, the order of temporal and spatial convolution operations can be set according to feature representation requirements. After convolution, feature mapping transformation is performed through a non-linear activation function to enhance the model's ability to fit complex flame dynamic features. The selection of the non-linear activation function must meet the requirements of gradient propagation stability and feature representation effectiveness. After spatiotemporal convolution processing, the spatiotemporal feature blocks of the left eye are transformed into a fused spatiotemporal feature map of the left eye, and the spatiotemporal feature blocks of the right eye are transformed into a fused spatiotemporal feature map of the right eye. Both types of feature maps retain the original spatial resolution and number of channels, while integrating temporal context information, thus possessing a stronger ability to represent dynamic flame features.

[0079] In one embodiment, based on a preset discrete depth assumption range, a cross-view spatiotemporal matching cost body is constructed according to the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, including:

[0080] S21. Based on the left-eye fusion spatiotemporal feature map, extract the left-eye feature vector of any pixel in the current frame image of the left eye.

[0081] Indicatively, the left-eye fusion spatiotemporal feature map is a multi-channel feature map processed by spatiotemporal convolution. Each spatial location corresponds to a pixel in the current frame image of the left eye, and each spatial location contains feature information from multiple channels. The channel information integrates the spatial semantic features and temporal evolution features of that pixel. Specifically, the extraction process of the left-eye feature vector involves locating the spatial coordinates of any specified pixel in the current frame image of the left eye in the left-eye fusion spatiotemporal feature map, extracting the feature values ​​of all channels at that coordinate location, and arranging them in a preset channel order to form a one-dimensional vector, which is the left-eye feature vector of that pixel. The dimension of the left-eye feature vector is consistent with the number of channels in the left-eye fusion spatiotemporal feature map, and its vector element values ​​quantify the feature expression of that pixel in the fusion spatiotemporal feature space.

[0082] S22. Based on the epipolar geometry of the binocular system and the preset depth assumption, the pixel corresponding to the feature vector of the left eye is projected onto the current frame image of the right eye to obtain the right eye projection coordinates corresponding to the pixel.

[0083] Optionally, the epipolar geometry of the binocular system is determined based on the binocular camera calibration results. The calibration process has already obtained the intrinsic and extrinsic parameter matrices of the left and right cameras, providing a geometric constraint basis for projection calculation. Specifically, based on the preset depth assumption, combined with the coordinates of the left-eye pixel and the camera's intrinsic parameter matrix, the pixel is converted from the image coordinate system to 3D point coordinates in the camera coordinate system. Through the rotation and translation relationships in the extrinsic parameter matrix, the 3D point coordinates in the camera coordinate system are converted to 3D point coordinates in the right-eye camera coordinate system. Based on the right-eye camera's intrinsic parameter matrix, the 3D point coordinates in the right-eye camera coordinate system are projected onto the image coordinate system of the current frame image of the right eye, obtaining the right-eye projected coordinates corresponding to that pixel. The projection process strictly follows the epipolar constraint, ensuring that the right-eye projected coordinates always fall on the corresponding epipolar line, significantly reducing the search range for subsequent feature matching and improving matching efficiency and accuracy.

[0084] S23. Extract the right eye feature vector at the right eye projection coordinates of the right eye fusion spatiotemporal feature map.

[0085] Furthermore, the right-eye fusion spatiotemporal feature map has the same number of channels and spatial resolution as the left-eye fusion spatiotemporal feature map, and the semantic meaning of its feature channels is consistent with that of the left eye, ensuring the isomorphism and comparability of the left and right eye feature vectors. The extraction process of the right-eye feature vector is completely consistent with that of the left-eye feature vector. That is, based on the obtained right-eye projection coordinates, its corresponding spatial position in the right-eye fusion spatiotemporal feature map is located, and the feature values ​​of all channels at that position are extracted and arranged in the same channel order as the left-eye feature vector to form a one-dimensional right-eye feature vector. The dimension of the right-eye feature vector is the same as that of the left-eye feature vector, and its vector element values ​​also integrate the spatial semantic features and temporal evolution features of the corresponding pixel, providing a matching basis for similarity calculation with the left-eye feature vector.

[0086] S24. Calculate the cosine similarity between the left and right feature vectors to obtain the basic matching metric, and obtain the left segmentation uncertainty value at the corresponding pixel of the left feature vector and the right segmentation uncertainty value at the right projection coordinate.

[0087] For example, cosine similarity is used to quantify the similarity between the left and right eye feature vectors. It measures the consistency of vector directions by calculating the cosine of the angle between the two vectors; the smaller the angle, the larger the cosine similarity value, indicating a higher degree of matching between the two feature vectors. Specifically, the left and right eye feature vectors are normalized to eliminate the influence of vector magnitude differences on similarity calculation. The cosine similarity result is then obtained through vector dot product operation. This result is the basic matching metric, with a value range of [-1, 1]. The closer the value is to 1, the better the matching between the left and right eye feature vectors. The segmentation uncertainty value is obtained based on the segmentation uncertainty map output by the aforementioned segmentation network. The left eye segmentation uncertainty value is the pixel value at the corresponding pixel position of the left eye feature vector in the left eye segmentation uncertainty map; the right eye segmentation uncertainty value is the pixel value at the right eye projection coordinate position in the right eye segmentation uncertainty map. Both types of uncertainty values ​​quantify the reliability of the segmentation results at the corresponding positions.

[0088] S25. Based on the adaptive weighted matching cost function, and combining the basic matching metric, the left-eye segmentation uncertainty value, and the right-eye segmentation uncertainty value, the matching cost of each pixel under the depth assumption is calculated; the expression of the adaptive weighted matching cost function is: ,in, For the left eye feature vector, The right eye feature vector, For left eye segmentation uncertainty value, For the uncertainty value of right eye segmentation, To balance the weighting coefficients, These are the pixel coordinates. The coordinates of the right eye projection. This is the depth assumption value.

[0089] in, This represents the inner product operation between the left and right feature vectors; The L2 norm of the left eye feature vector; The L2 norm of the right eye feature vector is represented; It represents the cosine similarity between the feature vectors of the left and right eyes, i.e., the basic matching metric; This represents the balancing weight coefficient, used to adjust the contribution ratio of the basic matching metric and the uncertainty penalty in the total cost; Coordinates in the left eye segmentation uncertainty graph Uncertainty value of left eye segmentation at the location; Coordinates in the right eye segmentation uncertainty graph The right eye segmentation uncertainty value at the location; This represents the pixel coordinates from which the left-eye feature vector is extracted from the current frame image of the left-eye view; This indicates the right-eye projection coordinates of the pixel after it has been projected onto the current frame image of the right eye; This represents the depth hypothesis value used in the current calculation. The function converts the matching degree into a basic cost by subtracting the cosine similarity from 1, and then applies a cost penalty to matches with low segmentation confidence by superimposing a weighted sum of the left and right segmentation uncertainty values. Finally, it obtains a comprehensive matching cost that reflects the feature matching quality and segmentation confidence.

[0090] S26. Traverse all pixels and all depth assumptions of the current frame image of the left eye, arrange each matching cost in the spatial and depth dimensions, and obtain the cross-view spatiotemporal matching cost body.

[0091] Specifically, the traversal process sequentially processes each pixel in the current frame image of the left-eye view in a preset order. For each pixel, all depth assumption values ​​within a preset discrete depth assumption range are applied to obtain the matching cost for each pixel under each depth assumption value. The spatial dimension corresponds to the pixel coordinate distribution of the current frame image of the left-eye view, including both height and width dimensions. The depth dimension corresponds to the preset set of discrete depth assumption values. All matching costs are arranged in the order of pixel height, pixel width, and depth assumption values ​​to construct a three-dimensional data structure for cross-view spatiotemporal matching cost volume. Each element value of this cost volume corresponds to the matching cost of a specific pixel under a specific depth assumption, fully representing the matching relationship of the spatiotemporal features of left and right eye fusion under different spatial locations and different depth assumptions.

[0092] In one embodiment, after regularizing the cross-view spatiotemporal matching cost volume using a pre-trained 3D convolutional neural network, a soft-maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution, including:

[0093] S31. By stacking three-dimensional convolutional layers and three-dimensional pooling layers in a three-dimensional convolutional neural network, the cross-view spatiotemporal matching cost body is filtered sequentially in the spatial and depth dimensions, and contextual information is aggregated to suppress noise and isolated high cost points caused by incorrect matching, so as to obtain a regularized cost body.

[0094] Indicatively, the core structure of a 3D convolutional neural network consists of several stacked alternating 3D convolutional layers and 3D pooling layers. The network input is a cross-view spatiotemporal matching cost volume, which is a 3D data structure containing spatial height, spatial width, and the number of depth hypotheses. The 3D convolutional layers use 3D convolutional kernels, the size of which can simultaneously capture the matching cost correlation between adjacent pixels in the spatial dimension and the cost change trend between adjacent depth hypothesis values ​​in the depth dimension. Specifically, the 3D convolutional kernels slide along the spatial and depth dimensions of the cross-view spatiotemporal matching cost volume, weightedly aggregating cost information in local regions. Through weight sharing, feature extraction and filtering of the global cost volume are achieved, effectively smoothing isolated high-cost points caused by incorrect matching and instantaneous flame flickering, and reducing noise interference. The 3D pooling layers use max pooling or average pooling strategies to compress the dimensions of the convolutional cost volume while preserving core cost features, improving network computational efficiency and further strengthening the global consistency of cost distribution. Through iterative processing of stacked 3D convolutional layers and 3D pooling layers, the noise component in the cross-view spatiotemporal matching cost volume is significantly suppressed, and the spatial continuity and depth correlation of the cost distribution are enhanced. Finally, a regularized cost volume with the same dimension as the original cost volume is output, which more accurately reflects the true relationship of left and right eye feature matching.

[0095] S32. Based on the depth dimension, apply the softmax function to each pixel of the regularized cost volume to calculate the probability distribution under different depth assumptions, thus obtaining the depth probability distribution; the expression for the softmax function is: ,in, Represents pixels The depth is the The probability of a depth hypothesis value The regularization cost volume at the pixel And the depth is the The value of the depth hypothesis. This represents the total number of depth hypotheses.

[0096] For example, This represents the natural exponential function, used to convert the regularization matching cost into a non-negative value, while amplifying the cost difference between different depth hypothesis values; An index variable representing the depth hypothesis value, used to iterate through all depth hypothesis values; This represents the total number of depth hypothesis values ​​within the preset discrete depth hypothesis range. The denominator is the sum of the exponential costs corresponding to all depth hypothesis values, used to normalize the numerator to ensure that the sum of the probabilities corresponding to all depth hypothesis values ​​is 1, satisfying the basic properties of probability distributions. After applying the softmax function, each pixel forms a probability distribution along the depth dimension. The depth hypothesis value with the highest probability value is the most likely depth estimate for that pixel. The dispersion of the probability distribution reflects the uncertainty of the depth estimate; the more concentrated the distribution, the higher the reliability of the depth estimate, and vice versa.

[0097] In one embodiment, a three-dimensional spatial probability volume of the flame is calculated based on the flame region segmentation probability map and the depth probability distribution, including:

[0098] S41. For any pixel in the current frame image of the left eye, according to the physical depth value corresponding to the intrinsic parameter matrix of the left eye camera and the depth assumption value, the pixel is back-projected into a three-dimensional spatial point in the camera coordinate system.

[0099] Indicatively, the left-eye camera intrinsic parameter matrix is ​​the core parameter matrix obtained during the binocular system calibration process. It includes key geometric parameters such as the camera's focal length (horizontal and vertical) and principal point coordinates (pixel coordinates corresponding to the camera's optical center in the image coordinate system). The physical depth value corresponding to the depth assumption value is the actual three-dimensional spatial distance corresponding to each depth assumption value within a preset discrete depth assumption range, and its unit is consistent with the length unit used in camera calibration. Specifically, the image coordinate system coordinates of the target pixel in the current frame image of the left eye are determined. This coordinate system has the upper left corner of the image as the origin, the horizontal axis as the u-axis, and the vertical axis as the v-axis. Based on the camera intrinsic parameter matrix, the image coordinate system coordinates are converted into normalized planar coordinates in the camera coordinate system to eliminate the distortion effects caused by the camera's optical system. The normalized planar coordinates are multiplied by the corresponding physical depth value. Combining this with the definition of the camera coordinate system (i.e., with the camera's optical center as the origin and the optical axis as the positive z-axis), the three-dimensional spatial coordinates of the pixel in the camera coordinate system are obtained, completing the mapping from a two-dimensional image pixel to a three-dimensional spatial point.

[0100] S42. Multiply the flame probability value at a pixel in the flame region segmentation probability map with the probability value of the depth probability distribution at the same pixel that has a depth assumption value to obtain the unnormalized joint probability density of the three-dimensional spatial points.

[0101] The flame probability value is the pixel value corresponding to the target pixel in the flame region segmentation probability map, ranging from [0,1]. It directly quantifies the likelihood that the pixel belongs to the flame region; a higher value indicates a stronger confidence that the pixel belongs to the flame region. The probability value of the same pixel having a depth hypothesis value is the probability output of the pixel corresponding to the current depth hypothesis value in the depth probability distribution. That is, the probability value corresponding to the target pixel coordinates and the current depth hypothesis value in the depth probability distribution, also ranging from [0,1]. It quantifies the confidence that the pixel's depth is the current depth hypothesis value. The calculation of the unnormalized joint probability density is based on the probability independence assumption, that is, whether a pixel is a flame and the pixel's depth assignment are two independent events. The joint probability density is obtained by multiplying the two events. This density value quantifies the probability of the joint event that the target pixel is a flame region and its depth is the current depth hypothesis value.

[0102] S43. Traverse all pixels in the current frame image of the left eye, fill the unnormalized joint probability density of the three-dimensional spatial points corresponding to each pixel into the three-dimensional spatial grid, and obtain the three-dimensional spatial probability volume of the flame.

[0103] Optionally, the traversal process processes each pixel in the current frame image of the left-eye view sequentially according to a preset pixel scanning order. For each pixel, it matches all depth assumption values ​​within a preset discrete depth assumption range, calculating the corresponding 3D spatial point and the corresponding unnormalized joint probability density for each pixel under different depth assumption values. The 3D spatial grid is a regular 3D grid divided according to the possible fire source spatial range in the camera coordinate system. The grid resolution is determined by the positioning accuracy requirements of the actual application scenario. Each cell of the grid corresponds to a fixed small spatial region in the camera coordinate system, and its 3D coordinate range is clear and non-overlapping. During the filling process, based on the coordinates of each 3D spatial point, the corresponding grid cell in the 3D spatial grid is determined, and the unnormalized joint probability density of the 3D spatial point is assigned to the corresponding grid cell. If multiple 3D spatial points are mapped to the same grid cell, these joint probability density values ​​are accumulated to ensure that the density value of the grid cell can comprehensively reflect the possibility of flames in the spatial region. After traversal, the unnormalized joint probability density of all grid cells together constitutes the three-dimensional spatial probability volume of the flame. This probability volume is a three-dimensional data structure, and its dimension is consistent with the division of the three-dimensional spatial grid. It fully quantifies the joint probability distribution of each region in the three-dimensional space under the camera coordinate system belonging to the fire source.

[0104] In one embodiment, the three-dimensional spatial probability volume of the flame and the historical estimated state are input into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance, including:

[0105] S51. The three-dimensional spatial probability volume of the flame is compressed into a feature vector of fixed dimension through a three-dimensional convolutional network.

[0106] Schematic illustration: The three-dimensional probabilistic volume of the flame, as a three-dimensional data structure, includes spatial height, spatial width, and depth, assuming corresponding three-dimensional grid dimensions. The three-dimensional convolutional layer uses miniaturized convolutional kernels, sliding operations along the three-dimensional spatial dimension to extract global features such as the spatial distribution pattern and density concentration trend of the flame region in the probabilistic volume, avoiding interference from local noise on feature representation. The global pooling layer performs global statistical aggregation on the convolutional feature map, converting the three-dimensional feature map into a one-dimensional feature vector. The dimension of this vector is a fixed value, ensuring the consistency of the input dimension of the subsequent temporal inference model, unaffected by changes in the size and depth range of the flame in the actual scene. This fixed-dimensional feature vector completely encodes the core information of the probability distribution of the flame in the current frame in three-dimensional space.

[0107] S52. Obtain the hidden state of the previous time step and the three-dimensional coordinates of the predicted fire source base point of the previous time step in the memory layer.

[0108] The memory layer is the core component in the temporal inference model that stores historical information. The hidden state stored within it is the internal state vector output by the gated loop unit after its computation at the previous time step. This vector encodes temporal context information such as the three-dimensional spatial probabilistic volume features of the flame and the trajectory of the fire source base point in all historical frames from the initial time step to the previous time step, reflecting the inertial trend and changing patterns of flame movement. The predicted three-dimensional coordinates of the fire source base point at the previous time step are the output of the temporal inference model at the previous time step, corresponding to the three-dimensional spatial coordinates in the camera coordinate system. These coordinate values ​​are stable values ​​inferred based on the observation information of the previous time step and all frames prior. Initially, the hidden state is initialized to an all-zero vector, and the predicted three-dimensional coordinates of the fire source base point at the previous time step can be set to initial values ​​according to the application scenario requirements to ensure the smooth start of the temporal inference process.

[0109] S53. The feature vector is concatenated with the three-dimensional coordinates of the predicted fire source base point of the previous time step, and then input into the gated loop unit along with the hidden state of the previous time step to obtain the hidden state of the current time step.

[0110] Furthermore, the concatenation operation between the feature vector and the predicted 3D coordinates of the fire source base point from the previous moment is achieved through dimensional superposition. This involves sequentially concatenating the fixed-dimensional feature vector with the corresponding 3D vectors of the 3D coordinates to form a new high-dimensional input vector. This concatenation process fuses the 3D spatial distribution features of the flame in the current frame with the positional information of the base point at historical moments, providing the gated recurrent unit (GRU) with complete input information on the current observation and historical positions. The GRU, as the core computational unit for time-series inference, selectively retains historical information and selectively integrates current information through an adaptive adjustment mechanism of update and reset gates. The update gate controls the contribution weight of the hidden state from the previous moment to the hidden state at the current moment, suppressing interference from redundant historical information; the reset gate adjusts the effectiveness of the current input information, strengthening the role of key observation features. The GRU receives the concatenated high-dimensional input vector and the hidden state from the previous moment. After internal nonlinear transformation and gate mechanism adjustment, it outputs the hidden state at the current moment. This state integrates the core features of the 3D spatial probability volume of the flame in the current frame and continues the motion patterns of the fire source base point accumulated in historical frames, achieving smooth transmission and dynamic updating of time-series information.

[0111] S54. Input the hidden state at the current moment into the multilayer perceptron to predict the displacement increment and covariance matrix of the fire source base point at the current moment relative to the previous moment.

[0112] Specifically, the multilayer perceptron consists of an input layer, several hidden layers, and an output layer, all connected via fully connected layers. The hidden layers employ nonlinear activation functions to enhance the model's ability to fit complex temporal changes. The current hidden state serves as the input to the multilayer perceptron, undergoing multiple nonlinear transformations in the hidden layers to map the temporal fusion features into an output that conforms to the motion characteristics of the fire source base point. The output layer has two branches: one outputs a three-dimensional displacement increment vector containing displacement components in the x, y, and z dimensions, corresponding to the positional changes of the fire source base point relative to the previous moment in the camera coordinate system along the three coordinate axes; the other outputs a 3×3 covariance matrix. The diagonal elements of this matrix correspond to the estimated variances of the three coordinate dimensions, while the off-diagonal elements correspond to the covariances between different coordinate dimensions, fully quantifying the uncertainty of the displacement increment and the final base point coordinate estimation. The magnitude of the variance and covariance directly reflects the reliability of the estimation results for the corresponding dimensions; smaller values ​​indicate higher estimation accuracy.

[0113] S55. Add the displacement increment to the predicted three-dimensional coordinates of the fire source base point at the previous moment to obtain the three-dimensional coordinates of the fire source base point at the current moment, and combine it with the covariance matrix to obtain the estimated covariance of the corresponding three-dimensional coordinates of the fire source base point.

[0114] The calculation of the three-dimensional coordinates of the fire source base point at the current moment follows the principle of temporal smoothness. Based on the inertia of the base point's motion, it avoids the occurrence of irregular abrupt changes in physical prior knowledge. It is obtained by vector addition of the predicted three-dimensional coordinates of the fire source base point at the previous moment and the predicted displacement increment at the current moment. That is, the current value of each coordinate dimension is equal to the sum of the corresponding dimension value at the previous moment and the displacement increment of that dimension. This calculation method can effectively filter out sudden interference caused by instantaneous flame flickering and observation noise, ensuring the temporal continuity and stability of the three-dimensional coordinates of the fire source base point. The estimated covariance of the three-dimensional coordinates of the fire source base point is directly obtained using the 3×3 covariance matrix output by the multilayer perceptron. This matrix serves as a quantitative representation of the uncertainty in the estimation of the base point coordinates at the current moment, providing crucial accuracy reference information for downstream fire extinguishing control devices. When the overall value of the covariance matrix is ​​small, it indicates that the current base point coordinate estimation has high reliability, and the fire extinguishing device can implement precise strikes based on these coordinates.

[0115] In one embodiment, the unnormalized joint probability density of the three-dimensional spatial points is obtained by multiplying the flame probability value of the flame region segmentation probability map at a pixel with the probability value of the depth probability distribution having a depth assumption at the same pixel, including:

[0116] S61. Based on the segmentation uncertainty map, obtain the left eye segmentation uncertainty value corresponding to the current pixel.

[0117] The segmentation uncertainty map is one of the two-branch results output by the segmentation network. It has the same spatial resolution as the current frame image in the left-eye view, and the numerical value of each pixel in the map represents the reliability of the flame segmentation result at the corresponding location. By locating the spatial coordinates of the currently processed left-eye pixel in the left-eye segmentation uncertainty map, the pixel value at that coordinate position is extracted, which is the left-eye segmentation uncertainty value corresponding to the current pixel. The magnitude of this value is negatively correlated with the segmentation reliability; the larger the value, the more unstable the flame morphology and the more blurred the boundary in the region where the pixel is located, and the lower the reliability of the segmentation result; the smaller the value, the more reliable the segmentation result.

[0118] S62. Based on the uncertainty value of left eye segmentation, calculate the confidence weight of each pixel using a confidence weight based on an exponential function; the expression for the confidence weight is: ,in, The preset positive scaling factor. The uncertainty value for left eye segmentation.

[0119] in, This indicates the coordinates in the current frame image of the left eye. The confidence weight of the current pixel is in the range of (0,1]. The larger the weight value, the more reliable the segmentation result of the pixel is, and the higher the contribution weight to the subsequent joint probability density calculation. The natural exponential function is used to convert the linear change of the uncertainty value of the left eye segmentation into a non-linear weight decay, thereby achieving rapid suppression of the weights of pixels with high uncertainty. This represents a preset positive scaling factor, used to adjust the strength of the influence of the left eye segmentation uncertainty value on the confidence weight. The larger the value, the more obvious the weight decay corresponding to the same uncertainty value, and the stronger the suppression effect on low confidence pixels. This represents the left-eye segmentation uncertainty value corresponding to the current pixel, and its value directly determines the decay rate of the confidence weight. This function uses the exponential decay characteristic to give pixels with stable segmentation and low uncertainty a high weight close to 1, while pixels with unstable segmentation and high uncertainty receive a low weight close to 0, thus achieving differentiated weighting of segmentation results with different confidence levels.

[0120] S63. Based on the confidence weight, the flame probability value of a pixel is multiplied by the probability value of the same pixel having a depth hypothesis value to obtain the unnormalized joint probability density.

[0121] The flame probability value of a pixel is taken from the value corresponding to the current pixel in the flame region segmentation probability map, quantifying the likelihood that the pixel belongs to the flame region. The probability value of the same pixel having a depth hypothesis value is taken from the probability output of the current pixel corresponding to the current depth hypothesis value in the depth probability distribution, quantifying the confidence that the pixel's depth is the current depth hypothesis value. The confidence weight acts as an adjustment factor, and the product of the two values ​​is weighted and corrected through multiplication. For pixels with reliable segmentation, the confidence weight is close to 1, and the product of its flame probability value and depth probability value remains basically unchanged, making a major contribution to the joint probability density. For pixels with unreliable segmentation, the confidence weight approaches 0, significantly weakening the influence of its product on the joint probability density and effectively suppressing the interference of false flame regions caused by segmentation errors on the 3D spatial probability volume. The product of these three factors is the unnormalized joint probability density of the three-dimensional space point corresponding to the current pixel. This density value comprehensively considers the probability of the flame's existence, the confidence level of the depth, and the confidence level of the segmentation, and more accurately quantifies the probability of the joint event that the current pixel is a flame region and the depth is the current assumed value.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] Based on the same inventive concept, this application also provides a deep learning-based fire source identification system for implementing the aforementioned deep learning-based fire source identification method. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more deep learning-based fire source identification system embodiments provided below can be found in the limitations of the deep learning-based fire source identification method described above, and will not be repeated here.

[0124] In an exemplary embodiment, as shown in Figure 2, a deep learning-based fire source identification system is provided, comprising:

[0125] The image acquisition module 201 is used to acquire the current frame image and the sequence of historical frame images of the flame scene; the current frame image includes the left-view current frame image and the right-view current frame image;

[0126] The feature extraction module 202 is used to extract the multi-scale spatial features of each frame image in the current frame image and the historical frame image sequence based on the weight-shared deep convolutional neural network, to obtain the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence, and to fuse the multi-scale spatial features of the current frame and the historical multi-scale spatial feature sequence in the time dimension to obtain the fused spatiotemporal feature map of the current frame image.

[0127] The flame determination module 203 is used to input the fused spatiotemporal feature map into the encoder-decoder-based segmentation network to obtain a flame region segmentation probability map and a segmentation uncertainty map; the segmentation probability map represents the likelihood that each pixel in the current frame image belongs to the flame.

[0128] The spatial module 204 is used to construct a cross-view spatiotemporal matching cost body based on the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, based on a preset discrete depth assumption range. After regularizing the cross-view spatiotemporal matching cost body through a pre-trained three-dimensional convolutional neural network, the soft maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution.

[0129] The flame location module 205 is used to calculate the three-dimensional spatial probability volume of the flame based on the flame region segmentation probability map and the depth probability distribution, and input the three-dimensional spatial probability volume of the flame and the historical estimated state into the time-series inference model based on the recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance.

[0130] In one embodiment, the feature extraction module 202 is further configured to:

[0131] Spatial features of the current frame image of the left eye and the current frame image of the right eye at multiple scales are extracted respectively to obtain the spatial feature map of the current frame of the left eye and the spatial feature map of the current frame of the right eye.

[0132] Extract left-eye historical spatial feature map sequences and right-eye historical spatial feature map sequences at the same scale based on historical frame image sequences;

[0133] The spatial feature map of the current frame of the left eye and the sequence of spatial feature maps of the left eye are concatenated in the time dimension to obtain the spatiotemporal feature block of the left eye;

[0134] The right eye's current frame spatial feature map and the right eye's historical spatial feature map sequence are concatenated in the time dimension to obtain the right eye's spatiotemporal feature block;

[0135] Spatiotemporal convolution is performed on the left-eye and right-eye spatiotemporal feature blocks to obtain a fused spatiotemporal feature map. The fused spatiotemporal feature map includes the left-eye fused spatiotemporal feature map and the right-eye fused spatiotemporal feature map. Spatiotemporal convolution corresponds to performing convolution operations in the time dimension to aggregate temporal context information and nonlinear activation function processing.

[0136] In one embodiment, the space module 204 is further configured to:

[0137] Based on the left-eye fusion spatiotemporal feature map, extract the left-eye feature vector of any pixel in the current frame image of the left eye;

[0138] Based on the epipolar geometry of the binocular system and the preset depth assumption, the pixel points corresponding to the feature vector of the left eye are projected onto the current frame image of the right eye to obtain the right eye projection coordinates corresponding to the pixel points.

[0139] Extract the right eye feature vector at the right eye projection coordinates of the right eye fusion spatiotemporal feature map;

[0140] Calculate the cosine similarity between the left and right feature vectors to obtain the basic matching metric, and obtain the left-eye segmentation uncertainty value at the corresponding pixel of the left-eye feature vector and the right-eye segmentation uncertainty value at the right-eye projection coordinates.

[0141] Based on the adaptive weighted matching cost function, and combining the basic matching metric, the left-eye segmentation uncertainty value, and the right-eye segmentation uncertainty value, the matching cost of a pixel under the depth assumption is calculated; the expression of the adaptive weighted matching cost function is: ,in, For the left eye feature vector, The right eye feature vector, For left eye segmentation uncertainty value, For the uncertainty value of right eye segmentation, To balance the weighting coefficients, These are the pixel coordinates. The coordinates of the right eye projection. This is the assumed value for depth;

[0142] By iterating through all pixels and depth assumptions of the current frame image of the left eye, and arranging each matching cost in the spatial and depth dimensions, a cross-view spatiotemporal matching cost volume is obtained.

[0143] In one embodiment, the space module 204 is further configured to:

[0144] By stacking three-dimensional convolutional layers and three-dimensional pooling layers in a three-dimensional convolutional neural network, the cross-view spatiotemporal matching cost volume is filtered sequentially in the spatial and depth dimensions, and contextual information is aggregated to suppress noise and isolated high-cost points caused by incorrect matching, thus obtaining a regularized cost volume.

[0145] Based on the depth dimension, the softmax function is applied to each pixel of the regularized cost volume to calculate the probability distribution under different depth assumptions, thus obtaining the depth probability distribution; the expression for the softmax function is: ,in, Represents pixels The depth is the The probability of a depth hypothesis value The regularization cost volume at the pixel And the depth is the The value of the depth hypothesis. This represents the total number of depth hypotheses.

[0146] In one embodiment, the flame position positioning module 205 is further configured to:

[0147] For any pixel in the current frame image of the left eye, the pixel is back-projected into a three-dimensional spatial point in the camera coordinate system according to the physical depth value corresponding to the left eye camera intrinsic parameter matrix and the depth assumption value.

[0148] The unnormalized joint probability density of the three-dimensional spatial points is obtained by multiplying the flame probability value of the flame region segmentation probability map at the pixel point with the probability value of the depth probability distribution at the same pixel point with the depth assumption.

[0149] Traverse all pixels in the current frame image of the left eye, and fill the three-dimensional space grid with the unnormalized joint probability density of the three-dimensional space points corresponding to each pixel to obtain the three-dimensional space probability volume of the flame.

[0150] In one embodiment, the flame position positioning module 205 is further configured to:

[0151] The three-dimensional spatial probability volume of the flame is compressed into a fixed-dimensional feature vector by using a three-dimensional convolutional network.

[0152] Obtain the hidden state of the previous time step and the three-dimensional coordinates of the predicted fire source base point of the previous time step in the memory layer;

[0153] The feature vector is concatenated with the three-dimensional coordinates of the predicted fire source base point at the previous time step, and then input into the gated loop unit along with the hidden state at the previous time step to obtain the hidden state at the current time step.

[0154] The hidden state at the current moment is input into the multilayer perceptron to predict the displacement increment and covariance matrix of the fire source base point at the current moment relative to the previous moment.

[0155] The displacement increment is added to the predicted three-dimensional coordinates of the fire source base point at the previous moment to obtain the three-dimensional coordinates of the fire source base point at the current moment. The estimated covariance of the corresponding three-dimensional coordinates of the fire source base point is obtained by combining the covariance matrix.

[0156] In one embodiment, an event probability module is also included, for:

[0157] Based on the segmentation uncertainty map, obtain the left-eye segmentation uncertainty value corresponding to the current pixel;

[0158] Based on the uncertainty value of left eye segmentation, the confidence weight of each pixel is calculated using a confidence weight based on an exponential function; the expression for the confidence weight is as follows: ,in, The preset positive scaling factor. For left eye segmentation uncertainty value;

[0159] Based on the confidence weight, the flame probability value of a pixel is multiplied by the probability value of the same pixel having a depth hypothesis value to obtain the unnormalized joint probability density.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0161] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0162] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0163] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A fire source identification method based on deep learning, characterized in that, The method includes: acquiring a current frame image and a sequence of historical frame images of a flame scene; the current frame image includes a left-view current frame image and a right-view current frame image; based on a weight-shared deep convolutional neural network, extracting multi-scale spatial features of each frame image in the current frame image and the sequence of historical frame images respectively, obtaining a current frame multi-scale spatial feature and a historical multi-scale spatial feature sequence, and fusing the current frame multi-scale spatial feature and the historical multi-scale spatial feature sequence in the time dimension to obtain a fused spatiotemporal feature map of the current frame image; inputting the fused spatiotemporal feature map into an encoder-decoder based segmentation network to obtain a flame region segmentation probability map and a segmentation uncertainty map; the segmentation probability... The graph represents the likelihood that each pixel in the current frame image belongs to the flame. Based on a preset discrete depth assumption range, a cross-view spatiotemporal matching cost body is constructed according to the left-eye and right-eye fusion spatiotemporal feature maps in the fusion spatiotemporal feature map. After regularizing the cross-view spatiotemporal matching cost body through a pre-trained 3D convolutional neural network, a soft-maximization probability distribution is calculated along the depth dimension to obtain the depth probability distribution. Based on the flame region segmentation probability map and the depth probability distribution, a flame 3D spatial probability body is calculated. The flame 3D spatial probability body and the historical estimated state are input into a time-series inference model based on a recurrent neural network to obtain the 3D coordinates of the fire source base point and the estimated covariance.

2. The method according to claim 1, characterized in that, The weight-shared deep convolutional neural network extracts multi-scale spatial features from each frame in the current frame image and the historical frame image sequence, respectively, to obtain the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence. The current frame multi-scale spatial features and the historical multi-scale spatial feature sequence are then fused along the temporal dimension to obtain the fused spatiotemporal feature map of the current frame image. This includes: extracting spatial features from the left-eye current frame image and the right-eye current frame image at multiple scales, respectively, to obtain the left-eye current frame spatial feature map and the right-eye current frame spatial feature map; and extracting the left-eye historical spatial feature map sequence at the same scale based on the historical frame image sequence. The left-eye current frame spatial feature map and the left-eye historical spatial feature map sequence are concatenated in the time dimension to obtain a left-eye spatiotemporal feature block; the right-eye current frame spatial feature map and the right-eye historical spatial feature map sequence are concatenated in the time dimension to obtain a right-eye spatiotemporal feature block; the left-eye spatiotemporal feature block and the right-eye spatiotemporal feature block are spatiotemporally convolved to obtain the fused spatiotemporal feature map; the fused spatiotemporal feature map includes the left-eye fused spatiotemporal feature map and the right-eye fused spatiotemporal feature map; the spatiotemporal convolution corresponds to performing convolution operations in the time dimension to aggregate temporal context information and nonlinear activation function processing.

3. The method according to claim 2, characterized in that, The method of constructing a cross-view spatiotemporal matching cost body based on the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, based on the preset discrete depth assumption range, includes: extracting the left-eye feature vector of any pixel in the current frame image of the left eye based on the left-eye fusion spatiotemporal feature map; projecting the pixel corresponding to the left-eye feature vector onto the current frame image of the right eye based on the epipolar geometry of the binocular system and the preset depth assumption value to obtain the right-eye projection coordinates corresponding to the pixel; extracting the right-eye feature vector of the right-eye fusion spatiotemporal feature map at the right-eye projection coordinates; calculating the cosine similarity between the left-eye feature vector and the right-eye feature vector to obtain the basic matching metric, and obtaining the left-eye segmentation uncertainty value of the segmentation uncertainty map at the pixel corresponding to the left-eye feature vector and the right-eye segmentation uncertainty value at the right-eye projection coordinates; and calculating the matching cost of the pixel under the depth assumption value based on the adaptive weighted matching cost function, combined with the basic matching metric, the left-eye segmentation uncertainty value and the right-eye segmentation uncertainty value; the expression of the adaptive weighted matching cost function is as follows: ,in, For the left eye feature vector, The right eye feature vector, For left eye segmentation uncertainty value, For the uncertainty value of right eye segmentation, To balance the weighting coefficients, These are the pixel coordinates. The coordinates of the right eye projection. The depth assumption value is used as the basis for traversing all pixels of the current frame image of the left eye and all the depth assumption values. The matching costs are arranged in the spatial and depth dimensions to obtain the cross-view spatiotemporal matching cost body.

4. The method according to claim 3, characterized in that, The process of regularizing the cross-view spatiotemporal matching cost body using a pre-trained 3D convolutional neural network and then calculating a soft-maximization probability distribution along the depth dimension to obtain a depth probability distribution includes: filtering the cross-view spatiotemporal matching cost body sequentially in both the spatial and depth dimensions using stacked 3D convolutional and pooling layers in the 3D convolutional neural network, aggregating contextual information to suppress noise and isolated high-cost points caused by incorrect matching, and obtaining a regularized cost body; and applying a softmax function to each pixel of the regularized cost body based on the depth dimension to calculate the probability distribution under different depth assumptions, thus obtaining the depth probability distribution; the expression for the softmax function is: ,in, Represents pixels The depth is the The probability of a depth hypothesis value The regularization cost volume at the pixel And the depth is the The value of the depth hypothesis. This represents the total number of depth hypotheses.

5. The method according to claim 3, characterized in that, The step of calculating the three-dimensional spatial probability volume of the flame based on the flame region segmentation probability map and the depth probability distribution includes: for any pixel in the current frame image of the left-eye camera, back-projecting the pixel into a three-dimensional spatial point in the camera coordinate system according to the physical depth value corresponding to the left-eye camera intrinsic parameter matrix and the depth assumption value; multiplying the flame probability value of the pixel in the flame region segmentation probability map with the probability value of the same pixel having the depth assumption value in the depth probability distribution to obtain the unnormalized joint probability density of the three-dimensional spatial point; traversing all pixels in the current frame image of the left-eye camera, filling the unnormalized joint probability density of the three-dimensional spatial points corresponding to each pixel into the three-dimensional spatial grid to obtain the three-dimensional spatial probability volume of the flame.

6. The method according to claim 1, characterized in that, The step of inputting the three-dimensional spatial probability volume of the flame and the historical estimated state into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance includes: compressing the three-dimensional spatial probability volume of the flame into a fixed-dimensional feature vector through a three-dimensional convolutional network; obtaining the hidden state of the previous time step and the predicted three-dimensional coordinates of the fire source base point of the previous time step in the memory layer; concatenating the feature vector with the predicted three-dimensional coordinates of the fire source base point of the previous time step, and inputting it together with the hidden state of the previous time step into a gated recurrent unit to obtain the hidden state of the current time step; inputting the hidden state of the current time step into a multilayer perceptron to predict the displacement increment and covariance matrix of the fire source base point of the current time step relative to the previous time step; adding the displacement increment with the predicted three-dimensional coordinates of the fire source base point of the previous time step to obtain the three-dimensional coordinates of the fire source base point of the current time step, and combining it with the covariance matrix to obtain the estimated covariance of the corresponding three-dimensional coordinates of the fire source base point.

7. The method according to claim 5, characterized in that, The step of multiplying the flame probability value of the flame region segmentation probability map at each pixel with the probability value of the depth probability distribution at the same pixel having the depth hypothesis value to obtain the unnormalized joint probability density of the three-dimensional spatial point includes: obtaining the left-eye segmentation uncertainty value corresponding to the current pixel based on the segmentation uncertainty map; calculating the confidence weight of the pixel based on the left-eye segmentation uncertainty value using a confidence weight based on an exponential function; the expression for the confidence weight is as follows: ,in, The preset positive scaling factor. The uncertainty value for left eye segmentation is determined; based on the confidence weight, the flame probability value of the pixel is multiplied by the probability value of the same pixel having the depth hypothesis value to obtain the unnormalized joint probability density.

8. A fire source identification system based on deep learning, characterized in that, The system includes: an image acquisition module for acquiring the current frame image and a sequence of historical frame images of a flame scene; the current frame image includes a left-view current frame image and a right-view current frame image; a feature extraction module for extracting multi-scale spatial features of each frame image in the current frame image and the sequence of historical frame images based on a weighted shared deep convolutional neural network, obtaining the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence, and fusing the current frame multi-scale spatial features and the historical multi-scale spatial feature sequence in a time dimension to obtain a fused spatiotemporal feature map of the current frame image; and a flame judgment module for inputting the fused spatiotemporal feature map into an encoder-decoder based segmentation network to obtain a flame region segmentation probability map and a segmentation uncertainty map; The segmentation probability map represents the likelihood that each pixel in the current frame image belongs to the flame; the spatial module is used to construct a cross-view spatiotemporal matching cost body based on a preset discrete depth assumption range, according to the left-eye fusion spatiotemporal feature map and the right-eye fusion spatiotemporal feature map in the fusion spatiotemporal feature map, and after regularizing the cross-view spatiotemporal matching cost body through a pre-trained three-dimensional convolutional neural network, calculate the soft-maximization probability distribution along the depth dimension to obtain the depth probability distribution; the flame location module is used to calculate the flame three-dimensional spatial probability body based on the flame region segmentation probability map and the depth probability distribution, and input the flame three-dimensional spatial probability body and the historical estimated state into a time-series inference model based on a recurrent neural network to obtain the three-dimensional coordinates of the fire source base point and the estimated covariance.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images