Fire point ranging and positioning method based on fusion of camera image and three-dimensional sparse point cloud

Abstract

The invention discloses a fire point ranging and positioning method based on fusion of a camera image and a three-dimensional sparse point cloud. The method specifically comprises the following steps: carrying out combined acquisition and preprocessing on a high-resolution event image and a sparse point cloud in an early forest fire scene; performing spatial registration on the event camera image and the sparse point cloud; mapping the newly collected point cloud to an event camera coordinate system and projecting the newly collected point cloud to an event image plane to form a projection point set; performing fire point local dense three-dimensional reconstruction based on event-point cloud fusion; carrying out fire point identification and three-dimensional distance measurement positioning on the dense point cloud; according to the method, sparse depth constraint and event edge guided smooth regularization are adopted to complement depth, and back projection is carried out to generate a local dense point cloud, so that high-precision three-dimensional reconstruction of an early fire point volume, a fire line boundary and a spatial relationship between the fire line boundary and surrounding earth surface / vegetation is realized.

Description

Technical Field

[0001] This invention belongs to the field of fire monitoring and early warning technology, and in particular relates to a fire point ranging and positioning method based on the fusion of camera images and three-dimensional sparse point clouds. Background Technology

[0002] Currently, early wildfire monitoring in forest and mountainous areas mainly employs fixed lookout towers, drones, or satellite-borne visible light cameras and infrared thermal imaging equipment. These devices identify smoke columns, flames, or temperature anomalies through two-dimensional or thermal infrared images to trigger automatic alarms. However, these methods are mostly limited to two-dimensional planar detection and struggle to provide high-precision three-dimensional ranging and spatial positioning of fire points. Their accuracy is easily affected by terrain undulations, smoke obstruction, and imaging geometric errors, and they lack the ability to continuously depict small-scale early fire points in three dimensions. Meanwhile, lidar and camera fusion technology has been widely used in fields such as autonomous driving for three-dimensional target detection and environmental perception. By aligning and jointly encoding sparse point clouds with two-dimensional image features, it effectively compensates for the lack of detail in point clouds and the lack of depth information in images. However, related research mainly focuses on rigid target scenarios such as vehicles and has not yet built suitable three-dimensional reconstruction and ranging models for flames, which are highly dynamic, deformably drastically, and small-scale targets.

[0003] In recent years, event cameras have emerged with advantages such as high temporal resolution, high dynamic range, and low latency. Some works have begun to explore flame detection datasets and recognition algorithms based on event cameras, but most are still limited to two-dimensional detection or fire presence / absence discrimination. They have not yet deeply fused high-resolution event camera images with three-dimensional sparse point clouds for dense three-dimensional reconstruction and high-precision ranging and localization of early wildfire fire points. Therefore, it is necessary to propose a fire point ranging and localization method that fuses high-resolution event camera images with three-dimensional sparse point clouds for early wildfire scenarios. Summary of the Invention

[0004] This invention proposes a fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds. The aim is to construct an anomaly score based on neighborhood density deviation, extract fire point candidate regions from dense point clouds, and improve the accuracy of judging the fire intensity, distance, etc. of fire points.

[0005] To achieve the above effects, the following technical solution is adopted:

[0006] The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds includes the following specific steps:

[0007] A joint observation system consisting of a high-resolution event camera and a 3D sparse point cloud sensor is constructed. The two types of sensors are fixedly installed on watchtowers, drones, or vehicle-mounted platforms, and collaborative acquisition is achieved through a unified trigger and time synchronization module. Intensity normalization and noise suppression are performed on each frame of event image acquired.

[0008] An unsupervised learning framework is used to spatially register event camera images that have undergone intensity normalization and noise suppression with sparse point clouds. Under the condition of lacking pixel-to-3D point ground truth correspondence, the extrinsic parameters between the two sensors are automatically solved and pixel-to-3D point correspondence is generated.

[0009] Using the extrinsic parameters obtained from training, the newly acquired point cloud is mapped to the event camera coordinate system and projected onto the event image plane to form a registered set of projection points.

[0010] In the event camera coordinate system, a depth map is constructed and cross-modal feature fusion is performed on the sparse point cloud, and a dense 3D reconstruction of the local area of ​​the fire point is achieved on the basis of the constructed and fused features.

[0011] After completing the reconstruction of the dense 3D point cloud of the fire point, the automatic identification and 3D ranging and positioning of the fire point area are achieved without relying on manual annotation.

[0012] The process of intensity normalization and noise suppression for each acquired event image frame specifically involves performing intensity normalization and noise suppression on each event image frame. Calculate its value in the image domain mean with standard deviation And standardize the process:

[0013] ,

[0014] in, The event image after normalization;

[0015] ,

[0016] The corresponding standard deviation, The total number of pixels in the image. To prevent tiny positive numbers with a denominator of zero.

[0017] The specific steps for spatial registration of event camera images and sparse point clouds using an unsupervised learning framework are as follows:

[0018] Given the current extrinsic parameters In this case, the three-dimensional points in the point cloud sensor coordinate system L are... By mapping the rigid body transformation onto the event camera coordinate system C, we obtain:

[0019] ,

[0020] in, Let be a rotation matrix. It is a translation vector. It is the first The three-dimensional coordinates of a point in the event camera coordinate system;

[0021] Using the intrinsic parameter matrix of the event camera can Projecting onto the event image plane, we obtain the pixel coordinates, expressed as:

[0022] ,

[0023] in, Let j be the pixel position of the j-th point in the event image coordinate system. For depth Relevant scaling factors;

[0024] From normalized event images Extract structural features to obtain gradient-based edge response maps. The expression is:

[0025] ,

[0026] Where (x, y) are the pixel coordinates in the event image. and They represent in and Gradient in direction;

[0027] Constructing a distance transform or cost field on an event image The distance from each pixel to the nearest structural edge is encoded as a scalar to measure the spatial deviation between the projected point and the real structure.

[0028] Based on this, the structural consistency loss of a single frame is defined as follows:

[0029] ,

[0030] in, Indicates the relationship with the first The number of valid projection points in the point cloud of frame event image matching. For projection point The value in the distance field characterizes the distance from the projection point to the nearest event edge;

[0031] Let the projection of this point in the k-th frame be... In the The projection in the frame is The expression for the multi-frame reprojection consistency loss is:

[0032] ,

[0033] in, This represents the number of static points that can establish a correspondence between two frames. Represents the L2 norm;

[0034] The overall loss function for unsupervised spatial registration of event camera images and sparse point clouds is constructed and calculated as follows:

[0035] ,

[0036] Where α, β, γ, and η are non-negative weighting coefficients. For structural consistency loss, For multi-frame reprojection consistency loss, For distributed / mutual information loss, This is the regularization term for the external parameters.

[0037] The specific process of constructing the depth map is as follows: using the obtained extrinsic parameters... and intrinsic parameter matrix Point cloud sensor coordinate system The three-dimensional points below Projected to the In the frame event image plane, obtain the pixel coordinates. and its depth in the camera coordinate system In the event image domain Within, a sparse depth map is constructed using pixel coordinates as indices. With effectiveness mask The expression is:

[0038] ,

[0039] Furthermore, the cross-modal feature fusion adopts a dual-branch coding structure to extract event images and sparse depth features respectively, and performs feature-level fusion at the pixel level to obtain three-dimensional reconstruction guidance features.

[0040] Furthermore, the dense 3D reconstruction of the local fire point region specifically involves constructing a depth completion decoding network, which uses fused features, sparse depth, and a mask as input to predict the dense depth map over the entire image domain.

[0041] Furthermore, the specific process for automatic identification of the fire point area is as follows:

[0042] Drawing upon the concept of local density deviation, we perform 3D fire point anomaly identification. Let the reconstructed local dense point cloud of the fire point be represented in the event camera coordinate system C as follows:

[0043] ,

[0044] in, For the first The coordinates of a 3D point in the event camera coordinate system The number of points in a dense point cloud;

[0045] Based on the aforementioned fusion feature extraction results, a feature vector containing geometric and event information is constructed, expressed as follows:

[0046] ,

[0047] in, Point Local geometric features, This represents the intensity or characteristics of an event obtained by projecting an event image. This represents the total feature dimension.

[0048] Furthermore, the unsupervised anomaly detection framework specifically comprises a feature set. As input, estimate the local density of each point in the 3D feature space and calculate anomaly scores. Taking the K-nearest neighbor-based local density deviation metric as an example, record the points... The K-nearest neighbor set is Let the distance be Euclidean, then the first... The distance from a point to its k-th nearest neighbor can be denoted as:

[0049] ,

[0050] Its local reachability is defined as follows:

[0051] ,

[0052] Estimated point The locally reachable density, expressed as:

[0053] ,

[0054] in, The number of nearest neighbors.

[0055] To measure points Anomaly scores are defined relative to the density deviation of their neighborhood, and are expressed as follows:

[0056] ,

[0057] Furthermore, when the abnormal score When the local density of a point is similar to its nearest neighbors, it can be considered a "normal background point". When the value is significantly greater than 1, it indicates that the local density at the location of the point is significantly lower than that of the neighborhood, and it belongs to the "locally sparse and abrupt" anomaly point, which can be regarded as a fire point or flame boundary candidate.

[0058] Furthermore, based on anomaly scoring, and through threshold segmentation and spatial connectivity analysis, three-dimensional anomaly regions of fire points are automatically extracted from dense point clouds. The anomaly threshold is set as follows: Then the initial set of outliers can be defined as:

[0059] ,

[0060] And in Perform three-dimensional connected component segmentation and clustering to obtain several spatially independent candidate fire points. For each candidate fire point... Its geometric center or highest point is taken as the representative point of the fire point, and its three-dimensional position expression is:

[0061] ,

[0062] in, Let be the number of points in the q-th fire cluster. That is, the three-dimensional representative position of the fire point in the event camera coordinate system.

[0063] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the fire point ranging and localization method based on the fusion of camera images and three-dimensional sparse point clouds as described in any of the preceding claims.

[0064] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the fire point ranging and localization method based on the fusion of camera images and three-dimensional sparse point clouds as described in any of the preceding claims.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] 1. This invention proposes a fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds. By utilizing the structural information of event images and the geometric distribution of sparse point clouds, this invention jointly optimizes the extrinsic parameters of the two sensors through unsupervised losses such as projection structure consistency, multi-frame reprojection consistency, and distribution correlation, thereby achieving automatic spatial alignment of high-resolution event camera images and 3D sparse point clouds without the need for calibration boards and manual ground truth.

[0067] 2. This invention integrates the high dynamic and high temporal resolution features of event images with the true depth information of sparse point clouds in the feature space. It uses sparse depth constraints and smooth regularization guided by event edges to complete the depth and back-project to generate local dense point clouds, thereby achieving high-precision three-dimensional reconstruction of the early fire point volume, fire line boundary and its spatial relationship with the surrounding surface / vegetation.

[0068] 3. On the reconstructed local dense point cloud, this invention constructs an unsupervised anomaly detection model based on geometric and event features, automatically identifies fire point anomaly clusters by local density deviation measurement, and transforms their representative points to the world coordinate system to complete the three-dimensional ranging and precise positioning of fire points. It can achieve automatic detection and spatial quantification of early fire points in complex mountainous backgrounds without the need for fire point annotation samples. Attached Figure Description

[0069] The accompanying drawings, which are included to provide a further understanding of the embodiments of the present invention and constitute a part of this application, are not intended to limit the embodiments of the present invention. In the drawings:

[0070] Figure 1 This is a flowchart of the ranging method in the embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0072] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0073] This invention provides a fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds, characterized by the following specific steps:

[0074] A joint observation system consisting of a high-resolution event camera and a 3D sparse point cloud sensor is constructed. The two types of sensors are fixedly installed on watchtowers, drones, or vehicle-mounted platforms, and collaborative acquisition is achieved through a unified trigger and time synchronization module. Intensity normalization and noise suppression are performed on each frame of event image acquired.

[0075] In the early stages of wildfire monitoring, a joint observation system consisting of a high-resolution event camera and a 3D sparse point cloud sensor is first constructed. The two types of sensors are fixedly installed on lookout towers, drones, or vehicle-mounted platforms, and collaborative acquisition is achieved through a unified triggering and time synchronization module. The event camera outputs a brightness change event stream in an asynchronous manner, which can maintain high temporal resolution and high dynamic range under conditions such as strong light, smoke, and rapid movement, making it very suitable for capturing the edge of the flame and the rapid changes in the fire. The 3D point cloud sensor (such as lidar) acquires the sparse 3D spatial structure of objects such as forest terrain, vegetation, and smoke columns in a scanning manner, providing a geometric basis for subsequent 3D ranging and localization of fire points.

[0076] On the event camera side, its output is modeled as a series of discrete events;

[0077] ,

[0078] in, This represents the pixel coordinates of the i-th event on the event camera image plane. Indicates the timestamp of the event. This indicates the polarity of the brightness change (brightening or darkening), where N is the total number of events within the current time window. Based on the principle of event imaging, this can be achieved within the time window. The event stream is then reconstructed into a high-resolution event image (or event intensity map). A typical cumulative form can be expressed as:

[0079] ,

[0080] In the formula, Indicates within the time window End of pixel The reconstructed logarithmic strength or event strength value, The cumulative value at the end of the previous window, where N is a contrast constant related to the event logarithmic brightness threshold;

[0081] The summation in the above formula iterates through all events falling on pixel (x,y) within the time window. Through the above reconstruction process, the highly sparse and asynchronous event stream can be restored into a high-resolution event image sequence that retains the high dynamic range of the flame and is easy to align with the point cloud.

[0082] To improve the usability of event images in subsequent networks, intensity normalization and noise suppression are performed on each frame of the event image. Specifically, for each frame of the event image... Calculate its value in the image domain mean with standard deviation And standardize the process:

[0083] ,

[0084] in, The event image after normalization;

[0085] ,

[0086] in, The corresponding standard deviation, The total number of pixels in the image. To prevent tiny positive numbers with a denominator of zero.

[0087] This normalization step can reduce the dynamic range differences in event intensity under different scenes and exposure conditions, allowing the subsequent feature extraction network to focus more stably on the structural information of the flame region rather than its absolute brightness.

[0088] On the sparse point cloud side, this invention represents the observations of the point cloud sensor in its body coordinate system as follows:

[0089] ,

[0090] in, This represents the three-dimensional coordinates of the j-th point in the coordinate system L of the point cloud sensor (e.g., LiDAR). These are its three components in this coordinate system, and M is the number of points obtained in the current scan.

[0091] To enable joint processing of subsequent event images and point clouds, the temporal and spatial references of the two types of sensors are unified during the preprocessing stage.

[0092] In terms of time, the event time is divided into a series of windows with no heavy snow or partial heavy snow. and the point cloud scan timestamp Alignment is performed by selecting point cloud frames that meet the criteria and pairing them with event images from the corresponding time windows. The expression is as follows:

[0093] ,

[0094] in Indicates the first The acquisition time of the frame point cloud To set the maximum allowable time deviation threshold, by setting it appropriately. Ensure that the paired point cloud corresponds to the same early fire state as the event image.

[0095] Spatially, extrinsic parameters are introduced from the point cloud sensor coordinate system L to the event camera coordinate system C. Transform the point cloud coordinates to the event camera coordinate system:

[0096] ,

[0097] in, Let j be the three-dimensional coordinates of the j-th point in the event camera coordinate system C. Let be the rotation matrix from the point cloud coordinate system to the event camera coordinate system. Translation vector.

[0098] An unsupervised learning framework is used to spatially register event camera images that have undergone intensity normalization and noise suppression with sparse point clouds. Under the condition of lacking pixel-to-3D point ground truth correspondence, the extrinsic parameters between the two sensors are automatically solved and pixel-to-3D point correspondence is generated.

[0099] The specific steps are as follows:

[0100] First, given the current extrinsic parameters In this case, the three-dimensional points in the point cloud sensor coordinate system L are... By mapping the rigid body transformation onto the event camera coordinate system C, we obtain:

[0101] ,

[0102] in, Let be a rotation matrix. It is a translation vector. It is the three-dimensional coordinate of the j-th point in the event camera coordinate system.

[0103] Further utilize the intrinsic parameter matrix of the event camera can Projecting onto the event image plane, the pixel coordinates are obtained as follows:

[0104] ,

[0105] in, Let j be the pixel position of the j-th point in the event image coordinate system. For depth Relevant scaling factors.

[0106] Through the above projection, given external parameters, Under the given conditions, the 3D point cloud is mapped to a set of projected points on the event camera image plane. And further construct a projection depth map or occupancy map.

[0107] To optimize extrinsic parameters under unsupervised conditions, a registration loss function based on structural consistency is constructed to align the projected point cloud with the flame outline and background structure as closely as possible in the event image. Specifically, this is achieved by analyzing the normalized event image... Extracting structural features, such as gradient-based edge response maps. The expression is:

[0108] ,

[0109] Where (x, y) are the pixel coordinates in the event image. and They represent in and Gradient in direction, edge response It can highlight strongly structured areas such as flame boundaries and terrain outlines.

[0110] Constructing a distance transform or cost field on an event image The distance from each pixel to the nearest structural edge is encoded as a scalar to measure the spatial deviation between the projected point and the real structure.

[0111] Based on this, the structural consistency loss of a single frame is defined as:

[0112] ,

[0113] in, This represents the number of valid projected points in the point cloud that match the event image of the Kth frame. For projection point The value in the distance field characterizes the distance from the projection point to the nearest event edge, achieved by minimizing... This can encourage the transformed 3D point cloud edge structure to observe the salient structure in the event image as much as possible, thereby driving the extrinsic parameter estimation to converge toward the real value.

[0114] Considering that the platform attitude may change slowly over time during early wildfire monitoring, a multi-frame reprojection consistency constraint is further introduced. This constraint is set within adjacent time windows. and In the process, the corresponding point cloud frames are used to obtain the platform pose transformation through odometry or induced estimation. Then, the projection of a static 3D point onto two event images should satisfy geometric consistency. Let the projection of this point on the first frame be... The projection in the frame is In the The projection in the frame is Then the expression for the multi-frame reprojection consistency loss is:

[0115] ,

[0116] in, This represents the number of static points that can establish a correspondence between two frames. Representing the L2 norm; by minimizing Redundant information in the time series can be used to suppress random noise in the extrinsic parameter estimation, so that the registration results remain consistent across multiple frames.

[0117] To enhance the consistency of the overall distribution across different modalities, this invention can also introduce registration terms based on distribution or mutual information. For example, the depth distribution of the projected point cloud can be represented by a probability density function. The description uses a probability density function to represent the event intensity distribution of an event image. Describe and construct the joint distribution mutual information As a measure of consistency:

[0118] ,

[0119] Where z is the quantization depth value, and i is the quantization event intensity value. , These are marginal distributions. Maximizing mutual information... This can make the projection depth highly correlated with the event intensity in a statistical sense, thereby improving the consistency of cross-modal structures globally.

[0120] In the optimized implementation, it can be transformed into an equivalent loss function. Minimize.

[0121] Considering the above constraints, the overall loss function for unsupervised spatial registration of event camera images and sparse point clouds is as follows:

[0122] ,

[0123] Where α, β, γ, and η are non-negative weighting coefficients. For structural consistency loss, For multi-frame reprojection consistency loss, For distributed / mutual information loss, Regularization terms for external parameters, such as constraint rotation matrices. The orthogonality of the loss function and the smooth variation of its extrinsic parameters within the prior range are also discussed. By minimizing the loss function on a large number of unlabeled event image-point cloud samples, the extrinsic parameters can be iteratively optimized under completely unsupervised conditions. Ultimately, a convergent and stable spatial transformation between sensors is obtained.

[0124] Using the extrinsic parameters obtained during training, the newly acquired point cloud Mapped to the event camera coordinate system and projected onto the event image plane, forming a registered set of projection points. .

[0125] By comparing with pixel coordinates in the event image The neighborhood matching relationship can be further used to generate pixel-3D point pairs, providing a precise spatial alignment basis for subsequent cross-modal feature fusion, local dense 3D reconstruction of fire points, and 3D ranging and localization of fire points.

[0126] Depth maps are constructed and cross-modal features are fused in the sparse point cloud under the event camera coordinate system C, and dense 3D reconstruction of the local area of ​​the fire point is achieved based on the constructed and fused features.

[0127] In this step, the obtained extrinsic parameters are first used. The intrinsic parameter matrix K represents the three-dimensional points in the point cloud sensor coordinate system L. Projecting the pixel coordinates onto the event image plane of the k-th frame yields the pixel coordinates. and its depth in the camera coordinate system In the event image domain Within, a sparse depth map is constructed using pixel coordinates as indices. With effectiveness mask :

[0128] ,

[0129] For pixel positions (x, y) where no projection point exists, set , The result is null. Through the above process, sparse depth observations with true depth only at the sparse point cloud projection locations can be obtained on the event image plane, providing constraints for subsequent depth completion and point cloud densification.

[0130] The cross-modal feature fusion employs a dual-branch coding structure to extract event images and sparse depth features respectively, and performs feature-level fusion at the pixel level to obtain 3D reconstruction guidance features.

[0131] In this embodiment, the specific implementation process is as follows: a dual-branch coding structure is used to extract features from the event image and sparse depth, respectively. Let the normalized k-th frame event image be... Network for extracting event features The expression for the multi-channel event feature map is as follows:

[0132] ,

[0133] in, This represents the event feature vector at pixel (x, y). For event feature dimensions;

[0134] At the same time, the depth map of dilute sulfur Its effectiveness mask Input together into the deep coding network The depth-side feature map is obtained, and its expression is:

[0135] ,

[0136] in, For depth feature vectors, For depth features;

[0137] At the pixel level, feature-level fusion is performed by concatenating event features and deep features, and then obtaining the fused features through linear mapping or an attention module. The expression is as follows:

[0138] ,

[0139] in, This serves as a guiding feature for the fused 3D reconstruction. For learnable feature fusion weight matrix, To fuse feature dimensions;

[0140] The fused features combine the high-resolution edge and texture information provided by event images with the realistic geometric constraints provided by point cloud depth, offering high-quality priors for subsequent depth completion and point cloud densification for local fire regions. Similar event-LiDAR fusion depth estimation and completion frameworks have been shown to significantly improve depth accuracy in highly dynamic environments, and based on this, a scenario-specific design was developed for small-scale features of wildfire flames.

[0141] Building a deep completion decoding network ,by sparse depth and mask Given the input, predict the entire image domain. Dense depth on The expression is:

[0142] ,

[0143] in, For pixels The estimated depth at that location, The parameter set for deep completion networks;

[0144] Introduce a training loss based on sparse depth consistency and edge-guided smoothness, let:

[0145] ,

[0146] For a set of pixels with true depth observations, the sparse depth consistency loss can be written as:

[0147] ,

[0148] in, For set The number of pixels;

[0149] The smooth term guided by event edges is expressed as follows:

[0150] ,

[0151] in, , Let x and y represent the estimated gradients of depth in the x and y directions, respectively. , To normalize the gradient in the corresponding direction of the event image, The hyperparameter for controlling the degree of edge weight decay, For the exponent term, This weakens the depth smoothing constraint at locations with strong responses at the event edges, thus preserving true depth abrupt changes such as the flame outline. The above two losses are combined, and a regularization term can be optionally added. The overall training objective is as follows:

[0152] ,

[0153] Where β and η are non-negative weighting coefficients;

[0154] By minimizing on a large number of "event image - sparse point cloud" samples It can train a robust depth completion model in early wildfire scenarios, generating a dense depth map with both realistic metric scale and clear boundaries. The idea of ​​combining depth completion with image guidance has been fully validated in the field of LiDAR-camera fusion. Based on this, high dynamic event features are introduced for 3D reconstruction of small-scale and high-contrast boundaries of flames.

[0155] The dense 3D reconstruction of the local fire point region specifically involves constructing a depth completion decoding network, which uses fused features, sparse depth, and a mask as inputs to predict a dense depth map over the entire image domain.

[0156] The specific process involves using the event camera intrinsic parameter matrix. Reconstructing it as a dense 3D point cloud in the event camera coordinate system C, for each pixel (x, y) in the image domain, according to the pinhole camera model, we can obtain:

[0157] ,

[0158] in, Let (x, y) be the coordinate vector of the 3D point corresponding to pixel (x, y) in the event camera coordinate system. This is the inverse of the event camera intrinsic parameter matrix;

[0159] For each point in the three-dimensional point cloud space Building a neighborhood The point positions are fine-tuned through local plane fitting or neighborhood mean smoothing, as expressed in the following expression:

[0160] ,

[0161] in, The position of the smoothed 3D point. For point The neighborhood index set, The number of neighboring points. This is the smoothing step size coefficient.

[0162] Through one or more iterations of smoothing, local burrs and isolated pseudo-points can be eliminated while maintaining the overall structure and boundary morphology of the flame, ultimately obtaining coverage of the flame volume, fire line boundary, and adjacent vegetation. The dense 3D point cloud of local fire points on the ground provides a fine 3D geometric basis for subsequent fire point feature recognition and world coordinate ranging and positioning based on the dense point cloud.

[0163] After completing the reconstruction of the dense 3D point cloud of the fire point, the automatic identification and 3D ranging and positioning of the fire point area are achieved without relying on manual annotation. The specific process of automatic identification of the fire point area is as follows:

[0164] Drawing upon the concept of local density deviation, we perform 3D fire point anomaly identification. Let the reconstructed local dense point cloud of the fire point be represented in the event camera coordinate system C as follows:

[0165] ,

[0166] in, For the first The coordinates of a 3D point in the event camera coordinate system The number of points in a dense point cloud;

[0167] Based on the aforementioned fusion feature extraction results, a feature vector containing geometric and event information is constructed, expressed as follows:

[0168] ,

[0169] in, Point Local geometric features, This represents the intensity or characteristics of an event obtained by projecting an event image. This represents the total feature dimension.

[0170] In the unsupervised anomaly detection stage, this invention uses feature sets As input, estimate the local density of each point in the 3D feature space and calculate the anomaly score.

[0171] Based on Taking the local density deviation metric of nearest neighbors as an example, the points are recorded. The K-nearest neighbor set is Let the distance be Euclidean, then the first... From point to its first The expression for the nearest neighbor distance is:

[0172] ,

[0173] Based on this, its local reachability distance is defined as follows:

[0174] ,

[0175] Based on this, points can be estimated. The formula for calculating the locally accessible density is:

[0176] ,

[0177] in This represents the number of nearest neighbors.

[0178] To measure points Based on the density deviation of its neighborhood, and drawing inspiration from the concept of local anomaly factors, the expression for its anomaly score is defined as follows:

[0179] ,

[0180] when When the local density of a point is similar to its nearest neighbors, it can be considered a "normal background point". When the value is significantly greater than 1, it indicates that the local density at the location of the point is significantly lower than that of the neighborhood, and it belongs to the "locally sparse and abrupt" anomaly point, which can be regarded as a fire point or flame boundary candidate.

[0181] Based on the above anomaly scoring, this invention automatically extracts three-dimensional anomaly regions of fire points in dense point clouds through threshold segmentation and spatial connectivity analysis. Let the anomaly threshold be... Then the initial set of outliers can be defined as:

[0182] ,

[0183] And in The above process performs three-dimensional connected component segmentation and clustering, resulting in several spatially independent candidate fire point clusters. For each candidate fire point cluster... Its geometric center or highest point is taken as the representative point of the fire point, and its three-dimensional position is calculated as follows:

[0184] ,

[0185] in, Let be the number of points in the q-th fire cluster. That is, the three-dimensional representative position of the fire point in the event camera coordinate system.

[0186] A second aspect of this invention discloses a computer device comprising a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface is used for wired or wireless communication with an external terminal; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a file synchronization method between a terminal device and a carrier module. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0187] A third aspect of this invention discloses a computer-readable storage medium having stored thereon computer program instructions, including:

[0188] A method for implementing any of the foregoing embodiments when computer program instructions are executed by a processor. The present invention can be a method, apparatus, system, and / or computer program product, which may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the present invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0189] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds, characterized in that, The specific steps include the following: A joint observation system consisting of a high-resolution event camera and a 3D sparse point cloud sensor is constructed. The two types of sensors are fixedly installed on watchtowers, drones, or vehicle-mounted platforms, and collaborative acquisition is achieved through a unified trigger and time synchronization module. Intensity normalization and noise suppression are performed on each frame of event image acquired. An unsupervised learning framework is used to spatially register event camera images that have undergone intensity normalization and noise suppression with sparse point clouds. Under the condition of lacking pixel-to-3D point ground truth correspondence, the extrinsic parameters between the two sensors are automatically solved and pixel-to-3D point correspondence is generated. Using the extrinsic parameters obtained from training, the newly acquired point cloud is mapped to the event camera coordinate system and projected onto the event image plane to form a registered set of projection points. In the event camera coordinate system, a depth map is constructed and cross-modal feature fusion is performed on the sparse point cloud, and a dense 3D reconstruction of the local area of ​​the fire point is achieved on the basis of the constructed and fused features. After completing the reconstruction of the dense 3D point cloud of the fire point, the automatic identification and 3D ranging and positioning of the fire point area are achieved without relying on manual annotation.

2. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The process of intensity normalization and noise suppression for each acquired event image frame specifically involves performing intensity normalization and noise suppression on each event image frame. Calculate its value in the image domain mean with standard deviation And standardize it: ， in, The event image after normalization; ， in, The corresponding standard deviation, The total number of pixels in the image. To prevent tiny positive numbers with a denominator of zero.

3. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The specific steps for spatial registration of event camera images and sparse point clouds using an unsupervised learning framework are as follows: Step 1: Given the current extrinsic parameters In this case, the point cloud sensor coordinate system The three-dimensional points below By mapping the rigid body transformation onto the event camera coordinate system C, we obtain: ， in, For rotation matrix, It is a translation vector. It is the first The three-dimensional coordinates of a point in the event camera coordinate system; Step 2: Utilize the intrinsic parameter matrix of the event camera can Projecting onto the event image plane, we obtain the pixel coordinates, expressed as: ， in, For the first The pixel position of each point in the event image coordinate system. For depth Relevant scaling factors; From normalized event images Extract structural features to obtain gradient-based edge response maps. The expression is: ， in These are the pixel coordinates in the event image. and They represent in and Gradient in direction; Step 3: Construct the distance transform or cost field on the event image The distance from each pixel to the nearest structural edge is encoded as a scalar to measure the spatial deviation between the projected point and the real structure. Based on this, the structural consistency loss of a single frame is defined as follows: ， in, Indicates the relationship with the first The number of valid projection points in the point cloud of frame event image matching. For projection point The value in the distance field characterizes the distance from the projection point to the nearest event edge; Step 4: Record the point at the [missing information] . The projection in the frame is In the The projection in the frame is The expression for the multi-frame reprojection consistency loss is: ， in, This represents the number of static points that can establish a correspondence between two frames. Represents the L2 norm; Step 5: Construct the overall loss function for unsupervised spatial registration of event camera images and sparse point clouds, calculated as follows: ， Where α, β, γ, and η are non-negative weighting coefficients. For structural consistency loss, For multi-frame reprojection consistency loss, For distributed / mutual information loss, This is the regularization term for the external parameters.

4. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The specific process of constructing the depth map is as follows: using the obtained extrinsic parameters ( , ) The intrinsic parameter matrix K represents the three-dimensional points in the point cloud sensor coordinate system L. Projecting the pixel coordinates onto the event image plane of the k-th frame yields the pixel coordinates. and its depth in the camera coordinate system In the event image domain Within, a sparse depth map is constructed using pixel coordinates as indices. With effectiveness mask The expression is: 。 5. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The cross-modal feature fusion employs a dual-branch coding structure to extract event images and sparse depth features respectively, and performs feature-level fusion at the pixel level to obtain 3D reconstruction guidance features.

6. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The dense 3D reconstruction of the local fire point region specifically involves constructing a depth completion decoding network, which uses fused features, sparse depth, and a mask as inputs to predict a dense depth map over the entire image domain.

7. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The specific process for automatic identification of the fire point area is as follows: Drawing upon the concept of local density deviation, we perform 3D fire point anomaly identification. Let the reconstructed local dense point cloud of the fire point be represented in the event camera coordinate system C as follows: ， in, For the first The coordinates of a 3D point in the event camera coordinate system The number of points in a dense point cloud; Based on the aforementioned fusion feature extraction results, a feature vector containing geometric and event information is constructed, expressed as follows: ， in, Point Local geometric features, This represents the intensity or characteristics of an event obtained by projecting an event image. This represents the total feature dimension.

8. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 1, characterized in that, The unsupervised anomaly detection framework described in S5 specifically uses a feature set. As input, estimate the local density of each point in the 3D feature space and calculate anomaly scores. Taking the K-nearest neighbor-based local density deviation metric as an example, record the points... The K-nearest neighbor set is Let the distance be Euclidean, then the first... The expression for the distance from a point to its Kth nearest neighbor is: ， Its local reachability is defined as follows: ， Estimated point The locally reachable density, expressed as: ， in The number of nearest neighbors is used to measure the number of points. The degree of density deviation relative to its neighborhood; Define anomaly scoring as follows: 。 9. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 8, characterized in that, When the abnormal score When the local density of a point is similar to its nearest neighbors, it can be considered a "normal background point". When the value is significantly greater than 1, it indicates that the local density at the location of the point is significantly lower than that of the neighborhood, and it belongs to the "locally sparse and abrupt" anomaly point, which can be regarded as a fire point or flame boundary candidate.

10. The fire point ranging and localization method based on the fusion of camera images and 3D sparse point clouds according to claim 8, characterized in that, Based on anomaly scoring, a three-dimensional anomaly region of fire points is automatically extracted from dense point clouds through threshold segmentation and spatial connectivity analysis. The anomaly threshold is set as follows: Then the initial set of outliers can be defined as: ， And in Perform three-dimensional connected component segmentation and clustering to obtain several spatially independent candidate fire points. For each candidate fire point... Its geometric center or highest point is taken as the representative point of the fire point, and its three-dimensional position expression is: , in, Let be the number of points in the q-th fire cluster. That is, the three-dimensional representative position of the fire point in the event camera coordinate system.

11. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the fire point ranging and localization method based on the fusion of camera images and three-dimensional sparse point clouds as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the fire point ranging and localization method based on the fusion of camera images and three-dimensional sparse point clouds as described in any one of claims 1 to 10.

Images