A fire scene three-dimensional reconstruction method, device and electronic equipment

Abstract

The application provides a fire scene three-dimensional reconstruction method and device and electronic equipment, and aims to improve the problem of low accuracy of fire scene three-dimensional reconstruction. The method comprises the following steps: extracting feature points from a plurality of to-be-processed images to obtain image feature points of each to-be-processed image; calculating the matching relationship of the image feature points to obtain a fundamental matrix, wherein the fundamental matrix represents the geometric relationship between any two matching images in the plurality of to-be-processed images; performing sparse reconstruction on the matching relationship of the image feature points, the image feature points and the fundamental matrix to obtain a sparse point cloud and camera parameters; converting the relative coordinate system of the sparse point cloud according to the geographical position and the camera parameters when the to-be-processed image is shot to obtain a real scale point cloud in a world coordinate system; and constructing a three-dimensional model of a target fire scene according to the real scale point cloud and the camera parameters.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and image processing, and in particular to a method, device and electronic equipment for three-dimensional reconstruction of a fire scene. Background Art

[0002] Current methods for 3D reconstruction of forest fire scenes typically use LiDAR and multi-view imaging. However, LiDAR equipment can only obtain solid measurement information and cannot effectively identify smoke and fire targets, resulting in an inability to reconstruct the flame scene. Multi-view imaging methods, on the other hand, are often ineffective due to the complex backgrounds and geographical environments of fire scenes. For example, traditional multi-view imaging methods are prone to failure in forest fire scenes with weak and repetitive textures due to lighting changes. As a result, current fire scene 3D reconstruction accuracy is low. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a method, device and electronic equipment for three-dimensional reconstruction of a fire scene, so as to improve the problem of low accuracy of three-dimensional reconstruction of a fire scene.

[0004] An embodiment of the present application provides a method for three-dimensional reconstruction of a fire scene, including: extracting feature points from multiple images to be processed to obtain image feature points of each image to be processed, where the images to be processed are obtained by photographing a target fire scene with an aircraft; calculating matching relationships between image feature points and image feature points to obtain a basic matrix, where the basic matrix represents the geometric relationship between any two matching images in the multiple images to be processed; sparsely reconstructing the matching relationships of the image feature points, the image feature points, and the basic matrix to obtain a sparse point cloud and camera parameters; transforming the relative coordinate system of the sparse point cloud according to the geographic location of the image to be processed at the time of shooting and the camera parameters to obtain a real-scale point cloud in a world coordinate system; and constructing a three-dimensional model of the target fire scene based on the real-scale point cloud and the camera parameters. In the implementation process of the above scheme, by extracting feature points of multiple images to be processed and representing the geometric relationship between two matching images based on the basic matrix calculated according to the matching relationship between image feature points and image feature points, the connection between different images can be accurately established. Moreover, the relative coordinate system of the sparse point cloud is converted into the world coordinate system through camera parameters, so that the reconstructed point cloud data can be combined with the actual space, so that the final three-dimensional model can accurately reflect the actual size and geometric relationship of the fire scene, and the images obtained from different perspectives can be accurately aligned and fused in the three-dimensional space, thereby effectively overcoming the interference factors of weak texture and repeated texture such as noise and illumination changes in the image, and effectively improving the accuracy of the three-dimensional reconstruction of the fire scene.

[0005] Optionally, in an embodiment of the present application, feature point extraction is performed on multiple images to be processed, including: for each of the multiple images to be processed, using a pre-trained neural network to extract feature points from the image to be processed to obtain first candidate feature points; using multiple Gaussian kernel standard deviations in a Gaussian difference algorithm to extract the image to be processed to obtain second candidate feature points; and fusing the first candidate feature points and the second candidate feature points to obtain image feature points. In the implementation of the above scheme, the pre-trained neural network can find more representative locations in the image as initial candidate feature points, and the Gaussian difference algorithm further ensures that these points are local extreme points, thereby improving the quality of the final determined image feature points.

[0006] Optionally, in an embodiment of the present application, before calculating the image feature points, the method further includes: estimating the image feature points in each of a plurality of images to be processed to obtain the direction and shape of the image feature points; generating a feature descriptor for the image to be processed based on the direction and shape of the image feature points; and matching the feature points of the plurality of images to be processed based on the feature descriptors of the image to be processed and the image feature points in the image to be processed to obtain a feature point matching relationship. In the implementation of the above scheme, by estimating the direction and shape of the image feature points in each image to be processed, the feature points are not only dependent on the position information, but also contain richer local structural information, which helps to accurately identify and match feature points in complex backgrounds, thereby enhancing the accuracy of feature point matching. Furthermore, by generating feature descriptors based on the direction and shape of the image feature points, a more unique and stable descriptor can be provided. This descriptor has high robustness to changes in illumination, changes in viewing angle, etc., thereby improving the success rate and accuracy of feature point matching between different images.

[0007] Optionally, in an embodiment of the present application, the relative coordinate system of the sparse point cloud is converted according to the camera parameters, including: obtaining the geographical location of the image to be processed when it was shot; converting the relative coordinate system of the sparse point cloud to the world coordinate system according to the geographical location and camera parameters of the image to be processed when it was shot, to obtain a real-scale point cloud. In the implementation process of the above scheme, by converting the relative coordinate system of the sparse point cloud to the world coordinate system, point cloud data directly corresponding to the real-world position can be obtained, so that each point in the point cloud can be accurately mapped to a specific position on the earth, greatly improving the accuracy of spatial positioning. Furthermore, by unifying the images and point cloud data acquired at different times and by different devices into the same world coordinate system, in this world coordinate system, the actual distance between two points can be directly calculated without considering the scale problem of the original image or point cloud, thereby improving the measurement accuracy.

[0008] Optionally, in an embodiment of the present application, a three-dimensional model of the target fire scene is constructed based on the real-scale point cloud and camera parameters, including: using the camera parameters to perform dedistortion processing on multiple images to be processed to obtain dedistorted images; projecting the three-dimensional Gaussian ellipsoid initialized for each point in the sparse point cloud to the image plane corresponding to the dedistorted image to obtain multiple projection points, and the projection points represent the two-dimensional Gaussian distribution on the image plane; performing differentiable rasterization rendering on the real-scale point cloud based on the multiple projection points to obtain a rendered image; and constructing a three-dimensional model of the target fire scene based on the three-dimensional Gaussian ellipsoid, the dedistorted image, and the rendered image. In the implementation process of the above scheme, the conversion from the real-scale point cloud to the high-quality rendered image is achieved by combining camera dedistortion processing, three-dimensional Gaussian ellipsoid projection, and differentiable rasterization rendering. Specifically, the multiple images to be processed are dedistorted to eliminate the geometric distortion caused by lens distortion, thereby ensuring the authenticity and consistency of the image. Furthermore, by initializing each point in the sparse point cloud as a three-dimensional Gaussian ellipsoid and projecting it onto the dedistorted image plane, the detail information of the original point cloud can be better preserved. Compared with the simple point projection method, the Gaussian ellipsoid can represent the uncertainty and diffusion of the point, making the projection result more natural and smooth, achieving higher rendering quality.

[0009] Optionally, in an embodiment of the present application, a three-dimensional model of the target fire scene is constructed based on a three-dimensional Gaussian ellipsoid, a dedistorted image, and a rendered image, including: determining a rendering depth map based on pixel depth values ​​calculated from the three-dimensional Gaussian ellipsoid, and performing depth estimation on the dedistorted image to obtain a dedistorted depth map; calculating a depth loss value between the rendering depth map and the dedistorted depth map; calculating an image loss value and a structural similarity loss value between the rendered image and the dedistorted image; determining a total loss value based on the depth loss value, the image loss value, and the structural similarity loss value; updating the parameters and point cloud density of the three-dimensional Gaussian ellipsoid based on the total loss value to obtain an updated Gaussian ellipsoid and point cloud; and determining the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene. In the implementation process of the above scheme, by combining multiple loss values ​​of depth loss, image loss, and structural similarity loss, the three-dimensional Gaussian ellipsoid parameters and point cloud density are optimized and updated, ensuring the high accuracy of the final three-dimensional model. Furthermore, by dynamically adjusting the parameters of the three-dimensional Gaussian ellipsoid and the point cloud density according to the total loss value, the system can obtain more accurate reconstruction results and maintain high robustness and adaptability even in the face of complex and changing fire scenes.

[0010] An embodiment of the present application also provides a three-dimensional reconstruction device for a fire scene, including: a feature point extraction module, used to extract feature points of multiple images to be processed, and obtain image feature points of each image to be processed, where the images to be processed are obtained by shooting a target fire scene by an aircraft; a basic matrix acquisition module, used to calculate the matching relationship between image feature points and image feature points, and obtain a basic matrix, where the basic matrix represents the geometric relationship between any two matching images in the multiple images to be processed; a point cloud sparse reconstruction module, used to sparsely reconstruct the matching relationship of image feature points, image feature points and basic matrix, and obtain a sparse point cloud and camera parameters; a coordinate system conversion module, used to convert the relative coordinate system of the sparse point cloud according to the camera parameters, and obtain a real-scale point cloud in the world coordinate system; and a model three-dimensional reconstruction module, used to construct a three-dimensional model of the target fire scene based on the real-scale point cloud and camera parameters.

[0011] Optionally, in an embodiment of the present application, feature point extraction is performed on multiple images to be processed, including: for each image to be processed in the multiple images to be processed, using a pre-trained neural network to extract feature points of the image to be processed to obtain first candidate feature points; using multiple Gaussian kernel standard deviations in a Gaussian difference algorithm to extract the image to be processed to obtain second candidate feature points; and fusing the first candidate feature points and the second candidate feature points to obtain image feature points.

[0012] Optionally, in an embodiment of the present application, the train three-dimensional reconstruction device also includes: a direction shape estimation submodule, which is used to estimate the image feature points in each of the multiple images to be processed, and obtain the direction and shape of the image feature points; a feature descriptor generation submodule, which is used to generate a feature descriptor of the image to be processed according to the direction and shape of the image feature points; and an image feature point matching submodule, which is used to perform feature point matching on multiple images to be processed based on the feature descriptors of the image to be processed and the image feature points in the image to be processed, and obtain a feature point matching relationship.

[0013] Optionally, in an embodiment of the present application, the coordinate system conversion module includes: a geographic location acquisition submodule, used to obtain the geographic location of the image to be processed when it was shot; a real scale acquisition submodule, used to convert the relative coordinate system of the sparse point cloud into a world coordinate system according to the geographic location and camera parameters of the image to be processed when it was shot, to obtain a real scale point cloud.

[0014] Optionally, in an embodiment of the present application, the three-dimensional model construction module includes: an image dedistortion processing submodule, which is used to use camera parameters to perform dedistortion processing on multiple images to be processed to obtain dedistorted images; an image plane projection submodule, which is used to project the three-dimensional Gaussian ellipsoid initialized for each point in the sparse point cloud to the image plane corresponding to the dedistorted image to obtain multiple projection points, and the projection points represent the two-dimensional Gaussian distribution on the image plane; a differentiable rasterization rendering submodule, which is used to perform differentiable rasterization rendering on the real-scale point cloud according to the multiple projection points to obtain a rendered image; and a three-dimensional model of the target fire scene is constructed based on the three-dimensional Gaussian ellipsoid, the dedistorted image and the rendered image.

[0015] Optionally, in an embodiment of the present application, the model three-dimensional reconstruction module includes: a rendering depth map determination submodule, which is used to determine the rendering depth map based on the pixel depth value calculated by the three-dimensional Gaussian ellipsoid, and perform depth estimation on the dedistorted image to obtain the dedistorted depth map; a depth loss value calculation submodule, which is used to calculate the depth loss value between the rendering depth map and the dedistorted depth map; an image loss value calculation submodule, which is used to calculate the image loss value and the structural similarity loss value between the rendered image and the dedistorted image; a total loss value calculation submodule, which is used to determine the total loss value based on the depth loss value, the image loss value and the structural similarity loss value; a Gaussian ellipsoid update submodule, which is used to update the parameters of the three-dimensional Gaussian ellipsoid and the point cloud density based on the total loss value to obtain an updated Gaussian ellipsoid and point cloud; and a three-dimensional model determination submodule, which is used to determine the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene.

[0016] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0017] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0018] An embodiment of the present application further provides a computer program product, including: a computer program or computer instructions, which executes the method described above when the computer program or computer instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only illustrate certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic flow chart of a fire scene 3D reconstruction method provided by an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of converting a relative coordinate system of a sparse point cloud into a world coordinate system according to an embodiment of the present application is shown;

[0022] Figure 3 A schematic structural diagram of a fire scene 3D reconstruction device provided in an embodiment of the present application is shown;

[0023] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the embodiments of the present application only serve the purpose of illustration and description and are not intended to limit the scope of protection of the embodiments of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the embodiments of the present application illustrate the operations implemented according to some embodiments of the embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the contents of the embodiments of the present application, can add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0025] In addition, the described embodiments are only a portion of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application generally described and shown in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but rather merely represents selected embodiments of the embodiments of the present application.

[0026] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and subsequent associated objects are in an "or" relationship. The term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two (including two).

[0027] It should be noted that the fire scene 3D reconstruction method provided in the embodiments of this application can be executed by electronic devices. Electronic devices herein refer to device terminals or servers capable of executing computer programs. Examples of device terminals include smartphones, personal computers, tablet computers, personal digital assistants, or mobile internet devices. Servers refer to devices that provide computing services over a network. Examples of servers include x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0028] In the prior art, 3D reconstruction of forest fire scenes primarily relies on laser detection and ranging (LiDAR) technology and multi-view imaging. However, these approaches each have significant limitations when dealing with complex fire environments. For example, LiDAR equipment can accurately acquire the 3D coordinates of solid objects, but struggles to effectively identify and measure non-solid objects such as smoke and flames. Due to limitations in its operating principle, LiDAR cannot capture the dynamic changes of flames, and therefore cannot reconstruct the complete fire scene, including its shape and spread. Multi-view image-based approaches are difficult due to the complex background and geographical environment of fire scenes. For example, forest fires often feature complex backgrounds and geographical features, such as dense trees, uneven terrain, and extensive vegetation cover, which increases the difficulty of image processing. Fire environments often have unstable lighting conditions, potentially with strong contrasts between light and dark or shadows, placing higher demands on image-based reconstruction algorithms. Furthermore, repetitive textures in forests (such as similar tree trunks and leaves) and weakly textured areas (such as large expanses of sky or grass) can easily cause traditional multi-view image reconstruction methods to fail, making feature point detection and matching difficult, and thus affecting reconstruction accuracy. Therefore, although lidar and multi-view image reconstruction are two commonly used fire scene 3D reconstruction technologies, their respective limitations limit the accuracy of the reconstruction results.

[0029] For the above questions, please see Figure 1The flowchart of the fire scene 3D reconstruction method provided by the embodiment of the present application is shown; the idea of ​​the fire scene 3D reconstruction method is to convert the relative coordinate system of the sparse point cloud into the world coordinate system based on the geographical location when the image to be processed was captured, so that the reconstructed point cloud data is combined with the actual geographic space, so that the final 3D model can accurately reflect the geometric relationship of the real size and position of the fire scene, so that images acquired from different perspectives can be accurately aligned and fused in 3D space to overcome interference factors such as weak texture and repeated texture in the image, such as noise and lighting changes, thereby effectively improving the accuracy of the fire scene 3D reconstruction. The implementation method of the above fire scene 3D reconstruction method may include:

[0030] Step S110: extracting feature points from a plurality of images to be processed to obtain image feature points of each image to be processed, where the images to be processed are obtained by photographing the target fire scene with an aircraft.

[0031] It is understandable that when the aircraft photographs the target fire scene, it can store the image to be processed and also needs to store the geographic location corresponding to the image, so as to convert the relative coordinate system of the sparse point cloud to the geographic location when the image to be processed was photographed into the world coordinate system. Among them, the above-mentioned aircraft can be a drone that photographs the target fire scene, or it can be a manned aircraft, and the people on the manned aircraft use cameras to manually photograph. In some scenarios, if the burning area of ​​the target fire scene is very large, the above-mentioned aircraft can also use low-Earth satellites, which can photograph the target fire scene to obtain images for three-dimensional reconstruction.

[0032] Step S120: Calculate the matching relationship between image feature points to obtain a basic matrix.

[0033] The fundamental matrix (Fundamental Matrix) is a matrix that represents the geometric relationship between any two matching images among multiple images to be processed, providing a solid mathematical foundation for subsequent matching and reconstruction. It is understandable that the two matching images above are also called image matching pairs.

[0034] Step S130: performing sparse reconstruction on the matching relationship of the image feature points, the image feature points and the basic matrix to obtain a sparse point cloud and camera parameters.

[0035] It's understandable that, in practice, incremental reconstruction methods (such as pycolmap) based on the matching relationships between image feature points and fundamental matrices can be used to quickly generate a sparse point cloud that represents the basic structure of the target fire scene. Compared to traditional global reconstruction methods, this incremental reconstruction method achieves a better balance between real-time performance and accuracy when processing large numbers of images.

[0036] Step S140: transforming the relative coordinate system of the sparse point cloud according to the camera parameters to obtain a true scale point cloud in the world coordinate system.

[0037] Step S150: constructing a three-dimensional model of the target fire scene based on the real-scale point cloud and camera parameters.

[0038] In the implementation process of the above scheme, by extracting feature points of multiple images to be processed and representing the geometric relationship between two matching images based on the basic matrix calculated according to the matching relationship between image feature points and image feature points, the connection between different images can be accurately established. Moreover, the relative coordinate system of the sparse point cloud is converted into the world coordinate system through camera parameters, so that the reconstructed point cloud data can be combined with the actual space, so that the final three-dimensional model can accurately reflect the actual size and geometric relationship of the fire scene, and the images obtained from different perspectives can be accurately aligned and fused in the three-dimensional space, thereby effectively overcoming the interference factors of weak texture and repeated texture such as noise and illumination changes in the image, and effectively improving the accuracy of the three-dimensional reconstruction of the fire scene.

[0039] Optionally, as an alternative implementation of the above-mentioned 3D fire scene reconstruction method, before extracting feature points from the multiple images to be processed, similarity matching can be performed on the images to facilitate finding indexes between image pairs. Feature point extraction can then be performed based on the indexed similar image pairs, thereby improving feature point extraction efficiency. For example, a determination is made as to whether the number of images to be processed is less than a preset maximum exhaustive matching number (exhaustive_if_less). If so, an exhaustive search algorithm is used to search for all similar image pairs from the multiple images to be processed. Otherwise, the DINOv2 model is used to perform similarity matching on the multiple images to obtain multiple similar image pairs, thereby improving feature point extraction efficiency. The DINOv2 model is a self-supervised deep learning model that has performed well on multiple image classification datasets and can learn rich visual feature representations from large amounts of unlabeled data. In complex environments such as forest fires, the DINOv2 model can capture deeper semantic information and maintain high matching accuracy and stability even in the face of lighting changes, smoke occlusion, or weak texture areas. This makes it more adaptable to complex natural environments than traditional methods based on hand-crafted features.

[0040] Step S101: Use the trained DINOv2 model to perform image embedding on multiple images to be processed to obtain multiple image embedding vectors.

[0041] An example implementation of step S101 is as follows: initializing an embedding vector list to obtain an initialized embedding vector list; traversing multiple images to be processed according to the path taken by the aircraft when photographing the target fire scene to obtain multiple traversed images; and performing standardized image embedding (Image Embedding) on ​​the multiple traversed images using the trained DINOv2 model to obtain multiple image embedding vectors. It is understood that the DINOv2 model includes an input layer, a hidden layer, a maximum pooling layer, and a normalized output layer. After processing the traversed images using the input layer to obtain an input feature map, the hidden layer is used to extract features from the input feature map to obtain a hidden feature map. Then, image embedding is performed on the hidden feature map using the maximum pooling layer and the normalized output layer, and the obtained image embedding vector is stored in the initialized embedding vector list. By utilizing the pre-trained DINOv2 model to generate high-dimensional embedding vectors and quickly screening similar image pairs based on vector distance, the computational complexity and time cost are significantly reduced. This method performs particularly well when processing large-scale image data, making it suitable for the processing needs of fire scene images collected in real time by aircraft.

[0042] Step S102: For each two image embedding vectors in the plurality of image embedding vectors as an embedding vector pair, a vector distance between each embedding vector pair is calculated.

[0043] As you can see, calculating the vector distance between image embeddings allows for rapid screening of similar image pairs without exhaustively searching all possible image combinations. This reduces computational effort and time, particularly when processing large amounts of image data. For fire scene images captured in real time by aircraft, rapid response and efficient processing are crucial for decision support in emergency situations.

[0044] Step S103: Determine whether the images corresponding to the two image embedding vectors are a similar image pair based on the vector distance.

[0045] For example, the first implementation of steps 102 to S103 is as follows: multiple image embedding vectors are read from the above-initialized embedding vector list, and for each two image embedding vectors in the multiple image embedding vectors, the vector distance between each embedding vector pair is calculated to obtain multiple vector distances. From the multiple vector distances, a vector distance greater than a preset distance threshold is screened out, and the two to-be-processed images corresponding to the vector distance are determined as the above-mentioned similar image pair. Optionally, the above-mentioned preset distance threshold can be dynamically adjusted according to the specific application scenario. For example, in different types of forest environments (such as coniferous forests and broad-leaved forests), or under different weather conditions (sunny and cloudy days), the system can automatically optimize the preset distance threshold setting according to the actual effect, thereby further improving the flexibility and accuracy of matching.

[0046] For example, the second implementation of the above steps 102 to S103 is as follows: multiple image embedding vectors are read from the above-initialized embedding vector list, and for each two image embedding vectors in the multiple image embedding vectors as an embedding vector pair, the vector distance between each embedding vector pair is calculated, and the vector distance is determined as the similarity between the embedding vector pairs. The embedding vector pairs with a similarity lower than the similarity threshold (Similarity_Threshold) are screened out from the multiple embedding vector pairs, and the two to-be-processed images corresponding to the embedding vector pair are determined as similar image pairs. In the implementation process of the above scheme, by using the high-dimensional embedding vector generated by DINOv2, the similarity between images can be measured more accurately. Using DINOv2, completely irrelevant image pairs can be filtered out, thereby saving the time of feature point matching and effectively improving the efficiency of three-dimensional reconstruction of the fire scene.

[0047] As an optional implementation of the above step S110, the above implementation of extracting feature points from multiple images to be processed may include:

[0048] Step S111: for each of the multiple images to be processed, extract feature points of the image to be processed using a pre-trained neural network to obtain multiple first candidate feature points.

[0049] An example of an implementation of step S111 is as follows: the pre-trained neural network may employ a convolutional neural network, such as a keynet neural network. Unlike traditional methods (such as SIFT and SURF), Keynet does not require manual definition of feature descriptors. The pre-trained weight parameters are first loaded into the keynet neural network to obtain a keynet model. Then, for each of a plurality of images to be processed, a convolutional neural network model, such as a keynet model, may be used to extract feature points from the image to obtain a plurality of first candidate feature points. The design of the Keynet model takes computational efficiency into consideration, minimizing the consumption of computing resources while ensuring performance, and is therefore suitable for use in real-time processing systems, such as processing forest fire images captured by aircraft.

[0050] Step S112: extracting feature points from the image to be processed using multiple Gaussian kernel standard deviations in a Difference of Gaussian algorithm to obtain second candidate feature points.

[0051] The implementation of step S112 is, for example: a series of Gaussian kernels of different scales are selected for the image to be processed, each Gaussian kernel is defined by its standard deviation σ, a series of gradually increasing σ values are usually selected to cover different scales. For example, σ = 1.6, 1.6xk, 1.6xk2,... can be selected, where k is the ratio between adjacent scales (usually 2^(1 / s), s is the number of generated scale layers, which can range from 1 to s, and 2 gradually increases). For each selected scale (i.e. each σ value), a Gaussian filter is applied to the original image to be processed, resulting in a series of images with different degrees of blur, which constitutes the scale space of the image. In the constructed scale space, for each pair of adjacent scale images, a difference operation is performed, i.e. the image at the larger scale is subtracted from the image at the smaller scale, resulting in a series of DoG images, which highlight structural features such as edges and corners at different scales. Then, local extreme points are searched in all DoG images, if a point is the maximum or minimum value in its 3x3 neighborhood at the current scale and its upper and lower adjacent scales, then this point can be considered as a local extreme point. The relative size of each pixel in its 26 neighbors at its scale and the two adjacent scales above and below can be checked (3x3 neighborhood x 3 layers = 27 points, excluding itself). Since the directly found extreme points may not be exact maximum or minimum values, Taylor expansion can be used to accurately locate the position of the extreme point at a sub-pixel level. Optionally, a contrast threshold is set empirically to remove points with too low contrast, because low-contrast points are often unstable and susceptible to noise. To further filter out high-quality feature points, the Hessian matrix can be used to evaluate whether the candidate point is located in an edge region. Specifically, the Hessian matrix at each candidate point is calculated, and based on the principal curvature ratio, points with strong edge response are removed, and points with higher principal curvature ratio (indicating that the point is more likely to be a smooth corner rather than an edge) are usually retained.

[0052] Step S113: merging the first candidate feature point and the second candidate feature point to obtain an image feature point.

[0053] There are many implementations for step S113, including: First, position-based fusion. For example, if two candidate feature points are very close in position (e.g., the distance between them is less than a certain threshold), they can be considered to represent the same actual feature point. In this case, one of the feature points (typically the one with the higher response value) can be retained, or their positions can be averaged to form the new feature point location. Second, descriptor-based fusion. These feature points are typically accompanied by their descriptors (e.g., SIFT, SURF, etc.). If the descriptors of two candidate feature points are highly similar (exceeding a certain threshold), they can be considered a match. In this case, the descriptors of the two feature points can be combined using some method (e.g., weighted averaging) to obtain a more robust descriptor. Third, multi-scale fusion. Feature points extracted at different scales may be sensitive to different image details. One strategy is to retain feature points at all scales but consider their scale information when comparing or matching. Another strategy is to select feature points at a specific scale based on application requirements or to combine information from different scales in some way.

[0054] As an optional implementation of the above step S120, before calculating the matching relationship between image feature points, the feature point matching relationship may be determined first. This implementation may include:

[0055] Step S121: for each image to be processed among the multiple images to be processed, estimate the image feature points in the image to be processed to obtain the direction and shape of the image feature points.

[0056] An implementation example of the above-mentioned step S121 is: using an Orinet neural network to estimate the image feature points in each of a plurality of images to be processed, obtaining the direction of the image feature points, and using an Affnet neural network to estimate the local affine (LAF) transformation matrix of the image feature points. This LAF transformation matrix contains the shape of the image feature points (such as the affine shape of the surrounding local area), which is used for extraction and matching of subsequent feature descriptors to enhance feature expression capabilities.

[0057] Step S122: generating a feature descriptor of the image to be processed according to the direction and shape of the image feature points.

[0058] The implementation of the step S122 is, for example, that for any image feature point in the image to be processed, a local image block of a preset size (e.g., 32*32 pixels) is extracted from the image to be processed using hardnet, and the region of the local image block is processed using a convolutional neural network, so as to learn the local description of the image feature point, obtain a high-dimensional vector, and determine the high-dimensional vector as the feature descriptor of the image to be processed.

[0059] The step S123 is to perform feature point matching on the plurality of images to be processed according to the feature descriptor of the image to be processed and the image feature points in the image to be processed, and obtain a feature point matching relationship.

[0060] The implementation of the step S123 is, for example, that the AdaLAM feature point matching algorithm is used to calculate the descriptor distance between the feature descriptors of different two images according to the feature descriptor of the image to be processed and the local affine (LAF) transformation matrix of the image feature points in the image to be processed, and then the feature point matching is performed on the plurality of images to be processed according to the descriptor distance, and the feature point matching relationship is obtained. Specifically, it is determined whether the descriptor distance between the feature descriptors of the two images is less than a distance threshold value. If yes, the similar matching relationship of the two images is determined as reliable matching, which means that the overall similarity between the image feature points of the two images is high. Otherwise, the similar matching relationship of the two images is determined as unreliable matching.

[0061] Optionally, in the case that the sparse view angle fire scene has a small number of feature point matches, which leads to unreliable matching, one of the two images can be rotated by a preset angle (e.g., 90°, 180°, 270°, etc.) to obtain a rotated image, and the feature point matching is performed between the rotated image and the unrotated image of the two images to obtain a rotation matching relationship. Finally, the feature point matching relationship above and the rotation matching relationship here are fused.

[0062] It is understood that in practice, the Random Sample Consensus (RANSAC) algorithm or the MAGSAC algorithm can be used to calculate the matching relationship between image feature points and obtain the fundamental matrix. The MAGSAC algorithm is an improved RANSAC algorithm that uses maximum likelihood estimation to improve robustness to noise. It can be implemented by calling the OpenCV code cv2.findFundmentalMat(cv2.USAC_MAGSAC) . By using the Random Sample Consensus (RANSAC) algorithm or the MAGSAC algorithm to calculate the fundamental matrix, robustness to noisy data is increased, the number of false matches is reduced, and the accuracy of the subsequent reconstruction process is ensured.

[0063] Optionally, as an optional implementation of step S130 above, the PyCOLMAP library can be used to perform incremental sparse reconstruction. For example, a new database is first created, and then the matching relationship of image feature points, image feature points, and basic matrix are added to the database. The Structure From Motion (SFM) method function in the PyCOLMAP library can be called to perform incremental sparse reconstruction on the matching relationship of image feature points, image feature points, and basic matrix in the database to obtain a sparse point cloud and camera parameters of the sparse point cloud during the sparse reconstruction process, as well as the pose information of the image to be processed. In the incremental sparse reconstruction process, it is possible to start with a pair of random images and gradually expand by registering new images and triangulating new points. Then, the two incrementally reconstructed models are screened and the best reconstructed model is selected as the preliminary sparse reconstruction result. 3D points whose reprojection error is greater than a preset error threshold can also be removed from the preliminary sparse reconstruction result to obtain the final sparse reconstruction result.

[0064] In the implementation process of the above scheme, compared with the traditional global reconstruction method, the incremental method is more suitable for dynamically changing scenes, such as fire scenes with complex backgrounds. By adopting incremental reconstruction methods (such as pycolmap), the above scheme can maintain high real-time performance when processing a large number of images while generating high-quality sparse point clouds.

[0065] See Figure 2 The schematic diagram of converting the relative coordinate system of the sparse point cloud into the world coordinate system provided by the embodiment of the present application is shown; as an optional implementation of the above step S140, the implementation of converting the relative coordinate system of the sparse point cloud according to the camera parameters may include:

[0066] Step S141: Obtain the geographical position of the to-be-processed image at the time of shooting and the camera parameters of the sparse point cloud in the sparse reconstruction process.

[0067] The implementation of the above step S141 is, for example: It can be understood that the above-mentioned geographical position of the to-be-processed image is recorded and stored in the unmanned aerial vehicle synchronously at the time of shooting the image, which can be stored in the data content of the to-be-processed image, or sent to the electronic device separately. In the process of using the PyCOLMAP library to incrementally reconstruct the similar matching relationship between multiple to-be-processed images, image feature points and fundamental matrices, the PyCOLMAP library will automatically calculate the camera parameters of the to-be-processed image at the time of shooting, so the camera parameters of the sparse point cloud in the sparse reconstruction process can be obtained from the intermediate calculation results of the PyCOLMAP library.

[0068] It can be understood that the high accuracy of point cloud conversion is ensured by the geographical position of the to-be-processed image at the time of shooting, and the geographical position here can be Global Positioning System (GPS) coordinate information. The above-mentioned geographical position can be embedded in the image file for storage, and when the geographical position is needed, the geographical position (such as GPS) can be directly extracted from the image file. Of course, the above-mentioned geographical position can not be embedded in the image file, but the to-be-processed image and its geographical position are stored in the database correspondingly, so that the database saves the correspondence of the two, and when the geographical position is needed, the corresponding geographical position of the to-be-processed image can be directly obtained from the database. The correspondence between the to-be-processed image and the geographical position is crucial for accurate measurement and evaluation of the fire scene, and the conversion of the sparse point cloud from the relative coordinate system to the world coordinate system gives it a real scale, so that the reconstruction result has a measuring significance.

[0069] Step S142: Convert the relative coordinate system of the sparse point cloud to the world coordinate system according to the geographical position of the to-be-processed image at the time of shooting and the camera parameters, and obtain a real scale point cloud.

[0070] The implementation of the above step S142 is, for example: In order to make the three-dimensional model of the fire scene three-dimensional reconstruction have a real scale, so that the reconstructed three-dimensional model has a measuring significance, the relative coordinate system of the sparse point cloud can be converted to the world coordinate system according to the geographical position (such as GPS coordinate information) of the to-be-processed image at the time of shooting and the camera parameters, and a real scale point cloud is obtained. Specifically, assuming that the camera center point of the to-be-processed image at the time of shooting is represented as , and the geographical position of the to-be-processed image at the time of shooting is represented as , then the formula: can be used to convert the relative coordinate system of the sparse point cloud to the world coordinate system to obtain a real scale point cloud. Wherein, They respectively represent the three-dimensional coordinates of the geographical location of the image to be processed when it was taken, represents the scaling factor, represents the scaling matrix, They represent the three-dimensional coordinates of the camera center point of the image to be processed when it was taken, Represent the three-dimensional components in the translation matrix. It can be understood that multiple groups and By substituting the specific values ​​of into the above formula, we can solve for the scaling factor, rotation matrix, and translation matrix. Then, using these scaling factors, rotation matrix, and translation matrix, we can transform the current relative coordinate system of the sparse point cloud into the real world coordinate system, thereby obtaining a true-scale point cloud. After obtaining a true-scale point cloud, we can also measure the width of the target fire, nearby trees, houses, and other scale information.

[0071] As an optional implementation of the above step S150, it may include:

[0072] Step S151: Dedistorting the plurality of images to be processed using camera parameters to obtain dedistorted images.

[0073] For example, in step S151, after obtaining the camera parameters for the sparse point cloud during sparse reconstruction, the camera parameters can be used to dedistort the multiple images to be processed to obtain dedistorted images. This approach eliminates the need for pre-calibration of aircraft camera parameters and allows for flexible scheduling of aircraft used to capture the fire scene based on the actual fire scene conditions, effectively reducing calibration workload and improving the efficiency of 3D fire scene reconstruction.

[0074] Step S152: Project the three-dimensional Gaussian ellipsoid initialized for each point in the sparse point cloud onto the image plane corresponding to the dedistorted image to obtain a plurality of projection points, where the projection points represent the two-dimensional Gaussian distribution on the image plane.

[0075] An example of an implementation of the above step S152 is as follows: after the SFM method function is called above to perform incremental sparse reconstruction to obtain the sparse point cloud and the pose information of the image to be processed, each point in the sparse point cloud can be initialized. The initialization here is a key step, and its purpose is to provide a reasonable starting point for 3D Gaussian sputtering so that it can efficiently optimize and reconstruct the three-dimensional scene. The result of the initialization is to obtain a three-dimensional Gaussian ellipsoid. Then, the three-dimensional Gaussian ellipsoid is projected onto the image plane corresponding to the dedistorted image using the pose information of the image to be processed, and a plurality of projection points can be obtained, wherein each projection point can be represented as a two-dimensional Gaussian distribution on the image plane, and this distribution describes the probability density of the point on the image plane. It can be understood that the pose information of the dedistorted image is the same as the pose information of the image to be processed, because the dedistorted image is obtained by dedistorting a plurality of images to be processed using camera parameters.

[0076] Step S153: performing differentiable raster rendering on the true-scale point cloud according to the multiple projection points to obtain a rendered image.

[0077] An example of an implementation of the above-mentioned step S153 is: each of the above-mentioned multiple projection points can be represented as a two-dimensional Gaussian distribution, which describes the probability density of the point on the image plane, and the real-scale point cloud can be differentiable rasterized according to the multiple projection points through rasterization rendering technology. In this way, the upstream parameters such as point cloud density and Gaussian parameters can be optimized through the back-propagation algorithm to obtain a rendered two-dimensional image containing all projection point information. This image is a visual representation of the original three-dimensional point cloud and can be used for subsequent analysis or comparison with other images. In the specific practice, deep learning frameworks such as PyTorch or TensorFlow can be used to implement differentiable rasterization rendering. These frameworks provide automatic differentiation functions to facilitate gradient calculation and parameter updates. The 3DGaussian Splatting library can also be used to call efficient differentiable rasterization rendering algorithms to simplify the development process.

[0078] Step S154: constructing a three-dimensional model of the target fire scene according to the three-dimensional Gaussian ellipsoid, the dedistorted image and the rendered image.

[0079] As an optional implementation of step S154, in the process of constructing the three-dimensional model of the target fire scene based on the three-dimensional Gaussian ellipsoid, the dedistorted image, and the rendered image, a 3D Gaussian Splatting algorithm may be used to reconstruct the three-dimensional model. The implementation of the algorithm may include:

[0080] Step S154a: determining a rendering depth map according to the pixel depth values ​​calculated by the three-dimensional Gaussian ellipsoid, and performing depth estimation on the dedistorted image to obtain a dedistorted depth map.

[0081] Depth estimation is the process of recovering the geometric structure of a three-dimensional scene from a two-dimensional image. It can be achieved through a variety of methods, including monocular depth estimation, stereo matching, structured light, etc.

[0082] An example implementation of step S154a is as follows: pixel depth values ​​calculated based on a three-dimensional Gaussian ellipsoid, i.e., calculating pixel depth values ​​during rasterization to obtain a rendered depth map, and performing depth estimation on the dedistorted image using a pre-trained depth estimation model to obtain a dedistorted depth map. Pre-trained depth estimation models that can be used include convolutional neural network models such as the MiDaS model, the DPT (Depth Prediction Transformer) model, or the Monodepth2 model.

[0083] Step S154b: Calculate the depth loss value between the rendered depth map and the dedistorted depth map.

[0084] Understandably, given the complex environment surrounding a fire, the need for real-time reconstruction, and the sparse perspective of the input aircraft imagery, the reconstructed 3D model may contain a large number of floating objects. To reduce this floating object, the 3D reconstruction process is constrained from a depth-geometric perspective by calculating the depth loss between the rendered depth map and the dedistorted depth map.

[0085] The implementation of the above step S154b can be, for example: the calculation formula of the depth loss value can be used: , that is, the covariance is divided by the square root of the product of the two variances to calculate the depth loss value between the rendered depth map and the dedistorted depth map. represents the dedistorted depth map, Represents the rendered depth map, Represents the depth loss value between the rendered depth map and the dedistorted depth map, Represents the covariance of the rendered depth map and the dedistorted depth map. represents the variance of the dedistorted depth map, Represents the variance of the rendered depth map. The variance (Var) here is used to standardize the denominator of the covariance (Cov).

[0086] Step S154c: Calculate the image loss value and the structural similarity loss value between the rendered image and the dedistorted image.

[0087] The implementation method of the above step S154c is, for example, to use the formula Calculate the image loss and structural similarity loss between the rendered image and the dedistorted image. Represents the weight coefficient of the image loss value, Represents the weight coefficient of the structural similarity loss value, Represents the rendered image, represents the dedistorted image, Represents the image loss value between the rendered image and the dedistorted image, Represents the structural similarity loss between the rendered image and the dedistorted image.

[0088] Step S154d: Determine a total loss value based on the depth loss value, the image loss value, and the structure similarity loss value.

[0089] The implementation method of the above step S154d is as follows: In the specific practice process, the formula Calculate the depth loss value, image loss value and structural similarity loss value to obtain the total loss value. represents the total loss value, Represents the weight coefficient of the image loss value, Represents the weight coefficient of the structural similarity loss value, Represents the weight coefficient of the depth loss value, Represents the image loss value between the rendered image and the dedistorted image, Represents the structural similarity loss between the rendered image and the dedistorted image, Represents the depth loss between the rendered depth map and the dedistorted depth map.

[0090] Step S154e: Update the parameters of the three-dimensional Gaussian ellipsoid and the point cloud density according to the total loss value to obtain an updated Gaussian ellipsoid and point cloud.

[0091] An example of an implementation of the above-mentioned step S154e is as follows: each three-dimensional Gaussian ellipsoid has several key parameter definitions, such as the center position, covariance matrix, and intensity color information. The back-propagation algorithm can be used to calculate the gradient of the total loss value relative to each Gaussian ellipsoid parameter, and the parameters of these three-dimensional Gaussian ellipsoids and the point cloud density are updated by the gradient descent method or other optimization algorithms (such as the Adam optimizer) to obtain an updated Gaussian ellipsoid and point cloud, and the updated 3D Gaussian ellipsoid and point cloud as well as the camera parameters are saved in a file. The above-mentioned parameter updates can specifically include: position update, shape update, and intensity color information update, wherein the position update is to adjust the center position of the Gaussian ellipsoid in three-dimensional space to better match the actual scene, the shape update is to modify the covariance matrix to change the shape of the Gaussian ellipsoid to make it more consistent with the actual geometric structure of the object, and the intensity color information update refers to updating the intensity or color parameters to ensure that the rendered color is consistent with the real image.

[0092] Step S154f: Determine the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene.

[0093] An implementation example of the above step S154f is: after saving the updated 3D Gaussian ellipsoid, point cloud and camera parameters in a file, the file can be determined as a three-dimensional model file of the target fire scene, and the above three-dimensional model file can be opened by a visualizer that can load Gaussian parameters.

[0094] See Figure 3 FIG2 shows a schematic diagram of the structure of a fire scene 3D reconstruction device provided in an embodiment of the present application. The present application provides a fire scene 3D reconstruction device 200, comprising:

[0095] The feature point extraction module 210 is used to extract feature points from a plurality of images to be processed to obtain image feature points of each image to be processed. The images to be processed are obtained by photographing a target fire scene by an aircraft.

[0096] The basic matrix obtaining module 220 is used to calculate the matching relationship between image feature points to obtain a basic matrix. The basic matrix represents the geometric relationship between any two matching images in the multiple images to be processed.

[0097] The point cloud sparse reconstruction module 230 is used to perform sparse reconstruction on the matching relationship of image feature points, image feature points and basic matrix to obtain sparse point cloud and camera parameters.

[0098] The coordinate system conversion module 240 is used to convert the relative coordinate system of the sparse point cloud according to the camera parameters to obtain the real-scale point cloud in the world coordinate system.

[0099] The model 3D reconstruction module 250 is used to construct a 3D model of the target fire scene based on the real-scale point cloud and camera parameters.

[0100] As an optional implementation of the above device, the feature point extraction module includes:

[0101] The candidate feature point extraction submodule is used to extract feature points from each of the multiple images to be processed using a pre-trained neural network to obtain a first candidate feature point.

[0102] The candidate feature point screening submodule is used to extract the image to be processed using multiple Gaussian kernel standard deviations in the Gaussian difference algorithm to obtain second candidate feature points;

[0103] The candidate feature point fusion submodule is used to fuse the first candidate feature point and the second candidate feature point to obtain the image feature point.

[0104] As an optional embodiment of the above device, the fire scene 3D reconstruction device further includes:

[0105] The direction and shape estimation submodule is used to estimate the image feature points in each of the multiple images to be processed to obtain the direction and shape of the image feature points.

[0106] The feature descriptor generation submodule is used to generate the feature descriptor of the image to be processed according to the direction and shape of the image feature points.

[0107] The image feature point matching submodule is used to perform feature point matching on multiple images to be processed based on the feature descriptors of the images to be processed and the image feature points in the images to be processed to obtain feature point matching relationships.

[0108] As an optional implementation of the above device, the coordinate system conversion module includes:

[0109] The camera parameter acquisition submodule is used to obtain the geographic location of the image to be processed when it was taken and the camera parameters of the sparse point cloud during the sparse reconstruction process.

[0110] The real-scale acquisition submodule is used to convert the relative coordinate system of the sparse point cloud into the world coordinate system according to the geographical location and camera parameters of the image to be processed when it was taken, so as to obtain the real-scale point cloud.

[0111] As an optional embodiment of the above device, the three-dimensional model construction module includes:

[0112] The image dedistortion processing submodule is used to perform dedistortion processing on multiple images to be processed using camera parameters to obtain dedistorted images.

[0113] The image plane projection submodule is used to project the three-dimensional Gaussian ellipsoid initialized for each point in the sparse point cloud onto the image plane corresponding to the dedistorted image to obtain multiple projection points, which represent the two-dimensional Gaussian distribution on the image plane.

[0114] A differentiable rasterization rendering submodule is used to perform differentiable rasterization rendering on the real-scale point cloud based on multiple projection points to obtain a rendered image;

[0115] The three-dimensional model construction submodule is used to construct a three-dimensional model of the target fire scene based on the three-dimensional Gaussian ellipsoid, the dedistorted image and the rendered image.

[0116] As an optional implementation of the above device, the three-dimensional model construction submodule includes:

[0117] The image depth estimation submodule is used to determine the rendering depth map based on the pixel depth value calculated by the three-dimensional Gaussian ellipsoid, and perform depth estimation on the dedistorted image to obtain the dedistorted depth map.

[0118] The depth loss value calculation submodule is used to calculate the depth loss value between the rendered depth map and the dedistorted depth map.

[0119] The image loss value calculation submodule is used to calculate the image loss value and structural similarity loss value between the rendered image and the dedistorted image.

[0120] The total loss value calculation submodule is used to determine the total loss value based on the depth loss value, the image loss value and the structural similarity loss value.

[0121] The Gaussian ellipsoid update submodule is used to update the parameters of the three-dimensional Gaussian ellipsoid and the point cloud density according to the total loss value to obtain the updated Gaussian ellipsoid and point cloud.

[0122] The three-dimensional model determination submodule is used to determine the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene.

[0123] It should be understood that this device corresponds to the aforementioned embodiment of the fire scene 3D reconstruction method and is capable of executing each of the steps involved in the aforementioned method embodiment. The specific functions of this device can be found in the description above, and a detailed description is omitted here. The device includes at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system (OS).

[0124] See Figure 4The electronic device 300 provided in the embodiment of the present application includes a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0125] The embodiment of the present application further provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program executes the above method when executed by the processor 310. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0126] An embodiment of the present application further provides a computer program product, including: a computer program or computer instructions, which executes the method described above when the computer program or computer instructions are executed by a processor.

[0127] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0128] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly based on the functions involved.

[0129] In addition, the functional modules of each embodiment in the embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in an appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0130] The above description is only an optional implementation method of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A fire scene 3D reconstruction method, characterized in that: include: Extracting feature points from a plurality of images to be processed to obtain image feature points of each of the images to be processed, wherein the images to be processed are obtained by photographing a target fire scene with an aircraft; Calculating the matching relationship between the image feature points and the image feature points to obtain a basic matrix, where the basic matrix represents the geometric relationship between any two matching images in the plurality of images to be processed; Performing sparse reconstruction on the matching relationship of the image feature points, the image feature points, and the basic matrix to obtain a sparse point cloud and camera parameters; Converting the relative coordinate system of the sparse point cloud according to the camera parameters to obtain a true-scale point cloud in a world coordinate system; Constructing a three-dimensional model of the target fire scene according to the real-scale point cloud and the camera parameters; The method comprises the following steps: constructing a three-dimensional model of the target fire scene according to the real-scale point cloud and the camera parameters, performing dedistortion processing on the multiple images to be processed using the camera parameters to obtain dedistorted images; projecting a three-dimensional Gaussian ellipsoid initialized from each point in the sparse point cloud onto an image plane corresponding to the dedistorted image to obtain multiple projection points, wherein the projection points represent a two-dimensional Gaussian distribution on the image plane; performing differentiable rasterization rendering on the real-scale point cloud according to the multiple projection points to obtain a rendered image; and constructing a three-dimensional model of the target fire scene according to the three-dimensional Gaussian ellipsoid, the dedistorted image, and the rendered image. The three-dimensional model of the target fire scene is constructed based on the three-dimensional Gaussian ellipsoid, the dedistorted image and the rendered image, including: determining a rendering depth map based on pixel depth values ​​calculated from the three-dimensional Gaussian ellipsoid, and performing depth estimation on the dedistorted image to obtain a dedistorted depth map; calculating a depth loss value between the rendering depth map and the dedistorted depth map; calculating an image loss value and a structural similarity loss value between the rendered image and the dedistorted image; determining a total loss value based on the depth loss value, the image loss value and the structural similarity loss value; updating the parameters and point cloud density of the three-dimensional Gaussian ellipsoid based on the total loss value to obtain an updated Gaussian ellipsoid and point cloud; and determining the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene.

2. The method according to claim 1, characterized in that The extracting feature points from the plurality of images to be processed includes: For each of the plurality of images to be processed, extract feature points of the image to be processed using a pre-trained neural network to obtain first candidate feature points; Extract the image to be processed using multiple Gaussian kernel standard deviations in the Gaussian difference algorithm to obtain second candidate feature points; The first candidate feature points and the second candidate feature points are fused to obtain the image feature points.

3. The method according to claim 1, characterized in that Before calculating the matching relationship between the image feature points and the image feature points, the method further includes: For each of the plurality of images to be processed, estimating image feature points in the image to be processed to obtain directions and shapes of the image feature points; generating a feature descriptor of the image to be processed according to the direction and shape of the image feature points; Feature point matching is performed on the multiple images to be processed according to the feature descriptors of the images to be processed and the image feature points in the images to be processed to obtain a matching relationship between the image feature points.

4. The method according to claim 1, wherein The converting the relative coordinate system of the sparse point cloud according to the camera parameters includes: Obtaining the geographic location of the image to be processed when it was taken; According to the geographical location of the image to be processed when it was captured and the camera parameters, the relative coordinate system of the sparse point cloud is converted into a world coordinate system to obtain the true-scale point cloud.

5. A fire scene 3D reconstruction device, characterized in that: include: A feature point extraction module is used to extract feature points from a plurality of images to be processed to obtain image feature points of each of the images to be processed, wherein the images to be processed are obtained by photographing a target fire scene by an aircraft; a basic matrix obtaining module, configured to calculate a matching relationship between the image feature points and the image feature points to obtain a basic matrix, wherein the basic matrix represents a geometric relationship between any two matching images in the plurality of images to be processed; A point cloud sparse reconstruction module is used to perform sparse reconstruction on the matching relationship of the image feature points, the image feature points and the basic matrix to obtain a sparse point cloud and camera parameters; A coordinate system conversion module, configured to convert the relative coordinate system of the sparse point cloud according to the camera parameters to obtain a true scale point cloud in a world coordinate system; A three-dimensional model reconstruction module, configured to construct a three-dimensional model of the target fire scene according to the real-scale point cloud and the camera parameters; The method comprises the following steps: constructing a three-dimensional model of the target fire scene according to the real-scale point cloud and the camera parameters, performing dedistortion processing on the multiple images to be processed using the camera parameters to obtain dedistorted images; projecting a three-dimensional Gaussian ellipsoid initialized from each point in the sparse point cloud onto an image plane corresponding to the dedistorted image to obtain multiple projection points, wherein the projection points represent a two-dimensional Gaussian distribution on the image plane; performing differentiable rasterization rendering on the real-scale point cloud according to the multiple projection points to obtain a rendered image; and constructing a three-dimensional model of the target fire scene according to the three-dimensional Gaussian ellipsoid, the dedistorted image, and the rendered image. The three-dimensional model of the target fire scene is constructed based on the three-dimensional Gaussian ellipsoid, the dedistorted image and the rendered image, including: determining a rendering depth map based on pixel depth values ​​calculated from the three-dimensional Gaussian ellipsoid, and performing depth estimation on the dedistorted image to obtain a dedistorted depth map; calculating a depth loss value between the rendering depth map and the dedistorted depth map; calculating an image loss value and a structural similarity loss value between the rendered image and the dedistorted image; determining a total loss value based on the depth loss value, the image loss value and the structural similarity loss value; updating the parameters and point cloud density of the three-dimensional Gaussian ellipsoid based on the total loss value to obtain an updated Gaussian ellipsoid and point cloud; and determining the updated Gaussian ellipsoid and point cloud as the three-dimensional model of the target fire scene.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is executed.

8. A computer program product, characterized in that include: A computer program or computer instruction, wherein the computer program or the computer instruction is executed by a processor to perform the method according to any one of claims 1 to 4.

Images