A remote control end edge-based 3D scene reconstruction system method under low communication bandwidth

Abstract

The present invention provides a system and method for edge-based three-dimensional scene reconstruction at a remote control end under low communication bandwidth. The system includes a control instruction receiving module, a data acquisition module, a data processing module, an algorithm processing module, and a data sending module at the robot end; and a control instruction sending module, a data receiving module, and an image display module at the remote control end. The data is processed on the robot end by an edge extraction algorithm based on deep learning to extract the image edges, and only the extracted image edges are subsequently transmitted to the remote control end; the received edge data is three-dimensionally reconstructed using a modeling tool at the remote control end, and a reconstructed three-dimensional map is output. The present invention screens and extracts the image edge contour and transmits the result to the control end for synchronous mapping, which solves the problem of efficient image transmission under low communication bandwidth conditions, improves the working efficiency of the remote control end, and meets the requirements of convenient and efficient construction of exploration maps.

Description

Technical Field

[0001] The present invention relates to a vision-based three-dimensional reconstruction technology, and specifically to an edge-based three-dimensional scene reconstruction system method at a remote control end under low communication bandwidth. Background Art

[0002] In the past decade, remote simultaneous positioning and mapping technology has developed rapidly. The traditional method is to use a local computer to remotely log in to a remote server, and then pass the processing results of the remote server to the local process through the network, and then present the created map on the local host. This has high requirements for network bandwidth. If you want a smoother experience, you need to occupy more bandwidth resources.

[0003] In urban, underground, border and other environments, there are problems such as communication signals being easily interfered, incomplete reconnaissance information, and slow response of personnel. Therefore, it is urgent to achieve effective 3D scene reconstruction under low bandwidth requirements. Zhao Cheng from Xiamen University suggested building a map through voice interaction with robots, returning the location of the sensor end to the client and recording it. This method transmits less data, but cannot intuitively reflect the location of the sensor in the entire environment. Summary of the invention

[0004] The purpose of the present invention is to propose an edge-based 3D scene reconstruction method for a remote control terminal under low communication bandwidth, creatively presenting the 3D scene map directly on the control terminal, adopting a C / S design mode and independently designing an interaction protocol, accelerating visualization, and improving the observer's ability to control the environment.

[0005] The technical solution to achieve the purpose of the present invention is: a three-dimensional scene reconstruction system based on the edge of a remote control end under low communication bandwidth, including a remote control end and a robot end, the robot end is used to control the movement of the robot according to the control instructions issued by the remote control end, and during the movement, edge image extraction is performed through an edge extraction algorithm based on deep learning and the image edge data is sent to the remote control end, and the remote control end uses the edge data sent by the robot end to perform three-dimensional reconstruction and display the three-dimensional reconstructed map.

[0006] Preferably, the robot terminal includes a control instruction receiving module, a data acquisition module, a data processing module, an algorithm processing module, and a data sending module, wherein:

[0007] The data acquisition module is used to collect image data within the range of the visual sensor;

[0008] The data processing module is used to pre-process the image data collected by the data collection module and adjust the size of the image;

[0009] The algorithm processing module is used to extract the edge of the image pre-processed by the data processing module, specifically: processing the data through an edge extraction algorithm based on deep learning to extract the image edge.

[0010] Preferably, the specific process of the algorithm processing module for performing edge extraction on the image preprocessed by the data processing module is:

[0011] Use the trained edge model to extract edges, including:

[0012] The contour point data is extracted through the canny edge detection algorithm, and 4 scale data blocks are selected around the candidate contour points to input into the five-layer KNet convolution layer. At the output of each convolution layer, only a sub-feature map of the feature map extracted around the center point is output; after the first layer of KNet convolution layer, the representation of the edge contour is obtained; the second convolution layer learns the edge structure; the third convolution layer obtains the complex texture structure, and the fourth layer obtains all texture structures corresponding to the shape of the prototype object; the fifth layer extracts the specific required edge texture, and the output data of the 5 convolution layers is averaged to produce the final contour prediction result.

[0013] Preferably, the remote control terminal includes an image display module, and the image display module is integrated and displayed using a three-dimensional modeling tool, specifically including:

[0014] The incremental SFM technology is used to sparsely reconstruct the edge data sent by the robot to obtain a sparse 3D point cloud. The GEM model is used to construct, accumulate and optimize the matching cost to obtain a depth estimation map.

[0015] For the estimated depth map, depth map fusion is performed through registration, and point cloud restoration is performed according to the projection method.

[0016] Preferably, the specific steps are:

[0017] S1: Collect the data set of the visual sensor in the actual scene on the robot side, clean and label the data set, and obtain the training data set with fixed structure labels;

[0018] S2: On the robot side, the labeled images are used to train the neural network model to obtain the edge model;

[0019] S3: The robot uses the trained edge model to perform edge extraction on the real-time collected image data and sends the edge extraction data to the remote control end;

[0020] S4: Use 3D modeling tools on the remote control end to perform 3D reconstruction on the edge extraction data.

[0021] Preferably, the training process of the neural network model is specifically as follows:

[0022] Import the cropped label pair images into the label file and start network training. Stop training when the loss function no longer decreases. During the training process, select all positive samples and one negative sample to randomly partition and train the network so that the negative / positive ratio is close to 2:1. The loss function uses the L1 norm.

[0023] Preferably, the specific process of edge model performing edge extraction on image data collected in real time is:

[0024] The contour point data is extracted through the canny edge detection algorithm, and 4 scale data blocks are selected around the candidate contour points to input into the five-layer KNet convolution layer. At the output of each convolution layer, only a sub-feature map of the feature map extracted around the center point is output; after the first layer of KNet convolution layer, the representation of the edge contour is obtained; the second convolution layer learns the edge structure; the third convolution layer obtains the complex texture structure, and the fourth layer obtains all texture structures corresponding to the shape of the prototype object; the fifth layer extracts the specific required edge texture, and the output data of the 5 convolution layers is averaged to produce the final contour prediction result.

[0025] Preferably, performing 3D reconstruction using a 3D modeling tool comprises the following steps:

[0026] S4.1: Use the edge image extracted by S3 as a data set; use the incremental SFM technique to perform sparse reconstruction on the edge data set;

[0027] S4.2: Use the GEM model to construct, accumulate and optimize the matching cost to obtain a depth estimation map; S4.3: For the estimated depth map, perform depth map fusion through registration, and perform point cloud recovery according to the projection method. Compared with the prior art, the present invention has the following significant advantages: 1. The present invention innovatively proposes to process the image through a deep learning edge extraction algorithm on the robot side, and only extracts the required key fixed structure data and transmits it to the control side, solving the problem of efficient image transmission under low communication bandwidth;

[0028] 2. The present invention uses an edge extraction algorithm based on deep learning to effectively screen and extract key information of the image, and more accurately and robustly extracts the buildings, tree trunks, and fixed structures required for outdoor roads or indoor spaces, paving the way for the remote control terminal to integrate and display three-dimensional images, thereby improving the working efficiency of the remote control terminal;

[0029] 3. The present invention realizes synchronous mapping on the control side, provides protection for emergency plans under harsh conditions and low communication bandwidth, and meets the requirements of convenient and efficient construction of exploration maps.

[0030] Other features and advantages of the present invention will be described in the following description, and part of them will become obvious from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0032] Figure 1 This is a framework diagram of the data transmission solution between the robot end and the remote control end.

[0033] Figure 2 Flowchart for collecting data for the image data collection unit.

[0034] Figure 3 The following is a flow chart of the system working method.

[0035] Figure 4 Flowchart of edge extraction algorithm based on deep learning.

[0036] Figure 5 Workflow diagram for 3D modeling tools.

[0037] Figure 6 This is the edge map effect after edge extraction. DETAILED DESCRIPTION

[0038] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can imagine various embodiments of the present invention. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or limitation of the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0039] The present invention is conceived as a remote control end edge-based three-dimensional scene reconstruction system under low communication bandwidth, including a remote control end and a robot end. The robot end receives control instructions issued by the remote control end through a local area network to control the movement of the robot, and constructs a three-dimensional scene through edge extraction based on deep learning while moving. Under extreme communication conditions, the bandwidth is low, and map construction needs to be constructed while exploring. As the exploration range expands, the generated three-dimensional scene will continue to expand. In the case of remote transmission, the bandwidth requirements for direct transmission will increase.

[0040] Therefore, the robot side compresses the data through an edge extraction algorithm based on deep learning to reduce the amount of transmitted data. Then, the image data is sent to the remote control terminal through the data sending module. After the remote control end receives the data through the data receiving module, it uses the three-dimensional modeling tool through the image display module to integrate the received data and display it on the remote control end.

[0041] The above system includes a control instruction receiving module, a data acquisition module, a data processing module, an algorithm processing module, and a data sending module on the robot side; and a control instruction sending module, a data receiving module, and an image display module on the remote control side.

[0042] The control instruction sending module and the control instruction receiving module, the data sending module and the data receiving module, that is, the robot end and the remote control end generate information interaction, and the robot end then performs corresponding operations such as movement, data collection, processing, and sending.

[0043] The data acquisition module includes an image data acquisition unit, which is used to collect image data within the range of the visual sensor at the robot sensor end.

[0044] The data processing module pre-processes the image data collected by the data collection module to prepare data for the algorithm processing module. It includes an image data pre-processing unit for adjusting the size of the input image.

[0045] The algorithm processing module receives the image data pre-processed by the data processing module for algorithm decision making. The data is compressed by the edge extraction algorithm based on deep learning, and a small amount of key point information is formed on the basis of sacrificing a small amount of image features, thereby reducing the amount of transmitted data and the demand for data link bandwidth for information transmission, and exponentially increasing the processing speed of the module.

[0046] The data sending module is used to send the image data to the remote control terminal.

[0047] The remote control terminal includes a control instruction sending module, a data receiving module, and an image display module. After the data receiving module receives the data, the image display module integrates and displays the received data using a three-dimensional modeling tool.

[0048] The control instruction sending module is used to send robot control instructions.

[0049] A three-dimensional scene reconstruction method based on edge extraction of deep learning under low communication bandwidth includes the following steps:

[0050] S1: Collect data sets of visual sensors in real-world scenarios, such as roads, indoors, and underground spaces. Preprocess the video, clean and manually annotate the image set, and obtain a training data set with fixed structure labels such as buildings, tree trunks, and door frames;

[0051] In some embodiments, the image is cropped to a size of 80×80 to reduce GPU memory usage, and each cropped image is placed at a corresponding position on the edge map as a marker of an edge reference label;

[0052] S2: Use the labeled images to train the neural network model and test its performance on the validation set to obtain the edge model to be extracted;

[0053] The training process of a neural network model consists of the following steps:

[0054] S2.1: Import these cropped label pair images into the label file and start network training.

[0055] S2.2: Train the CNN of the present invention and stop training when the loss function no longer decreases. For a large number of negative samples, in order to balance the ratio between negative samples and positive samples, all positive samples and a small negative sample are randomly partitioned to train the network so that the negative / positive ratio is close to 2:1. The loss function uses the L1 norm, trains for 100 generations, obtains 100 mats, and finally obtains the network model result. The image results extracted by the trunk and fixed structure edge are as follows Figure 6 shown.

[0056] S3: edge extraction using the trained edge model;

[0057] like Figure 2 As shown in the figure, the overall network model of DeepEdge contour extraction is divided into two separately trained network branches, the upper branch is the training classification, and the lower branch is the training regression factor. The scalar outputs of these two sub-networks are averaged to obtain the final output data. First, the contour point data is extracted by the canny edge detection algorithm, and then 4 scale data blocks are selected around the candidate contour points, which are 55×55, 27×27, 13×13 and 6×6, respectively, and put into five convolutional layers. In order to extract high-level features, at the output of each convolutional layer, only a small sub-feature map of the feature map extracted around the center point is output. After the first layer of KNet convolutional layer, a simple representation of the edge contour is obtained; the second convolutional layer learns the edge structure; the third convolutional layer obtains the complex texture structure. The fourth layer obtains all texture structures corresponding to the shape of the prototype object; and the fifth layer extracts the specific required edge texture. The output data of these 5 layers are averaged to produce the final contour prediction result.

[0058] S4: Use 3D modeling tools to perform 3D reconstruction on the edge extraction data to check the clarity and completeness of the results.

[0059] Among them, the three-dimensional reconstruction using the three-dimensional modeling tool in step S4 includes the following steps:

[0060] S4.1: Data acquisition, using the edge image extracted by S3 as the data set; using the incremental SFM technology to sparsely reconstruct the data set. The specific process is: Incremental SFM selects unordered images for feature matching, performs geometric correction and triangulation to restore the sparse point cloud structure, re-estimates the relative posture through the existing point cloud, and then performs local and global BA optimization. Then gradually add perspectives or images to the existing structure, perform triangulation and posture estimation, and then perform BA optimization to correct the structural data, and finally output all camera parameters and sparse 3D point clouds.

[0061] S4.2: Perform dense reconstruction based on depth map fusion. Use the GEM model to construct, accumulate and optimize the matching cost to obtain the depth estimation map. The matching cost construction uses NCC as a measure of optical consistency between images. The optimal fitting plane of feature l or the depth and normal vector of this feature must satisfy:

[0062]

[0063] in, is the feature depth, is the characteristic normal vector.

[0064] Use Patch Match to perform depth transfer and filter the initial depth map. If the pixels corresponding to feature l on representative i are stable between multiple views and satisfy:

[0065]

[0066] q(α)q(β)q(κ) refers to the distance value between the same feature point in different frames under the same viewing angle.

[0067] It can be considered that the geometric relationship between multiple views is stable, and the depth map calculation result is output.

[0068] S4.3: The time of depth estimation increases with the number of images. The estimated depth map is fused by registration and point cloud is restored according to the projection method.

[0069] Example 1

[0070] In combat, disaster, remote mountainous areas and other environments, the communication bandwidth is low. Based on this situation, it is urgent to develop a method to extract the edges of important parts of video image data and reconstruct the three-dimensional scene graph, so that the robot end and the control end can accurately and quickly transmit and reconstruct the scene.

[0071] To solve this problem, the present invention provides a remote control end-based edge-based 3D scene reconstruction method under low communication bandwidth, and the robot end uses a ZED binocular stereo camera as a visual sensor. Video acquisition is performed through the movement of the visual sensor, and the map is built synchronously.

[0072] The following is an introduction to the edge-based three-dimensional scene reconstruction method of the remote control end of the present invention.

[0073] Figure 1 The structural diagram of this system is as follows: Figure 1 As shown, the system includes a control instruction receiving module, a data acquisition module, a data processing module, an algorithm processing module, and a data sending module on the robot side; and a control instruction sending module, a data receiving module, and an image display module on the remote control side.

[0074] Among them, each module of the edge-based 3D scene reconstruction system of the remote control end can be independent and encapsulated into a separate functional module. For example, a personal notebook can be used to call a data acquisition module to collect data. In addition, the functions of each module in the system provided by the present invention can also be integrated together. The present invention does not set the specific form of each module, as long as the function of each module can be realized.

[0075] Specifically, the data acquisition module includes an image data acquisition unit, which is used to collect image data within the range of the robot's visual sensor. First, open the video file, query the video device capabilities, and set the parameters of video acquisition, such as the size and format of the acquired image; next, apply for the frame buffer of the video stream data from the driver, and then map the address space of the frame buffer to the user space through memory mapping, put all the applied frame buffers into the video capture output queue to start data collection, and finally repeat the cycle of taking out the frame buffer from the video capture output queue, and the frame buffer is rearranged into the input queue to obtain the latest image data in real time.

[0076] The data processing module preprocesses the image data collected by the data acquisition module and prepares data for the algorithm processing module. It includes an image data preprocessing unit for adjusting the size of the input image and ultimately achieving time subdivision at high frequencies. Frequency subdivision at low frequencies can automatically adapt to the requirements of the signal, thereby focusing on details and eliminating noise.

[0077] The algorithm processing module receives the image data pre-processed by the data processing module for algorithm decision-making. By compressing the data through the edge extraction algorithm based on deep learning, a small amount of key point information is formed on the basis of sacrificing a small amount of image features, reducing the amount of transmitted data, reducing the demand for data link bandwidth for information transmission, and multiplying the processing speed of the module.

[0078] The image display module, i.e. the remote control end, receives the data through the data receiving module, integrates and displays the received data using the 3D modeling tool, and finally outputs the reconstructed 3D image at the remote control end.

[0079] like Figure 3 As shown, a method for edge-based 3D scene reconstruction at a remote control end under low communication bandwidth includes the following steps:

[0080] S1: Collect data sets of visual sensors in real-world scenarios, such as roads, indoors, and underground spaces. Preprocess the video, clean and manually annotate the image set, and obtain a training data set with fixed structure labels such as buildings, tree trunks, and door frames;

[0081] S2: Use the labeled images to train the neural network model and test its performance on the validation set to obtain the edge model to be extracted;

[0082] S3: Use the trained edge model to perform edge extraction and generate data to be used;

[0083] S4: Use 3D modeling tools to reconstruct the above results in 3D to check the clarity and completeness of the results.

[0084] Specifically, Figure 4 As shown, the network training process in step S2 includes the following steps:

[0085] S2.1: Import these cropped label pair images into the label file and start network training. Use the DeepEdge contour extraction algorithm to obtain the edge model required for extraction.

[0086] S2.2: Train the CNN of the present invention and stop training when the loss function no longer decreases. For a large number of negative samples, in order to balance the ratio between negative samples and positive samples, all positive samples and a small negative sample are randomly partitioned to train the network so that the negative / positive ratio is close to 2:1. The loss function uses the L1 norm, trains for 100 generations, obtains 100 mats, and finally obtains the network model result. The image results extracted by the trunk and fixed structure edge are as follows Figure 6 shown.

[0087] like Figure 5As shown, the three-dimensional reconstruction using the three-dimensional modeling tool in step S4 includes the following steps:

[0088] S4.1: Data acquisition, using the edge image extracted by S3 as the data set; using the incremental SFM technology to sparsely reconstruct the data set. The specific process is: Incremental SFM selects unordered images for feature matching, performs geometric correction and triangulation to restore the sparse point cloud structure, re-estimates the relative posture through the existing point cloud, and then performs local and global BA optimization. Then gradually add perspectives or images to the existing structure, perform triangulation and posture estimation, and then perform BA optimization to correct the structural data, and finally output all camera parameters and sparse 3D point clouds.

[0089] S4.2: Perform dense reconstruction based on depth map fusion. Use the GEM model to construct, accumulate and optimize the matching cost to obtain the depth estimation map. The matching cost construction uses NCC as a measure of optical consistency between images. The optimal fitting plane of feature l or the depth and normal vector of this feature must satisfy:

[0090]

[0091] Wherein, NCC represents the NCC of feature l, I and the reference image.

[0092] Use Patch Match to perform depth transfer and filter the initial depth map. If the pixels corresponding to feature l on representative i are stable between multiple views and satisfy:

[0093]

[0094] It can be considered that the geometric relationship between multiple views is stable, and the depth map calculation result is output.

[0095] S4.3: The time of depth estimation increases with the number of images. The estimated depth map is fused by registration and point cloud is restored according to the projection method.

[0096] This invention presents the three-dimensional scene map directly on the control end for the first time, and innovatively proposes to effectively screen and extract key image information through a deep learning edge extraction algorithm on the robot end, so as to more accurately and robustly extract the buildings, tree trunks, and fixed structures required for outdoor roads or indoor spaces. It solves the problem of efficient image transmission under low communication bandwidth, improves the work efficiency of the remote control end, and meets the requirements of convenient and efficient construction of three-dimensional scene maps.

[0097] The above is only a preferred embodiment of the present invention. However, the present invention is not limited to the above embodiment, and any equivalent changes and modifications made according to the present invention, as long as the resulting functions do not exceed the scope of this solution, belong to the protection scope of the present invention.

[0098] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0099] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as the features included in the exemplary embodiments are all necessary technical features of the patent claims.

[0100] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be set in a device different from the embodiment. The different modules, units, or components included in the device of the embodiment can be combined into one module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. A 3D scene reconstruction system based on edge at remote control end under low communication bandwidth, It is characterized in that Including remote control end and robot end; The remote control terminal includes an image display module, which is integrated and displayed using a three-dimensional modeling tool, specifically including: Use incremental SFM technology to sparsely reconstruct the edge data sent by the robot to obtain a sparse 3D point cloud; The GEM model is used to construct, accumulate and optimize the matching cost to obtain a depth estimation map; For the estimated depth map, depth map fusion is performed through registration, and point cloud recovery is performed according to the projection method; The robot end is used to control the movement of the robot according to the control command issued by the remote control end, and during the movement, edge image extraction is performed through an edge extraction algorithm based on deep learning and the image edge data is sent to the remote control end. The remote control end uses the edge data sent by the robot end to perform three-dimensional reconstruction and display the three-dimensional reconstructed map. The robot end includes a control command receiving module, a data acquisition module, a data processing module, an algorithm processing module, and a data sending module, wherein: The data acquisition module is used to collect image data within the range of the visual sensor; The data processing module is used to pre-process the image data collected by the data collection module and adjust the size of the image; The algorithm processing module is used to extract the edge of the image preprocessed by the data processing module, specifically: processing the data through the edge extraction algorithm based on deep learning to extract the image edge. The specific process is: Use the trained edge model to extract edges, including: The contour point data is extracted through the canny edge detection algorithm, and 4 scale data blocks are selected around the candidate contour points to input into the five-layer KNet convolution layer. At the output of each convolution layer, only a sub-feature map of the feature map extracted around the center point is output; after the first layer of KNet convolution layer, the representation of the edge contour is obtained; the second convolution layer learns the edge structure; the third convolution layer obtains the complex texture structure, and the fourth layer obtains all texture structures corresponding to the shape of the prototype object; the fifth layer extracts the specific required edge texture, and the output data of the 5 convolution layers is averaged to produce the final contour prediction result.

2. A 3D scene reconstruction system based on edge at remote control end under low communication bandwidth, It is characterized in that The specific steps are: S1: Collect the data set of the visual sensor in the actual scene on the robot side, clean and label the data set, and obtain the training data set with fixed structure labels; S2: On the robot side, the labeled images are used to train the neural network model to obtain the edge model; S3: On the robot side, use the trained edge model to perform edge extraction on the real-time collected image data and send the edge extraction data to the remote control side; S4: Using a 3D modeling tool on the remote control end to perform 3D reconstruction on the edge extraction data; The training process of the neural network model is as follows: Import the cropped label pair images into the label file and start network training. Stop training when the loss function no longer decreases. During the training process, select all positive samples and one negative sample to randomly partition the network so that the negative / positive ratio is close to 2:

1. The loss function uses the L1 norm. The specific process of edge model extracting edge of real-time collected image data is as follows: The contour point data is extracted through the canny edge detection algorithm, and four scale data blocks are selected around the candidate contour points and input into the five-layer KNet convolution layer. At the output of each convolution layer, only a sub-feature map of the feature map extracted around the center point is output; After the first KNet convolution layer, the edge contour representation is obtained; the second convolution layer learns the edge structure; the third convolution layer obtains the complex texture structure, and the fourth layer obtains all the texture structures corresponding to the shape of the prototype object; the fifth layer extracts the specific required edge texture, and averages the output data of the five convolution layers to produce the final contour prediction result; The 3D reconstruction using 3D modeling tools includes the following steps: S4.1: Use the edge image extracted by S3 as a data set; use the incremental SFM technology to perform sparse reconstruction on the edge data set; S4.2: Use the GEM model to construct, accumulate and optimize the matching cost to obtain a depth estimation map; S4.3: For the estimated depth map, depth map fusion is performed through registration, and point cloud recovery is performed according to the projection method.

Images