Three-dimensional monomer method, device and equipment based on image segmentation and storage medium

Abstract

The invention provides a three-dimensional monomer method and device based on image segmentation, equipment and a storage medium, and particularly relates to the technical field of live-action three-dimensional modeling. According to the method, feature points of aerial images are extracted and matched, aerial triangulation encrypted data are constructed by using a bundle block adjustment technology, buildings in the images are accurately identified and classified by using an advanced two-dimensional image segmentation technology, monomeric modeling is realized, image segmentation information is synchronously fused in a depth map generation process, and the depth map generation efficiency is improved. And the definition and modeling precision of the building boundary are improved. In addition, for a blank map area caused by dynamic artifacts (such as vehicles), intelligent completion is performed by using a multi-view fusion technology. According to the method, the problems of large resource consumption and high calculation difficulty of traditional three-dimensional model data segmentation are solved, and the blank map area of the artifact is filled by using the two-dimensional image segmentation data, so that the precision and efficiency of live-action three-dimensional modeling are improved, and the accuracy of building segmentation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of real-scene three-dimensional modeling, and in particular to a three-dimensional monomerization method, device, equipment and storage medium based on image segmentation. Background Art

[0002] In the current field of real-life 3D modeling, accurately segmenting buildings and removing dynamic artifacts (such as vehicles) is a critical and challenging step. Traditional 3D building modeling methods mainly rely on the processing of point cloud data. However, existing technologies often have problems such as high computing resource requirements, high processing difficulty, and insufficient accuracy when dealing with complex 3D scene segmentation and removal of motion artifacts.

[0003] Among the traditional 3D modeling methods, on the one hand, the existing 3D point cloud segmentation technologies mainly include the following methods: (1) Data segmentation processing based on point cloud features: processing and modeling point cloud data obtained by laser radar or multi-view stereo vision. Although this type of method can handle simple scenes, it is prone to errors when segmenting buildings and removing artifacts, and has high requirements for computing resources. (2) Segmentation based on texture and color information: using the texture and color information of the image to segment buildings, but the effect is limited when dealing with complex scenes and dynamic objects, and the segmentation accuracy is insufficient for areas with complex textures or similar colors. (3) Methods based on geometric features: This method uses the geometric properties of the point cloud (such as curvature, normal vector, flatness, etc.) for segmentation. Although this method works well for objects with regular shapes, it is sensitive to noise and has difficulty in processing objects with complex and irregular shapes. (4) Methods based on graph theory: treating the point cloud as a graph structure and using the graph's connectivity and minimum cut properties for segmentation. This type of method has a relatively fine segmentation effect, but the computational complexity is high and it is difficult to process large-scale point cloud data in real time. (5) Deep learning-based methods: point cloud features are automatically learned through neural networks for end-to-end segmentation. Although the segmentation accuracy is high and it can process complex and diverse point cloud data, it requires a large amount of labeled data for training and has high computing resource requirements. On the other hand, based on two-dimensional image segmentation technology, drone aerial images face many challenges in image segmentation tasks, including the following aspects: (1) Multi-scale objects: Aerial images contain objects of different sizes, such as high-rise buildings, roads, vegetation, etc. The differences in scale of these objects make it difficult for traditional image segmentation algorithms to accurately segment large-scale and small-scale objects. (2) Complex background: Aerial images often have complex backgrounds. Due to the wide shooting range, the contrast between different areas is not obvious, which easily leads to blurred segmentation boundaries between objects and backgrounds, thereby affecting the accuracy of segmentation. (3) Variable viewing angles: The shooting angle of drones is highly flexible, resulting in a variety of shapes and proportions of objects in the image. As the shooting angle changes, the same object may present significantly different appearances in different images, which brings additional difficulties to traditional image segmentation. Therefore, although two-dimensional image segmentation technology has made significant progress in the field of computer vision, its application in the field of drone aerial photography and three-dimensional modeling is still limited. The problem of high resource consumption and high computational difficulty in three-dimensional model data segmentation still exists.

[0004] In summary, how to solve the problem of traditional 3D model data segmentation consuming large resources and high computational difficulty is a difficult problem that needs to be solved urgently in this technical field. Summary of the invention

[0005] The main purpose of the present invention is to provide a three-dimensional monomerization method, device, equipment and storage medium based on image segmentation to solve the problem that traditional three-dimensional model data segmentation consumes large resources and has high calculation difficulty, thereby improving the accuracy and efficiency of real-scene three-dimensional modeling, improving the accuracy of building segmentation, reducing the need for manual correction of modeling results, and making three-dimensional modeling in complex scenes more efficient and automated.

[0006] In order to achieve the above-mentioned objectives, the present invention provides a three-dimensional monomerization method, device, equipment and storage medium based on image segmentation, which improves the accuracy and efficiency of real-scene three-dimensional modeling, improves the accuracy of building segmentation, reduces the need for manual correction of modeling results, and makes three-dimensional modeling in complex scenes more efficient and automated.

[0007] In a first aspect, the present invention provides a three-dimensional monomerization method based on image segmentation, the method comprising:

[0008] Acquire first aerial image data, reconstruct and optimize the geometric position relationship between the first aerial image data by using an incremental motion restoration algorithm and a bundle adjustment technique, and obtain second aerial image data;

[0009] Using a two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data;

[0010] Performing depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data;

[0011] According to the second aerial image data, fifth aerial image data is obtained through sub-block division and data reverse pulling, wherein the sub-block division is used to divide the fourth aerial image data into a plurality of sub-blocks, and the data reverse pulling is used to extract the camera position and image data in the sub-blocks;

[0012] According to the fifth aerial image data and the fourth aerial image data, three-dimensional individualized data is acquired and outputted through image preprocessing and motion artifact removal.

[0013] Optionally, the acquiring of the first aerial image data, reconstructing and optimizing the geometric position relationship between the first aerial image data by an incremental motion restoration algorithm and a bundle adjustment technique, and acquiring the second aerial image data comprises:

[0014] The first aerial image data is obtained by preprocessing and extracting feature points of the image data collected by the drone aerial photography, wherein the preprocessing includes: image denoising, color correction and resolution adjustment;

[0015] Reconstructing the geometric position relationship between the first aerial image data by using the incremental motion recovery algorithm to obtain the reconstructed first aerial image data;

[0016] The reconstructed first aerial image data is optimized by using bundle adjustment technology to obtain second aerial image data.

[0017] Optionally, the using a two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data includes:

[0018] The first aerial image data is accurately segmented through the DeepLab v3 model to obtain third aerial image data, wherein the DeepLab v3 model includes a dilated convolution module and a dilated spatial pyramid pooling module, and the DeepLab v3 model is used to specifically implement two-dimensional image segmentation technology.

[0019] Optionally, the accurately segmenting the first aerial image data by using the DeepLab v3 model to obtain the third aerial image data includes:

[0020] The atrous convolution module and the atrous spatial pyramid pooling module are used to extract multi-scale features of the first aerial image data to obtain the segmented first aerial image data;

[0021] The boundaries of different objects in the segmented first aerial image data are optimized by a fully connected conditional random field to obtain third aerial image data.

[0022] Optionally, the first aerial image data includes a reference view, the second aerial image data includes a reference view, and performing depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data includes:

[0023] Acquire depth maps of a reference view and a referenced view, and project depth values ​​of the depth maps into a three-dimensional space to obtain the three-dimensional point cloud data;

[0024] Color information fusion and classification are performed on the three-dimensional point cloud data to obtain fourth aerial image data.

[0025] Optionally, acquiring fifth aerial image data by sub-block division and reverse data pulling according to the second aerial image data includes:

[0026] Dividing the second aerial image data into a plurality of sub-blocks by sub-block division;

[0027] Generate sparse point cloud data corresponding to the sub-block by multi-view stereo matching and structure recovery algorithm, wherein the sparse point cloud data includes camera position and image data, and the sparse point cloud data is used to indicate the position of each point in the three-dimensional scene and the camera viewing angle information;

[0028] The camera position and image data corresponding to the sub-block are processed by reverse data pulling to obtain fifth aerial image data.

[0029] Optionally, acquiring and outputting three-dimensional individualized data by image preprocessing and removing motion artifacts according to the fifth aerial image data and the fourth aerial image data includes:

[0030] According to the fifth aerial image data, images meeting the screening standard parameters are screened to obtain the screened fifth aerial image data, wherein the screening standard parameters include: image clarity, overlap, exposure, and integrity of image content;

[0031] Detecting the motion area in the filtered fifth aerial image data by an optical flow method;

[0032] The detected motion area is repaired to obtain fifth aerial image data after motion artifacts are removed, the fifth aerial image data after motion artifacts are removed is fused, and the fused data is mapped to the fourth aerial image data to obtain and output three-dimensional monomerized data.

[0033] In a second aspect, the present invention provides a three-dimensional individualization device based on image segmentation, wherein the device is applied to the three-dimensional individualization method based on image segmentation in the first aspect, and comprises:

[0034] An acquisition module, the acquisition module is used to acquire first aerial image data;

[0035] A processing module, the processing module is used to reconstruct and optimize the geometric position relationship between the first aerial image data through an incremental motion recovery algorithm and a bundle adjustment technology to obtain second aerial image data;

[0036] A processing module, the processing module is further used to accurately segment the first aerial image data using a two-dimensional image segmentation technology to obtain third aerial image data;

[0037] A processing module, the processing module is further used to perform depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data;

[0038] A processing module, the processing module is further used to obtain fifth aerial image data through sub-block division and data reverse pulling according to the second aerial image data;

[0039] A processing module, the processing module is further used to obtain three-dimensional monomer data through image preprocessing and motion artifact removal according to the fifth aerial image data;

[0040] An output module, wherein the output module is used to output the three-dimensional monomerized data.

[0041] Optionally, the acquisition module is specifically used to obtain first aerial image data by preprocessing and extracting feature points of image data collected by drone aerial photography, wherein the preprocessing includes: image denoising, color correction and resolution adjustment; reconstructing the geometric position relationship between the first aerial image data through the incremental motion recovery algorithm to obtain the reconstructed first aerial image data; optimizing the reconstructed first aerial image data through the bundle adjustment technology to obtain the second aerial image data.

[0042] Optionally, the processing module is specifically used to accurately segment the first aerial image data through the DeepLab v3 model to obtain third aerial image data, wherein the DeepLab v3 model includes a dilated convolution module and a dilated spatial pyramid pooling module, and the DeepLab v3 model is used to specifically implement two-dimensional image segmentation technology.

[0043] Optionally, the processing module is specifically used to use the void convolution module and the void spatial pyramid pooling module to extract multi-scale features of the first aerial image data to obtain the segmented first aerial image data; optimize the boundaries of different objects in the segmented first aerial image data through a fully connected conditional random field to obtain third aerial image data.

[0044] Optionally, the acquisition module is specifically used to acquire depth maps of the reference view and the referenced view, and project depth values ​​of the depth maps into a three-dimensional space to obtain the three-dimensional point cloud data;

[0045] The processing module is specifically used to perform color information fusion classification on the three-dimensional point cloud data to obtain fourth aerial image data.

[0046] Optionally, the processing module is specifically used to divide the second aerial image data into multiple sub-blocks through sub-block division; generate sparse point cloud data corresponding to the sub-blocks through multi-view stereo matching and structure recovery algorithm, wherein the sparse point cloud data includes camera position and image data, and the sparse point cloud data is used to indicate the position and camera viewing angle information of each point in the three-dimensional scene; and obtain fifth aerial image data by reverse pulling the camera position and image data corresponding to the sub-block.

[0047] Optionally, the processing module is specifically used to screen out images that meet screening standard parameters based on the fifth aerial image data, and obtain the screened fifth aerial image data, wherein the screening standard parameters include: image clarity, overlap, exposure and integrity of image content; detect the motion area in the screened fifth aerial image data by optical flow method; repair the detected motion area to obtain the fifth aerial image data after removing motion artifacts, fuse the fifth aerial image data after removing motion artifacts, and map the fused data to the fourth aerial image data to obtain three-dimensional monomer data.

[0048] In a third aspect, the present invention provides a three-dimensional individualization device based on image segmentation, the device comprising:

[0049] Memory;

[0050] processor;

[0051] Wherein, the memory stores computer-executable instructions;

[0052] The processor executes the computer-executable instructions stored in the memory to implement the above first aspect and / or various possible implementations of the first aspect or to implement the above second aspect and / or various possible implementations of the second aspect.

[0053] In a fourth aspect, the present application provides a computer storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation methods of the first aspect as above, or to implement the second aspect and / or various possible implementation methods of the second aspect as above.

[0054] The present application provides a three-dimensional singularization method, apparatus, device and storage medium based on image segmentation. The method first acquires first aerial image data, reconstructs and optimizes the geometric position relationship between the first aerial image data through an incremental motion recovery algorithm and a bundle adjustment technique, and obtains second aerial image data, wherein the first aerial image data is used to indicate the camera posture of the aerial image data and the three-dimensional point data of different objects, and the second aerial image data is used to indicate the camera posture of the reconstructed and optimized first aerial image data and the three-dimensional point data of different objects; the first aerial image data is accurately segmented by a two-dimensional image segmentation technique to obtain third aerial image data, wherein the third aerial image data The invention relates to a method for obtaining a three-dimensional point cloud data by using a sub-block partitioning method and a reverse data extraction method. The method is used to indicate the buildings that can be accurately identified and singulated in the three-dimensional singulation process; the second aerial image data and the third aerial image data are subjected to a depth map fusion to obtain the fourth aerial image data, wherein the fourth aerial image data is used to indicate the three-dimensional point cloud data after the depth fusion; the fifth aerial image data is obtained by sub-block partitioning and data reverse extraction according to the second aerial image data, wherein the sub-block partitioning is used to divide the fourth aerial image data into a plurality of sub-blocks, and the data reverse extraction is used to extract the camera position and image data in the sub-blocks; the three-dimensional singulation data is obtained and output by image preprocessing and removing motion artifacts according to the fifth aerial image data and the fourth aerial image data. The method solves the problem that the traditional three-dimensional model data segmentation consumes large resources and has high computational difficulty, thereby improving the accuracy and efficiency of real-scene three-dimensional modeling, improving the accuracy of building segmentation, reducing the need for manual correction of modeling results, and making three-dimensional modeling in complex scenes more efficient and automated. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0056] Figure 1 A schematic diagram of the process of the three-dimensional singularization method based on image segmentation provided in this application;

[0057] Figure 2 A schematic diagram of the structure of a three-dimensional individualization device based on image segmentation provided in this application;

[0058] Figure 3 A schematic diagram of the structure of a three-dimensional individualization device based on image segmentation provided in this application.

[0059] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings, if present, are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0062] In the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0063] In the current field of real-life 3D modeling, accurately segmenting buildings and removing dynamic artifacts (such as vehicles) is a critical and challenging step. Traditional 3D modeling methods mainly rely on the processing of point cloud data. However, existing technologies often have problems such as high computing resource requirements, high processing difficulty, and insufficient accuracy when dealing with complex scene segmentation and removing motion artifacts.

[0064] Among the traditional 3D modeling methods, on the one hand, the existing 3D point cloud segmentation technologies mainly include the following methods: (1) Data segmentation processing based on point cloud features: processing and modeling point cloud data obtained by laser radar or multi-view stereo vision. Although this type of method can handle simple scenes, it is prone to errors when segmenting buildings and removing artifacts, and has high requirements for computing resources. (2) Segmentation based on texture and color information: using the texture and color information of the image to segment buildings, but the effect is limited when dealing with complex scenes and dynamic objects, and the segmentation accuracy is insufficient for areas with complex textures or similar colors. (3) Methods based on geometric features: This method uses the geometric properties of the point cloud (such as curvature, normal vector, flatness, etc.) for segmentation. Although this method works well for objects with regular shapes, it is sensitive to noise and has difficulty in processing objects with complex and irregular shapes. (4) Methods based on graph theory: treating the point cloud as a graph structure and using the graph's connectivity and minimum cut properties for segmentation. This type of method has a relatively fine segmentation effect, but the computational complexity is high and it is difficult to process large-scale point cloud data in real time. (5) Deep learning-based methods: point cloud features are automatically learned through neural networks for end-to-end segmentation. Although the segmentation accuracy is high and it can process complex and diverse point cloud data, it requires a large amount of labeled data for training and has high computing resource requirements. On the other hand, drone aerial images face many challenges in image segmentation tasks, including the following aspects: (1) Multi-scale objects: Aerial images contain objects of different sizes, such as high-rise buildings, roads, vegetation, etc. The differences in scale of these objects make it difficult for traditional image segmentation algorithms to accurately segment large-scale and small-scale objects. (2) Complex background: Aerial images often have complex backgrounds. Due to the wide shooting range, the contrast between different areas is not obvious, which easily leads to blurred segmentation boundaries between objects and backgrounds, thereby affecting the accuracy of segmentation. (3) Variable viewing angles: The shooting angles of drones are highly flexible, resulting in a variety of shapes and proportions of objects in the image. As the shooting angle changes, the same object may have significantly different appearances in different images, which brings additional difficulties to traditional image segmentation. Therefore, although two-dimensional image segmentation technology has made significant progress in the field of computer vision, its application in the field of drone aerial photography and three-dimensional modeling is still limited. The problem of high resource consumption and high computational difficulty in three-dimensional model data segmentation still exists.

[0065] In summary, how to solve the problem of traditional 3D model data segmentation consuming large resources and high computational difficulty is a difficult problem that needs to be solved urgently in this technical field.

[0066] In response to the above problems, the present application provides a three-dimensional singularization method, device, equipment and storage medium based on image segmentation. This method combines two-dimensional image segmentation technology, multi-view fusion technology, and sub-block division and reverse pulling technology to achieve accurate singularization of buildings in the three-dimensional modeling process, while removing dynamic artifacts and completing blank areas. This method effectively improves the accuracy and efficiency of real-life three-dimensional modeling, significantly improves the display effect of the final model, and has broad application prospects.

[0067] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0068] Figure 1 The flowchart of the three-dimensional singularization method based on image segmentation provided in this application is provided, and the three-dimensional singularization method based on image segmentation is described in detail, such as Figure 1 As shown, the three-dimensional monomerization method based on image segmentation provided in this embodiment includes:

[0069] S101: Acquire first aerial image data, reconstruct and optimize the geometric position relationship between the first aerial image data through an incremental motion restoration algorithm and a bundle adjustment technique, and obtain second aerial image data.

[0070] Among them, the first aerial image data is used to indicate the original aerial image data, and the second aerial image data is used to indicate the generation of aerial triangulation encrypted data (i.e. the original aerial image data's position and posture and the three-dimensional point data of key features and the correlation relationship data between images) through the original aerial image data. This step first pre-processes the image data collected by the drone aerial photography, uses the incremental motion recovery structure algorithm to estimate the internal and external parameters of the camera and preliminarily estimate the depth range, and selects multiple views of stereo matching. The principle of incremental SFM is to first extract and match feature points of all images, calculate the geometric position relationship between images; secondly, select the two views of the best matching pair for initial reconstruction, and use the random sampling consensus algorithm (RANSAC) to remove erroneous matching points; and perform BA optimization on the initial camera pose and 3D points again. On this basis, new images are continuously added to solve the camera pose and triangulate the feature points. At the same time, each time a new image is added and the reconstruction is completed, a beam adjustment is required to reduce error accumulation and improve the robustness of SFM until all images are added and the entire scene is reconstructed.

[0071] Furthermore, the step of acquiring the first aerial image data and reconstructing and optimizing the geometric position relationship between the first aerial image data by using an incremental motion restoration algorithm and a bundle adjustment technique to obtain the second aerial image data includes:

[0072] The first aerial image data is obtained by preprocessing the image data collected by the drone aerial photography and extracting feature points, wherein the preprocessing includes: image denoising, color correction and resolution adjustment. In specific implementation, the image data collected by the drone aerial photography can be preprocessed first, including steps such as image denoising, color correction and resolution adjustment, to ensure the quality of the input data. Then, feature points are extracted for the processed image data, usually using algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Rapid Oriented Binarization Features) to obtain key feature points in the image;

[0073] The geometric position relationship between the first aerial image data is reconstructed by the incremental motion recovery algorithm to obtain the reconstructed first aerial image data. In specific implementation, after preprocessing and feature extraction, the incremental motion recovery structure (Structure from Motion, SFM) algorithm can be used to estimate the geometric relationship between the images and restore the three-dimensional structure. Feature points are matched for all images, and the geometric position relationship between the images is established by calculating the associated matching relationship between the images. This step ensures the accuracy of the spatial position and direction relationship between the images; the two views of the best matching pair are selected from all images for initial three-dimensional reconstruction. In this process, the random sampling consensus algorithm (RANSAC) is used to remove erroneous matching points to ensure the accuracy of the initial reconstruction. The RANSAC algorithm iteratively identifies and excludes erroneous matching points that do not meet the model assumptions, thereby improving the reliability of the initial reconstruction. Two images with sufficient overlapping areas are selected as the initial image pair, and the matching feature point pairs estimate the basic matrix F, and the erroneous matches are eliminated by RANSAC. The initial camera matrices P1 and P2 are decomposed from the basic matrix F. The three-dimensional points are calculated by the initial camera matrix and the matching feature points:

[0074] X=triangulate(P1,P2,x1,x2)

[0075] Among them, X is a 3D point, x1 and x2 are matching feature points on the image.

[0076] The reconstructed first aerial image data is optimized by the bundle adjustment technology to obtain the second aerial image data. In specific implementation, after the initial reconstruction is completed, BA (bundle adjustment technology) is used to optimize the initial position of the camera and the position of the three-dimensional points to reduce the accumulation of errors and further improve the accuracy of three-dimensional reconstruction. New images are gradually added to the estimated model. For each newly added image, the perspective n-point (PnP) problem is solved using the known three-dimensional points and their projections in the new image to estimate the camera position:

[0077] P i =PnP(X,x i )

[0078] Among them, P i is the camera matrix of the new image, X is the 3D point, x i is the projected point in the new image.

[0079] It can be understood that on the basis of the above, new images are added incrementally, and the camera pose is solved and the new feature points are triangulated step by step. After each new image is added and the corresponding 3D reconstruction is completed, a bundle adjustment is performed to further optimize the global model, reduce cumulative errors, and enhance the robustness of the SFM process. Given the camera matrices P1 and P2 and the feature points X1 and X2, new images and existing images are used for triangulation to calculate new 3D points. After each new image is added, global optimization is performed to simultaneously optimize all camera poses and 3D point positions through bundle adjustment to minimize the reprojection error:

[0080] min∑ i ∑ j ||x ij -P i X j || 2

[0081] Among them, x ij is the projection of the jth feature point in the i-th image, P i is the camera matrix of the ith image, X j is the jth three-dimensional point.

[0082] This incremental process continues until all images have been processed and included in the 3D reconstruction. With each image added, the 3D model of the entire scene is gradually improved until the entire scene is reconstructed.

[0083] S102: Using a two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data.

[0084] Wherein, the third aerial image data is used to indicate the classification data generated by the original aerial image data through the two-dimensional segmentation technology based on deep learning. In this step, in order to cope with these difficult drone aerial photography complex backgrounds, multi-scale and variable perspective features, the DeepLab v3 model shows significant advantages in the drone aerial image segmentation task through its multi-scale feature extraction capability. Specifically, DeepLab v3 uses atrous convolution (AtrousConvolution) and atrous spatial pyramid pooling (ASPP, Atrous Spatial Pyramid Pooling) modules to extract multi-scale features, so that the semantic information of objects of different scales can be effectively captured. This structural design enables DeepLabv3 to maintain a high segmentation accuracy in complex scenes. In the application process, the drone aerial images are first preprocessed to improve the segmentation effect. The preprocessing steps include geometric correction, color adjustment and enhancement of the image. These operations can reduce the distortion generated during the shooting process, and improve the contrast and clarity of the image, laying a good foundation for subsequent segmentation tasks. Next, the preprocessed image is input into the DeepLab v3 model. Through its dilated convolution and ASPP structure, the model extracts multi-scale features in the image layer by layer and finally generates semantic segmentation results. Since DeepLab v3 can process information of different scales at the same time, it is particularly suitable for dealing with multi-scale object problems in aerial images. In order to further improve the accuracy of the segmentation results, especially in the processing of object boundaries, conditional random fields (CRF) are used to optimize the preliminary segmentation results. CRF can refine the segmentation edges by modeling the local dependencies between pixels, thereby improving the accuracy of the segmentation boundaries and making the segmentation results more consistent with the actual object contours. In short, DeepLab v3 effectively copes with the challenges of multi-scale objects, complex backgrounds, and changing perspectives in drone aerial images through its multi-scale feature extraction capabilities and CRF optimization strategies, making it perform well in the semantic segmentation tasks of such images.

[0085] Furthermore, the use of two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data includes: accurately segmenting the first aerial image data through a DeepLab v3 model to obtain third aerial image data, wherein the DeepLab v3 model includes a dilated convolution module and a dilated spatial pyramid pooling module, and the DeepLab v3 model is used to specifically implement the two-dimensional image segmentation technology.

[0086] Specifically, the first aerial image data is accurately segmented by the DeepLab v3 model to obtain the third aerial image data, including:

[0087] The atrous convolution module and the atrous spatial pyramid pooling module are used to extract the multi-scale features of the first aerial image data, and obtain the segmented first aerial image data. In the specific implementation, atrous convolution is used to expand the receptive field without increasing the amount of calculation by introducing the dilation rate, thereby capturing a wider range of contextual information. Atrous convolution is a key component of DeepLab v3. It expands the receptive field by inserting holes in the standard convolution kernel (i.e., inserting "empty" positions between adjacent kernel elements). The atrous convolution formula is as follows:

[0088] y[i]=Σ k x[i+r·k]·ω[k]

[0089] Among them, y[i] is the value of position i in the output feature map, x[i+r·k] is the value of position i+r·k in the input feature map, ω[k] is the convolution kernel weight, and r is the dilation rate, which represents the spacing between adjacent kernel elements.

[0090] By changing the dilation rate r, the size of the receptive field can be controlled, thereby capturing features of different scales without increasing the computational complexity.

[0091] Spatial Pyramid Pooling (ASPP) is used to apply multiple dilated convolutions of different dilation rates to feature maps in parallel to capture multi-scale information. The ASPP structure extracts multi-scale contextual information by applying dilated convolutions of different dilation rates in parallel. ASPP can be expressed as:

[0092]

[0093] Among them, ASPP(x) is the ASPP output feature, AtrousConv(x,rate i ) is the i-th void rate rate i The dilated convolution results under , N is the number of different dilation rates.

[0094] By fusing features from these different scales, DeepLab v3 can more effectively segment objects with complex boundaries.

[0095] The boundaries of different objects in the first aerial image data after segmentation are optimized by a fully connected conditional random field to obtain the third aerial image data. In the specific implementation, a fully connected conditional random field (CRF) is used for post-processing to optimize the segmentation boundary. DeepLab v3 usually applies a conditional random field (CRF) to the segmentation result for post-processing to further optimize the boundary. The optimization goal of CRF is to make adjacent pixels more semantically consistent. The mathematical expression of CRF is:

[0096]

[0097] Where E[x] is the energy function, x i represents the label of pixel i, is the single-point energy function of pixel i, usually related to the class probability of the pixel, φ p (x i ,x j ) is the pairing energy function of pixels i and j, which is usually related to the distance and color difference between the pixels.

[0098] It can be understood that the application of CRF helps to suppress artifacts and isolated pixels, thereby obtaining smoother and more accurate segmentation results.

[0099] S103: Perform depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data.

[0100] Among them, the fourth aerial image data is used to indicate the depth map generated by the original aerial image obtained in step S101 and the association relationship data generated in step S101 and the classification data of the original aerial image data obtained in step S102 during the depth map fusion process. In this step, when generating the point cloud data of the reference image, the incremental motion recovery structure (SFM) algorithm is first used to obtain the association matching relationship between multiple views. Through these matching relationships, the depth maps of the reference view and the referenced view can be obtained, and their depth values ​​can be projected into three-dimensional space to estimate the position of the three-dimensional point. After the three-dimensional point cloud data is fused, the three-dimensional point cloud data of the current point is subjected to color information fusion classification, and the multi-threshold fusion strategy is performed on the referenced associated view depth map and the current reference view depth map to achieve the fusion of the two-dimensional segmented image data.

[0101] Furthermore, the first aerial image data includes a referenced view, the second aerial image data includes a reference view, and the second aerial image data and the third aerial image data are subjected to depth map fusion to obtain the fourth aerial image data, including: obtaining depth maps of the reference view and the referenced view, projecting the depth values ​​of the depth maps into a three-dimensional space to obtain the three-dimensional point cloud data; and performing color information fusion classification on the three-dimensional point cloud data to obtain the fourth aerial image data. In specific implementation, it can be achieved through the following steps:

[0102] (1) For the reference view, first obtain the depth estimation value d of its depth map ref The reference view depth map and the reference view depth map are projected into the three-dimensional space, and the intrinsic parameter matrix K of the camera and the pixel point p on the image plane are used. ref , project the depth value into the world coordinate system and calculate the 3D point X ref , the specific projection formula is as follows:

[0103] X ref =d ref (u,v)·K -1 p ref

[0104] Among them, d ref is the depth value of the reference view depth map, K is the camera intrinsic parameter matrix, p ref Is the pixel point of the image plane. Calculation of adjacent image 3D points: For each reference image, obtain the 3D points of its adjacent images and calculate the 3D coordinates and normal vector of each pixel point.

[0105] (2) For each reference image, obtain the 3D points of its adjacent images. These 3D points are calculated using the depth map of the adjacent views. For each pixel, calculate its 3D coordinates X in 3D space. adj and the normal vector N adj ,This information will be used for the fusion and subsequent processing of point cloud data.

[0106] (3) Project the 3D points in the reference view and the adjacent view into a unified 3D space, and fuse the 3D points at the same position. In this process, a multi-threshold fusion strategy is used to optimize the fusion accuracy of the 3D points, taking into account the matching relationship between multiple views and the credibility of the depth map. The fused 3D point cloud data not only contains the position coordinates, but also combines the normal vector information to improve the geometric consistency and detail expression of the point cloud data. Since multiple reference views repeatedly calculate the same point in space, it is necessary to remove the redundant points near the point and perform depth fusion in space. Figure 3 The fusion constraints of dimensional point cloud data are:

[0107] If the depth of the point is less than the depth of the adjacent image, the point is deleted.

[0108] d ref (u,v) <d neigh (u,v)

[0109] The relative depth difference of the points to be fused is less than 0.01.

[0110] d ref (u,v)-d neigh (u,v)<0.01

[0111] Normal difference is less than 10 degrees,

[0112]

[0113] The reprojection error is less than 5 pixels,

[0114] ||p ref -P neigh X ref ||<5

[0115] If the point to be fused exists in at least 5 reference view depth maps, the point will be retained, otherwise it will be deleted.

[0116] ∑ k {X ref ∈X neigh,k}≥3

[0117] The spatial coordinates (x, y, z) of the fusion point are the centroid positions calculated from all points.

[0118]

[0119] (4) Color information fusion and classification: After the fusion of 3D point cloud data is completed, the color information of the point cloud data is further fused and classified. By analyzing the color information of each point in the point cloud data and using the color features of the image, the point cloud is classified and marked. This process can help distinguish different objects or areas, such as buildings, roads, vegetation, etc., thereby improving the semantic hierarchy and visualization effect of the 3D model. Fusion of joint multi-view depth maps: Finally, by combining the depth maps of the reference view and the adjacent associated views, a multi-threshold fusion strategy is adopted to further optimize the fusion processing of the segmented image data. This process can more accurately integrate the depth information of different views and improve the consistency and accuracy of the segmentation results in 3D space. The viewing angle information is all the reference views that can be associated, and the color information is the average value of RGB of each point (r, g, b).

[0120]

[0121] It can be understood that through the above steps, the quality of the generated point cloud data is significantly improved. In practical applications, the accuracy and efficiency of point cloud data can be further improved by adjusting the feature detection and matching methods and optimizing the algorithm parameters. Through the above steps, this method realizes the high-precision generation and optimization of the reference image point cloud data, ensuring the integrity of the final 3D model and the accurate presentation of details.

[0122] S104: Acquire fifth aerial image data according to the second aerial image data through sub-block division and data reverse pulling.

[0123] Among them, the sub-block pulling technology is to use the three-dimensional point data of the key features generated in step S101 to divide the blocks, and then pull the original aerial image data to remove artifacts. The reverse pulling of the data is used to extract the camera position and image data in the sub-block. In this step, in the three-dimensional reconstruction process, especially when dealing with complex scenes, in order to improve the consistency of data and the accuracy of reconstruction, the entire scene is usually divided into multiple sub-blocks, and each sub-block is processed in detail. According to the sub-block division result, the camera position distribution position and image data corresponding to the sparse point cloud in each sub-block are reversely pulled. The reverse pulling is to calculate the feature points contained in the corresponding camera acquisition image in the current block sparse point, and the corresponding image data is pulled and selected accordingly, and the image is pre-processed to ensure the consistency of data quality in the sub-block. This step includes screening, alignment and format conversion of image data. In the three-dimensional reconstruction process, reverse pulling of the camera position distribution and image data corresponding to the sparse point cloud of the sub-block can improve the consistency of data and reconstruction accuracy.

[0124] Furthermore, the fifth aerial image data is obtained by sub-block division and data reverse pulling according to the second aerial image data, including: dividing the second aerial image data into multiple sub-blocks by sub-block division; generating sparse point cloud data corresponding to the sub-blocks by multi-view stereo matching and structure recovery algorithm, wherein the sparse point cloud data includes camera position and image data, and the sparse point cloud data is used to indicate the position and camera viewing angle information of each point in the three-dimensional scene; and processing the camera position and image data corresponding to the sub-block by data reverse pulling to obtain the fifth aerial image data. In specific implementation, it can be realized by the following steps:

[0125] (1) Sub-block division: First, the entire 3D scene is divided into multiple sub-blocks according to the complexity and features of the scene. This division is usually based on spatial geometric distribution to ensure that each sub-block contains relatively uniform point cloud data and features. For each sub-block, the sparse point cloud data corresponding to the sub-block is generated through multi-view stereo matching and structure recovery algorithm (SFM). The sparse point cloud contains the position of each point in the 3D scene and the related camera view information.

[0126] (2) Reversely pull the camera position and image data corresponding to the sparse point cloud. Reverse pulling means calculating the camera position and image data associated with the sparse point cloud generated in the current sub-block, and extracting and processing these image data to ensure the integrity and consistency of the data. For the sparse point cloud data of each sub-block, first calculate the image feature points associated with each 3D point. Based on these feature points, determine the corresponding relationship under different camera perspectives, and reversely track the images to which these feature points belong. Extract the image data to which the identified feature points belong from the original data set as the basis for subsequent processing. Reversely pull the camera position and image data corresponding to the sparse point cloud to calculate the feature point distribution: For the sparse point cloud data in each sub-block, calculate the feature point distribution of these points in the image. Select the camera position: According to the distribution of the feature points in the image, select the camera position that shoots these feature points. Reversely pull the image data: Reversely pull the image data corresponding to the selected camera to ensure the data quality in each sub-block.

[0127] Assume that the sparse point cloud data in a sub-block is The feature points of the corresponding image are Calculate the distribution of these feature points in the image:

[0128] p i =KRP i +t

[0129] Among them, K is the camera intrinsic parameter matrix, R and t are the rotation and translation matrices of the camera.

[0130] It can be understood that through the above steps, it is possible to ensure that the point cloud data in each sub-block has high consistency and accuracy, thereby improving the overall quality of three-dimensional reconstruction.

[0131] S105: Acquire and output three-dimensional individualized data through image preprocessing and motion artifact removal according to the fifth aerial image data and the fourth aerial image data.

[0132] Furthermore, the method of obtaining and outputting three-dimensional individualized data through image preprocessing and removing motion artifacts according to the fifth aerial image data and the fourth aerial image data includes: screening images that meet the screening standard parameters according to the fifth aerial image data to obtain the fifth aerial image data after screening, wherein the screening standard parameters include: image clarity, overlap, exposure and integrity of image content; detecting the motion area in the fifth aerial image data after screening by the optical flow method; repairing the detected motion area to obtain the fifth aerial image data after removing motion artifacts, fusing the fifth aerial image data after removing motion artifacts, and mapping the fused data to the fourth aerial image data to obtain and output three-dimensional individualized data. In specific implementation, it can be achieved through the following steps:

[0133] (1) Image preprocessing: The pulled image data is screened to retain images with high quality and high overlap. The screening criteria include image clarity, exposure, and integrity of image content. The screened images are aligned to ensure spatial consistency of images in the same sub-block. Feature point matching and homography transformation can be used for image alignment. The aligned images are format converted to unify the resolution, color space, and file format for subsequent processing. The pulled image data is strictly screened to remove images that are not suitable for reconstruction due to perspective deviation or quality problems. The screened images are accurately aligned to ensure that all images in the same sub-block are geometrically consistent, thereby reducing possible errors in the reconstruction process. If needed, the image data is converted into a unified format for subsequent processing and analysis. Based on image alignment and format conversion, the color and contrast of the image data are further optimized to enhance the detail performance during 3D reconstruction.

[0134] (2) Removing motion artifacts: Motion artifacts are the phenomenon of image blur or repetition caused by the movement of the camera or object. The steps to remove motion artifacts include: Motion detection: Detect the moving area in the image through optical flow method or other motion detection algorithms. Artifact removal: Repair the detected moving area, such as using image interpolation, deblurring algorithm, etc. Image fusion: Fusion of the image after removing motion artifacts to ensure the smoothness and consistency of the image.

[0135] (3) Mapping the fused data to the fourth aerial image data: Mapping the fused data to the fourth aerial image data by means of UV mapping coordinates (UV Coordinates). UV mapping coordinates (UV Coordinates) refer to texture coordinates on the mesh surface, which are used to map the texture to the mesh surface. Each vertex has a corresponding UV mapping coordinate.

[0136] It can be understood that, through the above steps, it is possible to further ensure that the point cloud data in each sub-block has high consistency and accuracy, thereby improving the overall quality of three-dimensional reconstruction.

[0137] The three-dimensional monomerization method based on image segmentation provided in the present application first obtains the first aerial image data, reconstructs and optimizes the geometric position relationship between the first aerial image data through the incremental motion recovery algorithm and the bundle adjustment technology, and obtains the second aerial image data; the first aerial image data is accurately segmented by the two-dimensional image segmentation technology to obtain the third aerial image data; the second aerial image data and the third aerial image data are subjected to depth map fusion to obtain the fourth aerial image data; according to the second aerial image data, the fifth aerial image data is obtained through sub-block division and reverse data pulling; according to the fifth aerial image data and the fourth aerial image data, the three-dimensional monomerization data is obtained and output through image preprocessing and removal of motion artifacts. The problem of high resource consumption and high computational difficulty of traditional three-dimensional model data segmentation is solved, thereby improving the accuracy and efficiency of real-scene three-dimensional modeling, improving the accuracy of building segmentation, reducing the need for manual correction of modeling results, and making three-dimensional modeling in complex scenes more efficient and automated.

[0138] Figure 2 A schematic diagram of the structure of a three-dimensional monomerization device based on image segmentation provided in this application; Figure 2 As shown, the present application provides a three-dimensional monomerization device 200 based on image segmentation, the device comprising:

[0139] An acquisition module 201, wherein the acquisition module 201 is used to acquire first aerial image data;

[0140] A processing module 202, the processing module 202 is used to reconstruct and optimize the geometric position relationship between the first aerial image data through an incremental motion recovery algorithm and a bundle adjustment technique to obtain second aerial image data;

[0141] The processing module 202 is further configured to use a two-dimensional image segmentation technique to accurately segment the first aerial image data to obtain third aerial image data;

[0142] The processing module 202 is further configured to perform depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data;

[0143] The processing module 202 is further configured to obtain fifth aerial image data through sub-block division and data reverse extraction according to the second aerial image data;

[0144] The processing module 202 is further configured to obtain three-dimensional monomer data by image preprocessing and removing motion artifacts according to the fifth aerial image data;

[0145] The output module 203 is used to output the three-dimensional monomerized data.

[0146] Optionally, the acquisition module 201 is specifically used to obtain first aerial image data by preprocessing and extracting feature points of image data collected by drone aerial photography, wherein the preprocessing includes: image denoising, color correction and resolution adjustment; reconstructing the geometric position relationship between the first aerial image data through the incremental motion recovery algorithm to obtain the reconstructed first aerial image data; optimizing the reconstructed first aerial image data through the bundle adjustment technology to obtain the second aerial image data.

[0147] Optionally, the processing module 202 is specifically used to accurately segment the first aerial image data through the DeepLab v3 model to obtain third aerial image data, wherein the DeepLab v3 model includes a dilated convolution module and a dilated spatial pyramid pooling module, and the DeepLab v3 model is used to specifically implement two-dimensional image segmentation technology.

[0148] Optionally, the processing module 202 is specifically used to use the void convolution module and the void spatial pyramid pooling module to extract the multi-scale features of the aerial image data to obtain the first aerial image data after segmentation; optimize the boundaries of different objects in the first aerial image data after segmentation through a fully connected conditional random field to obtain the third aerial image data.

[0149] Optionally, the acquisition module 201 is specifically used to acquire depth maps of the reference view and the referenced view, and project depth values ​​of the depth maps into a three-dimensional space to obtain the three-dimensional point cloud data;

[0150] The processing module 202 is specifically used to perform color information fusion classification on the three-dimensional point cloud data to obtain fourth aerial image data.

[0151] Optionally, the processing module 202 is specifically used to divide the second aerial image data into multiple sub-blocks through sub-block division; generate sparse point cloud data corresponding to the sub-blocks through multi-view stereo matching and structure recovery algorithm, wherein the sparse point cloud data includes camera position and image data, and the sparse point cloud data is used to indicate the position and camera viewing angle information of each point in the three-dimensional scene; and obtain fifth aerial image data by reverse pulling the camera position and image data corresponding to the sub-block.

[0152] Optionally, the processing module 202 is specifically used to screen out images that meet screening standard parameters based on the fifth aerial image data to obtain the screened fifth aerial image data, wherein the screening standard parameters include: image clarity, overlap, exposure and integrity of image content; detect the motion area in the screened fifth aerial image data through the optical flow method; repair the detected motion area to obtain the fifth aerial image data after removing motion artifacts, fuse the fifth aerial image data after removing motion artifacts, and map the fused data to the fourth aerial image data to obtain three-dimensional monomer data.

[0153] Figure 3 A schematic diagram of the structure of a three-dimensional monomer device based on image segmentation provided in this application, such as Figure 3 As shown, the three-dimensional individualization device 300 based on image segmentation provided in this embodiment includes:

[0154] Memory 302;

[0155] Processor 301;

[0156] Wherein, the memory 302 stores computer-executable instructions;

[0157] The processor 301 executes the computer-executable instructions stored in the memory to implement the three-dimensional singulation method based on image segmentation as shown in the above embodiment.

[0158] The present application also provides a computer storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the three-dimensional monomerization method based on image segmentation shown in the above embodiment.

[0159] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0160] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0161] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A three-dimensional monomerization method based on image segmentation, characterized in that: The method comprises: Acquire first aerial image data, reconstruct and optimize the geometric position relationship between the first aerial image data by using an incremental motion restoration algorithm and a bundle adjustment technique, and obtain second aerial image data; Using a two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data; Performing depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data; According to the second aerial image data, fifth aerial image data is obtained by sub-block division and data reverse pulling; According to the fifth aerial image data and the fourth aerial image data, three-dimensional individualized data is acquired and outputted through image preprocessing and motion artifact removal.

2. The method according to claim 1, characterized in that The step of acquiring the first aerial image data, reconstructing and optimizing the geometric position relationship between the first aerial image data by using an incremental motion restoration algorithm and a bundle adjustment technique, and obtaining the second aerial image data comprises: The first aerial image data is obtained by preprocessing and extracting feature points of the image data collected by the drone aerial photography, wherein the preprocessing includes: image denoising, color correction and resolution adjustment; Reconstructing the geometric position relationship between the first aerial image data by using the incremental motion recovery algorithm to obtain the reconstructed first aerial image data; The reconstructed first aerial image data is optimized by using bundle adjustment technology to obtain second aerial image data.

3. The method according to claim 1, characterized in that The step of using a two-dimensional image segmentation technology to accurately segment the first aerial image data to obtain third aerial image data includes: The first aerial image data is accurately segmented through the DeepLab v3 model to obtain third aerial image data, wherein the DeepLab v3 model includes a dilated convolution module and a dilated spatial pyramid pooling module, and the DeepLab v3 model is used to specifically implement the two-dimensional image segmentation technology.

4. The method according to claim 3, characterized in that The method of accurately segmenting the first aerial image data by using the DeepLab v3 model to obtain the third aerial image data includes: The atrous convolution module and the atrous spatial pyramid pooling module are used to extract multi-scale features of the first aerial image data to obtain the segmented first aerial image data; The boundaries of different objects in the segmented first aerial image data are optimized by a fully connected conditional random field to obtain third aerial image data.

5. The method according to claim 1, characterized in that The first aerial image data includes a reference view, the second aerial image data includes a reference view, and the second aerial image data and the third aerial image data are subjected to depth map fusion to obtain fourth aerial image data, including: Acquire depth maps of the reference view and the referenced view, and project depth values ​​of the depth maps into a three-dimensional space to obtain three-dimensional point cloud data; Color information fusion and classification are performed on the three-dimensional point cloud data to obtain fourth aerial image data.

6. The method according to claim 1, characterized in that The step of obtaining fifth aerial image data by sub-block division and reverse data pulling according to the second aerial image data includes: Dividing the second aerial image data into a plurality of sub-blocks by sub-block division; Generate sparse point cloud data corresponding to the sub-block by multi-view stereo matching and structure recovery algorithm, wherein the sparse point cloud data includes camera position and image data, and the sparse point cloud data is used to indicate the position of each point in the three-dimensional scene and the camera viewing angle information; The camera position and image data corresponding to the sub-block are processed by reverse data pulling to obtain fifth aerial image data.

7. The method according to claim 1, characterized in that The method of obtaining and outputting three-dimensional monomerized data by image preprocessing and removing motion artifacts according to the fifth aerial image data and the fourth aerial image data includes: According to the fifth aerial image data, images meeting the screening standard parameters are screened to obtain the screened fifth aerial image data, wherein the screening standard parameters include: image clarity, overlap, exposure, and integrity of image content; Detecting the motion area in the filtered fifth aerial image data by an optical flow method; The detected motion area is repaired to obtain fifth aerial image data after removing motion artifacts, the fifth aerial image data after removing motion artifacts is fused, and the fused data is mapped to the fourth aerial image data to obtain and output three-dimensional monomerized data.

8. A three-dimensional singularization device based on image segmentation, characterized in that: The device is applied to the three-dimensional monomerization method based on image segmentation according to any one of claims 1 to 7, and the device comprises: An acquisition module, the acquisition module is used to acquire first aerial image data; A processing module, the processing module is used to reconstruct and optimize the geometric position relationship between the first aerial image data through an incremental motion recovery algorithm and a bundle adjustment technology to obtain second aerial image data; A processing module, the processing module is further used to accurately segment the first aerial image data using a two-dimensional image segmentation technology to obtain third aerial image data; A processing module, the processing module is further used to perform depth map fusion on the second aerial image data and the third aerial image data to obtain fourth aerial image data; A processing module, the processing module is further used to obtain fifth aerial image data through sub-block division and data reverse pulling according to the second aerial image data; A processing module, the processing module is further used to obtain three-dimensional monomer data through image preprocessing and motion artifact removal according to the fifth aerial image data; An output module, wherein the output module is used to output the three-dimensional monomerized data.

9. A three-dimensional singularization device based on image segmentation, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the three-dimensional monomerization method based on image segmentation as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer execution instructions, which, when executed by a processor, are used to implement the three-dimensional monomerization method based on image segmentation as described in any one of claims 1 to 7.

Images