Landscape three-dimensional reconstruction method and system based on multi-view assistance

Abstract

The invention relates to the technical field of image processing, in particular to a landscape three-dimensional reconstruction method and system based on multi-view assistance, and the method comprises the steps: obtaining a plurality of landscape images which comprise the landscape images of each view angle in a plurality of view angles at different moments; determining a matching point of each pixel point in the landscape image at adjacent moments of the same visual angle; based on the matching point of each pixel point, determining a motion vector of each pixel point and an overall motion vector of the landscape image where each pixel point is located; determining a moving interference point in the landscape image of each view angle based on the moving vector of each pixel point and the overall moving vector of the landscape image where each pixel point is located; modeling is carried out based on the target landscape image of each view angle, a target landscape model is obtained, and the target landscape image of one view angle is a landscape image with the highest information entropy and the mobile interference points are screened out. The method can obtain an accurate and effective landscape model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and system for three-dimensional landscape reconstruction based on multi-view assistance. Background Technology

[0002] The 3D reconstruction technology of landscape is used to digitize a landscape with high precision and obtain its corresponding 3D landscape model, thereby accurately displaying the actual terrain, vegetation, building distribution and other aspects of the landscape.

[0003] In related technologies, it is usually necessary to acquire multi-view images of the landscape area to be tested, and then match the multi-view images based on feature point matching to obtain a landscape model.

[0004] However, in the above methods, since the drone itself is not completely stable during operation, and plants, light and shadow are also subject to dynamic shaking interference, the landscape model obtained based on the actual multi-view images may have local blurring and splicing misalignment, resulting in low accuracy of the landscape model. Summary of the Invention

[0005] To address the technical problem of low accuracy in landscape models obtained from multi-view images acquired in reality, the present invention aims to provide a method and system for three-dimensional landscape reconstruction based on multi-view assistance. The specific technical solution adopted is as follows:

[0006] This application provides a method for three-dimensional landscape reconstruction based on multi-view assistance, including:

[0007] Multiple landscape images are acquired, including landscape images from each viewpoint at different times. Matching points for each pixel in each landscape image are determined in adjacent landscape images from the same viewpoint. Based on the matching points for each pixel, the movement vector of each pixel and the overall movement vector of the landscape image containing that pixel are determined. Based on the movement vectors of each pixel and the overall movement vector of the landscape image containing that pixel, moving interference points in the landscape images of each viewpoint are determined. A target landscape model is obtained by modeling the target landscape image for each viewpoint. The target landscape image for a given viewpoint is the landscape image with the highest information entropy after filtering out moving interference points.

[0008] Optionally, the above-mentioned determination of the matching point of each pixel in each landscape image at adjacent times in the same viewpoint includes: determining the gray-level distribution feature of each pixel based on the gray-level value of each pixel in the landscape image at adjacent times, the gray-level distribution feature being used to characterize the gray-level performance of the pixel in the landscape image; determining the matching degree of the first pixel with each pixel in the second landscape image based on the gray-level distribution feature, similarity, and Euclidean distance of the first pixel and each pixel in the second landscape image, the first pixel being any pixel in the first landscape image, the first landscape image being the landscape image at an adjacent time of the second landscape image; and determining the pixel with the highest matching degree with the first pixel as the matching point of the first pixel.

[0009] Optionally, the above-mentioned determination of the overall motion vector of the landscape image where each pixel is located based on the matching point of each pixel includes: determining the cosine similarity of the motion vector of each pixel based on the matching point of each pixel and the motion vector of each pixel, wherein the cosine similarity of the motion vector of a pixel is the cosine similarity between the motion vector of the pixel and the motion vector of its matching point; determining the stable point of the landscape image at each viewpoint based on the cosine similarity of the motion vector of each pixel; and determining the sum of the motion vectors of the stable points in each landscape image as the overall motion vector of each landscape image.

[0010] Optionally, determining stable points of landscape images from each viewpoint based on the cosine similarity of the movement vectors of each pixel includes: determining the corresponding point of the first pixel in each landscape image from the same viewpoint; determining the stability of the first pixel based on the range of the cosine similarity of the movement vectors of the first pixel and the corresponding point, the mean of the cosine similarity of the movement vectors of the first pixel and the corresponding point, the mean of the magnitude of the movement vectors of the first pixel and the corresponding point, and the mean of the magnitude of the movement vectors of all pixels in the multiple landscape images; and determining the first pixel as a stable point if the stability of the first pixel is greater than a stability threshold.

[0011] Optionally, the above-mentioned determination of motion interference points in the landscape image of each viewpoint based on the motion vector of each pixel and the overall motion vector of the landscape image where each pixel is located includes: determining the mobility of the first pixel based on the sum of the cosine similarity of the overall motion vectors of the first pixel and its corresponding point, the mean of the matching degree of the first pixel and its corresponding point, and the mean of the matching degree of all pixels and their respective corresponding points in all landscape images of the first viewpoint. The overall motion vector cosine similarity of a pixel is: the cosine similarity between the motion vector of the pixel and the overall motion vector of the landscape image where the pixel is located; the matching degree of a pixel is the matching degree between the pixel and its matching point; if the mobility of the first pixel is greater than the mobility threshold, the first pixel is determined as a motion interference point.

[0012] Optionally, the above-mentioned modeling of the target landscape image based on each viewpoint to obtain the target landscape model includes: determining edge points in each target landscape image; determining the feature point probability of each edge point based on the regional stability index and geometric contribution index of each edge point, wherein the regional stability index of an edge point is used to describe the displacement stability of the region where the edge point is located, and the geometric contribution index of an edge point is used to characterize the gray-scale change of the edge point; determining feature points in each target landscape image based on the feature point probability of each edge point; and modeling based on the feature points in each target landscape image to obtain the target landscape model.

[0013] Optionally, the method further includes: determining the regional stability index of the first edge point based on the sum of the cosine similarity of the overall movement vectors of the center points of the regions where the first edge point and its corresponding point are located, and the mean of the magnitudes of the movement vectors of the first edge point and its corresponding point.

[0014] Optionally, the method further includes: determining all edge points within the region where the first edge point is located; and determining the geometric contribution index of the first edge point based on the gray-level gradient magnitude of the first edge point, the mean of the gray-level gradient magnitudes of all edge points, and the curvature of the first edge point.

[0015] Optionally, the above-mentioned determination of feature points in each target landscape image based on the feature point probability of each edge point includes: determining the overlapping region between the first target landscape image and the second target landscape image; determining the initial number of feature points in the overlapping region of the first target landscape image and the initial number of feature points in the overlapping region of the second target landscape image, wherein the initial feature points are edge points whose feature point probability is greater than a feature point probability threshold; if the initial number of feature points in the overlapping region of the first target landscape image is less than the initial number of feature points in the overlapping region of the second target landscape image, determining a target number, wherein the target number is the initial number of feature points in the overlapping region of the second target landscape image; and determining the feature points in the overlapping region of the first target landscape image in descending order of feature point probability based on the target number.

[0016] This application also proposes a multi-view-assisted landscape 3D reconstruction system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any of the above-described multi-view-assisted landscape 3D reconstruction methods.

[0017] This application has the following beneficial effects:

[0018] In this embodiment, pixel matching is performed on landscape images at adjacent times from the same viewpoint to determine the movement trend of each pixel in the temporal sequence and the overall movement trend of the landscape image. Based on the movement trend, unstable moving interference points with large movement trends in the landscape image are identified. Finally, by filtering out these moving interference points, the target landscape image with the highest information entropy is determined. Since the target landscape image does not have moving interference points, the three-dimensional model obtained by modeling based on the target landscape image is an effective and accurate landscape model. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a multi-view-assisted landscape 3D reconstruction method provided in one embodiment of the present invention.

[0021] Figure 2 This is a flowchart of another landscape 3D reconstruction method based on multi-view assistance provided in an embodiment of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multi-view-assisted landscape 3D reconstruction method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] Landscape models obtained through 3D landscape reconstruction technology can facilitate planning decisions for landscape areas or be used for teaching demonstrations. In actual modeling, it is necessary to acquire actual detection images of a specific area of ​​the landscape (usually multi-view images), and then perform 3D modeling of the landscape area based on these images.

[0025] Existing methods typically require multi-view images of a landscape area to be tested, acquired using drones, to be matched using methods such as feature point matching, in order to obtain a landscape model.

[0026] In actual landscape modeling, the complex structure of the landscape scene itself (e.g., complex vegetation or similar structures, such as similar and regular ground paving areas) makes it difficult for existing methods to accurately determine effective feature points when matching them, affecting the matching effect. In addition, when actually acquiring multi-view images, the operation of the drone itself is usually not completely stable, and the plants also have dynamic shaking interference, which makes it impossible to obtain an accurate and effective 3D model, resulting in local blurring of the point cloud, and the cumulative error may cause misalignment when stitching images from different perspectives.

[0027] The following description, in conjunction with the accompanying drawings, details the specific scheme of a multi-view-assisted landscape 3D reconstruction method and system provided by the present invention.

[0028] Please see Figure 1 The diagram shows a flowchart of a multi-view-assisted landscape 3D reconstruction method provided by an embodiment of the present invention.

[0029] like Figure 1 As shown, the multi-view-assisted landscape 3D reconstruction method includes S101-S105.

[0030] S101. Acquire multiple landscape images.

[0031] The multiple landscape images include landscape images from each of multiple perspectives at different times.

[0032] In one alternative implementation, the multiple landscape images can be acquired through a drone inspection process.

[0033] Optionally, multiple data collection points can be set up along the drone inspection route. Each data collection point can be understood as a viewpoint, and multiple landscape images can be continuously collected at each data collection point.

[0034] It should be understood that each landscape image includes its corresponding acquisition time.

[0035] Optionally, the drone can perform inspections by flying in a grid pattern, where each grid point can be understood as a data collection point.

[0036] It should be noted that landscape images acquired from adjacent viewpoints should contain at least 30% overlapping areas.

[0037] In one alternative implementation, after acquiring multiple landscape images, these images can be preprocessed, and subsequent operations can be performed based on the preprocessed landscape images.

[0038] Optionally, the preprocessing can be filtering and denoising, grayscale processing, etc.

[0039] Optionally, the multi-view assisted landscape 3D reconstruction system includes an edge processing module, which allows the drone to wirelessly transmit and store multiple landscape images to the system's edge processing module.

[0040] For example, the multiple perspectives can be 6 perspectives, and each perspective can capture landscape images at 3 time points.

[0041] S102. Determine the matching points of each pixel in each landscape image at adjacent times in the same viewpoint.

[0042] It should be understood that the area collected from each perspective is the same. Due to the movement of plants or drones, there are certain differences in the landscape images at different times from each perspective. The same pixel may be displaced in different landscape images. At this time, the matching point corresponding to each pixel in the landscape images at adjacent times (hereinafter referred to as adjacent images) can be determined, and then the displacement of each pixel in the adjacent images can be determined.

[0043] In one alternative implementation, landscape images from the same viewpoint can be arranged chronologically to obtain adjacent images for each landscape image.

[0044] In one implementation of this application, the matching point of each pixel can be determined based on the grayscale distribution characteristics of each pixel.

[0045] It should be understood that grayscale distribution features are used to characterize the grayscale representation of pixels in landscape images.

[0046] The following uses any pixel in any landscape image (such as the first landscape image) as an example to illustrate the process of determining the matching point of the first pixel in the landscape image (hereinafter referred to as the second landscape image) at an adjacent time.

[0047] Step 1: Determine the grayscale distribution characteristics of each pixel based on the grayscale values ​​of each pixel in the landscape images at adjacent time points.

[0048] Specifically, the grayscale distribution characteristics of the first pixel are determined based on the grayscale value of the first pixel and the overall grayscale value of the first landscape image.

[0049] Optionally, the average grayscale value of each pixel in the first landscape image can be determined as the overall grayscale value of the first landscape image.

[0050] Optionally, the ratio of the gray value of the first pixel to the overall gray value of the first landscape image can be determined as the gray distribution feature of the first pixel.

[0051] Similarly, the ratio of the gray value of each pixel in the second landscape image to the overall gray value of the second landscape image can be used to determine the gray distribution feature of each pixel in the second landscape image.

[0052] Step 2: Based on the grayscale distribution features, similarity, and Euclidean distance between the first pixel and each pixel in the second landscape image, determine the matching degree between the first pixel and each pixel in the second landscape image.

[0053] Specifically, the gray-level distribution difference between the first pixel and each pixel in the second landscape image can be determined based on the difference in gray-level distribution features between the first pixel and each pixel in the second landscape image. Based on this gray-level distribution difference, cosine similarity, and Euclidean distance, the matching degree of each pixel can be determined.

[0054] It should be understood that the matching degree between two pixels is used to characterize the probability that the two pixels belong to the same landscape structure. The higher the matching degree, the greater the probability that the two pixels belong to the same landscape structure.

[0055] Understandably, the cosine similarity between two vectors can be used to characterize the degree of similarity between the two vectors.

[0056] In this embodiment of the application, the similarity between two pixels can be determined based on the cosine similarity between the gradient vectors of the two pixels.

[0057] It should be understood that the higher the cosine similarity of the vectors of two pixels, the more similar the gradient vectors of the two pixels are.

[0058] Understandably, Euclidean distance is used to characterize the spatial distance between two pixels. The closer the spatial distance, the greater the likelihood that the two pixels belong to the same landscape structure.

[0059] Optionally, the matching degree between two pixels satisfies the following formula:

[0060]

[0061] in, Represents pixels and pixels The degree of matching, Represents pixels and pixels similarity, Representing landscape images medium pixel The grayscale distribution characteristics, Representing landscape images medium pixel The grayscale distribution characteristics, Represents pixels and pixels Euclidean distance.

[0062] It should be noted that the image With images The denominator of the above formula is 0.01 to avoid a denominator of 0, which represents landscape images at adjacent times.

[0063] Optionally, the grayscale distribution characteristics and Euclidean distance mentioned above can be standardized to avoid the influence of dimensions.

[0064] Based on the above formula, it should be understood that if the pixel point With pixels The closer the spatial positions, the smaller the overall grayscale variation, and the more similar the gradient directions of grayscale at the two points, the better the pixel... With pixels The higher the match, the better.

[0065] Step 3: Determine the pixel with the highest matching degree to the first pixel as the matching point of the first pixel.

[0066] It should be understood that the higher the matching degree, the greater the probability that the two pixels belong to the same landscape structure. Therefore, when the matching degree between the first pixel and a certain pixel (such as the second pixel) is the highest, it means that the landscape structure of the second pixel is the most likely to be the same as the landscape structure of the first pixel in the adjacent image. At this time, the second pixel can be determined as the matching point of the first pixel.

[0067] In one alternative implementation, based on steps one through three above, matching points for each pixel in the first landscape image can be determined sequentially.

[0068] Optionally, after a pair of matching points is determined, the pair of matching points can be marked as invalid pixels, and these invalid pixels will not be included in the calculation when other matching points are determined later.

[0069] S103. Based on the matching points of each pixel, determine the movement vector of each pixel and the overall movement vector of the landscape image where each pixel is located.

[0070] It should be understood that the movement vector of a pixel is used to characterize the positional change of that pixel in adjacent time intervals.

[0071] Optionally, taking the first pixel as an example, the direction pointing from the first pixel to the coordinates of the matching point of the first pixel can be determined as the movement vector of the first pixel.

[0072] It should be understood that the overall movement vector is used to characterize the overall offset trend of the landscape image.

[0073] Optionally, the sum of the movement vectors of all pixels in a landscape image can be used to determine the overall movement vector.

[0074] It is understandable that due to drone movement or plant movement, landscape images taken at adjacent moments may exhibit overall and local shifts. In such cases, determining the overall positional trend of the landscape based on local shifts may be inaccurate. It is advisable to first identify stable points in the landscape image, and then determine the overall movement vector based on the movement vectors of these stable points.

[0075] In one alternative implementation, the cosine similarity of the movement vector of each pixel can be determined based on the matching point of each pixel and the movement vector of each pixel. Based on the cosine similarity of the movement vector of each pixel, the stable point of the landscape image from each viewpoint can be determined. Finally, the sum of the movement vectors of the stable points in each landscape image is determined as the overall movement vector of each landscape image.

[0076] It should be understood that the cosine similarity of the movement vector of a pixel is the cosine similarity between the movement vector of that pixel and the movement vector of its matching point, and the stable point is the pixel with the smaller movement amplitude in adjacent time moments.

[0077] Optionally, since drone attitude deviations may cause overall displacement of objects in the image, while local displacements or local imaging afterimages caused by factors such as plant swaying usually do not conform to the overall displacement trend of the image. For example, the same building may have coordinate deviations in different images, but the displacement trend and grayscale performance are usually relatively stable. Therefore, combining the displacement of stable points in the landscape image to analyze the overall displacement trend of the landscape image, the obtained overall movement vector is more referential and effective.

[0078] In one implementation of this application, the stable point of the landscape image from each viewpoint is determined based on the cosine similarity of the movement vector of each pixel, specifically including steps four to six.

[0079] Step 4: Determine the corresponding point of the first pixel in each landscape image from the same viewpoint.

[0080] It should be understood that the corresponding point of a pixel is the pixel in other landscape images that corresponds to the landscape structure of that pixel. The corresponding point of a pixel includes: the matching point of each pixel, the matching point of the matching point in another adjacent landscape image, etc.

[0081] Optionally, the corresponding point of the first pixel in each landscape image from the same viewpoint can be determined sequentially according to the temporal order of the landscape images.

[0082] For example, assuming the images are acquired in sequence as image 1, image 2, and image 3, the matching point of pixel 1 in image 1 in image 2 is pixel 2, and the matching point of pixel 2 in image 2 in image 3 is pixel 3, then the corresponding points of pixel 1 include pixel 2 and pixel 3.

[0083] Step 5: Determine the stability of the first pixel based on the range of the cosine similarity between the first pixel and its corresponding point, the mean of the cosine similarity between the first pixel and its corresponding point, the mean of the magnitude of the movement vector between the first pixel and its corresponding point, and the mean of the magnitude of the movement vector of all pixels in multiple landscape images.

[0084] It should be understood that after determining the movement vector of a pixel, the magnitude of that movement vector can be obtained.

[0085] Optionally, the stability of a pixel satisfies the following formula:

[0086]

[0087] in, Represents pixels stability, Represents pixels The mean of the cosine similarity between the movement vectors of the points and their corresponding points. Represents pixels The range of the cosine similarity between the movement vectors of the corresponding points and their corresponding points. This represents the mean of the magnitudes of the motion vectors of all pixels in multiple landscape images. Represents pixels The mean of the magnitude of the movement vector of the corresponding point.

[0088] Based on the above formula, it should be understood that, The larger the value, the more pixels there are. The more stable the offset direction, The larger the value, the more pixels there are. The smaller the degree of offset.

[0089] Step 6: If the stability of the first pixel is greater than the stability threshold, then the first pixel is determined as a stable point.

[0090] It is understandable that if the probability of a stable point for the first pixel is greater than the stability threshold, it indicates that the offset direction of the first pixel is relatively stable and the degree of offset is small.

[0091] Optionally, stability can be normalized.

[0092] For example, suppose that after normalization of the stabilization, the stability threshold can be 0.5.

[0093] Optionally, the stability of each pixel can be determined based on the methods described in steps four to six above.

[0094] It should be understood that determining the stability of a pixel based on the stability and degree of its offset direction, and identifying pixels with stable offset direction and smaller degree of offset as stable points, is accurate and reliable.

[0095] S104. Based on the movement vector of each pixel and the overall movement vector of the landscape image where each pixel is located, determine the moving interference points in the landscape image of each viewpoint.

[0096] It should be understood that moving interference points are pixels with a large degree of offset and whose movement trend does not conform to the overall offset trend.

[0097] In one implementation of this application, taking a first pixel as an example, the mobility of the first pixel can be determined based on the sum of the cosine similarity of the overall motion vectors of the first pixel and its corresponding point, the mean of the matching degree of the first pixel and its corresponding point, and the mean of the matching degree of all pixels and their respective corresponding points in all landscape images of the first viewpoint. If the mobility of the first pixel is greater than the mobility threshold, the first pixel is determined as a motion interference point.

[0098] The cosine similarity of the overall movement vector of a pixel is: the cosine similarity between the movement vector of the pixel and the overall movement vector of the landscape image in which the pixel is located; the matching degree of a pixel is the matching degree between the pixel and its matching point.

[0099] Alternatively, the mobility of a pixel satisfies the following formula:

[0100]

[0101] in, Represents pixels Mobility Represents pixels and pixels The sum of the cosine similarities of the overall movement vectors of the corresponding points. This indicates the pixel point The mean of the matching degree between all pixels and their corresponding points in all landscape images from the given viewpoint. Represents pixels and pixels The mean of the matching degree of the corresponding points.

[0102] Based on the above formula, it can be understood that, The larger the value, the more pixels there are. The worse the match with the rest of the image, the worse the pixel's appearance. The less reliable it is.

[0103] Optionally, the mobility can be normalized by setting the mobility threshold to 0.5. If the mobility of a pixel is greater than 0.5, the pixel is identified as a motion interference point.

[0104] Understandably, identifying unreliable pixels as moving interference points can lead to a more effective landscape image when these moving interference points are subsequently removed.

[0105] S105. Model the target landscape image based on the target landscape image from each viewpoint to obtain the target landscape image.

[0106] Among them, the target landscape image from one perspective is the landscape image with the highest information entropy after filtering out moving interference points.

[0107] In one alternative implementation, the information entropy of each landscape image among multiple landscape images from each viewpoint can be determined, and then the landscape image with the highest information entropy can be identified. The moving interference points identified above are then filtered out from this landscape image, thus obtaining the target landscape image for each viewpoint.

[0108] It should be understood that the higher the entropy value of a landscape image, the more uniform the grayscale distribution of the landscape image, the more details and texture information it contains, and the "richer" the image. Therefore, 3D modeling of the target landscape image determined based on this entropy can yield a clearer landscape model with more details and textures.

[0109] In this embodiment, pixel matching is performed on landscape images at adjacent times from the same viewpoint to determine the movement trend of each pixel in the temporal sequence and the overall movement trend of the landscape image. Based on the movement trend, unstable moving interference points with large movement trends in the landscape image are identified. Finally, by filtering out these moving interference points, the target landscape image with the highest information entropy is determined. Since the target landscape image does not have moving interference points, the three-dimensional model obtained by modeling based on the target landscape image is an effective and accurate landscape model.

[0110] Combination Figure 1 ,like Figure 2 As shown, in one implementation of this application embodiment, the above-mentioned modeling based on the target landscape image from each viewpoint to obtain the target landscape model specifically includes S201-S204.

[0111] S201. Determine the edge points in each target landscape image.

[0112] It should be understood that edge points in a landscape image are points of abrupt changes in grayscale.

[0113] Alternatively, the edge point can be determined based on the grayscale value of each pixel in the target landscape image.

[0114] In one alternative implementation, the Sobel operator can be used to segment the target landscape image, morphological closing operations can be used to fill in the holes, the closed shapes within can be identified as the region to be measured, and edge points can be determined from the region to be measured.

[0115] It should be understood that a target landscape image may contain one or more regions to be measured, and a region to be measured may contain one or more edge points.

[0116] S202. Based on the regional stability index and geometric contribution index of each edge point, determine the probability of feature points for each edge point.

[0117] Among them, the regional stability index of an edge point is used to describe the displacement stability of the region where the edge point is located, and the geometric contribution index of an edge point is used to characterize the gray-scale change of the edge point.

[0118] It should be understood that the feature point probability of an edge point is used to characterize the likelihood that the edge point is a feature point.

[0119] In one alternative implementation, the displacement stability of a region can be determined by the displacement of the center point of the region where the edge point is located. Specifically, when the coordinate position of the center point of the region changes little and the movement conforms to the overall motion trend, the region stability index of the edge point is high.

[0120] Optionally, taking the first edge point as an example, the regional stability index of the first edge point can be determined based on the sum of the cosine similarity of the overall movement vectors of the first edge point and the center point of the region where the first edge point and its corresponding point are located, and the mean of the magnitude of the movement vectors of the first edge point and its corresponding point.

[0121] Optionally, the regional stability index of an edge point satisfies the following formula:

[0122]

[0123] in, Represents edge points The regional stability index, Represents edge points in multiple landscape images from the same viewpoint and edge points The sum of the cosine similarities of the overall movement vectors of the center points of the regions where the corresponding points are located. Represents edge points in multiple landscape images from the same viewpoint and edge points The mean of the magnitude of the movement vector of the corresponding point.

[0124] Based on the above formula, it should be understood that, The smaller the value, the smaller the displacement of the center point of the region, and the greater the stability of the region where the edge point is located.

[0125] It should be noted that before performing calculations based on the above formula, the parameters in the formula can be standardized.

[0126] It should be understood that the gray-level gradient magnitude can reflect the gray-level change of the edge point, and the curvature of the edge point can reflect the degree of geometric structural change of the edge point. Therefore, the geometric contribution index of the edge point can be determined based on the gray-level gradient magnitude and curvature of the edge point.

[0127] In one alternative implementation, taking the first edge point as an example, all edge points within the region where the first edge point is located can be determined; based on the gray-level gradient magnitude of the first edge point, the mean of the gray-level gradient magnitudes of all edge points, and the curvature of the first edge point, the geometric contribution index of the first edge point can be determined.

[0128] Optionally, the geometric contribution index of an edge point satisfies the following formula:

[0129]

[0130] in, Represents edge points Geometric contribution index, Represents edge points grayscale gradient magnitude, Represents edge points The mean of the gray-level gradient magnitudes of all edge points within the region. Represents edge points The curvature.

[0131] Optionally, the chain code mutation value can also reflect the degree of geometric structural change of the edge point. Alternatively, the curvature of the first edge point can be replaced by the chain code diagram of the first edge point. That is, the geometric contribution index of the first edge point can be determined based on the gray-level gradient magnitude of the first edge point, the gray-level gradient magnitude of all edge points, and the chain code mutation value of the first edge point.

[0132] Optionally, the chain code values ​​of the 8-neighborhood of the edge where the first edge point is located are extracted, and the chain code values ​​are subjected to first-order difference according to the chain code acquisition order. The absolute value of the first-order difference between the first edge point and the previous chain code value is determined as the chain code mutation value.

[0133] In one alternative implementation, the probabilities of a feature point can be determined based on the edge point's regional stability index, geometric contribution index, mobility, and the range of Euclidean distances between the edge point and the center point of the region where the edge point is located in multiple landscape images from the same viewpoint.

[0134] Optionally, the feature point probability of an edge point satisfies the following formula:

[0135]

[0136] in, Represents edge points The probability of feature points, Represents edge points The regional stability index, Represents edge points Geometric contribution index, Represents edge points Mobility Represents edge points Edge points in multiple landscape images from the same viewpoint The range of the Euclidean distance from the center point of the region.

[0137] Based on the above formula, it should be understood that, The smaller the value, the more important the edge point is. The smaller the displacement relative to its region.

[0138] It should be noted that the parameters in the above formula can be standardized to eliminate the influence of dimensions.

[0139] Optionally, the probability of the feature point can be normalized.

[0140] S203. Determine the feature points in each target landscape image based on the feature point probability of each edge point.

[0141] It should be understood that feature points should be pixels that contribute significantly to the local landscape structure, meaning that their grayscale features and geometric representations differ significantly from other points in the vicinity, and that the area in which they are located has a high degree of spatial stability under the corresponding viewpoint.

[0142] In one alternative implementation, a feature point probability threshold can be set, and edge points that are greater than the feature point probability threshold can be identified as feature points.

[0143] For example, after normalizing the probability of feature points, the probability threshold of feature points can be set to 0.5.

[0144] In another alternative implementation, the structural complexity varies among different target landscape images. To prevent the image matching accuracy from being reduced due to an insufficient number of feature points, the feature points in each target landscape image can be adjusted based on the difference in the number of feature points in the overlapping areas of different target landscape images.

[0145] In one implementation of this application, taking a first target landscape as an example, the overlapping area between the first target landscape image and the second target landscape image can be determined; then, the initial number of feature points in the overlapping area of ​​the first target landscape image and the initial number of feature points in the overlapping area of ​​the second target landscape image are determined. If the initial number of feature points in the overlapping area of ​​the first target landscape image is less than the initial number of feature points in the overlapping area of ​​the second target landscape image, the target number is determined. Finally, based on the target number, the feature points in the overlapping area of ​​the first target landscape are determined in descending order of feature point probability.

[0146] It should be understood that the initial feature points are edge points whose feature point probability is greater than the feature point probability threshold; the target number is the initial number of feature points in the overlapping area of ​​the second target landscape image.

[0147] In one alternative implementation, a sliding window can be set up, and then the sliding window can be started from the upper left corner of the first target landscape image and the second target landscape image respectively. The perceptual hash values ​​in the two sliding windows are traversed, and the position of the sliding window with the most similar perceptual hash values ​​(i.e., the smallest Hamming distance) is determined as the overlapping area of ​​the first target landscape image and the second target landscape image.

[0148] Optionally, the sliding window can be an n×n rectangular sliding window with an area that occupies 30% of the target landscape image area.

[0149] Optionally, the initial number of feature points in two overlapping regions can be determined based on a feature point probability threshold.

[0150] It should be understood that if the number of initial feature points in the overlapping region of the first target landscape image is less than the number of initial feature points in the overlapping region of the second target landscape image, the number of feature points in the overlapping region of the first target landscape image can be increased. Specifically, the same number of feature points as the initial number of feature points in the overlapping region of the second target landscape image can be determined.

[0151] Understandably, sorting feature points from highest to lowest probability and determining the target number of feature points can provide the validity of feature points within overlapping areas, thus making landscape modeling more accurate.

[0152] It should be understood that when the number of feature points in the overlapping areas of different target landscape images is different, adjusting the feature point density to make them the same can improve the matching accuracy of different target landscape images.

[0153] S204. Model the target landscape based on the feature points in each target landscape image to obtain the target landscape model.

[0154] Specifically, target landscape images from different perspectives can be matched by combining spatial distribution characteristics and identified feature points.

[0155] In one alternative implementation, sampling can be performed based on the weights of the determined feature points and the random sample consensus (RANSAC) algorithm, and the target landscape images from different perspectives can be stitched together based on the principle of RANSAC to obtain a three-dimensional model of the landscape region, i.e., the target landscape model.

[0156] Optionally, when stitching two target landscape images from different perspectives using the RANSAC algorithm, if a pair of matching points selected does not belong to the same test area, the pair of matching points can be filtered out, and new matching points can be selected. Based on the matching points determined by the algorithm, the transformation matrix between the two target landscape images can be obtained, thereby achieving image stitching and obtaining the target landscape model for this inspection.

[0157] It should be understood that when the center points of two regions are the closest in coordinate distance within their overlapping range, the two regions are considered to be the same region.

[0158] Optionally, based on the above method, target landscape models that have been inspected multiple times can be obtained. Based on the projection error of the target landscape model corresponding to each inspection, the target landscape model with the smallest projection error can be determined as the final target landscape model.

[0159] It is understood that the technical solution of this application embodiment, by combining the difference between the local movement trend (e.g., plant swaying interference) and the overall movement trend (UAV pose change) in multi-view landscape images, filters out moving interference points, and determines the matching feature points of each view image based on the regional stability index and geometric contribution index of each point. Based on the feature point differences in the matched images, the density of the corresponding feature points of each image is adjusted, thereby improving the matching accuracy of multi-view landscape images and obtaining an accurate three-dimensional model of the landscape area.

[0160] This application also proposes a multi-view-assisted landscape 3D reconstruction system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the steps of the multi-view-assisted landscape 3D reconstruction method described above.

[0161] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for three-dimensional landscape reconstruction based on multi-view assistance, characterized in that, The method includes: Acquire multiple landscape images, which include landscape images from each of multiple viewpoints at different times; Identify matching points for each pixel in each landscape image at adjacent times within the same viewpoint; Based on the matching points of each pixel, determine the movement vector of each pixel and the overall movement vector of the landscape image where each pixel is located; Based on the movement vector of each pixel and the overall movement vector of the landscape image where each pixel is located, the moving interference points in the landscape image of each viewpoint are determined. Modeling is performed on the target landscape image from each viewpoint to obtain the target landscape model. The target landscape image from one viewpoint is the landscape image with the highest information entropy after filtering out moving interference points.

2. The landscape 3D reconstruction method based on multi-view assistance according to claim 1, characterized in that, The process of determining the matching point of each pixel in each landscape image at adjacent times within the same viewpoint includes: Based on the gray value of each pixel in the landscape image at adjacent time points, the gray distribution feature of each pixel is determined, and the gray distribution feature is used to characterize the gray level performance of the pixel in the landscape image. Based on the grayscale distribution features, similarity, and Euclidean distance between the first pixel and each pixel in the second landscape image, the matching degree between the first pixel and each pixel in the second landscape image is determined. The first pixel is any pixel in the first landscape image, and the first landscape image is a landscape image at an adjacent time of the second landscape image. The pixel with the highest matching degree to the first pixel is determined as the matching point of the first pixel.

3. The landscape 3D reconstruction method based on multi-view assistance according to claim 1, characterized in that, Based on the matching points of each pixel, the overall movement vector of the landscape image containing each pixel is determined, including: Based on the matching point of each pixel and the movement vector of each pixel, the cosine similarity of the movement vector of each pixel is determined. The cosine similarity of the movement vector of a pixel is the cosine similarity between the movement vector of the pixel and the movement vector of its matching point. Based on the cosine similarity of the movement vector of each pixel, the stable point of the landscape image from each viewpoint is determined; The sum of the movement vectors of the stable points in each landscape image is used to determine the overall movement vector of each landscape image.

4. The landscape 3D reconstruction method based on multi-view assistance according to claim 3, characterized in that, Based on the cosine similarity of the movement vector of each pixel, the stable points of the landscape image from each viewpoint are determined, including: Determine the corresponding point of the first pixel in each landscape image from the same viewpoint; The stability of the first pixel is determined based on the range of the cosine similarity between the movement vectors of the first pixel and the corresponding point, the mean of the cosine similarity between the movement vectors of the first pixel and the corresponding point, the mean of the magnitude of the movement vectors of the first pixel and the corresponding point, and the mean of the magnitude of the movement vectors of all pixels in the multiple landscape images. If the stability of the first pixel is greater than the stability threshold, the first pixel is determined to be a stable point.

5. The landscape 3D reconstruction method based on multi-view assistance according to claim 1, characterized in that, The process of determining motion interference points in the landscape image from each viewpoint based on the motion vector of each pixel and the overall motion vector of the landscape image containing each pixel includes: Based on the sum of the cosine similarity of the overall movement vectors of the first pixel and its corresponding point, the mean of the matching degree of the first pixel and its corresponding point, and the mean of the matching degree of all pixels and their respective corresponding points in all landscape images from the first perspective, the mobility of the first pixel is determined. The cosine similarity of the overall movement vector of a pixel is: the cosine similarity between the movement vector of the pixel and the overall movement vector of the landscape image in which the pixel is located. The matching degree of a pixel is the matching degree between the pixel and its matching point. If the mobility of the first pixel exceeds the mobility threshold, the first pixel is identified as a motion interference point.

6. The landscape 3D reconstruction method based on multi-view assistance according to claim 1, characterized in that, The target landscape model is obtained by modeling the target landscape image based on each viewpoint, including: Identify edge points in each target landscape image; Based on the regional stability index and geometric contribution index of each edge point, the probability of feature points of each edge point is determined. The regional stability index of an edge point is used to determine the displacement stability of the region where the edge point is located, and the geometric contribution index of an edge point is used to characterize the gray-scale change of the edge point. Feature points in each target landscape image are determined based on the feature point probability of each edge point; The target landscape model is obtained by modeling based on the feature points in each target landscape image.

7. The landscape 3D reconstruction method based on multi-view assistance according to claim 6, characterized in that, The method further includes: The regional stability index of the first edge point is determined based on the sum of the cosine similarity of the overall movement vectors of the center points of the regions where the first edge point and its corresponding points are located, and the mean of the magnitudes of the movement vectors of the first edge point and its corresponding points.

8. The landscape 3D reconstruction method based on multi-view assistance according to claim 6, characterized in that, The method further includes: Identify all edge points within the region containing the first edge point; The geometric contribution index of the first edge point is determined based on the gray-level gradient magnitude of the first edge point, the mean gray-level gradient magnitude of all edge points, and the curvature of the first edge point.

9. The landscape 3D reconstruction method based on multi-view assistance according to claim 6, characterized in that, The process of determining feature points in each target landscape image based on the feature point probability of each edge point includes: Determine the overlapping area between the first target landscape image and the second target landscape image; Determine the initial number of feature points in the overlapping region of the first target landscape image and the initial number of feature points in the overlapping region of the second target landscape image. The initial feature points are edge points whose feature point probability is greater than the feature point probability threshold. If the number of initial feature points in the overlapping region of the first target landscape image is less than the number of initial feature points in the overlapping region of the second target landscape image, the target number is determined, wherein the target number is the number of initial feature points in the overlapping region of the second target landscape image. Based on the number of targets, feature points in the overlapping areas of the first target landscape are determined in descending order of the probability of feature points.

10. A landscape 3D reconstruction system based on multi-view assistance, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of a multi-view-assisted landscape 3D reconstruction method as described in any one of claims 1-9.

Images