Computer program product for 3D modeling and method for removing moving objects therefrom

Abstract

To provide a computer program product for use in 3D modeling and a mobile-body removing method therefor.SOLUTION: The present invention is directed to a mobile body removing method for use in 3D modeling including a step of detecting a plurality of feature points in each of original images in an original image sequence by a feature point detection process, a step of dividing each original image into a plurality of regions by a region division process, a step of determining a target region and one or more non-target regions from the regions of each of the original images, a step of, based on these feature points in the original image of two consecutive frames in the original image sequence, determining, by a feature point matching process, whether or not each of the non-target regions of the original image of the two consecutive frames is a motion region, and a step of obtaining a series of still images by replacing, with a featureless region, a region determined as the motion region in the non-target region of each of the original image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to 3D modeling technology, and more particularly to a computer program product used in 3D modeling and a method for removing moving objects therefrom. [Background technology]

[0002] Unmanned aerial vehicles (UAVs) have the ability to overcome terrain constraints to carry out missions, assist professional personnel in carrying out their tasks, and effectively improve efficiency, which has led to a booming market in recent years. They are known to be developing rapidly in fields such as construction, engineering, mining, energy, transportation, public facilities, and precision agriculture, and in particular, 3D modeling of terrain, landscapes, and buildings using UAVs has become one of the mainstream applications in recent years.

[0003] Conventional 3D modeling of unmanned aerial vehicles uses an unmanned aerial vehicle equipped with a high-resolution camera to take continuous photographs of the stationary object or scenery to be measured from different positions and angles, and estimates the 3D coordinates of the unmanned aerial vehicle using feature points on each image. The measured 3D coordinates are then inversely estimated to create a 3D model.

[0004] FIG. 1 is a schematic diagram of 3D modeling photography using an unmanned aerial vehicle. As shown in FIG. 1, the unmanned aerial vehicle flies from a first position 100 in the air along a flight path 102 to a second position 104, and then along a flight path 106 to a third position 108. Images of a target object 110 are taken at these three positions, resulting in a first image 120, a second image 130, and a third image 140, respectively, which are used for modeling. Feature points A 112 and B 114 on the target object 110 are projected by multiple light rays 116 to points a1 122 and b1 124 in the first image 120, points a2 132 and b2 134 in the second image 130, and points a3 142 and b3 144 in the third image 140, respectively. Then, by calculating the characteristic similarity of each image and back-projecting the rays of multiple viewpoint angles into 3D space, the 3D coordinates of feature point A 112 and feature point B 114 can be estimated, which can be expressed in the form of a 3D point cloud.

[0005] The above-mentioned 3D coordinate calculation is based on the principle that the projections of conjugate points of images taken at different angles intersect at the same point in 3D space. However, since the above-mentioned shooting process needs to continue for a certain period of time (taking a 30-meter-long bridge as an example, it takes about 10 minutes of shooting), if a moving object (e.g., a train or high-speed railway) passes by during shooting, the ray projections of these moving objects in different frame images cannot intersect at an accurate position, resulting in noise in the constructed 3D model.

[0006] FIG. 2 illustrates the source of noise in 3D modeling. As shown in FIG. 2, an unmanned aerial vehicle (UAV) flies from one location 200 to another location 204 along a flight path 202, photographing the same object at these two locations to obtain a first image 214 and a second image 224 for modeling, respectively. During this period, the object moves from point A 210 to point A' 220. Before the UAV moves, point A 210 is projected onto point a 216 in the first image 214 via ray 212. Before the UAV moves, point A' 220 is projected onto point a' 226 in the second image 224 via ray 222. Based on the 3D modeling principles described above, point a 216 and point a' 226 have similar image characteristics, so when they are back-projected into 3D space, they intersect at point A* 230. However, since the position of point A* 230 is neither point A 210 nor point A' 220, noise occurs.

[0007] In reality, drone photography makes it impossible to restrict the movement of objects into and out of the scene. To ensure the accuracy of the final 3D model, noise must be manually removed one by one after 3D modeling. This process is tedious and time-consuming. While there are currently methods for performing noise removal on 3D models after modeling, these methods rely on the projection relationship obtained in the initial 3D modeling, and therefore fail in certain situations. For example, if the input image contains a moving object with a particular appearance (e.g., too large), this can cause an initial modeling error, rendering the noise removal process ineffective for subsequent models.

[0008] As the demand for various 3D modeling using unmanned aerial vehicles is expected to increase in the future, designing a moving object removal solution that can be used for 3D modeling is becoming an increasingly important challenge. Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present disclosure is to provide a computer program product for use in 3D modeling and a method for removing moving objects therefrom. [Means for solving the problem]

[0010] The moving object removal method of the present disclosure is a moving object removal method used in 3D modeling, and includes the steps of detecting a plurality of feature points for each of the original images in the original image sequence by a feature point detection process, dividing each of the original images into a plurality of regions by a region division process, identifying a target region and one or more non-target regions from the plurality of regions in each of the original images, determining whether each of the non-target regions in the original images of two consecutive frames in the original image sequence is a moving region by a feature point matching process based on the plurality of feature points in the original images of two consecutive frames, and obtaining a still image sequence by replacing the regions determined to be moving regions in each of the non-target regions in the original images with featureless regions.

[0011] In one embodiment of the moving object removal method according to the present disclosure, the step of determining whether each of the non-target regions in the original images of two consecutive frames of the original image sequence is a moving region through a feature point matching process based on a plurality of feature points in the original images of two consecutive frames includes the steps of obtaining a plurality of match pairs for the feature points through a feature point matching process based on the plurality of feature points in the original images of the two consecutive frames; estimating a transformation matrix based on the plurality of match pairs in the target region in the original images of the two consecutive frames; calculating an average projection error for each of the non-target regions in the original images of the two consecutive frames based on the plurality of match pairs in the non-target region and the transformation matrix; and determining whether each of the non-target regions in the original images of the two consecutive frames is a moving region by comparing the average projection error with a threshold for each of the non-target regions in the original images of the two consecutive frames.

[0012] One embodiment of the moving object removal method according to the present disclosure further includes, before the step of estimating the transformation matrix, a step of checking the homography relationship of the target region in the original images of the two consecutive frames based on the number of multiple matching pairs within the target region in the original images of the two consecutive frames.

[0013] In one embodiment of the moving object removal method according to the present disclosure, the step of calculating an average projection error for each non-target region of an original image of two consecutive frames based on a plurality of matching pairs in the non-target region and a transformation matrix includes the steps of: for each matching pair in the non-target region, projecting a first feature point in the matching pair onto a projection point using the transformation matrix, and calculating the Euclidean distance between the projection point and a second feature point in the matching pair; and setting the average of the Euclidean distances of the matching pairs as the average projection error.

[0014] One embodiment of the moving object removal method according to the present disclosure further includes, before the step of comparing the average projection error with a threshold, a step of calculating the average projection error of the target region and a standard deviation corresponding to the target region based on matching pairs and transformation matrices of the target region of the original images of two consecutive frames, and a step of determining a threshold based on the average projection error and standard deviation of the target region of the original images of two consecutive frames.

[0015] In one embodiment of the moving object removal method according to the present disclosure, each feature point has a feature descriptor.

[0016] In one embodiment of the moving object removal method according to the present disclosure, the feature point matching procedure includes a step of comparing these feature descriptors in the original images of two consecutive frames to obtain multiple matching pairs of feature points.

[0017] In one embodiment of the moving object removal method according to the present disclosure, the feature point detection process includes using a scale invariant feature transform (SIFT) algorithm.

[0018] In one embodiment of the moving object removal method according to the present disclosure, the region segmentation process includes a step of using a Vision Transformer Adapter (ViT-Adapter).

[0019] In one embodiment of the moving object removal method according to the present disclosure, the feature point matching process includes a step of using a brute force matcher (BFMatcher).

[0020] An embodiment of the present disclosure further provides a computer program product for use in 3D modeling, including a user interface module, a 3D modeling module, and a moving object removal module. The user interface module provides a user interface. The 3D modeling module creates a 3D model. The moving object removal module executes a moving object removal method. After being loaded into a computer, the computer program product can execute the following steps: obtaining an original image sequence; upon receiving a moving object removal command from the user interface, calling the moving object removal module to execute the moving object removal method and driving the 3D modeling module to create a 3D model using the series of still images obtained by executing the moving object removal method; and upon receiving a direct modeling command from the user interface, driving the 3D modeling module to create a 3D model using the original image sequence.

[0021] In one embodiment of a computer program product according to the present disclosure, the user interface is a graphical user interface (GUI) that presents segmentation results of the segmentation process and allows a computer user to select a target region on the segmentation results.

[0022] In one embodiment of a computer program product according to the present disclosure, the moving object removal module further detects the area proportion of moving regions in the original image sequence and checks the number of images determined not to contain moving regions in the original image sequence, and in response to the area proportion of moving regions in the original image sequence exceeding a first specified threshold and the number of images determined not to contain moving regions in the original image sequence not reaching a second specified threshold, the moving object removal module notifies the user interface module to present an exception message on the user interface.

[0023] Another embodiment of the present disclosure further provides a computer program product for use in 3D modeling. After being loaded by a computer, the computer program product can provide a graphical user interface (GUI). The graphical user interface includes an image import unit, a moving object removal unit, and a modeling unit. The image import unit is for a user to input a specified path to import an original image sequence. The moving object removal unit is for a user to input a removal command. The modeling unit is for a user to input a modeling command. In response to a received removal command, the computer program product causes the computer to perform a moving object removal method on the original image sequence to obtain a still image sequence. In response to a received modeling command, the computer program product causes the computer to select one of the original image sequence and the still image sequence based on the removal command and use it to create a 3D model.

[0024] In one embodiment of the computer program product according to the present disclosure, the graphical user interface further includes a removal progress display section that presents the processing progress of the mobile object removal method.

[0025] In one embodiment of the computer program product according to the present disclosure, the graphical user interface further includes a target region selection portion that presents the segmentation result for a user to select a target region on the segmentation result.

[0026] In one embodiment of a computer program product according to the present disclosure, the graphical user interface further includes an image display portion for presenting at least one of the original image sequence and the still image sequence.

[0027] In one embodiment, the graphical user interface further includes a removal result display unit that displays an exception message in response to the area percentage of moving regions in the original image sequence exceeding a first specified threshold and the number of images in the original image sequence determined not to include moving regions not reaching a second specified threshold, and that displays a success message in response to the captured still image sequence.

[0028] In one embodiment, the graphical user interface further includes a manual removal portion for a user to manually remove the mobile object after the exception message is presented.

[0029] In one embodiment, the exception message is configured to guide the user to add more source images to the source image sequence.

[0030] The solution for removing moving objects in 3D modeling provided by various embodiments of the present disclosure simultaneously recognizes space (objects) and time (motion), removing only moving objects in an image while preserving information about stationary objects, ensuring the integrity and accuracy of the 3D model. The user interface of the present disclosure also provides the flexibility to enable or disable moving object removal, and offers user interactive features that differ from traditional 3D modeling, such as allowing users to select target areas on the segmentation results and display the removal results. Compared to some noise removal processes performed after modeling, removing objects before modeling can further prevent noise from being mixed into the 3D model. The removed images are compatible with various commercial modeling software, making them applicable in the market.

[0031] The present disclosure can be better understood from the following description of exemplary embodiments and the accompanying drawings. Furthermore, it should be understood that in the flowcharts of the present disclosure, the execution order of each block may be changed, and / or some blocks may be changed, deleted, or combined. [Brief explanation of the drawings]

[0032] [Figure 1] Figure 1 is a schematic diagram of 3D modeling photography using an unmanned aerial vehicle. [Figure 2] Figure 2 shows the sources of noise in 3D modeling. [Figure 3] FIG. 3 is a flowchart illustrating a moving object removal method for 3D modeling according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram illustrating two frames of original images taken by an unmanned aerial vehicle flying from right to left, according to one embodiment of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram illustrating a feature point detection result of an original image according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram illustrating a result of segmenting an original image according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram illustrating matching pairs of feature points according to one embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart illustrating more detailed steps of determining a moving region according to one embodiment of the present disclosure. [Figure 9A] FIG. 9A is a conceptual diagram illustrating determining a moving region using a transformation matrix of a target region, according to one embodiment of the present disclosure. [Figure 9B] FIG. 9B is a conceptual diagram illustrating determining a moving region using a transformation matrix of a target region, according to one embodiment of the present disclosure. [Figure 10] FIG. 10 shows an example of a two-frame still image according to one embodiment of the present disclosure. [Figure 11] FIG. 11 is a system block diagram illustrating a computer system for 3D modeling, according to one embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart illustrating the basic operation of a 3D modeling program, according to one embodiment of the present disclosure. [Figure 13]FIG. 13 is an interface block diagram illustrating a user interface that may be provided by a 3D modeling program after being loaded by a computer, according to one embodiment of the present disclosure. [Figure 14] FIG. 14 is a schematic example of a user interface provided by a 3D modeling program when the above-mentioned exception event is triggered. DETAILED DESCRIPTION OF THE INVENTION

[0033] The following description lists various embodiments of the present invention, but is not intended to limit the scope of the invention, the actual scope of the invention being defined by the scope of the patent application.

[0034] In each of the embodiments listed below, the same or similar elements or components are designated by the same reference numerals.

[0035] Ordinal terms such as "first," "second," and "third" used in the claims are used for convenience of description only and do not imply any priority relationship between them.

[0036] Any discussion below regarding apparatus or system embodiments also applies to method embodiments, and vice versa.

[0037] 3 is a flowchart illustrating a moving object removal method 300 for 3D modeling according to one embodiment of the present disclosure. As shown in FIG. 3, the method 300 may include steps S302 to S310.

[0038] In step S302, a feature point detection process detects a number of feature points in each original image in the sequence of original images.

[0039] The original image sequence may be a series of images taken from different angles by a camera mounted on the unmanned aerial vehicle, without removing any moving objects. The present disclosure does not limit the type of unmanned aerial vehicle or camera, or the captured scene.

[0040] A feature point can be understood as the most distinguishable location or region in an image, and is typically a point where characteristics such as brightness change, texture, edge, color, or shape in the image are significant (e.g., have local extrema). It should be understood that due to changes in the viewing angle (or the position of the unmanned aerial vehicle), the location distribution of feature points in different original images in the original image sequence will also be different, regardless of whether the captured scene is completely static. In one embodiment, each feature point has a feature descriptor, which can be represented in the form of a feature vector that symbolizes the gradient, direction, and feature strength around the feature point.

[0041] 4 is a schematic diagram illustrating two original images, i.e., a first original image 400 and a second original image 410, taken by an unmanned aerial vehicle flying from right to left, according to one embodiment of the present disclosure. As shown in FIG. 4, the first original image 400 includes a train 402, a bridge 404, and ground 406, while the second original image 410 includes a train 412, a bridge 414, and ground 416. In this example, it is assumed that the trains 402 and 412 are the same moving train taken at different times, and the bridge and ground are stationary. From the first original image 400 and the second original image 410, it can be observed that the appearance of the train 412 has changed significantly compared to the electric train 402.

[0042] 5 is a schematic diagram illustrating a first feature point detection result 500 and a second feature point detection result 510 obtained after step S302 for the first original image 400 and the second original image 410 according to an embodiment of the present disclosure. As shown in FIG. 5, the upper left portion of the first feature point detection result 500 includes feature points on the stationary bridge (such as feature point 502) and feature points located on the moving train (such as feature point 504). However, in the second feature point detection result 510, the upper left portion does not include feature points on the stationary bridge, but includes feature points on the moving train, such as feature point 512. That is, the feature points on the stationary bridge are partially occluded by the moving train.

[0043] The feature point detection process in step S302 can be implemented using various feature detection algorithms or corner detectors, such as the speeded-up robust features (SURF) algorithm, the Accelerated-KAZE algorithm, the Harris corner detector, Features from Accelarated Segment Test (FAST), Binary Robust Invariant Scalable Keypoints (BRISK), etc. In one embodiment, the feature point detection process is implemented using the Scale Invariant Feature Transform (SIFT) algorithm. The concept is to convolve the image with Gaussian filters of different scales, obtain Gaussian variances based on successive Gaussian blur images, and thereby obtain feature points that are invariant to scaling and rotation. Each feature point has a feature descriptor, which can be represented in the form of a feature vector, which symbolizes the gradient, direction, and feature strength around the feature point. In actual operation, approximately 50,000 SIFT feature points can be detected from a single 3840x2160 image, and a single SIFT feature point contains 128 dimensions of information representing the magnitude and direction of the gradient within the area surrounding the feature point.

[0044] Referring again to FIG. 3, in step S304, a region segmentation process divides each original image into multiple regions.

[0045] More specifically, the segmentation process assigns an index value to each pixel in the original image, and pixels with the same index value form a single region. For example, pixels with an index value of "1" form a first region, pixels with an index value of "2" form a second region, and so on. In other words, pixels in the first region have an index value of "1," pixels in the second region have an index value of "2," and so on. This allows subsequent steps to identify each region based on the index value and perform corresponding processing.

[0046] FIG. 6 is a schematic diagram illustrating a first segmentation result 600 and a second segmentation result 610 of the first original image 400 and the second original image 410, respectively, obtained after step S304 according to an embodiment of the present disclosure. As shown in FIG. 6, the first segmentation result 600 includes a train region 602, a bridge region 604, and a ground region 606, while the second segmentation result 610 includes a train region 612, a bridge region 614, and a ground region 616. Note that the terms "train region," "bridge region," and "ground region" are used to help readers easily understand the correspondence between the segmentation results and the original images, and that these regions are actually identified by index values. The segmentation process does not identify or distinguish between types of objects, such as trains, bridges, or ground, that these regions belong to.

[0047] The segmentation process in step S304 may be performed using a segmentation algorithm such as Simple Linear Iterative Clustering (SLIC), Graph-Based Segmentation (Felzenszwalb's), Quick Shift, or Region Growing. Alternatively, a segmentation process built using a convolutional neural network (CNN)-based machine learning model such as U-Net, SegNet, or a fully convolutional network (FCN) may be used. The training process may include obtaining labeled data, selecting a loss function, and configuring an optimization algorithm. Various conventional methods may be employed, but the present disclosure is not limited thereto. Furthermore, the segmentation model may be trained locally, or may be trained on another computer device (e.g., a server) first, and then the trained segmentation model may be obtained via a network (e.g., downloaded from the cloud), a storage medium (e.g., external hardware), or another communication interface (e.g., USB). The present disclosure is not limited thereto. In one embodiment, the segmentation process is performed using the Visual Transformer Adapter (ViT-Adapter), a deep learning-based classifier whose concept is to improve the performance and generality of a pre-trained visual playing card model by adding small neural network modules called "adapters" within different layers.

[0048] Referring again to FIG. 3, in step S306, within these regions of each original image, a target region and one or more non-target regions (ie, regions other than the target region) are determined.

[0049] In one embodiment, the target region can be defined by a user's selection from the region segmentation results presented in a graphical user interface (GUI). In another embodiment, a machine learning model for identifying target regions, i.e., a target region identification model, can be pre-trained. Then, in step S306, the trained target region identification model can be used to identify target regions within these regions of each original image. The target region identification model can be a convolutional neural network (CNN)-based model, and the training process can include obtaining label data, selecting a loss function, and configuring an optimization algorithm. Various conventional methods can be used, but the present disclosure is not limited thereto. Furthermore, the target region identification model can be trained locally, or it can be trained on another computer device (e.g., a server) and then obtained via a network (e.g., downloaded from the cloud), a storage medium (e.g., external hardware), or another communication interface (e.g., USB). The present disclosure is not limited thereto.

[0050] In a single modeling task, the target object of modeling is known in advance. For example, the target of bridge modeling is the bridge itself and the surrounding terrain. Taking FIG. 6 as an example, a bridge region 604 can be selected as the target region. Because the original images in the original image sequence have a high degree of overlap with each other (generally more than 50%), the index value of the original bridge region 604 is further extended to the bridge region 614 of the next original image based on the overlap ratio (e.g., only regions that overlap more than 50% are considered), and the bridge region 614 is considered as the target region of the second original image 410. In other words, by performing step S306 once, the remaining original images in the original image sequence can know their target regions by inference.

[0051] Referring again to FIG. 3, in step S308, based on the feature points in the original images of two consecutive frames in the original image sequence, a feature point matching process is used to determine whether each non-target region in the original images of the two consecutive frames is a moving region (i.e., a region corresponding to a moving object).

[0052] In one embodiment, the feature point matching process involves comparing feature descriptors in two images to find similar feature points in the two images. Two feature points with matching feature descriptors form a matching pair. As described above, the feature descriptors can be represented in the form of feature vectors that represent the gradient, direction, and feature strength around the feature points. Therefore, the feature point matching process can further include calculating a distance or similarity between the two feature vectors, such as Euclidean distance, Manhattan distance, cosine similarity, or other measures used to represent distance or similarity, although the present disclosure is not limited thereto.

[0053] 7 is a schematic diagram illustrating a feature point matching pair of a first original image 700 and a second original image 710 according to one embodiment of the present disclosure. The first original image 700 and the second original image 710 correspond to the first feature point detection result 500 and the second feature point detection result 510 in FIG. 5, respectively, although not all feature points depicted in FIG. 5 appear in FIG. 7. In the example of Figure 7, only nine similar feature points, i.e., nine matching pairs, found in the feature point matching process of step S308 are shown, including a first matching pair consisting of feature point 701 in the first original image 700 and feature point 711 in the second original image 710, a second matching pair consisting of feature point 703 in the first original image 700 and feature point 713 in the second original image 710, a third matching pair consisting of feature point 705 in the first original image 700 and feature point 715 in the second original image 710, a fourth matching pair consisting of feature point 706 in the first original image 700 and feature point 716 in the second original image 710, a fifth matching pair consisting of feature point 708 in the first original image 700 and feature point 718 in the second original image 710, and four other matching pairs without symbols. Of these matching pairs, the first, second, and third matching pairs are located on a moving train 702, while the fourth and fifth matching pairs are located on a stationary bridge, so there is a larger displacement between feature point 701 and feature point 711, between feature point 703 and feature point 713, and between feature point 705 and feature point 715 than between feature point 706 and feature point 716, or between feature point 708 and feature point 718. Therefore, it can be determined that the area corresponding to train 702 is a moving area.

[0054] The feature point matching process in step S308 can be performed using nearest neighbor matching, Random Sample Consensus (RANSAC), Kanade-Lucas-Tomasi feature tracker, or other similar algorithms. In one embodiment, the feature point matching process is performed using a brute force matcher (BFMatcher), which searches for the best match by calculating the distance or similarity between two sets of feature descriptors (or feature vectors).

[0055] 8 is a flowchart showing more detailed steps of the moving region determination in step S308 according to an embodiment of the present disclosure. As shown in FIG. 8, step S308 may further include steps S802 to S808.

[0056] In step S802, based on the feature points in two consecutive frames of original images, a feature point matching process is performed to obtain multiple matching pairs of these feature points. Taking Fig. 7 as an example, after step S802, nine matching pairs of the first original image 700 and the second original image 710 can be obtained.

[0057] In step S804, a transformation matrix is ​​estimated based on matching pairs within the target region of two consecutive frames of the original image.

[0058] The transformation matrix can represent the transformation relationship between the positions of feature points and other pixels in each matching pair between the two original images. Therefore, step S804 can be understood as determining the transformation relationship between pixel positions in the target region of two consecutive frames of the original image. In a subsequent step, the moving region can be found by checking whether the transformation relationship of each non-target region is significantly different from that of the target region.

[0059] In step S806, for each non-target region of the original images of two consecutive frames, an average projection error is calculated based on the matching pairs and transformation matrices in the non-target region.

[0060] The average projection error can represent the dissimilarity between the pixel position transformation relationship of the non-target region and the pixel position transformation relationship of the target region. In other words, the larger the average projection error, the greater the difference between the pixel position transformation relationship representing the non-target region and the pixel position transformation relationship of the target region.

[0061] In step S808, for each non-target region of the original images of two consecutive frames, the average projection error is compared with a threshold to determine whether the non-target region is a moving region.

[0062] Specifically, if the average projection error exceeds a threshold, it means that the difference between the pixel position transformation relationship of the non-target region and the pixel position transformation relationship of the target region is significantly large, and therefore the non-target region is determined to be a moving region. The threshold may be a predefined numerical value or a variable determined by a specific calculation.

[0063] 9A and 9B illustrate a transformation matrix H of a target region according to one embodiment of the present disclosure. 3×3 9A and 9B, a first original image 900 includes an area 902, an area 904, and an area 906, which correspond to an area 912, a second area 914, and a third area 916, respectively, of a second original image 910. The areas 902 (and 912), 904 (and 914), and 906 (and 916) have corresponding index values ​​0, 1, and 2, respectively, indicated by k. In this example, the areas 902 and 912 with index value k=0 are set as target areas, and the other areas are set as non-target areas. Also, M kLet represent the number of matching pairs in the region with index value k, and the i-th feature point in the region with index value k in the first original image 900 and the second original image 910 is

number

number

number

number

[0064] Transformation matrix H 3x3 The estimation of is conceptually a single optimal transformation matrix H 3x3 Find the feature points in the target region (i.e., i=1 to M k ) to satisfy the following formula as much as possible.

number

[0065] Next, in step S806, feature points in each non-target region (i.e., a region with index value k≠0) on the first original image 900 are

number

number

[0066] As shown in FIG. 9B, the feature points in the non-target region 904 of the first original image 900 are transformed by the transformation matrix H.

number

[0067] Next, the projection point in the second original image 910

number

number

[0068] Taking FIG. 9B as an example, the average projection error value E 1 is calculated as follows:

number

[0069] In one embodiment, the transformation matrix H 3×3 The estimation is performed using the least squares method, least absolute deviations (LAD), least squares support vector machine (LS-SVM), This may include using polynomial fitting or any other function fitting and related algorithms, although the present disclosure is not limited thereto.

[0070] In one embodiment, before step S808, the average projection error and corresponding standard deviation of the target area can be calculated based on the matching pairs and transformation matrices in the target area of ​​the original images of two consecutive frames. More specifically, the standard deviation σ can be calculated as follows:

number

[0071] However, based on the calculated average projection error and standard deviation, a threshold value can be determined to be used for comparison with the average projection error in step S808. In a preferred embodiment, the threshold value is the average projection error E 0 and twice the standard deviation. For example, the average projection error E 0 9A and 9B, the average projection error E 0 and the corresponding standard deviation is calculated, and the corresponding threshold T p The average projection error value E for the non-target regions 904 and 914 can be determined. 1 T p If so, the non-target regions 904 and 914 are determined to be moving regions.

[0072] In one embodiment, before step S804, the homography relationship of the target region in the original images of two consecutive frames can be checked based on the number of matching pairs in the target region in the original images of the two consecutive frames. If it is determined that the target region has a homography relationship in the original images of the two consecutive frames, the condition for estimating the transformation matrix in step S804 is met. Otherwise, steps S804 to S808 are skipped. That is, it is considered that no moving region has been detected in the original images of the two consecutive frames.

[0073] The above-mentioned homography relationship refers to the reversible transformation from a real projective plane to a projective plane. When shooting a stationary object at a long distance, the original images of two consecutive frames can be considered to be almost on the same plane, so theoretically, the stationary region (represented by the target region) of the original image of the first frame should be able to be transformed into the corresponding region of the original image of the second frame based on the projective relationship.

[0074] Referring again to FIG. 3, in step S310, regions determined to be moving regions within the non-target regions of each original image are replaced with featureless regions to obtain a series of still images.

[0075] The above-mentioned featureless region means that the pixels in this region do not have feature values. Generally, a region of pure color (such as white) can be used as a featureless region.

[0076] 10 shows example still images 1000 and 1010 of two frames obtained after step S310, according to one embodiment of the present disclosure. As shown in FIG. 10, in still images 1000 and 1010, areas presenting the train outline (e.g., the train's front and passenger cars) have been replaced with pure white, featureless areas 1002 and 1012.

[0077] 11 is a system block diagram illustrating a computer system 1100 for 3D modeling, according to one embodiment of the present disclosure. As shown in FIG. 11, the computer system 1100 can include a processing unit 1102 and a storage unit 1104.

[0078] The computer system 1100 can be a personal computer (e.g., a desktop computer or a laptop computer) running an operating system (e.g., Windows, Mac OS, Linux, UNIX, etc.), or a server computer, or a mobile device such as a tablet computer or smartphone, although the present disclosure is not limited thereto.

[0079] The processing unit 1102 may include any one or more general-purpose or special-purpose processors, and combinations thereof, for executing instructions. In various embodiments of the present disclosure, the processing unit 1102 is configured to execute the aforementioned moving object removal methods, such as method 300. In one embodiment, the processing unit 1102 may further include a central processing unit (CPU) and a graphics processing unit (GPU), although not shown in FIG. 11 . A GPU is an electronic circuit specifically designed for computer graphics calculations and image processing, and is therefore more efficient in computer graphics calculations and image processing than a general-purpose CPU. Therefore, in various embodiments of the present disclosure, appropriate tasks may be assigned to the CPU and the GPU depending on their characteristics. For example, tasks related to acquiring data or communicating with other devices may be assigned to the CPU, and tasks related to computer graphics and image processing may be assigned to the GPU. In a further embodiment, the processing unit 1102 may further include a neural network processing unit (NPU), although not shown in FIG. 11 , that is specifically optimized for deep learning. Compared to a GPU, an NPU may have a greater computing performance advantage in operating the aforementioned region segmentation model and / or target region identification model. Therefore, in this embodiment, tasks involving these machine learning models can be assigned to the NPU.

[0080] The storage device 1104 may include, but is not limited to, volatile memory (such as random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM)), and / or any one or more types of non-volatile memory (such as read-only memory, electrically erasable programmable read-only memory (EEPROM), flash memory, non-volatile random access memory (NVRAM)) devices (such as hard disks (HDDs), solid-state drives (SSDs), or optical disks), and combinations thereof. In various embodiments of the present disclosure, the storage device 1104 is used to store a program corresponding to the aforementioned mobile object removal method. When the processing device 1102 loads this program from the storage device 1104, the aforementioned mobile object removal method can be executed.

[0081] 11 , the storage device 1104 stores a computer program product for 3D modeling, referred to herein as a “3D modeling program” 1110. As shown in FIG. 11 , the 3D modeling program 1110 can include a user interface module 1112, a 3D modeling module 1114, and a moving object removal module 1116. The user interface module 1112 is used to provide a user interface, the 3D modeling module 1114 is used to create a 3D model, and the moving object removal module is used to perform the above-described moving object removal method, such as method 300. The processing device 1102 can load the 3D modeling program 1110 from the storage device 1104 and execute it to perform the user interface module 1112, the 3D modeling module 1114, and the moving object removal module 1116, as well as other basic operations of the 3D modeling program 1110.

[0082] In one embodiment, the processing unit 1102 may be coupled to a display device on which to display the user interface provided by the user interface module 1112. The display device may be any device used to display visual information, such as an LCD display, an LED display, an OLED display, or a plasma display, although the invention is not limited thereto.

[0083] The user interface provided by the user interface module 1112 can be a graphical user interface (GUI), a command line interface (CLI), a touch interface, or a voice interface, although the present disclosure is not limited thereto.

[0084] The 3D modeling module 1114 may include any known 3D modeling techniques, such as, for example, stereo correspondence, point cloud reconstruction, point cloud processing, 3D rendering, etc., but the present disclosure is not limited thereto.

[0085] 12 is a flowchart of a basic operation 1200 of the 3D modeling program 1110 according to one embodiment of the present disclosure. As shown in FIG. 12, the basic operation 1200 of the 3D modeling program 1110 may include steps S1202 to S1210.

[0086] In step S1202, the original image sequence is obtained.

[0087] As described above, the original image sequence is a series of images taken by the image capture device mounted on the unmanned aerial vehicle from different viewpoints, from which moving objects have not yet been removed. The original image sequence can be obtained via a network, a storage medium, or various other communication interfaces, but the present disclosure is not limited thereto.

[0088] In step S1204, the user interface module 1112 receives an instruction from a user of the computer system 1100 via a user interface. In response to receiving a moving object removal instruction, the process proceeds to step S1206. In response to receiving a direct modeling instruction, the process proceeds to step S1210.

[0089] In step S1206, the mobile removal module 1116 is called to perform the mobile removal method (method 300) described above.

[0090] In step S1208, the still image sequence obtained by performing the moving object removal method is also used to drive the 3D modeling module 1114 to create a 3D model.

[0091] In step S1210, the 3D modeling module 1114 is driven to create a 3D model using the original images of this sequence.

[0092] In one embodiment, the user interface provided by the user interface module 1112 is a graphical user interface (GUI) that presents segmentation results of the segmentation process (e.g., the first segmentation result 600 and the second segmentation result 610 shown in FIG. 6 ) and allows a user of the computer system 1100 to select a target region in the segmentation results. This allows the moving object removal module 1116 to determine the target region in step S306 and determine other regions other than the target region as non-target regions. Additionally, the graphical user interface may provide check boxes, toggle switches, drop-down lists, or other similar GUI elements or widgets to allow a user of the computer system 1100 to select moving object removal instructions or direct modeling instructions, although the present disclosure is not limited to a specific graphic design arrangement of the graphical user interface.

[0093] If the moving speed of an object is too slow, it may frequently cover the target area in the original image sequence, which may affect the accuracy of 3D modeling. In view of this, in one embodiment, the moving object removal module 1116 further detects the area ratio of moving regions in the original image sequence and checks the number of images determined to not contain moving regions in the original image sequence. In response to the area ratio of moving regions in the original image sequence exceeding a first specified threshold and the number of images determined to not contain moving regions in the original image sequence not reaching a second specified threshold, the moving object removal module 1116 notifies the user interface module 1112 to present an exception message on the user interface. The content of the exception message can be set to guide the user to manually remove the moving object or add more additional original images (e.g., images of non-existent slow-moving moving objects taken at other times) to facilitate 3D modeling.

[0094] Figure 13 is an interface block diagram illustrating a user interface 1300 that can be provided by a 3D modeling program after being loaded by a computer, according to one embodiment of the present disclosure. Similarly, Figure 14 is a schematic example of a user interface 1400 provided by the 3D modeling program when the above-mentioned exception event is triggered. User interface 1300 includes at least an image import unit 1301, a moving object removal unit 1302, and a modeling unit 1305 shown in Figure 13. Similarly, user interface 1400 includes at least an image import unit 1401, a moving object removal unit 1402, and a modeling unit 1405 shown in Figure 14.

[0095] The image import units 1301 and 1401 allow a user to input a specified path, for example, "C: / Users / 3DModeling / RawData" or other similar path, and import the original image sequence from this path. The image import units 1301 and 1401 can be implemented, for example, with a file selection dialog, a text box, a file drag and drop, a tree view, or other similar GUI elements or widgets, but the present disclosure is not limited thereto.

[0096] The moving object removal units 1302 and 1402 are used by the user to input removal instructions, which indicate whether a moving object removal method (e.g., moving object removal method 300 of FIG. 3 ) is enabled and trigger execution of the moving object removal method. In response to receiving the removal instruction input by the user via the moving object removal units 1302 and 1402, the 3D modeling program causes the computer to execute the moving object removal method on the original image sequence imported from the specified path (input by the user via the image import units 1302 and 1402). The moving object removal units 1302 and 1402 can be implemented, for example, as a check box, a toggle switch, a drop-down menu, or other similar GUI elements or widgets combined with a confirmation button and a start button, although the present disclosure is not limited thereto.

[0097] The modeling units 1305 and 1405 allow a user to input modeling instructions, which are used to trigger the 3D modeling process. More specifically, if the removal instructions indicate that the moving object removal method is enabled, a 3D model is created using the still image sequence. Conversely, if the removal instructions indicate that the moving object removal method is not enabled, a 3D model is created using the original image sequence. The modeling units 1305 and 1405 can be implemented, for example, with a confirmation button, a start button, a toolbar button, a menu option, or other similar GUI elements or widgets, although the present disclosure is not limited thereto.

[0098] In one embodiment, user interfaces 1300 and 1400 may further include removal progress indicators 1303 and 1403. Removal progress indicators 1303 and 1403 are for presenting the processing progress of the mobile object removal method, e.g., a completion percentage such as 20%, 50%, or 90%, or a text representation of a status such as "processing" or "completed." Removal progress indicators 1303 and 1403 may be implemented, for example, as a progress bar, a circular progress bar, a digital display, a status label, a timeline, or other similar GUI element or widget, but the present disclosure is not limited thereto.

[0099] In one embodiment, user interface 1300 may further include a target region selection portion 1306. Target region selection portion 1306 is used to present segmentation results (such as first segmentation result 600 and second segmentation result 610 shown in FIG. 6 ) and allow a user to select a target region on the segmentation results. As an illustrative example, selection removal portion 1407 of user interface 1400 is used to present the above-mentioned segmentation results and allow a user to select a target region on the segmentation results.

[0100] In one embodiment, the user interfaces 1300 and 1400 may further include image display portions 1308 and 1408. The image display portions 1308 and 1408 are used to present the original image sequence and / or the still image sequence.

[0101] In one embodiment, the user interfaces 1300 and 1400 may further include removal result display units 1304 and 1404. When the moving object removal method is successfully executed and a still image sequence is acquired, the removal result display units 1304 and 1404 present a success message. The success message is used to notify the user that moving objects (or moving regions) in the original image sequence have been removed. The success message may be, for example, "13 moving regions out of 105 images were successfully removed." In response to an exception event, i.e., when the area percentage of moving regions in the original image sequence exceeds a first specified threshold and the number of original images determined not to contain moving regions does not reach a second specified threshold, the removal result display units 1304 and 1404 present an exception message. The exception message is used to notify the user that the moving object removal method encountered an exception event (e.g., a moving object obscures the target area) and was not successful. In a further embodiment, the content of the exception message may be set to guide the user to add more additional original images (e.g., images taken at other times that do not contain slowly moving objects) to the original image sequence. In another embodiment, the content of the exception message may be set to guide the user to manually remove the moving object. After this exception message is presented, the user interface 1300 may further include a manual removal portion 1307 that allows the user to manually remove the moving object. As an illustrative example, the selective removal portion 1407 of the user interface 1400 may be used to allow the user to manually remove the moving object after this exception message is presented.

[0102] The solution for removing moving objects in 3D modeling provided by various embodiments of the present disclosure simultaneously recognizes space (objects) and time (motion), removing only moving objects in an image while preserving information about stationary objects, ensuring the integrity and accuracy of the 3D model. The user interface of the present disclosure also provides the flexibility to enable or disable moving object removal, and offers user interactive features that differ from traditional 3D modeling, such as allowing users to select target areas on the segmentation results and display the removal results. Compared to some noise removal processes performed after modeling, removing objects before modeling can further prevent noise from being mixed into the 3D model. The removed images are compatible with various commercial modeling software, making them applicable in the market.

[0103] The above paragraphs are described in multiple aspects. Obviously, the teachings of this specification can be implemented in various ways. Any specific structure or function disclosed in the examples is merely a representative example. According to the teachings of this specification, those skilled in the art should note that any disclosed aspect can be implemented individually, or two or more aspects can be implemented in combination.

[0104] Although the present disclosure has been described above as an embodiment, the present disclosure is not limited thereto. Those skilled in the art can modify and adjust the present disclosure without departing from the spirit and scope of the present disclosure. The scope of protection of the present invention is subject to the claims. [Explanation of symbols]

[0105] 100 1st position 102 Flight Path 104 Second Position 106 Flight Path 108 Third Position 110 Target Body 112 Feature point A 114 Feature point B 116 Ray of light 120 First Image 122 a1 point 124 b1 point 130 Second Image 132 a2 points 134 b2 points 140 Third Image 142 a3 points 144 b3 points 200 positions 202 Flight Path 204 position 210 Point A 212 Ray of light 214 First Image 216 point a 220 A' point 222 Ray of light 224 Second Image 226 point a' 230 A* point 300 Moving object removal method S302~S310 steps 400 First original image Train 402 404 Bridges 406 Ground 410 Second original image Train 412 414 Bridges 416 Ground 500 First feature point detection result 502, 504 feature points 510 Second feature point detection result 512 minutiae 600 First region segmentation result 602 Train area 604 Bridge area 606 Ground area 610 Second region segmentation result 612 Train area 614 Bridge area 616 Ground area 700 First original image Train 702 701, 703, 705, 706, 708, 711, 713, 715, 716, 718 feature points 710 Second original image S802~S808 steps 900 First original image 902 target area 904, 906 Non-target areas 910 Second original image 912 target area 914, 916 Non-target areas 1000, 1010 still images 1002, 1012 featureless areas 1100 Computer Systems 1102 Processing equipment 1104 Storage device 1110 3D modeling program 1112 User Interface Module 1114 3D Modeling Module 1116 Moving Object Removal Module 1200 Basic Operation of 3D Modeling Programs S1202~S1210 steps 1300 User Interface 1301 Image Import Section 1302 Mobile Object Removal Department 1303 Removal progress display 1304 Removal result display area 1305 Modeling Department 1306 Target Area Selection Section 1307 Manual removal section 1308 Image display unit 1400 User Interface 1401 Image Import Section 1402 Mobile Object Removal Department 1403 Removal progress display 1404 Removal result display area 1405 Modeling Department 1407 Manual removal section 1408 Image display unit

Claims

1. A moving object removal method for use in 3D modeling, comprising: detecting a plurality of feature points for each of the source images in the sequence of source images through a feature point detection process; dividing each of the original images into a plurality of regions through a region segmentation process; identifying a target region and one or more non-target regions from the plurality of regions in each of the source images; determining whether each of the non-target regions in the original images of the two consecutive frames of the original image sequence is a moving region through a feature point matching process based on the plurality of feature points in the original images of the two consecutive frames of the original image sequence; and replacing the determined moving regions in the non-target regions of each of the source images with regions of a single color to obtain a sequence of still images. How to remove moving objects.

2. The step of determining whether each of the non-target regions in the original images of the two consecutive frames of the original image sequence is the moving region by the feature point matching process based on the plurality of feature points in the original images of the two consecutive frames of the original image sequence, includes: obtaining a plurality of match pairs of feature points through the feature point matching process based on the plurality of feature points in the original images of the two consecutive frames; estimating a transformation matrix based on the plurality of matching pairs within the target region in the original images of the two consecutive frames; For each of the non-target regions of the original images of the two consecutive frames, calculating an average projection error based on the plurality of matching pairs in the non-target region and the transformation matrix; determining whether each of the non-target regions in the original images of the two consecutive frames is the moving region by comparing the average projection error with a threshold value; Including, The moving body removal method according to claim 1 .

3. Before estimating the transformation matrix, checking a homography relationship of the target region in the original images of the two consecutive frames based on the number of the plurality of matching pairs within the target region in the original images of the two consecutive frames; further comprising: The moving body removal method according to claim 2 .

4. The step of calculating the average projection error for each of the non-target regions of the original images of the two consecutive frames based on the plurality of matching pairs in the non-target region and the transformation matrix includes: For each matching pair in the non-target region, projecting a first feature point in the matching pair to a projection point using the transformation matrix, and calculating a Euclidean distance between the projection point and a second feature point in the matching pair; determining the average of the Euclidean distances of the matching pairs as the average projection error; Including, The moving body removal method according to claim 2 .

5. before the step of comparing the average projection error with the threshold value, Calculating the average projection error of the target area and a standard deviation corresponding to the target area based on the matching pairs of the target areas of the original images of the two consecutive frames and the transformation matrix; determining the threshold value based on the average projection error and the standard deviation of the target region of the original images of the two consecutive frames; further comprising: The moving body removal method according to claim 2 .

6. Each of the feature points has a feature descriptor. The moving body removal method according to claim 1 .

7. The feature point matching process includes a step of comparing the feature descriptors in the original images of the two consecutive frames to obtain a plurality of matching pairs of the feature points. The moving body removal method according to claim 6.

8. The feature point detection process includes using a scale invariant feature transform (SIFT) algorithm. The moving body removal method according to claim 6.

9. The segmentation process includes using a Vision Transformer Adapter (ViT-Adapter). The moving body removal method according to claim 1 .

10. The feature point matching process includes using a brute force matcher (BFMatcher). The moving body removal method according to claim 1 .

11. a user interface module that provides a user interface; a 3D modeling module for creating a 3D model; A computer program product for use in 3D modeling, comprising: a moving object removal module that executes the moving object removal method according to any one of claims 1 to 10; After the computer program product is loaded into a computer, obtaining an original image sequence; upon receiving a moving object removal command from the user interface, calling the moving object removal module to execute the moving object removal method and driving the 3D modeling module to create the 3D model using the still image sequence obtained by executing the moving object removal method; upon receiving a direct modeling command from the user interface, driving the 3D modeling module to create the 3D model using the source image sequence; causing the computer to execute Computer program products.

12. the user interface is a graphical user interface (GUI) that presents the segmentation results of the segmentation process so that a user of the computer can select the target region for the segmentation results.

12. The computer program product of claim 11.

13. The moving object removal module further comprises: Detecting the area ratio of the moving region in the original image sequence, and checking the number of original images in the original image sequence that are determined not to include the moving region; If the area ratio of the moving region in the original image sequence exceeds a first specified threshold and the number of original images in the original image sequence that are determined not to include the moving region does not reach a second specified threshold, the moving object removal module notifies the user interface module to present an exception message on the user interface.

12. The computer program product of claim 11.

Images