Data processing method, electronic device, storage medium, and computer program product

Abstract

The application provides a data processing method, an electronic device, a storage medium and a computer program product. The method comprises the following steps: projecting a first 3D point contained in first point cloud data to a two-dimensional image corresponding to the first 3D point, to obtain a first re-projection point of the first 3D point in the two-dimensional image; and obtaining a first observation point of the first 3D point in the corresponding two-dimensional image; performing filtering processing on the first point cloud data according to distance information between the first observation point and the first re-projection point, to obtain second point cloud data; determining a confidence degree corresponding to a second 3D point contained in the second point cloud data; performing filtering processing on the second point cloud data based on the confidence degree, to obtain third point cloud data; and the third point cloud data is used for three-dimensional reconstruction. The filtering effect of the point cloud data can be improved, and the effect of the three-dimensional reconstruction is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] Three-dimensional (3D) point clouds refer to discrete sets of points on the surface of an object in three-dimensional space, obtained through various sensors or computing methods. These points typically contain coordinate information as well as attributes such as color and intensity. They are widely used in fields such as autonomous driving, virtual reality, robot navigation, and 3D modeling. Optimization of sparse 3D point clouds is usually required by using adaptive filtering data that incorporates prior knowledge or filtering methods based on geometric constraints. However, the filtering effect is limited, resulting in poor filtering of point cloud data and consequently, poor 3D reconstruction results. Summary of the Invention

[0003] This application provides a data processing method, electronic device, storage medium, and computer program product that can improve the filtering effect of point cloud data, thereby improving the effect of 3D reconstruction.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, the method comprising: Project the first 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point to obtain the first super-projection point of the first 3D point in the two-dimensional image; and obtain the first observation point of the first 3D point in the corresponding two-dimensional image. Based on the distance information between the first observation point and the first reprojection point, the first point cloud data is filtered to obtain the second point cloud data. Determine the confidence level corresponding to the second 3D point contained in the second point cloud data; The second point cloud data is filtered based on the confidence level to obtain the third point cloud data; the third point cloud data is used for at least three-dimensional reconstruction.

[0005] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method provided in the embodiments of this application.

[0006] This application provides a storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the data processing method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: By projecting the first 3D point contained in the first point cloud data onto the corresponding two-dimensional image, the first reprojection point of the first 3D point in the two-dimensional image is obtained, and the first observation point of the first 3D point in the corresponding second image is obtained. Based on the distance information between the first observation point and the first reprojection point, the first point cloud data is filtered to obtain the second point cloud data. Based on the confidence level corresponding to the second 3D point contained in the second point cloud data, the second point cloud data is filtered to obtain the third point cloud data. The third point cloud data is used at least for three-dimensional reconstruction. By performing double filtering on the first point cloud data sequentially through the distance information between the first observation point and the first reprojection point, and the confidence level corresponding to the second 3D point, the filtering effect of the point cloud data can be improved, thereby improving the effect of three-dimensional reconstruction. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 This is an exemplary schematic diagram illustrating the calculation of the three-dimensional coordinates of a corresponding 3D point based on a pair of 2D feature points, as provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] 3D point clouds refer to discrete sets of points on the surface of an object in three-dimensional space, acquired through various sensors or computational methods. These points typically contain coordinate information as well as attributes such as color and intensity. They are widely used in fields such as autonomous driving, virtual reality, robot navigation, and 3D modeling. Core technologies include LiDAR scanning, structured light, stereo vision, and deep learning-based generation methods. Sparse 3D point cloud optimization refers to the structural processing and quality improvement of acquired 3D point cloud data through methods such as filtering out noise points, filling in missing regions, and increasing point cloud density and geometric accuracy. This aims to meet the high-precision requirements of subsequent applications such as 3D reconstruction, scene understanding, autonomous driving, and robot navigation. Its core lies in preserving key geometric features and optimizing overall data quality while maintaining the sparsity of the point cloud. This typically involves point cloud filtering, registration, completion, and optimization based on geometric constraints, statistical models, or deep learning methods.

[0014] Currently, in the field of sparse 3D point cloud optimization, relevant technical solutions include geometrically constrained filtering methods, statistical model-driven point cloud optimization techniques, deep learning-assisted point cloud completion and denoising techniques, voxel mesh-based downsampling and resampling techniques, multi-scale feature fusion filtering algorithms, graph-based point cloud registration and optimization techniques, and adaptive filtering techniques that combine prior knowledge.

[0015] For example, by obtaining the set of edge pixels corresponding to the target line segment in a 2D image, a first set of spatial points can be determined. Based on this set, the first 3D point cloud can be filtered to obtain a second 3D point cloud with a reduced range, from which the 3D point cloud corresponding to the target line segment can be extracted. However, this approach relies solely on a single set of spatial points for filtering, which, while improving efficiency, may result in the loss of some detailed information, affecting the final reconstruction result. Alternatively, by introducing key technologies such as ground segmentation, clustering, and isolated cluster filtering, ground interference and noise points can be effectively removed, simplifying the point cloud data structure, improving data quality, and combining key point extraction and feature description calculation to achieve high-precision coarse and fine two-level registration. However, this approach relies excessively on ground segmentation and clustering operations in the preprocessing stage, exhibiting poor robustness to point cloud data with complex terrain or severe occlusion, and has limited ability to eliminate noise and outliers, thus restricting the reconstruction accuracy of the 3D Gaussian Splatting (3DGS) algorithm in complex scenes.

[0016] Initializing the point cloud is crucial for the reconstruction effect of 3DGS. However, due to the limited performance of the Structure from Motion (SFM) algorithm, the calculated 3DGS initial point cloud often contains a lot of noise. The aforementioned related technologies have attempted to solve the problem of initial point cloud noise through various schemes, but their filtering effect is limited, or they require the introduction of stringent prior conditions, resulting in poor filtering effect and robustness.

[0017] This application provides a data processing method. Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Project the first 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point to obtain the first super-projection point of the first 3D point in the two-dimensional image; and obtain the first observation point of the first 3D point in the corresponding two-dimensional image.

[0018] It should be noted that the first point cloud data can be understood as a sparse 3D point cloud. The first 3D point can be understood as one or more 3D points contained in the first point cloud data, or it can be illustrated as one or more 3D points (or discrete points) contained in the sparse 3D point cloud.

[0019] In this embodiment of the application, before projecting the first 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point, the method further includes: acquiring a multi-view two-dimensional image; determining one or more feature point pairs corresponding to the multi-view two-dimensional image; and determining the first point cloud data based on the one or more feature point pairs.

[0020] It should be noted that multi-view 2D images can be understood as multiple 2D images of the same scene taken from multiple locations by one camera or multiple cameras with identical parameters, each with a different perspective. The camera parameters and number of cameras can be determined based on the actual situation and are not limited here. The scene can be understood as the scene requiring 3D reconstruction, and its specific determination is based on the actual situation and is not limited here. The number of perspectives can also be determined based on the actual situation and is not limited here. In multi-view 2D images, the perspective corresponding to each 2D image must form a certain parallax angle with the perspective corresponding to adjacent 2D images. That is, the camera positions used to capture each 2D image need to form a certain parallax angle, which can be determined based on the actual situation and is not limited here. As an example, the overlap between two adjacent images is generally between 60% and 85%. This parallax angle is beneficial for subsequent calculation of coefficients to generate 3D point clouds based on triangulation.

[0021] It should be noted that feature point pairs can be understood as two-dimensional (2D) feature point pairs; the number of feature point pairs can be determined according to the actual situation and is not limited here. Determining one or more feature point pairs corresponding to a multi-view 2D image can be understood as performing feature point detection and matching on the multi-view image using a feature point detection and matching algorithm to obtain one or more feature point pairs. The feature point detection and matching algorithm can be any algorithm capable of feature point detection and matching, determined according to the actual situation and is not limited here; as an example, the feature point detection and matching algorithm can be the XFEAT algorithm. Determining the first point cloud data based on one or more feature point pairs can be understood as calculating the three-dimensional coordinates of the corresponding 3D point for each 2D feature point pair; a stereo matching algorithm can be used to calculate the three-dimensional coordinates of the corresponding 3D point; the calculation is performed on all 2D feature point pairs, and the resulting three-dimensional coordinates of all corresponding 3D points are used as the first point cloud data.

[0022] In the scheme of this application embodiment, the coordinates of each 3D point in the first point cloud data are jointly constrained by the camera views from multiple perspectives. The more cameras there are, the more accurate the spatial position of the point, thereby effectively improving the overall accuracy of the acquired first point cloud data.

[0023] It should be noted that the first observation point can be understood as the actual observation point of the first 3D point in the corresponding 2D image, or it can be understood as the first 3D point calculated based on the first observation point in the corresponding 2D image. Projecting the first 3D point contained in the first point cloud data onto the corresponding 2D image can be understood as reprojecting the first 3D point back into the corresponding 2D image to obtain the first reprojection point. There is a one-to-one correspondence between the 3D points contained in the first 3D point, the reprojection points contained in the first reprojection point, and the observation points contained in the first observation point, and the number of each is the same; that is, when the first 3D point contains multiple 3D points, it is necessary to obtain the reprojection point and observation point corresponding to each 3D point separately.

[0024] S102. Based on the distance information between the first observation point and the first reprojection point, the first point cloud data is filtered to obtain the second point cloud data.

[0025] It should be noted that the distance information between the first observation point and the first reprojection point can be understood as the Euclidean distance between the first observation point and the first reprojection point.

[0026] In this embodiment of the application, the process of filtering the first point cloud data based on the distance information between the first observation point and the first reprojection point to obtain the second point cloud data specifically includes: filtering the first point cloud data based on the distance information and the reprojection error threshold to obtain the second point cloud data.

[0027] It should be noted that the reprojection error threshold can be set according to the actual situation and is not limited here; in practical applications, the reprojection error can be denoted as... The first point cloud data is filtered based on distance information and a reprojection error threshold to obtain the second point cloud data. This can be understood as follows: if the distance between the first reprojection point and the first observation point corresponding to a certain 3D point in the first 3D point data is greater than the reprojection error threshold, that 3D point is determined to be a noise point, and it is filtered (or removed) from the first point cloud data. The same method is used to determine whether to filter for each 3D point in the first point cloud data, resulting in the second point cloud data. If the distance between the first reprojection point and the first observation point corresponding to a certain 3D point is less than or equal to the reprojection error threshold, that 3D point does not need to be removed from the first point cloud data.

[0028] In this embodiment of the application, the process of filtering the first point cloud data based on the distance information between the first observation point and the first reprojection point to obtain the second point cloud data specifically includes: determining the mean information and variance information corresponding to the first reprojection point based on the distance information; and filtering the first point cloud data based on the distance information, mean information, and variance information to obtain the second point cloud data.

[0029] It should be noted that the mean and variance information corresponding to the first projection point can be understood as the mean and variance information corresponding to all 3D points contained in the first point cloud data. Determining the mean and variance information corresponding to the first projection point based on distance information can be understood as determining the mean and variance information based on the distance information corresponding to each of the 3D points (the first 3D point) contained in the first point cloud data and the number of all 3D points. In practical applications, the mean information can be denoted as... Variance information can be denoted as The first point cloud data is filtered based on distance, mean, and variance information to obtain the second point cloud data. This can be understood as follows: if the distance information of a 3D point in the first 3D points contained in the first point cloud data is greater than a first value or less than a second value, that 3D point is determined to be a noise point, and is filtered (or removed) from the first point cloud data. The same method is used to determine whether to filter each 3D point in the first point cloud data to obtain the second point cloud data. The first and second values ​​are obtained based on the mean, variance, and a first threshold. The first threshold is a preset value, and its specific value can be determined according to actual conditions and is not limited here. As an example, the mean information is denoted as... Variance information is denoted as When the first threshold is denoted as K, the first value can be denoted as... The second value can be denoted as Specifically, if the distance between the first reprojection point and the first observation point corresponding to a certain 3D point is less than or equal to the first value or greater than or equal to the second value, then it is not necessary to filter out that 3D point from the first point cloud data.

[0030] The solution in this application embodiment reprojects the 3D points in the first point cloud data onto the original 2D image, calculates the deviation between the 3D points and the actual observation points, and the smaller the error, the more reliable the 3D point is. Effective filtering of the point cloud data is then performed to select high-precision point clouds from the first point cloud data.

[0031] S103. Determine the confidence level of the second 3D point contained in the second point cloud data.

[0032] It should be noted that the second point cloud data can be understood as the point cloud data obtained after reprojecting and filtering the first point cloud data (sparse 3D point cloud). The second 3D point can be understood as all 3D points contained in the second point cloud data; a second 3D point may contain one or more 3D points. Determining the confidence level corresponding to the second 3D points contained in the second point cloud data can be understood as determining the confidence level corresponding to each individual 3D point contained in the second point cloud data.

[0033] In this embodiment of the application, the process of determining the confidence level of the second 3D point contained in the second point cloud data specifically includes: obtaining the coordinate information corresponding to the second 3D point; and determining the confidence level of the second 3D point based on the coordinate information and the variance information corresponding to the first reprojection point.

[0034] It should be noted that coordinate information can be understood as the three-dimensional coordinate information corresponding to each 3D point among all the 3D points contained in the second 3D point, which can be denoted as (x, y, z). The z value can also be exemplified as... , representing the depth coordinates of a 3D point; obtaining the coordinate information corresponding to the second 3D point can be understood as obtaining the 3D coordinate information corresponding to each 3D point among all 3D points. Determining the confidence level of the second 3D point based on the coordinate information and the variance information corresponding to the first reprojection point can be understood as determining the confidence level of the corresponding 3D point based on the z-value in the coordinate information of each 3D point in the second point cloud data, the number of cameras that observed the corresponding 3D point, the variance information of all reprojection points corresponding to all 3D points in the first point cloud data, and empirical parameters; where the value of the empirical parameter can be determined according to the actual situation and is not limited here; as an example, the empirical parameter can be 0.1; in practical applications, determining the confidence level of the second 3D point based on the coordinate information and the variance information corresponding to the first reprojection point can be achieved through... To represent; among which, represents the confidence level of a given 3D point; N represents the total number of cameras that observed and calculated the value of that 3D point. It is the variance of the reprojection error of all 3D points calculated in the previous step (reprojection filtering); This is an empirical parameter (set to 0.1 here). It is the depth coordinate of a certain 3D point, which is equal to z in (x,y,z).

[0035] S104. Filter the second point cloud data based on the confidence level to obtain the third point cloud data; the third point cloud data is used for at least three-dimensional reconstruction.

[0036] It should be noted that the third point cloud data is used at least for 3D reconstruction, which can be understood as the third point cloud data being used at least for the reconstruction of ultra-large scenes based on 3DGS.

[0037] In this embodiment of the application, the process of filtering the second point cloud data based on confidence level to obtain the third point cloud data specifically includes: filtering the second point cloud data based on confidence level to obtain the fourth point cloud data; obtaining the first quantity information corresponding to the adjacent 3D points of each third 3D point contained in the fourth point cloud data; and filtering the fourth point cloud data according to the first quantity information to obtain the third point cloud data.

[0038] It should be noted that the fourth point cloud data is obtained by filtering the second point cloud data based on confidence levels. This can be understood as follows: if the confidence level of a 3D point in the second point cloud data is less than a confidence threshold, that 3D point is considered noise and is filtered out. This method is then applied to the confidence levels of all 3D points in the second point cloud data to obtain the fourth point cloud data. The value of the confidence threshold can be determined based on the specific circumstances and is not limited here.

[0039] It should be noted that if a certain 3D point in the second point cloud data is observed by fewer viewpoints (or cameras) than the third value, the 3D point is judged to be a noise point and is filtered out; the third value can be 3.

[0040] It should be noted that the first quantity information can be understood as the quantity information of adjacent 3D points within a specified radius for each third 3D point contained in the fourth point cloud data; the specified radius can be determined according to the actual situation and is not limited here. Obtaining the first quantity information corresponding to the adjacent 3D points of each third 3D point contained in the fourth point cloud data can be understood as obtaining the quantity information of adjacent 3D points within a specified radius for each third 3D point contained in the fourth point cloud data.

[0041] In this embodiment of the application, the process of filtering the fourth point cloud data according to the first quantity information to obtain the third point cloud data specifically includes: filtering the fourth point cloud data according to the first quantity information to obtain the fifth point cloud data; dividing the fifth point cloud data into one or more point cloud units, and obtaining the second quantity information corresponding to the fourth 3D point contained in each point cloud unit; and filtering the fifth point cloud data according to the second quantity information to obtain the third point cloud data.

[0042] It should be noted that filtering the fourth point cloud data based on the first quantity information to obtain the fifth point cloud data can be understood as follows: if the first quantity information corresponding to the adjacent 3D points of a certain third 3D point in the fourth point cloud data is less than a set quantity threshold, then that third 3D point is filtered from the fourth point cloud data. The same filtering method is then applied to all third 3D points in the fourth point cloud data to obtain the fifth point cloud data. The quantity threshold can be determined based on actual circumstances and is not limited here.

[0043] It should be noted that a point cloud unit can be understood as a cube unit; its number can be determined according to the actual situation and is not limited here. Dividing the fifth point cloud data into one or more point cloud units can be understood as dividing the entire point cloud space corresponding to the fifth point cloud data into one or more cube units. The second quantity information can be understood as the quantity information corresponding to the fourth 3D point (all 3D points) contained in each point cloud unit (or cube unit). Filtering the fifth point cloud data according to the second quantity information to obtain the third point cloud data can be understood as follows: if the second quantity information contained in a point cloud unit is greater than or equal to the point count threshold, the centroid of all 3D points contained in that point cloud unit is retained; if the second quantity information contained in a point cloud unit is less than the point count threshold, that point cloud unit is ignored, and the same filtering method is applied to all point cloud units corresponding to the fifth point cloud data to obtain the third point cloud data; the value of the point count threshold can be determined according to the actual situation and is not limited here; as an example, the point count threshold can be 5.

[0044] The solution in this application embodiment filters the number of adjacent 3D points of each 3D point in the fourth point cloud data to obtain the noise points present in the fourth point cloud data, and then filters them to determine the purity of the obtained fifth point cloud data. By dividing the fifth point cloud data into point cloud units and retaining only the confidence level or ignoring the point cloud units, it is possible to ensure that the obtained third point cloud data has good uniformity and representativeness in spatial distribution, thereby improving the effect of subsequent rendering and modeling.

[0045] The solution in this application embodiment performs dual filtering on the first point cloud data by sequentially filtering it with the distance information between the first observation point and the first reprojection point, as well as the confidence level corresponding to the second 3D point. This can improve the filtering effect of the point cloud data and thus improve the effect of 3D reconstruction.

[0046] To facilitate understanding, the above method is illustrated with an example. In practical applications, the data processing method described above can be exemplified as a multi-level filtering optimization algorithm for sparse 3D point clouds in 3DGS ultra-large scene reconstruction. This filtering algorithm exhibits excellent filtering performance, does not rely on stringent prior conditions, and demonstrates strong adaptability and robustness, making it suitable for various complex scenarios.

[0047] 1. By analyzing the principle of point cloud generation, a dual filtering mechanism is adopted, which uses multi-camera constraints and reprojection error filtering strategies to filter out a large part of the noise; 2. Based on the results of the previous step, there will still be some false positive noise point clouds. That is, the noise satisfies both the multi-camera constraint and the reprojection error strategy. Therefore, an outlier filtering algorithm is introduced for secondary filtering. 3. Based on the above filtering, a uniform sampling strategy is used for resampling, which makes the final 3DGS initial point cloud more uniform and representative, and helps the 3DGS to reconstruct better.

[0048] This application provides a sparse 3D point cloud filtering optimization method based on multi-view images, aiming to improve the quality and stability of 3D point cloud data to meet the input data requirements of subsequent 3DGS scene reconstruction algorithms.

[0049] First, in the point cloud generation stage, the system simultaneously acquires multi-view 2D images of the target scene using multiple cameras. A deep learning-based 2D keypoint detection algorithm is used to extract 2D keypoints from each viewpoint. Then, an epipolar geometry algorithm is used to calculate the fundamental and essential matrices between the images, thereby achieving triangulation of the 3D points. The spatial coordinates (x, y, z) of each 3D point are determined by constraints from multiple camera views. The more cameras present, the more accurate the spatial position of the point, effectively improving the overall accuracy of the point cloud.

[0050] Secondly, in the point cloud filtering stage, a dual filtering mechanism is adopted: one is a filtering strategy based on multi-camera constraints, which assigns weights according to the number of cameras involved in each 3D point and removes low-confidence points; the other is a filtering strategy based on reprojection error, which reprojects the 3D points onto the original 2D image and calculates the deviation between the 3D points and the actual pixel coordinates. The smaller the error, the more reliable the point is, thereby further filtering out high-precision point clouds.

[0051] However, even after the above two processing steps, some outlier noise points may still exist in the point cloud. These points are usually caused by false detections or occlusion. To further remove noise, outlier detection algorithms are introduced, such as outlier identification methods based on statistical distance or density, to identify and remove abnormal points that still exist after the above two filtering steps, ensuring the purity of the point cloud data.

[0052] Finally, to make the point cloud more in line with the input requirements of the 3DGS algorithm, a uniform sampling strategy is adopted to resample the final sparse 3D point cloud, ensuring that the point cloud has good uniformity and representativeness in spatial distribution, thereby improving the effect of subsequent rendering and modeling.

[0053] The embodiments of this application effectively improve the quality and stability of sparse 3D point clouds through multi-algorithm collaborative optimization, providing a reliable data foundation for high-precision 3DGS scene reconstruction and visualization.

[0054] To make it easier to understand, the specific steps of the above method are illustrated here.

[0055] 1. Data processing.

[0056] The data source for this application embodiment is a multi-view 2D image sequence, acquired by one or more cameras with identical parameters at multiple locations. The camera should possess high resolution, a wide-angle lens, and good exposure control capabilities to ensure high-quality image data is acquired under varying lighting conditions. The image format is the common RGB image, with a minimum resolution generally required to be 1920×1080 pixels. Each captured image should be taken from a camera position that creates a certain parallax angle; generally, the overlap between two adjacent images should be between 60% and 85% to facilitate subsequent triangulation calculations (triangulation calculations are as follows). Figure 2 (The process). Furthermore, the photographs taken should completely cover the entire scene to prevent incomplete reconstruction due to missing areas.

[0057] 2. Sparse 3D point cloud generation.

[0058] The generation of sparse 3D point clouds mainly consists of two modules: 2D feature point detection and matching, and stereo matching.

[0059] Firstly, regarding the 2D feature point detection algorithm, this proposal abandons the Scale-Invariant Feature Transform (SIFT) + Fast Library for Approximate Nearest Neighbors (FLANN) algorithm, which has low performance, especially in weak texture regions. Instead, it uses the XFEAT feature point detection and matching algorithm. (Among related technologies, SIFT + FLANN is a relatively old algorithm; XFEAT is a lightweight image feature extraction and matching algorithm designed specifically for resource-constrained devices. Its core goal is to achieve extreme computational efficiency while maintaining high matching accuracy, and its performance is far superior to SIFT + FLANN. Moreover, XFEAT can detect and match more 2D feature points, especially in weak texture regions.)

[0060] Secondly, there's the stereo matching algorithm. This proposal uses a purely visual method for point cloud generation, meaning it only uses images and lacks point clouds generated by LiDAR equipment. Therefore, the solution used is a stereo matching algorithm. Simply put, stereo matching calculates the 3D coordinates of the corresponding 3D points by detecting pairs of 2D feature points from two images. Figure 2 This application provides an exemplary schematic diagram illustrating the calculation of the three-dimensional coordinates of corresponding 3D points based on 2D feature point pairs; as shown in the embodiment of this application. Figure 2 As shown, O1 and O2 are the optical center positions of the two cameras, the two boxes are two 2D images, and p1 and p2 are the 2D feature points matched in the previous step, with coordinates u1 and u2. For the pinhole camera model, we have s1u1=Kp, s2u2=K (Rp+t); where K is the camera intrinsic parameter, R and t are the rotation matrix and translation vector, respectively, representing the camera motion. The values ​​of R and t can be obtained through the epipolar geometry algorithm, which is the camera pose; s1 and s2 are the depths of the two points. After obtaining the values ​​of R and t, substituting them into the above formulas yields the 3D spatial coordinates (x, y, z) of point p. Point p is the point cloud data. Performing the same operation on all 2D keypoint matching pairs yields a complete sparse 3D point cloud for 3DGS scene reconstruction.

[0061] 3. Sparse 3D point cloud filtering.

[0062] (1) Reprojection error filtering.

[0063] Reprojection error is a key metric for measuring the consistency between 3D points and the original 2D image. For each 3D point P, it is projected back to the i-th camera view to obtain the reprojected point. And calculate its relationship with the actual observation point. The Euclidean distance, i.e. ;in, This indicates Euclidean distance.

[0064] Filtering can be performed in the following two ways: Set reprojection error threshold Filtering is performed when the reprojection error exceeds a certain threshold. If so, it is determined to be noise filtering; First, calculate the mean of all 3D point reprojection errors. and variance Where n is the total number of 3D points, and a threshold k is set, if the point or If the point is identified as noise, it will be filtered out.

[0065] (2) Multi-camera constrained filtering.

[0066] In the multi-camera constrained filtering stage, the spatial coordinates (x, y, d) of each 3D point are obtained by the joint constraints of N camera views. Let the 2D point corresponding to a certain 3D point p in the i-th camera view be... The confidence level of this point is defined as It can be done To represent; among which, represents the confidence level of a given 3D point; N represents the total number of cameras that observed and calculated the value of that 3D point. It is the variance of the reprojection error of all 3D points calculated in the previous step (reprojection filtering); This is an empirical parameter (set to 0.1 here). It is the depth coordinate of a certain 3D point, which is equal to z in (x,y,z).

[0067] Filtering can be constrained using the following two methods: If the number of cameras at point p is less than 3, then the point is directly identified as a noise point and filtered out. This is mainly because the more cameras are calculated for each 3D point, the smaller the error will be, and vice versa. It may even be a false positive point (i.e., reprojection error, or high confidence but large actual error). Set confidence threshold If the confidence level of point p is less than the threshold, then the noise point is determined to be filtered out.

[0068] (3) Outlier filtering.

[0069] Even after the two filtering steps described above, some outliers may still exist. These outliers typically exhibit extremely low local density or significant differences from other points. Therefore, a radius-based outlier filtering method is used for further point cloud filtering. Radius-based outlier filtering detects outliers by calculating the number of neighbors of each point within a specified radius. If a point has fewer than a set threshold of neighbors within the specified radius, it is identified as an outlier.

[0070] (4) Uniform sampling strategy.

[0071] To make the point cloud more consistent with the input requirements of the 3DGS algorithm, a spatially uniform sampling strategy is adopted to eliminate the problem of uneven point cloud density. Specifically, the entire point cloud space is divided into several cubic units, and only one representative point is retained in each unit, which is the centroid of all points in the unit. The sampling process follows these rules: If the number of points in a cell is greater than or equal to a set threshold (e.g., 5), then the centroid point is retained. Otherwise, ignore the unit.

[0072] This sampling strategy not only improves the uniformity of the point cloud, but also reduces the computational complexity of the subsequent 3DGS algorithm.

[0073] This application provides a data processing apparatus. Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; as shown below. Figure 3 As shown, the data processing device 300 includes: The first processing unit 301 is configured to project the first 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point to obtain the first reprojection point of the first 3D point in the two-dimensional image; and to obtain the first observation point of the first 3D point in the corresponding two-dimensional image. The first filtering unit 302 is used to filter the first point cloud data according to the distance information between the first observation point and the first reprojection point to obtain the second point cloud data. The determining unit 303 is used to determine the confidence level corresponding to the second 3D point contained in the second point cloud data; The second filtering unit 304 is used to filter the second point cloud data based on the confidence level to obtain the third point cloud data; the third point cloud data is used for at least three-dimensional reconstruction.

[0074] In one embodiment, the first filtering unit 302 is further configured to filter the first point cloud data according to the distance information and the reprojection error threshold to obtain the second point cloud data.

[0075] In one embodiment, the first filtering unit 302 is further configured to determine the mean information and variance information corresponding to the first reprojection point based on the distance information; and to perform filtering processing on the first point cloud data according to the distance information, the mean information and the variance information to obtain the second point cloud data.

[0076] In one embodiment, the determining unit 303 is further configured to obtain coordinate information corresponding to the second 3D point; and determine the confidence level corresponding to the second 3D point based on the coordinate information and the variance information corresponding to the first reprojection point.

[0077] In one embodiment, the second filtering unit 304 is further configured to filter the second point cloud data based on the confidence level to obtain fourth point cloud data; obtain first quantity information corresponding to the adjacent 3D points of each third 3D point contained in the fourth point cloud data; and filter the fourth point cloud data according to the first quantity information to obtain the third point cloud data.

[0078] In one embodiment, the second filtering unit 304 is further configured to filter the fourth point cloud data according to the first quantity information to obtain fifth point cloud data; divide the fifth point cloud data into one or more point cloud units, and obtain the second quantity information corresponding to the fourth 3D point contained in each point cloud unit; and filter the fifth point cloud data according to the second quantity information to obtain the third point cloud data.

[0079] In one embodiment, before projecting the first three-dimensional 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point, the data processing device 300 further includes a second processing unit for acquiring a multi-view two-dimensional image; determining one or more feature point pairs corresponding to the multi-view two-dimensional image; and determining the first point cloud data based on the one or more feature point pairs.

[0080] This application also provides an electronic device. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 4 As shown, the electronic device 400 includes a processor 401 and a memory 403. Optionally, the electronic device 400 may also include a communication bus 402.

[0081] In specific embodiments, the processor 401 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not impose specific limitations.

[0082] In this embodiment, the communication bus 402 is used to establish a connection and communication between the processor 401 and the memory 403; when the processor 401 executes the running program stored in the memory 403, it implements the following data processing method: The first 3D point contained in the first point cloud data is projected onto the corresponding two-dimensional image to obtain the first reprojection point of the first 3D point in the two-dimensional image; and the first observation point of the first 3D point in the corresponding two-dimensional image is obtained; the first point cloud data is filtered according to the distance information between the first observation point and the first reprojection point to obtain the second point cloud data; the confidence level corresponding to the second 3D point contained in the second point cloud data is determined; the second point cloud data is filtered based on the confidence level to obtain the third point cloud data; the third point cloud data is used for at least three-dimensional reconstruction.

[0083] Furthermore, the processor 401 is also used to filter the first point cloud data according to the distance information and the reprojection error threshold to obtain the second point cloud data.

[0084] Furthermore, the processor 401 is also configured to determine the mean information and variance information corresponding to the first reprojection point based on the distance information; and to perform filtering processing on the first point cloud data according to the distance information, the mean information and the variance information to obtain the second point cloud data.

[0085] Furthermore, the processor 401 is also used to obtain the coordinate information corresponding to the second 3D point; and to determine the confidence level corresponding to the second 3D point based on the coordinate information and the variance information corresponding to the first reprojection point.

[0086] Furthermore, the processor 401 is also configured to perform filtering processing on the second point cloud data based on the confidence level to obtain fourth point cloud data; obtain first quantity information corresponding to the adjacent 3D points of each third 3D point contained in the fourth point cloud data; and perform filtering processing on the fourth point cloud data according to the first quantity information to obtain the third point cloud data.

[0087] Furthermore, the processor 401 is also configured to filter the fourth point cloud data according to the first quantity information to obtain the fifth point cloud data; divide the fifth point cloud data into one or more point cloud units, and obtain the second quantity information corresponding to the fourth 3D point contained in each point cloud unit; and filter the fifth point cloud data according to the second quantity information to obtain the third point cloud data.

[0088] Furthermore, before projecting the first three-dimensional 3D point contained in the first point cloud data onto the two-dimensional image corresponding to the first 3D point, the processor 401 is further configured to acquire a multi-view two-dimensional image; and determine one or more feature point pairs corresponding to the multi-view two-dimensional image; and determine the first point cloud data based on the one or more feature point pairs.

[0089] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the data processing method described above.

[0090] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the data processing method described above.

[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0093] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Project the first 3D point contained in the first point cloud data onto the corresponding 2D image of the first 3D point to obtain the first super-projection point of the first 3D point in the 2D image; and obtain the first observation point of the first 3D point in the corresponding 2D image. Based on the distance information between the first observation point and the first reprojection point, the first point cloud data is filtered to obtain the second point cloud data. Determine the confidence level corresponding to the second 3D point contained in the second point cloud data; The second point cloud data is filtered based on the confidence level to obtain the third point cloud data; the third point cloud data is used for at least three-dimensional reconstruction.

2. The method according to claim 1, characterized in that, The step of filtering the first point cloud data based on the distance information between the first observation point and the first reprojection point to obtain the second point cloud data includes: The first point cloud data is filtered based on the distance information and the reprojection error threshold to obtain the second point cloud data.

3. The method according to claim 1, characterized in that, The step of filtering the first point cloud data based on the distance information between the first observation point and the first reprojection point to obtain the second point cloud data includes: Based on the distance information, determine the mean information and the variance information corresponding to the first reprojection point; The first point cloud data is filtered based on the distance information, the mean information, and the variance information to obtain the second point cloud data.

4. The method according to claim 1, characterized in that, Determining the confidence level corresponding to the second 3D point contained in the second point cloud data includes: Obtain the coordinate information corresponding to the second 3D point; The confidence level of the second 3D point is determined based on the coordinate information and the variance information corresponding to the first reprojection point.

5. The method according to claim 1, characterized in that, The step of filtering the second point cloud data based on the confidence level to obtain the third point cloud data includes: The second point cloud data is filtered based on the confidence level to obtain the fourth point cloud data; Obtain the first quantity information corresponding to the adjacent 3D points of each third 3D point contained in the fourth point cloud data; The fourth point cloud data is filtered based on the first quantity information to obtain the third point cloud data.

6. The method according to claim 5, characterized in that, The step of filtering the fourth point cloud data based on the first quantity information to obtain the third point cloud data includes: The fourth point cloud data is filtered based on the first quantity information to obtain the fifth point cloud data. The fifth point cloud data is divided into one or more point cloud units, and the second quantity information corresponding to the fourth 3D points contained in each point cloud unit is obtained. The fifth point cloud data is filtered based on the second quantity information to obtain the third point cloud data.

7. The method according to claim 1, characterized in that, Before projecting the first 3D point contained in the first point cloud data onto the corresponding 2D image, the method further includes: Acquire multi-view two-dimensional images; and determine one or more feature point pairs corresponding to the multi-view two-dimensional images; The first point cloud data is determined based on the one or more feature point pairs.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A storage medium having a computer program or computer-executable instructions stored thereon, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images