Point cloud data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a point cloud data processing method and device, electronic equipment and storage medium, which are related to the fields of artificial intelligence, three-dimensional modeling, cloud technology and the like. The method comprises: performing feature extraction on original point cloud data to obtain feature representations of a plurality of first points, the feature representations comprising three-dimensional coordinates and point features; for each first point, determining a first number of neighborhood points of the point from the plurality of first points according to the coordinates of the first points, determining the relevance between the point and each neighborhood point of the point according to the feature representation of the point and the feature representation of each neighborhood point of the point, fusing the feature representations of each neighborhood point of the point according to the relevance to obtain corresponding fused features, and obtaining second feature representations of a plurality of second points based on the fused features corresponding to each first point; and generating target point cloud data based on the feature representations of each second point. The method based on the embodiments of the present application can effectively improve the completion effect of point cloud data.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of artificial intelligence, cloud technology, big data and computer technology, in particular, the present application relates to a point cloud data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] A point cloud is a set of points, each point in the set contains three-dimensional coordinates, and some may contain color information or reflectivity information. Point cloud is one of the most commonly used three-dimensional object representation models. The point cloud completion (similar to the completion / repair of two-dimensional images) task aims to restore incomplete and sparse scanning data into complete and dense point clouds. As shown in FIG. (a) in the figure, Figure 1 Figure 1 The incomplete point cloud data corresponds to a three-dimensional image, and the purpose of point cloud completion is to obtain the complete point cloud data shown in FIG. (b) in the figure. Point cloud completion can be applied to various application scenarios such as repair and completion of three-dimensional scanning data. Figure 1

[0003] With the development of artificial intelligence technology, in recent years, deep learning technology has also begun to be applied to point cloud data, and point cloud completion can be achieved by training neural networks. At present, although some neural networks can achieve point cloud completion to a certain extent, the effect of most network models on point cloud completion is not good, and still needs to be improved. How to further improve the effect of point cloud completion has always been an important problem in the study of point cloud completion tasks. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a point cloud data processing method, device, electronic equipment and storage medium which can effectively improve the point cloud completion. In order to achieve the above purpose, the technical scheme provided by the embodiments of the present application is as follows:

[0005] On the one hand, the present application provides a point cloud data processing method, which comprises:

[0006] Obtaining original point cloud data to be completed, the original point cloud data comprising three-dimensional coordinates of a plurality of initial points;

[0007] Performing feature extraction on the original point cloud data to obtain feature representation of a plurality of first points, the feature representation of each first point comprising three-dimensional coordinates and point features of the first point;

[0008] According to the three-dimensional coordinates of each first point, a first number of neighborhood points of each first point is determined from the plurality of first points;

[0009] ​​For each first point, determine the relevance between the first point and each neighbor point of the first point according to the feature representation of the first point and the feature representation of each neighbor point of the first point, and fuse the feature representation of each neighbor point of the first point according to the corresponding relevance of each neighbor point of the first point, to obtain the fusion feature corresponding to the first point;

[0010] Based on the fusion feature of each first point, obtain the second feature representation of a plurality of second points;

[0011] Based on the feature representation of the plurality of second points, generate the target point cloud data after completion.

[0012] In another aspect, an embodiment of the present application provides a point cloud data processing device, the device comprising:

[0013] A to-be-completed data acquisition module is configured to acquire original point cloud data to be completed, the original point cloud data comprising three-dimensional coordinates of a plurality of initial points;

[0014] A data completion module is configured to obtain target point cloud data after completion by performing a data completion operation, the data completion operation comprising the following steps:

[0015] Feature extraction is performed on the original point cloud data to obtain feature representation of a plurality of first points, the feature representation of each first point comprising three-dimensional coordinates and point features of the first point;

[0016] According to the three-dimensional coordinates of each first point, a first number of neighbor points of each first point are determined from the plurality of first points;

[0017] For each first point, determine the relevance between the first point and each neighbor point of the first point according to the feature representation of the first point and the feature representation of each neighbor point of the first point, and fuse the feature representation of each neighbor point of the first point according to the corresponding relevance of each neighbor point of the first point, to obtain the fusion feature corresponding to the first point;

[0018] Based on the fusion feature of each first point, obtain the second feature representation of a plurality of second points;

[0019] Based on the feature representation of the plurality of second points, generate the target point cloud data after completion.

[0020] Optionally, the data completion module can obtain the fusion feature corresponding to each first point by performing the following operations:

[0021] For each first point, concatenate the point features and three-dimensional coordinates of the first point to obtain the first feature of the first point;

[0022] For each neighborhood point of each first point, determine the relevance between the first point and the neighborhood point according to the first feature of the first point and the first feature of the neighborhood point;

[0023] For each first point, weight and fuse the first features of the neighborhood points of the first point by taking the relevance corresponding to each neighborhood point of the first point as a weight, to obtain the fusion feature corresponding to the first point.

[0024] Optionally, the data completion module can obtain the relevance between each first point and the neighborhood points of the first point and the fusion feature corresponding to the first point by using at least one first attention module based on a channel attention mechanism; wherein, when there are at least two first attention modules, the fusion feature corresponding to each first point can be obtained by the following way:

[0025] For each first point, take the first feature of the first point as a query vector, and take the first feature of each neighborhood point of the first point as a key vector respectively, and execute the following operations by each first attention module respectively:

[0026] For each neighborhood point of the first point, determine the first attention weight between the first point and the neighborhood point based on the first feature of the first point and the first feature of the neighborhood point, and take the first attention weight as the relevance;

[0027] Weight and fuse the first features of the respective neighborhood points by taking the first attention weight corresponding to each neighborhood point as a weight, to obtain the new feature corresponding to the first point;

[0028] Wherein, the fusion feature corresponding to each first point includes the new feature corresponding to each first attention module of the first point;

[0029] Correspondingly, when the data completion module obtains the second feature representation of a plurality of second points based on the fusion feature corresponding to each first point, it can be used for:

[0030] Take the new feature corresponding to each first attention module of each first point as the point feature of a second point respectively, and generate the three-dimensional coordinates of each second point based on the point features of a plurality of second points, wherein the feature representation of each second point includes the three-dimensional coordinates and the point feature of the second point.

[0031] Optionally, the data completion module is further used for: for each first point, generating the spatial coordinate feature of the first point according to the three-dimensional coordinates of the first point;

[0032] Correspondingly, for each neighborhood point of each first point, the data completion module can be configured to:

[0033] determine the relevance between the first point and the neighborhood point according to the first feature of the first point and the first feature of the neighborhood point, and obtain the fusion feature corresponding to the first point;

[0034] for each neighborhood point of the first point, perform feature extraction based on the first feature and the spatial coordinate feature of the neighborhood point to obtain the second feature of the neighborhood point;

[0035] weight the second features of each neighborhood point of the first point using the weights corresponding to the neighborhood points to obtain the fusion feature corresponding to the first point.

[0036] Optionally, the data completion module can be configured to, when generating the completed target point cloud data based on the feature representations of the plurality of second points:

[0037] based on the feature representations of the plurality of second points, obtain the point features of a plurality of target points by performing at least one upsampling process, and obtain the three-dimensional coordinates of the plurality of target points according to the point features of the plurality of target points, wherein the target point cloud data comprises the three-dimensional coordinates of the plurality of target points.

[0038] Optionally, the data completion module can be configured to, when generating the target point cloud data:

[0039] determine an initial point cloud input based on the three-dimensional coordinates of the plurality of initial points and the three-dimensional coordinates of each second point, wherein the initial point cloud input comprises the three-dimensional coordinates of a plurality of input points;

[0040] use the initial point cloud input as the point cloud input of the first upsampling process, repeatedly perform the upsampling process until a set number of times is reached, and determine the three-dimensional coordinates of a plurality of output points obtained by the last upsampling process as the three-dimensional coordinates of a plurality of target points;

[0041] wherein the upsampling process comprises the following steps:

[0042] for each input point in the point cloud input, determine the distance between the input point and each second point according to the three-dimensional coordinates of the input point and the three-dimensional coordinates of each second point, and determine the second number of neighborhood points of the input point from each second point according to the distance;

[0043] for each input point, obtain the point feature of the input point by fusing the point features of each neighborhood point of the input point;

[0044] based on the three-dimensional coordinates and the point features of the input points, obtaining the point features of the output points through feature upsampling;

[0045] based on the point features of the output points, obtaining the three-dimensional coordinates of each output point, and taking the three-dimensional coordinates of each output point as the point cloud input of the next upsampling processing.

[0046] Optionally, for each input point, when obtaining the point feature of the input point by fusing the point features of each neighborhood point of the input point, the data completion module can be used for:

[0047] determining the weight corresponding to each neighborhood point of the input point according to the distance between the input point and each neighborhood point of the input point, wherein the weight corresponding to each neighborhood point is negatively correlated with the distance corresponding to the neighborhood point;

[0048] weighting and fusing the point features of each neighborhood point of the input point according to the weight corresponding to each neighborhood point of the input point, to obtain the point feature of the input point.

[0049] Optionally, for each upsampling processing, when obtaining the point features of the output points through feature upsampling based on the three-dimensional coordinates and the point features of the input points, the data completion module can be used for:

[0050] concatenating the point feature and the three-dimensional coordinates of each input point respectively to obtain the third feature of each input point;

[0051] determining the third number of neighborhood points of each input point from each input point according to the three-dimensional coordinates of each input point;

[0052] for each input point, obtaining the point features of at least two output points corresponding to the input point through feature upsampling according to the third feature of the input point and the historical features of each neighborhood point of the input point;

[0053] wherein for the first upsampling processing, the historical feature of a neighborhood point is the third feature of the neighborhood point, and for each upsampling processing except the first upsampling processing, the historical feature of a neighborhood point is the point feature of the neighborhood point as an output point in the last upsampling processing.

[0054] Optionally, for each input point, when obtaining the point features of at least two output points corresponding to the input point through feature upsampling according to the third feature of the input point and the historical features of each neighborhood point of the input point, the data completion module can be used for:

[0055] performing the following operations through each attention module in the at least two second attention modules respectively:

[0056] determine a second attention weight between the third feature of the input point and the historical feature of each neighbor point of the input point;

[0057] weight and fuse the historical features of the neighbor points of the input point based on the second attention weights corresponding to the neighbor points of the input point, to obtain a point feature of an output point corresponding to the input point.

[0058] Optionally, when obtaining the three-dimensional coordinates of each output point based on the point features of the plurality of output points, the data completion module can be configured to:

[0059] perform feature extraction based on the point features of the plurality of output points, to obtain a three-dimensional coordinate offset of each output point;

[0060] obtain the three-dimensional coordinates of each output point based on the three-dimensional coordinates of each input point and the three-dimensional coordinate offset of each output point.

[0061] Optionally, when obtaining the target point cloud data after completion based on the original point cloud data, the data completion module is configured to implement the operation by invoking a trained point cloud completion network, the point cloud completion network is obtained by iteratively training a neural network model based on a training data set, the training data set includes a plurality of training samples, each loss training sample includes a point cloud data pair, and the point cloud data pair includes sample point cloud data to be completed and corresponding sample point cloud data after completion.

[0062] In another aspect, an electronic device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method provided in any of the optional embodiments of the present application by executing the computer program.

[0063] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided in any of the optional embodiments of the present application.

[0064] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the method provided in any of the optional embodiments of the present application.

[0065] The technical scheme provided by the embodiments of the present application has the following beneficial effects:

[0066] The point cloud data processing method provided in this application, during the completion processing of original point cloud data, after extracting features from the original point cloud data to obtain initial point cloud feature representations (i.e., the three-dimensional coordinates and point features of multiple first points), obtains feature representations of multiple second points through feature fusion based on the semantic relationships between each point and its neighboring points in the initial point cloud feature representations. Since feature fusion considers not only the information of the point itself but also the information of its neighboring points based on the semantic relationships between the point and its neighboring points, the feature representations of the multiple second points obtained not only contain more detailed features but also better maintain the semantic consistency between each point and its neighboring points. Therefore, the target point cloud data generated based on the feature representations of the above multiple second points can more realistically restore the structure and details of the object itself, effectively improve the point cloud completion effect, and better meet the needs of practical applications. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0068] Figure 1 A diagram showing the difference between the effects before and after completing point cloud data;

[0069] Figure 2 A flowchart illustrating a point cloud data processing method provided in an embodiment of this application;

[0070] Figure 3 This is a schematic diagram of the structure of a point cloud data processing system applicable to the embodiments of this application;

[0071] Figure 4 A schematic diagram illustrating the model structure and data processing principle of a point cloud completion network provided in this application embodiment;

[0072] Figure 5 A schematic diagram illustrating the structure and data processing principle of a generator provided in an embodiment of this application;

[0073] Figure 6 and Figure 7 A schematic diagram showing the effect comparison of point cloud data obtained by different schemes in the embodiments of this application;

[0074] Figure 8 This is a schematic diagram of the structure of a point cloud data processing device provided in the embodiments of this application;

[0075] Figure 9 This is a schematic diagram of the structure of an electronic device to which this application applies. Detailed Implementation

[0076] The embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0077] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms, unless specifically stated otherwise. It should be further understood that the terms "include" and "contain" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or their combinations supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein means that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B". When describing a plurality of (two or more) items, if the relationship between the plurality of items is not explicitly limited, the plurality of items can mean one, more or all of the plurality of items, for example, for the description of "parameters A include A1, A2, A3", the parameters A can include A1 or A2 or A3, or the parameters A can include at least two of the three parameters A1, A2, A3.

[0078] The present application is a point cloud data processing method proposed to improve the point cloud data completion effect. Based on the method, the completion effect of point cloud data can be effectively improved.

[0079] Optionally, the scheme provided by the embodiments of the present application can be implemented based on artificial intelligence (AI) technology, and specifically can involve training of a neural network model and application of the neural network model. For example, a point cloud completion network meeting the requirements can be trained based on a training data set in a machine learning (ML) manner, and then for original point cloud data that needs to be completed, the trained point cloud completion network can be called to obtain completed target point cloud data, thereby providing a good data basis for three-dimensional repair or other applications of a target object (such as a target object).

[0080] Artificial intelligence is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is also the design principle and implementation method of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making. With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous vehicle, unmanned aerial vehicle, robot, intelligent medical treatment, intelligent customer service, Internet of vehicles, autonomous driving, intelligent transportation and the like. It is believed that with the development of technology, the technology will be applied in more fields and play an increasingly important value.

[0081] Optionally, the data processing involved in the method provided by the embodiment of the application can be realized based on cloud technology. For example, the training of the point cloud completion network (i.e., the initial neural network model for point cloud data completion) can be based on cloud technology, and various data calculations (such as calculation of training loss, adjustment of model parameters, etc.) involved in the training process can be realized by cloud computing. Optionally, the storage of each training set used in the training process can also be stored by cloud storage. Optionally, when the original point cloud data is processed by using the trained point cloud completion network, the data processing steps involved can also be realized by cloud computing. For example, the point cloud data processing method provided by the embodiment of the application can be executed by a cloud server.

[0082] Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool, and can be used as needed, flexibly and conveniently. Cloud computing refers to the delivery and use mode of IT infrastructure, which refers to obtaining the required resources in a demand-driven and easily expandable manner through a network; broad sense of cloud computing refers to the delivery and use mode of service, which refers to obtaining the required service in a demand-driven and easily expandable manner through a network. Such services can be IT and software, Internet related, or other services. With the development of Internet, real-time data flow, and diversified connected devices, and the promotion of search services, social networks, mobile commerce and open collaboration, the generation of cloud computing will revolutionize the entire Internet model and enterprise management model from the concept.

[0083] Optionally, the point cloud data processing method provided by the embodiments of the present application can be implemented as an independent application program or a functional module / plug-in in an application program (such as a target application). By running the independent application program or the module / plug-in with the corresponding function, the point cloud data can be completed. Optionally, the application program can also render a three-dimensional image of the target object corresponding to the target point cloud data based on the completed target point cloud data.

[0084] The method provided by the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server. As an optional mode, the method can be executed by a server (such as a cloud server). The server can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The user terminal (also referred to as user equipment) can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device (such as a smart speaker), a wearable electronic device (such as a smart watch), a vehicle-mounted terminal, a smart home appliance (such as a smart television), an AR / VR device, etc., but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the present application.

[0085] The technical solutions of the embodiments of the present application and the technical effects of the technical solutions of the present application will be described below through the description of several optional embodiments. It should be pointed out that the embodiments described below can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0086] Figure 2 A flowchart of a point cloud data processing method provided by the embodiments of the present application is shown. The method can be executed by a server, such as Figure 2 As shown, the method includes the following steps S110 to S160.

[0087] Step S110: obtaining original point cloud data to be completed, wherein the original point cloud data includes three-dimensional coordinates of a plurality of initial points.

[0088] The original point cloud data, i.e., sparse incomplete point cloud data, is the original input of the method. The application embodiments do not limit the acquisition mode of the original point cloud data. The original point cloud data can be obtained in various ways. For example, the scanning data of a target object in a real scene can be obtained by using a point cloud data acquisition device. For example, the point cloud data can be obtained by scanning the surface of an object by a laser scanner. In actual application, due to human factors or light, environment and other factors, the obtained point cloud data is not complete, and therefore needs to be completed. The purpose of point cloud completion is to repair the incomplete point cloud data to obtain repaired target point cloud data.

[0089] Step S120: performing feature extraction on the original point cloud data to obtain a feature representation of each of a plurality of first points, wherein the feature representation of each first point includes a three-dimensional coordinate of the first point and a point feature.

[0090] This step is used to realize extraction of incomplete point cloud data, so as to realize subsequent point cloud completion based on the feature representation of the extracted plurality of points. The application embodiments do not limit the specific mode of feature extraction on the original point cloud data. Optionally, the original point cloud data can be extracted by using a point cloud feature extraction network. The specific network model can be selected according to the requirement. Optionally, a hierarchical point cloud feature extraction network can be used for feature extraction. For example, an encoder in a feature extraction network based on PointNet++ or based on point transformer can be used to extract the feature of the original point cloud data to obtain the point feature and the three-dimensional coordinate of each of the plurality of first points.

[0091] As an exemplary description, it is assumed that the original point cloud data includes three-dimensional coordinates of 2000 points. The three-dimensional coordinates of the 2000 points can be input as input data of a neural network into a trained encoder. Based on the encoder, the point feature and the three-dimensional coordinate of each of the plurality of points can be extracted. For example, N p = 500 points of coordinates and point features wherein, represents a set of three-dimensional coordinates of the 500 first points extracted, represents a set of point features of the 500 first points, and the feature dimension of the point feature of each first point is C p .

[0092] Step S130: determining a first number of neighborhood points of each first point from the plurality of first points according to the three-dimensional coordinate of each first point.

[0093] Step S140: For each first point, determining the relevance between the first point and each neighbor point of the first point according to the feature representation of the first point and the feature representation of each neighbor point of the first point, and fusing the feature representation of each neighbor point of the first point according to the relevance corresponding to each neighbor point of the first point to obtain the fused feature corresponding to the first point.

[0094] Step S150: Obtaining the second feature representation of the plurality of second points based on the fused feature corresponding to each first point.

[0095] Step S160: Generating the completed target point cloud data based on the feature representation of the plurality of second points.

[0096] Wherein, for each first point, the relevance between the first point and any neighbor point of the first point represents the semantic correlation degree between the first point and the neighbor point of the first point.

[0097] In order to make the completed point cloud data not only maintain the overall structure of the target object, but also have more real detailed information, the method provided in the embodiment of the application further combines the semantic relationship between the first point and its neighbor points after obtaining the three-dimensional coordinates and point features of the plurality of first points based on the original point cloud data, generates the fused feature corresponding to each first point, so that the fused feature not only includes the information of the point itself, but also incorporates the information of the neighboring points. Therefore, compared with the features extracted by step S120, the fused feature contains more detailed features, which provides better data support for subsequent target point cloud data.

[0098] In the embodiment of the application, the neighbor points of each first point can be determined based on the distance between the first points, and the specific value (positive integer) of the number of neighbor points (i.e. the first number) is not limited in the embodiment of the application, and can be configured according to actual application requirements, experience values or experimental values. Assuming that the number of neighbor points is m, the distance between each two first points can be calculated according to the three-dimensional coordinates of the plurality of first points. For each first point, the first points corresponding to the m smallest distances between the point and each first point can be selected as the neighbor points of the point in ascending order. Optionally, in actual application, the neighbor points of each first point can include the point itself or not.

[0099] After determining the neighbor points of each first point, for each first point, the semantic relevance between the point and each neighbor point can be calculated according to the feature representation of the point and the feature representation of the neighbor point, that is, the relevance between the feature representations of the two points. The specific calculation method of the relevance of the first point and its neighbor points is not limited in the embodiment of the application.

[0100] For example, for each first point, a new feature of the point can be obtained by fusing the point feature and the three-dimensional coordinates (such as stitching), and for each neighborhood point of the point, the correlation between the new feature of the point and the new feature of the neighborhood point can be taken as the correlation between the point and the neighborhood point; further feature extraction can also be performed on the new feature of each first point to obtain a feature with better semantic expression ability, and the correlation between the further extracted feature of the first point and the further extracted feature of its neighborhood point is obtained.

[0101] Optionally, the correlation between any two points (a first point and a neighborhood point of the point) can be a coarse-grained correlation or a fine-grained correlation. For example, the correlation can be a feature vector, which can be a vector with only one numerical value, and the larger the numerical value, the greater the semantic correlation between the two points. The vector can also be a vector including multiple numerical values (the size of the numerical value is the same as the dimension of the feature of the first point), and each numerical value in the vector represents the correlation of one dimension in the features of the two points.

[0102] After obtaining the correlation between each first point and each neighborhood point thereof, the feature representations of the neighborhood points can be fused according to the correlation to obtain the fused feature corresponding to each point. For example, for a first point, after obtaining the correlation between the point and each of the m neighborhood points thereof, the feature of each neighborhood point (which can be based on the point feature and the three-dimensional coordinates of the neighborhood point) can be weighted according to the correlation corresponding to the neighborhood point as the weight, and the weighted features of the neighborhood points can be fused to obtain the fused feature corresponding to the first point.

[0103] As an optional solution, the fused feature corresponding to each first point can be obtained by the following way:

[0104] For each first point, the point feature and the three-dimensional coordinates of the first point are stitched to obtain a first feature of the first point;

[0105] For each neighborhood point of each first point, the correlation between the first point and the neighborhood point is determined according to the first feature of the first point and the first feature of the neighborhood point;

[0106] For each first point, the first features of the neighborhood points of the first point are weighted and fused according to the correlations corresponding to the neighborhood points as the weights to obtain a fused feature corresponding to the first point.

[0107] Optionally, for each first point, in the execution of the above-mentioned weighted fusion, the weight of each neighbor point of the first point can be directly used to perform weighted summation on the first feature of each corresponding neighbor point to obtain the fusion feature, or the first feature of each neighbor point can be further extracted, and then the weight of each neighbor point of the first point can be used to perform weighted summation on the extracted feature of each corresponding neighbor point.

[0108] In order to better learn the semantic association between adjacent points, as an optional solution, an attention module based on an attention mechanism can be used to obtain the fusion feature corresponding to each first point based on the feature representation of each first point. Optionally, the input feature of each first point can be obtained based on the feature representation of each first point. For each first point, the input feature of the point can be used as the query vector (Q) and the key vector (K) of the attention module. The input feature can be further extracted to obtain the value vector (V) of the point. Based on the input feature of the point and the input feature of each neighbor point of the point, the attention weight (i.e., the association) corresponding to the point and each neighbor point can be obtained through a self-attention module, and the value vector of each neighbor point can be weighted and fused to obtain the fusion feature corresponding to the point.

[0109] For the point cloud completion task, since the purpose is to obtain the completed point cloud, that is, the three-dimensional coordinates of the completed points, the coordinate information of the points is particularly important in the point cloud completion task. In order to further improve the completion effect of the point cloud data, as an optional solution, the method of the present application can further include: for each first point, generating a spatial coordinate feature of the first point according to the three-dimensional coordinates of the first point;

[0110] For each neighbor point of each first point, the above-mentioned determination of the association between the first point and the neighbor point based on the first feature of the first point and the first feature of the neighbor point can include: determining the association between the first point and the neighbor point based on the first feature of the first point, the first feature of the neighbor point, and the spatial coordinate feature of the neighbor point;

[0111] For each first point, the above-mentioned weighted fusion of the first feature of each neighbor point of the first point by taking the association of each neighbor point of the first point as the weight to obtain the fusion feature corresponding to the first point includes:

[0112] For each neighbor point of the first point, a second feature of the neighbor point is obtained by performing feature extraction based on the first feature and the spatial coordinate feature of the neighbor point.

[0113] The second feature of each neighbor point of the first point is weighted and fused by taking the weight corresponding to each neighbor point of the first point to obtain the fusion feature corresponding to the first point.

[0114] In this optional solution, when calculating the relevance between the first point and its neighborhood points and obtaining the fusion feature corresponding to the first point by fusing the features of the neighborhood points, the coordinate information of each neighborhood point is further considered. By incorporating the spatial coordinate features of the three-dimensional coordinates of each neighborhood point as supplementary information into the calculation of the relevance between the points and the fusion feature, the calculated result can be more consistent with the actual situation, and the real effect of the finally obtained target point cloud data can be improved.

[0115] The specific manner in which the second feature of the neighborhood point is obtained based on the first feature and the spatial coordinate feature of the neighborhood point is not limited by the embodiments of the application. Optionally, for each neighborhood point, the first feature of the neighborhood point can be extracted, and the extracted feature can be fused (such as splicing or adding corresponding elements, etc.) with the spatial coordinate feature of the neighborhood point to obtain the second feature.

[0116] After obtaining the fusion feature corresponding to each first point, the feature representation of a plurality of second points can be obtained based on the fusion feature corresponding to each first point, wherein the feature representation of each second point includes the point feature and the three-dimensional coordinates of the second point. Optionally, the fusion feature corresponding to each first point can be one, at this time, each first point can be a second point, and the three-dimensional coordinates of a first point are the three-dimensional coordinates of a second point. The fusion feature corresponding to the first point can be used as the point feature of the second point. The fusion feature corresponding to each first point can also be multiple. In actual application, in order to obtain a more complete completed point cloud structure, the fusion feature corresponding to each first point can also be multiple, that is, in addition to obtaining fusion features containing more information, up-sampling of point cloud data can also be realized. At this time, each fusion feature corresponding to each point can be used as the point feature of a second point.

[0117] In order to learn the more fine-grained semantic relevance between each first point and its neighborhood points, in the optional embodiments of the application, the relevance between each first point and the neighborhood points of the first point, and the fusion feature corresponding to the first point can be realized by at least one first attention module based on a channel attention mechanism. When there are at least two first attention modules, the fusion feature corresponding to each first point can be obtained by the following steps:

[0118] For each first point, the first feature of the first point is used as a query vector, and the first feature of each neighborhood point of the first point is used as a key vector, and the following operations are performed by each first attention module:

[0119] For each neighborhood point of the first point, a first attention weight between the first point and the neighborhood point is determined based on the first feature of the first point and the first feature of the neighborhood point, and the first attention weight is taken as the relevance;

[0120] The first features of the respective neighborhood points are weighted and fused by using the first attention weights corresponding to the respective neighborhood points of the first point to obtain a new feature corresponding to the first point, wherein the fused feature corresponding to each first point includes the new feature corresponding to each first attention module of the first point;

[0121] Correspondingly, obtaining the second feature representation of the plurality of second points based on the fused feature corresponding to each first point includes:

[0122] The new feature corresponding to each first attention module of each first point is taken as the point feature of a second point, and the three-dimensional coordinates of each second point are generated based on the point features of the plurality of second points.

[0123] For each first point and its neighborhood point, a first attention module based on a channel attention mechanism can learn the semantic association between the features of each dimension of the two points. For example, if the feature dimension of the first feature of each first point is d, then the attention weight between a first point and one of its neighborhood points is a d-dimensional weight vector. When weighting the first feature of the neighborhood point by using the attention weight, the weight value of each dimension in the d-dimensional weight vector is multiplied by the feature value of the corresponding dimension, that is, the corresponding elements are multiplied (Element-wise product). For example, the weight vector and the first feature of the neighborhood point are multiplied by corresponding elements, or the first feature of the neighborhood point is extracted, and the extracted feature and the weight vector are multiplied by corresponding elements to obtain the weighted feature. Then, the weighted features corresponding to each neighborhood point of the first point are fused to obtain the fused feature.

[0124] When multiple attention modules are used, different attention modules can learn the relevance of the first point and its neighborhood points at multiple different levels (different feature spaces). It can be understood that the model parameters of different attention modules are different, and each attention module can learn the relevance of the two points in the feature space corresponding to the attention module. By learning the relevance of each first point and its neighborhood points at multiple different levels, the fused feature corresponding to multiple levels of each first point can be obtained. The semantic information fusion between the features of adjacent points is realized, and the upsampling of the point cloud data is also realized, more feature representations of points are obtained, and thus a sparse but more complete point cloud structure is obtained.

[0125] Since the feature representation of the second point contains not only the coordinate data of the second point, but also the point features of the second point fused with the local detail information, the point cloud data can be completed based on the feature representations of multiple second points to obtain the target point cloud data of the target object containing more details and having a more realistic completion effect. It can be understood that the process of generating the target point cloud data based on the feature representations of multiple second points is an upsampling process, and through one or more upsampling processes, point cloud data of more points, that is, three-dimensional coordinates of more points, can be obtained.

[0126] The specific process of generating the target point cloud data based on the feature representations of multiple second points is not limited in the embodiments of the present application, and can be optionally implemented in the existing manner of completing point cloud based on point cloud features (the feature representations of multiple points in the embodiments of the present application). Since the feature representations of multiple points obtained in the method of the embodiments of the present application can provide more detailed features for point cloud data completion, even if the existing manner is used to generate the target point cloud data subsequently, the target point cloud data can better restore the details and overall structure of the target object, and can better meet the actual application requirements. In order to further improve the effect of the target point cloud data obtained finally, in the optional embodiments of the present application, the completed target point cloud data can be obtained by the following manner:

[0127] Based on the feature representations of multiple second points, at least one upsampling process is performed to obtain point features of multiple target points, and three-dimensional coordinates of the multiple target points are obtained according to the point features of the multiple target points, and the target point cloud data includes the three-dimensional coordinates of the multiple target points.

[0128] Optionally, the upsampling process can be implemented through a trained neural network, the feature representations of multiple second points containing more local detail information can be input into the trained neural network, one or more feature upsampling processes can be performed through the neural network, and the three-dimensional coordinates of the multiple target points can be predicted based on the upsampled features.

[0129] As an optional solution, the three-dimensional coordinates of the multiple target points can be obtained by the following manner:

[0130] Based on the three-dimensional coordinates of the multiple initial points and the three-dimensional coordinates of each second point, an initial point cloud input is determined, and the initial point cloud input includes three-dimensional coordinates of multiple input points.

[0131] The initial point cloud input is used as the point cloud input of the first upsampling process, and the upsampling process is repeatedly performed until a set number of times is reached, and the three-dimensional coordinates of the multiple output points obtained through the last upsampling process are determined as the three-dimensional coordinates of the multiple target points; wherein the upsampling process includes the following steps:

[0132] For each input point in the point cloud input, a distance between the input point and each second point is determined according to the three-dimensional coordinates of the input point and the three-dimensional coordinates of each second point, and a second number of neighborhood points of the input point are determined from the second points according to the distance;

[0133] For each input point, a point feature of the input point is obtained by fusing point features of each neighborhood point of the input point;

[0134] Based on the three-dimensional coordinates and the point features of each input point, point features of a plurality of output points are obtained through feature upsampling;

[0135] Based on the point features of the plurality of output points, three-dimensional coordinates of each output point are obtained, and the three-dimensional coordinates of each output point are taken as point cloud input for the next upsampling processing.

[0136] It can be seen that the input data for each upsampling processing includes a point cloud input and a feature representation (i.e., a local feature representation) of each second point, the feature representation including three-dimensional coordinates of points and point features, and the point cloud input including three-dimensional coordinates of a plurality of input points. The input data used for the first upsampling processing can be obtained based on the original point cloud data and the three-dimensional coordinates of each second point, and the point cloud input used for each upsampling processing except the first one is the three-dimensional coordinates of a plurality of points obtained after the last upsampling processing. It can be understood that the number of points in the point cloud input used for each upsampling processing is increasing according to the execution order of each upsampling processing. For example, the number of points in the point cloud input for the nth upsampling processing is N1, and the number of points in the point cloud input used for the n+1th upsampling processing is 2N1 if the upsampling rate is 2 when performing feature upsampling.

[0137] For the specific manner of determining the point cloud input used for the first upsampling processing based on the three-dimensional coordinates of the plurality of initial points and the three-dimensional coordinates of each second point, embodiments of the present application are not limited. Optionally, a preset number of input points can be selected from the plurality of initial points and the plurality of second points through a sampling manner, or the plurality of input points can be determined through an interpolation manner on the plurality of initial points and the plurality of second points. For example, a reference point can be randomly selected from the plurality of initial points and the plurality of second points, and then a plurality of points closest to the reference point are found from the remaining points, the three-dimensional coordinates of the plurality of points are averaged (or weighted summation is performed according to the distance from the reference point) to obtain the three-dimensional coordinates of an input point, then a new reference point is selected (for example, a point farthest from the last reference point is selected as the new reference point from the points other than the last reference point and the plurality of closest points corresponding to the last reference point), and the three-dimensional coordinates of another input point are obtained by repeating the above steps. The three-dimensional coordinates of a preset number of input points can be obtained by continuously repeating the above operations.

[0138] After determining the three-dimensional coordinates of each input point used in the first upsampling process, for each input point, a number of neighboring points, i.e., the above-mentioned neighborhood points, of the input point can be determined from a plurality of second points according to the distances between the input point and the second points, where the number of neighborhood points is k (where the second number and the first number can be the same or different and can be configured according to requirements). In the order of the distances from small to large, the k second points corresponding to the top k distances are determined as the neighborhood points of the input point. Since each neighborhood point is a neighboring point of the input point, the point feature of the input point is obtained by fusing the point features of the neighborhood points. Wherein, the specific way of fusing the point features of each neighborhood point is not limited by the embodiments of the present application, and can be, for example, averaging the point features of each neighborhood point (i.e., averaging the feature values at the same position in the point features of each neighborhood point) to obtain the point feature of the input point. As another optional solution, for each input point, the above-mentioned obtaining the point feature of the input point by fusing the point features of each neighborhood point of the input point can include:

[0139] determining the weight corresponding to each neighborhood point of the input point according to the distance between the input point and each neighborhood point of the input point, where the weight corresponding to a neighborhood point is negatively correlated with the distance corresponding to the neighborhood point;

[0140] weighting and fusing the point features of each neighborhood point of the input point according to the weight corresponding to each neighborhood point of the input point to obtain the point feature of the input point.

[0141] The scheme provided by the embodiments of the present application realizes the fusion of the features of a plurality of neighborhood points of an input point by feature interpolation. For each input point, the distances between the different neighborhood points of the input point and the input point are different. Since the farther apart two points are, the less likely they are to influence each other, the optional solution of the present application further considers the distance between each neighborhood point and the input point when fusing the point features of a plurality of neighborhood points of the input point, so that the features of neighborhood points closer to the input point can have a relatively greater influence on the point feature of the input point, thereby obtaining a more accurate point feature of the input point.

[0142] After the point features and the three-dimensional coordinates of the input points are determined, the point features of the output points can be obtained through feature upsampling based on the three-dimensional coordinates and the point features of the input points, wherein the number of the output points is determined by the number of the input points and the upsampling rate. After the point features of the output points are obtained, the three-dimensional coordinates of the output points can be predicted based on the point features, that is, the point cloud data used in the next upsampling process. If the current upsampling process is the last upsampling process, the three-dimensional coordinates of the output points obtained at this time are the three-dimensional coordinates of the target points.

[0143] Optionally, for each upsampling process, after the point features of the output points corresponding to the process are obtained, the three-dimensional coordinates of each point can be predicted through a neural network (such as a plurality of perceptrons), and the neural network can generate the three-dimensional coordinates of each point based on the association between the point features of the output points.

[0144] In an optional embodiment of the present application, for each upsampling process, the point features of the output points are obtained through feature upsampling based on the three-dimensional coordinates and the point features of the input points, which can include:

[0145] The point feature and the three-dimensional coordinates of each input point are concatenated respectively to obtain a third feature of each input point.

[0146] According to the three-dimensional coordinates of each input point, a third number of neighborhood points of each input point are determined from the input points.

[0147] For each input point, the point features of at least two output points corresponding to the input point are obtained through feature upsampling based on the third feature of the input point and the historical features of the neighborhood points of the input point.

[0148] For the first upsampling process, the historical feature of each neighborhood point is the third feature (i.e., the concatenation of the point feature and the three-dimensional coordinates) of the neighborhood point. For each upsampling process other than the first upsampling process, the historical feature of each neighborhood point is the point feature of the neighborhood point as an output point in the last upsampling process.

[0149] In the optional solution, for each upsampling processing, when calculating the point feature of the output point corresponding to each input point of the processing, the historical features of each neighborhood point of each input point, that is, the point features of each output point obtained in the last upsampling processing, are effectively utilized. Since each output point obtained in the last upsampling processing is each input point of the current upsampling processing, the historical feature of each input point is the feature information of the input point that has been generated. By using the optional solution, the feature information of each point that has been generated can be retained in the current upsampling processing, so that the point features of each output point obtained in the current upsampling processing contain richer feature information and have better feature expression capability, and the target point cloud data obtained finally can express the target object more detailedly and realistically. The specific value of the third quantity can be configured according to actual needs, and the value of the third quantity can be the same as or different from the values of the first quantity and the second quantity.

[0150] In the optional embodiment of the present application, for each input point, the point features of at least two output points corresponding to the input point are obtained through feature upsampling according to the third feature of the input point and the historical features of each neighborhood point of the input point, including:

[0151] Each of the at least two second attention modules respectively performs the following operations:

[0152] Second attention weights between the third feature of the input point and the historical features of each neighborhood point of the input point are respectively determined;

[0153] The historical features of each neighborhood point of the input point are weighted and fused by using the attention weights corresponding to the neighborhood points to obtain the point feature of one output point corresponding to the input point.

[0154] It can be understood that the model parameters of different second attention modules are different, and different attention modules can learn the semantic correlation, that is, the attention weight, of the input point and its neighborhood points in different feature spaces. The greater the weight value is, the greater the semantic correlation is. Then, based on the attention weights corresponding to each neighborhood point of the input point, the historical features of the corresponding neighborhood points are fused, so that the point feature of the output point is obtained, and the information of multiple neighborhood points of the output point is integrated into the point feature, and the size of the semantic correlation between each neighborhood point and the input point is also considered when the information is integrated.

[0155] Optionally, for each input point, when the second attention weight corresponding to each neighborhood point of the input point (i.e., the relevance between the input point and the neighborhood point) is adopted, the feature of the three-dimensional coordinates of each neighborhood point, i.e., the spatial coordinate feature, can be further considered. For example, for each neighborhood point, the relevance between the input point and the neighborhood point can be determined according to the third feature of the input point, the third feature of the neighborhood point, and the spatial coordinate feature of the neighborhood point. When the historical features of each neighborhood point of the input point are weighted and fused, the spatial coordinate feature of each neighborhood point can also be integrated. For example, for each neighborhood point of the input point, the historical feature of the neighborhood point can be extracted, and the extracted feature can be fused with the spatial coordinate feature of the neighborhood point to obtain a fused feature. When the historical features of each neighborhood point are fused by using the second attention weight corresponding to each neighborhood point, the fused features of each neighborhood point can be fused.

[0156] Optionally, the second attention module can be an attention module based on a channel attention mechanism, which can learn the relevance between each dimension of the third feature of the input point and the historical feature of the neighborhood point, i.e., a more fine-grained relevance between point pairs. The number of the second attention module is equal to the up-sampling rate of the up-sampling process.

[0157] In an optional embodiment of the present application, the point feature based on the plurality of output points to obtain the three-dimensional coordinates of each output point includes:

[0158] The point feature based on the plurality of output points is extracted to obtain the three-dimensional coordinate offset of each output point.

[0159] Based on the three-dimensional coordinates of each input point and the three-dimensional coordinate offset of each output point, the three-dimensional coordinates of each output point are obtained.

[0160] Optionally, for each up-sampling process, after obtaining the point feature of the plurality of output points corresponding to the process, the three-dimensional coordinate offset of each output point can be predicted based on the point feature of the output points. The three-dimensional coordinate offset of each output point can be predicted by a trained neural network structure, for example, a multi-layer perceptron. The three-dimensional coordinate offset of the output point is the three-dimensional coordinate of the output point relative to the three-dimensional coordinates of the input point in the point cloud input of the up-sampling process. The three-dimensional coordinates of the input point in the point cloud input can be copied by a corresponding multiple according to the up-sampling rate of the up-sampling process. The number of the copied input points is equal to the number of the output points. The three-dimensional coordinate offset of each output point is added to the three-dimensional coordinates of the corresponding input point after copying, and the three-dimensional coordinates of each output point are obtained.

[0161] Of course, as another alternative, a neural network structure capable of directly obtaining point features based on multiple output points and obtaining three-dimensional coordinates of each output point can also be obtained through training.

[0162] Optionally, the point cloud data processing method provided in the embodiments of the present application can be implemented by calling the trained point cloud completion network. The point cloud completion network is obtained by iteratively training a neural network model based on a training data set. The training data set includes multiple training samples. Each loss training sample includes a point cloud data pair. The point cloud data pair includes sample point cloud data to be completed and corresponding completed sample point cloud data.

[0163] The specific model architecture of the point cloud completion network is not limited in the embodiments of the present application and can be configured according to requirements. Optionally, the point cloud completion network can include an initial feature extraction network, a seed feature generation network, and a target data generation network. The initial feature extraction network is used to extract features of the original point cloud data to obtain feature representations of multiple first points. The feature representations of the multiple first points can be used as input data of the seed feature network. The seed feature network can perform feature upsampling based on the semantic association relationship between each first point and its neighborhood points and generate feature representations of multiple second points. The feature representations of the multiple second points can be used as input data of the target data generation network. The target generation network generates the final target point cloud data according to the feature representations of the multiple second points.

[0164] The specific calculation method of the target point cloud data is not limited in the embodiments of the present application. An end-to-end training method can be used, or one or more parts can be pre-trained and then the entire network can be trained. For example, the initial feature extraction network can be trained first to obtain a trained network. Then, the model parameters of the initial feature extraction network can be fixed, and the entire point cloud completion network can be iteratively trained in an end-to-end manner to obtain a trained point cloud completion network.

[0165] The method provided in the embodiments of the present application can be applied to any application scenario with point cloud data completion requirements, such as three-dimensional scene reconstruction scenarios, to realize repair and completion of three-dimensional scanning data. To better illustrate the scheme provided in the embodiments of the present application and the practicability of the method, the scheme will be described below in combination with a specific scene embodiment.

[0166] Figure 3 The structure schematic diagram of a point cloud data processing system to which the scene embodiment is applicable is shown in FIG. 1. Figure 3As shown, the point cloud data processing can include the collection device 10, the terminal device 20, the three-dimensional reconstruction server 30 and the training server 40. Among them, the training server 40 can be used to perform the training operation of the neural network involved in the embodiments of the present application, such as iterative training of the point cloud completion network to be trained based on the training data set, and after obtaining the trained point cloud completion network, the point cloud completion network can be deployed to the three-dimensional reconstruction server 30.

[0167] In the present embodiment, the collection device 10 is a point cloud data collection device, which is used to collect original point cloud data to be completed. The collection device can be a laser scanner. When three-dimensional reconstruction of a target object is needed, the target object surface can be scanned by the laser scanner to obtain the original point cloud data to be completed, that is, the incomplete point cloud data. The original point cloud data obtained by the collection device 10 can be sent to the three-dimensional reconstruction server 30 through the terminal device 20. The three-dimensional reconstruction server 30 can complete the original point cloud data based on the trained point cloud completion network by performing the method provided in the embodiments of the present application, obtain the completed target point cloud data, and send the target point cloud data to the terminal device 20. The terminal device 20 can render the three-dimensional structure of the target object based on the target point cloud data.

[0168] Figure 4 The structure of the point cloud completion network provided in the embodiments of the present application and the data processing original schematic diagram are shown. The following will be described in combination with Figure 3 and Figure 4 An optional implementation of the point cloud data processing method provided in the present application is described.

[0169] As Figure 4 shown, the overall structure of the point cloud completion network provided in the embodiments of the present application adopts an encoder-decoder structure. The data processing flow of the network can include the following three steps:

[0170] Step one: feature extraction is performed on the incomplete point cloud input.

[0171] This step can be implemented by the encoder of Figure 4 The specific model structure of the encoder is not limited in the embodiments of the present application. Optionally, a hierarchical point cloud feature extraction network (such as based on PointNet++ or pointtransformer) is used to extract features. The features extracted in this step correspond to the feature representation of the plurality of first points in the foregoing.

[0172] Step two: generate local feature representation.

[0173] This step can input the features extracted in step one into, for example, Figure 4In the seed generator shown (which can be a transformer-based generator), a sparse point cloud structure and a corresponding local feature representation (corresponding to the feature representation of the plurality of second points in the foregoing) are obtained

[0174] Step three: complete point cloud generation and detail recovery.

[0175] This step is based on the generated local feature representation, and a plurality of upsampling layers are used to complete the sparse point cloud, and a complete point cloud output is gradually obtained, i.e., based on the feature representation of the plurality of second points, the target point cloud data is obtained by performing multiple upsampling processing. Optionally, as shown in Figure 4 In each upsampling layer, a transformer-based generator can be used for feature conversion and feature point generation.

[0176] The implementation of each of the above steps will be described in detail below in conjunction with the point cloud completion network in Figure 4

[0177] Step one: feature extraction is performed on the incomplete point cloud input to obtain an initial point cloud feature representation, i.e., a plurality of first point features and three-dimensional coordinates.

[0178] As an example, assuming that the incomplete point cloud input P, i.e., the original point cloud data, includes 2000 three-dimensional coordinates, an encoder can be used to perform feature extraction on the incomplete point cloud input original point cloud data to obtain an initial point cloud feature representation, which includes incomplete point cloud coordinates and the feature of each point corresponding thereto wherein N p represents the number of first points, 3 represents the coordinate dimension of each first point, C p represents the feature dimension of each point, and F p represents the point feature of the first point.

[0179] Step two: generate a local feature representation.

[0180] In the embodiments of the present application, a local feature representation specially used for a point cloud completion task is proposed, which can provide more rich detail features for the point cloud completion task.

[0181] In this step, the incomplete point cloud coordinates and the feature of each point corresponding thereto are used to generate a sparse but complete point cloud structure by a seed generator. As shown in Figure 4 ​The seed generator can be a transformer-based generator, including a transformer structure for implementing up-sampling and a shared multi-layer perception (Shared MLP), as shown in the network structure of part (a) in FIG. 7. In the scheme provided by the embodiments of the present application, the transformer structure in natural language processing is extended to the point cloud generation task, and a transformer-based generator is designed. The generator can process and up-sample the input point cloud by combining the semantic information in the local neighborhood, and finally obtain the newly generated point coordinates. The generator can be used in the process of steps two and three.

[0182] The process of obtaining the local feature representation (i.e., the point features and three-dimensional coordinates of the second points) using the transformer-based generator will be described below in conjunction with an example.

[0183] In step two, the input of the seed generator is the initial point cloud feature representation extracted in step one, i.e., the incomplete point cloud coordinates and the features of each point Assuming that the original point cloud data includes three-dimensional coordinates of 2000 initial points, after extraction in step one, N p = 500 first point coordinates and the features of each point Assuming that the up-sampling rate of the transformer-based seed generator is 2, the incomplete point cloud coordinates and the features of each point are input into the seed generator, and N s = 1000 second point coordinates and the point features corresponding to each point may be generated.

[0184] Step 1: The input data of N p *(C p + 3) (the features after splicing the point features and coordinates of each first point, i.e., the first features of each first point) can be obtained after feature extraction (such as after MLP processing), which is the query vector of each first point q i represents the query vector of the i-th first point, and in each first point, the key vector key can be equal to the query, i.e. k i = q i .

[0185] Step 2: Based on the three-dimensional coordinates of each first point, determine the neighborhood points of each first point, assuming the number of neighborhood points is 4, taking the ith first point as an example, the distance between the three-dimensional coordinates of this point and the three-dimensional coordinates of each first point can be calculated, and the nearest 4 first points are taken as the neighborhood points of this point. After finding the nearest 4 neighborhood points, for each neighborhood point in these 4 points, taking the jth point as an example, based on q i and k j of this point, the channel-wise self-attention is used to calculate the weight a between the ith and jth points, assuming the up-sampling multiple is 2, then two different attention weight parameters (i.e. attention mechanism with different model parameters) are used to calculate two weights, denoted as a1 and a2, then a1 and a2 can be normalized to obtain two normalized attention weights a1 and a2 corresponding to the jth point. Similarly, the two weights corresponding to the other three neighborhood points can be calculated.

[0186] Step 3: Optionally, each q i in the query can be extracted to obtain the value vector v i corresponding to each first point (or q i is directly taken as v i , or q i and k i are concatenated and then extracted to obtain v i ), for each first point, a1 corresponding to the 4 neighborhood points of this point can be weighted with the value vectors of the 4 neighborhood points, and the sum of the weighted 4 value vectors is obtained to obtain a new feature (i.e. fusion feature) corresponding to the first point. Similarly, a2 corresponding to the 4 neighborhood points of this point can be weighted with the value vectors of the 4 neighborhood points, and the sum of the weighted 4 value vectors is obtained to obtain another new feature (i.e. fusion feature) corresponding to the first point, thereby realizing feature fusion and feature up-sampling based on the semantic association between points. Each first point corresponds to two new features, therefore, based on the features and coordinates of the 500 first points, the new features of 1000 points can be obtained, i.e. After that, the Shared MLP can obtain the three-dimensional coordinates corresponding to the 100 points. Optionally, the three-dimensional coordinates can be the three-dimensional coordinate offset corresponding to the 1000 points, which can be obtained by copying the coordinates of the N p = 500 points of the input part of the seed generator, and then copying the three-dimensional coordinates of the 1000 points and N s= 1000 points of three-dimensional coordinate offset addition, get the final 1000 points of coordinates The 1000 points are the second points in the foregoing, The 1000 points are the second points in the foregoing, The 1000 points are the second points in the foregoing,

[0187] At this point, the output of the seed generator is obtained, that is And That is Figure 4 Patch Seeds in the local feature representation, wherein N s The 1000 points represent the second points in the foregoing, The 1000 points are the second points in the foregoing, The 1000 points are the second points in the foregoing,

[0188] As can be seen from the generation process of The local feature representation generated by the scheme provided in the embodiment of the application can retain more details of the local area, that is, the feature corresponding to each point in S covers the range of its neighborhood, and can provide effective regional information for subsequent point cloud generation.

[0189] Step three: complete point cloud generation and detail recovery, that is, generation of target point cloud data.

[0190] In the embodiment, the generation of the target point cloud data can be realized by multiple upsampling processes, such as the multiple upsampling layers shown in Figure 4 Each upsampling layer realizes one upsampling process. Figure 4 In the structure shown in l The input point cloud of the l-th upsampling layer (the point cloud input in the foregoing) is represented by P

[0191] The generation of the input point cloud of the first layer of the upsampling layer (that is, the initial point cloud input) will be introduced below.

[0192] Optionally, based on the coordinates (assuming N s = 1000) and the features output by the seed generator and the original point cloud data (assuming 2000 points of three-dimensional coordinates, represented by ), the coordinates of the 1000 points and ​The coordinates of these 2000 points are fused (the fusion method is not limited) to obtain the point cloud input corresponding to the first upsampling layer. N0 can be configured with specific values ​​as needed; it represents the number of input points in the first upsampling layer. In step three, p... i This represents the three-dimensional coordinates of the i-th input point.

[0193] After obtaining the point cloud input of the first upsampling layer Subsequently, based on this point cloud input and the local feature representation obtained in step two, and The first upsampling layer performs a first upsampling process to obtain the point cloud input for the next upsampling process. Based on this new point cloud input and local feature representations, a second upsampling layer performs a second upsampling process to obtain the point cloud input for the next upsampling process. This process is repeated until the point cloud input output from the last upsampling layer is obtained; this point cloud data is the target point cloud data. Figure 4 As shown in part (b) of the network structure, the input point cloud P is based on the upsampling layer of layer l-1. l And local feature representations (Patch Seeds), through this upsampling layer, the input point cloud P of the l-th upsampling layer can be obtained. l+1 The following section uses layer l-1 as an example to introduce the upsampling process for this layer.

[0194] First, for a point cloud input N is the number of points, which can be processed by seed feature interpolation. Figure 4 The interpolation seed features in the model extend the local feature representation F, i.e., the patch seeds, to each point in the input point cloud. Specifically, for the point cloud input... And the local feature representation F can be propagated to every point using interpolation. For each point p i (The 3D coordinates of the i-th point in the point cloud input) can first be determined based on point p. i Calculate point p using the 3D coordinates of each point in S (the points output by the seed generator). i The distances between points in S and point p are found in S. i Let point p be the k nearest points. i The neighborhood N(i) = {x1,…,x k}∈S, meaning the number of neighboring points is k, then we can use point p i The interpolated feature s is calculated using the following expression to determine the distances between each neighboring point. i (point p) i Point features):

[0195] wherein,

[0196] In the above expression, d ij represents the distance between the point p i and the neighborhood point x j . By this method, the local feature can be extended to each point in the point cloud input, and the corresponding s is obtained. i is the feature of the input point with three-dimensional coordinates p i .

[0197] For the l-th up-sampling layer, the point cloud input of this layer is i.e. the three-dimensional coordinates of N l input points, through the above feature interpolation processing, the interpolated local feature corresponding to this layer can be obtained. After that, based on the point cloud input and the local feature Figure 5 , a transformer-based generator is used to generate P l+1 .

[0198] Step 1: The input of the generator is the point cloud input of the l-1-th layer and the local feature obtained by interpolation First, the two features are spliced, and the spliced feature (Seed features in Figure 4 ) is processed by an MLP to obtain new query feature (as the query of the transformer, Figure 5 Q in , represents the feature extracted from the spliced feature (i.e. the third feature) of the i-th input point.

[0199] Step 2: In order to retain the feature information of the generated points, the output feature of the previous layer (i.e. the historical feature, corresponding to the Skip connections in Figure 4 ) is used as the key feature of the transformer ( Figure 5 K in Figure 4 ), as shown in , the output feature N l+1 ×C of the l-1-th layer will be used as the key of the l-th layer. If there is no such existing feature (the first layer up-sampling has no output feature of the previous layer), then

[0200] Step 3: Calculate the distance between input points according to their three-dimensional coordinates, and determine the neighborhood of each input point, i.e., multiple neighborhood points, according to the distance. The neighborhood N(i) of a given point p i (i.e., S neighborhood points), for each neighborhood point j, channel-wise self-attention can be applied to calculate the weight of each dimension between each pair of points (p i and neighborhood point j), and the calculation principle is as follows:

[0201]

[0202] This formula corresponds to the relationship calculation in Figure 5 , the fusion of δ, and the processing steps of multi-layer perception MLP, where α m , β, γ are feature transformation functions (MLPs on the left side of Figure 5 , model parameters), used to calculate the weight vector δ j is the spatial coordinate encoding of neighborhood point j, i.e., the positional encoding. For a given upsampling rate r l , the generator will input the point cloud and upsample it by r l times, i.e., r l N l = N l+1 . Correspondingly, m = 1, 2,..., r l indicates the process of each upsampling, which is used to generate a new set of points, for example, r l = 2, m takes values of 1 and 2, and are weight vectors (i.e., attention scores for each dimension) calculated by different feature transformation functions.

[0203] Step 4: Normalize the generated weights using the softmax function:

[0204]

[0205] Step 5: Multiply the weights with the corresponding Value features (i.e., value features / value vectors) to obtain the final generated new point features, i.e., fuse the features of each neighborhood point using the corresponding weights of each neighborhood point. Optionally, the new point features can be calculated by the following expression:

[0206]

[0207] where Value features It can be obtained by concatenating the key and query of the neighboring point j and passing them through an MLP layer. φ is also a feature transformation function (model parameter), which can be used to further extract features from the concatenated features. * indicates element-wise product.

[0208] Through the weighted fusion described above, for each input point in the point cloud input, the point features of m output points corresponding to that input point can be obtained, where m is equal to the upsampling rate of the sampling layer.

[0209] Step 6: Combine all the features obtained in Step 5 to obtain the output feature H. l ={h im |i=1,…,N l m = 1, ..., r l},Right now Figure 4 N in l+1 ×C. Output feature H l After passing through a Shared MLP, the coordinates of the corresponding generated points can be obtained. Specifically, the MLP can be used to obtain the 3D coordinate offset ΔP of each output point. l , will △P l And the copied input P l By fusion, P is obtained. l+1 .

[0210] At this point, the upsampling process of layer l-1 is complete, and the obtained P l+1 As the point cloud input for the upsampling layer of layer l.

[0211] It should be noted that, in practical applications, the implementation of step three above can be achieved using the transformer-based generator proposed in the embodiments of this application, or it can be achieved using other common backbone networks, such as graph convolution, PointNet++, etc.

[0212] Furthermore, for those skilled in the art, the above-described embodiments of this application are... Figure 4 and Figure 5 The generator structure shown is not unique and immutable; it can be modified according to application needs. However, for those skilled in the art, other network structures with the same data processing principles as the methods provided in the embodiments of this application also fall within the protection scope of this application. For example, Figure 4 The number of sampling layers shown can be set according to requirements, and other feature extraction networks can also be used for the MLP used for further feature extraction.

[0213] The point cloud completion method based on local feature representation provided in the embodiments of the present application can achieve better detail recovery and feature preservation effects, and is beneficial to semantic understanding of the point cloud. The point cloud completion scheme in the embodiments of the present application has at least the following advantages and innovations:

[0214] 1) Details and important structures in the missing point cloud can be effectively recovered, the missing part of the object can be completed, the complete point cloud data with consistent semantics can be recovered, the structure and details of the object itself can be more truly restored, and a more realistic completion result can be obtained.

[0215] 2) The point cloud completion network provided in the embodiments of the present application can be extended to scanning data of a real scene, has good generalization performance, realizes general three-dimensional point cloud completion, and can be extended to real scanning data and data of unknown object categories. For sample categories that do not exist in network training, there is still good effect.

[0216] 3) The transformer structure in natural language processing is extended to three-dimensional point cloud generation operation, and a generator based on transformer is designed.

[0217] 4) A point cloud completion network based on local feature representation is designed. By refining the previous global feature into a localized local / region feature, local details can be preserved, and not only the information of the point itself but also the semantic relationship between adjacent points can be considered in the completion process, so that a more realistic completion effect can be achieved.

[0218] In order to test the effect of the scheme provided in the embodiments of the present application, the method provided in the embodiments of the present application and the existing point cloud completion scheme are tested on sample data, Figure 6 and Figure 7 The completion effect comparison diagram of the present application scheme and the prior art is shown, wherein the input is the object structure corresponding to the incomplete point cloud data, the baseline is the object structure of the completed point cloud data obtained by using the existing scheme, and the true value is the complete point cloud data corresponding to the input in the sample, Figure 6 and Figure 7 It can be seen that, compared with the prior art, the completion effect obtained by using the point cloud completion scheme provided in the embodiments of the present application is better and more realistic, and the detail information of the object is richer.

[0219] Corresponding to the point cloud data processing method provided in the embodiments of the present application, the point cloud data processing apparatus provided in the embodiments of the present application is also provided. Optionally, the point cloud data processing apparatus 100 can be any electronic device, such as a server, as Figure 8 shown, the point cloud data processing apparatus 100 can include a to-be-completed data acquisition module 110 and a data completion module 120. Wherein,

[0220] The data completion data acquisition module 110 is configured to acquire original point cloud data to be completed, wherein the original point cloud data comprises three-dimensional coordinates of a plurality of initial points.

[0221] The data completion module 120 is configured to obtain target point cloud data after completion by performing a data completion operation, wherein the data completion operation comprises the following steps:

[0222] Feature extraction is performed on the original point cloud data to obtain feature representations of a plurality of first points, wherein each feature representation of the first points comprises three-dimensional coordinates and point features of the first point.

[0223] According to the three-dimensional coordinates of each first point, a first number of neighborhood points of each first point are determined from the plurality of first points.

[0224] For each first point, the correlation between the first point and each neighborhood point of the first point is determined according to the feature representation of the first point and the feature representation of each neighborhood point of the first point, and the feature representation of each neighborhood point of the first point is fused according to the corresponding correlation of each neighborhood point of the first point to obtain the fusion feature corresponding to the first point.

[0225] Based on the fusion feature of each first point, a second feature representation of a plurality of second points is obtained.

[0226] Based on the feature representation of the plurality of second points, the target point cloud data after completion is generated.

[0227] Optionally, the data completion module can obtain the fusion feature corresponding to each first point by performing the following operations:

[0228] For each first point, the point feature and the three-dimensional coordinates of the first point are spliced to obtain the first feature of the first point.

[0229] For each neighborhood point of each first point, the correlation between the first point and the neighborhood point is determined according to the first feature of the first point and the first feature of the neighborhood point.

[0230] For each first point, the correlation corresponding to each neighborhood point of the first point is taken as a weight to perform weighted fusion on the first feature of each neighborhood point of the first point to obtain the fusion feature corresponding to the first point.

[0231] Optionally, the data completion module can obtain the correlation between each first point and the neighborhood point of the first point and the fusion feature corresponding to the first point by adopting at least one first attention module based on a channel attention mechanism; wherein when the first attention module has at least two, the fusion feature corresponding to each first point can be obtained by the following way:

[0232] For each first point, taking the first feature of the first point as a query vector, and taking the first feature of each neighbor point of the first point as a key vector respectively, the following operations are performed by each first attention module respectively:

[0233] For each neighbor point of the first point, based on the first feature of the first point and the first feature of the neighbor point, determining a first attention weight between the first point and the neighbor point, and taking the first attention weight as the relevance;

[0234] Using the first attention weight corresponding to each neighbor point of the first point, the first feature of each corresponding neighbor point is weighted and fused to obtain a new feature corresponding to the first point;

[0235] The fusion feature corresponding to each first point includes the new feature corresponding to each first attention module of the first point.

[0236] Correspondingly, when the data completion module obtains the second feature representation of a plurality of second points based on the fusion feature corresponding to each first point, it can be used for:

[0237] Taking the new feature corresponding to each first attention module of each first point as a point feature of a second point respectively, and based on the point features of a plurality of second points, generating a three-dimensional coordinate of each second point, wherein the feature representation of each second point includes the three-dimensional coordinate and the point feature of the second point.

[0238] Optionally, the data completion module is further used for: for each first point, generating a spatial coordinate feature of the first point according to the three-dimensional coordinate of the first point.

[0239] Correspondingly, for each neighbor point of each first point, when the data completion module determines the relevance between the first point and the neighbor point based on the first feature of the first point and the first feature of the neighbor point, and obtains the fusion feature corresponding to the first point, it can be used for:

[0240] Determining the relevance between the first point and the neighbor point according to the first feature of the first point, the first feature of the neighbor point, and the spatial coordinate feature of the neighbor point.

[0241] For each neighbor point of the first point, performing feature extraction based on the first feature and the spatial coordinate feature of the neighbor point to obtain a second feature of the neighbor point.

[0242] Using the weight corresponding to each neighbor point of the first point, the second feature of each neighbor point of the first point is weighted and fused to obtain the fusion feature corresponding to the first point.

[0243] Optionally, the data completion module, when generating the completed target point cloud data based on the feature representations of the plurality of second points, can be configured to:

[0244] based on the feature representations of the plurality of second points, obtain point features of a plurality of target points by performing at least one upsampling process, and obtain three-dimensional coordinates of the plurality of target points according to the point features of the plurality of target points, wherein the target point cloud data comprises the three-dimensional coordinates of the plurality of target points.

[0245] Optionally, the data completion module, when generating the target point cloud data, can be configured to:

[0246] based on the three-dimensional coordinates of the plurality of initial points and the three-dimensional coordinates of each second point, determine an initial point cloud input, wherein the initial point cloud input comprises three-dimensional coordinates of a plurality of input points;

[0247] use the initial point cloud input as the point cloud input of the first upsampling process, repeatedly perform the upsampling process until a set number of times is reached, and determine the three-dimensional coordinates of a plurality of output points obtained by the last upsampling process as the three-dimensional coordinates of the plurality of target points;

[0248] wherein the upsampling process comprises the following steps:

[0249] for each of the input points in the point cloud input, determine the distance between the input point and each second point according to the three-dimensional coordinates of the input point and the three-dimensional coordinates of each second point, and determine the second number of neighbor points of the input point from each second point according to the distance;

[0250] for each of the input points, obtain the point feature of the input point by fusing the point features of each neighbor point of the input point;

[0251] based on the three-dimensional coordinates and the point features of each of the input points, obtain the point features of a plurality of output points by feature upsampling;

[0252] based on the point features of the plurality of output points, obtain the three-dimensional coordinates of each output point, and use the three-dimensional coordinates of each output point as the point cloud input of the next upsampling process.

[0253] Optionally, for each of the input points, the data completion module, when obtaining the point feature of the input point by fusing the point features of each neighbor point of the input point, can be configured to:

[0254] determine the weight corresponding to each neighbor point of the input point according to the distance between the input point and each neighbor point of the input point, wherein the weight corresponding to each neighbor point is negatively correlated with the distance corresponding to the neighbor point;

[0255] The point features of the neighbor points of the input point are fused by weighting according to the weights corresponding to the neighbor points of the input point, to obtain the point feature of the input point.

[0256] Optionally, for each upsampling processing, when the data completion module obtains the point features of the plurality of output points by feature upsampling based on the three-dimensional coordinates and the point features of the input points, the data completion module can be configured to:

[0257] The point feature and the three-dimensional coordinates of each input point are spliced respectively to obtain a third feature of each input point.

[0258] The third number of neighbor points of each input point are determined from the input points according to the three-dimensional coordinates of the input points.

[0259] For each input point, the point features of at least two output points corresponding to the input point are obtained by feature upsampling based on the third feature of the input point and the historical features of the neighbor points of the input point.

[0260] For the first upsampling processing, the historical feature of a neighbor point is the third feature of the neighbor point, and for each upsampling processing other than the first upsampling processing, the historical feature of a neighbor point is the point feature of the neighbor point as an output point in the last upsampling processing.

[0261] Optionally, for each input point, when the data completion module obtains the point features of at least two output points corresponding to the input point by feature upsampling based on the third feature of the input point and the historical features of the neighbor points of the input point, the data completion module can be configured to:

[0262] Each of the at least two second attention modules is configured to perform the following operations respectively:

[0263] Second attention weights between the third feature of the input point and the historical feature of each neighbor point of the input point are determined respectively.

[0264] The historical features of the neighbor points of the input point are fused by weighting according to the second attention weights corresponding to the neighbor points of the input point, to obtain the point feature of an output point corresponding to the input point.

[0265] Optionally, when the data completion module obtains the three-dimensional coordinates of the output points based on the point features of the plurality of output points, the data completion module can be configured to:

[0266] Feature extraction is performed based on the point features of the plurality of output points to obtain three-dimensional coordinate offsets of the output points.

[0267] The three-dimensional coordinates of the output points are obtained based on the three-dimensional coordinates of the input points and the three-dimensional coordinate offsets of the output points.

[0268] Optionally, the data completion module is configured to obtain the completed target point cloud data based on the original point cloud data by invoking the trained point cloud completion network, the point cloud completion network is obtained by iteratively training a neural network model based on a training data set, the training data set includes a plurality of training samples, each loss training sample includes a point cloud data pair, and the point cloud data pair includes sample point cloud data to be completed and corresponding completed sample point cloud data.

[0269] It can be understood that the apparatus provided in the embodiments of the present application can perform the method provided in the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the apparatus of the embodiments of the present application correspond to the steps in the method of the embodiments of the present application. For the detailed function description of each module of the apparatus, refer to the description of the corresponding method in the foregoing description, which will not be repeated here.

[0270] In the embodiments of the present application, an electronic device is provided, which includes a memory, a processor and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any optional embodiment of the present application can be implemented.

[0271] Figure 9 The structure of an electronic device to which the embodiments of the present application are applied is shown in a structural schematic diagram, as shown in Figure 9 The electronic device can be a server or a user terminal, and the electronic device can be used to implement the method provided in any embodiment of the present application.

[0272] As shown in Figure 9 The electronic device 2000 can mainly include at least one processor 2001 (one is shown in Figure 9 The electronic device 2000 can mainly include at least one processor 2001 (one is shown in Figure 9 The structure of the electronic device 2000 shown in

[0273] The memory 2002 can be used to store an operating system and application programs, etc. The application programs can include a computer program which, when invoked by the processor 2001, implements the method shown in the embodiments of the present application, and can also include programs for implementing other functions or services. The memory 2002 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and computer programs, and can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disc storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto.

[0274] The processor 2001 is connected to the memory 2002 through the bus 2005, and implements corresponding functions by invoking the application programs stored in the memory 2002. The processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof, which can implement or execute various exemplary logical blocks, modules and circuits described in connection with the present disclosure. The processor 2001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0275] The electronic device 2000 can be connected to a network through the communication module 2003 (which can include, but is not limited to, a network interface and the like) to communicate with other devices (such as user terminals or servers) through the network to exchange data, such as sending data to other devices or receiving data from other devices. Among them, the communication module 2003 can include a wired network interface and / or a wireless network interface, that is, the communication module can include at least one of a wired communication module or a wireless communication module.

[0276] The electronic device 2000 can be connected to the required input / output devices such as a keyboard, a display device, etc. through the input / output interface 2004, and the electronic device 2000 itself can have a display device, and other display devices can also be connected through the interface 2004. Optionally, the storage device such as a hard disk can also be connected through the interface 2004, so that the data in the electronic device 2000 can be stored in the storage device, or the data in the storage device can be read, and the data in the storage device can also be stored in the storage 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. According to different actual application scenarios, the devices connected with the input / output interface 2004 can be a component of the electronic device 2000, or an external device connected with the electronic device 2000 when needed.

[0277] The bus 2005 for connecting various components can include a channel for transmitting information between the above components. The bus 2005 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 2005 can be divided into an address bus, a data bus, a control bus, etc.

[0278] Optionally, for the scheme provided by the embodiment of the application, the storage 2002 can be used to store a computer program for executing the scheme of the application, and the processor 2001 runs the computer program to realize the actions of the method or device provided by the embodiment of the application.

[0279] Based on the same principle as the method provided by the embodiment of the application, the embodiment of the application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the corresponding content of the foregoing method embodiment.

[0280] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program can realize the corresponding content of the foregoing method embodiment when executed by a processor.

[0281] It should be noted that the terms "first", "second", "third", "fourth", "1", "2", and the like (if any) in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described.

[0282] It should be understood that, although the flowcharts of the embodiments of the present application indicate the respective operation steps by arrows, the implementation order of the steps is not limited to the order indicated by the arrows. Unless otherwise specified herein, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders as required. In addition, part or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on the actual implementation scenario. Part or all of these sub-steps or stages can be executed at the same time, and each of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of the present application do not limit this.

[0283] The above is only an optional implementation manner of some implementation scenarios of the present application, and it should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation manners based on the technical idea of the present application without departing from the technical concept of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. A point cloud data processing method, characterized by, The method comprises: obtaining original point cloud data to be completed, the original point cloud data comprising three-dimensional coordinates of a plurality of initial points; performing feature extraction on the original point cloud data to obtain feature representations of a plurality of first points, each feature representation of the first points comprising three-dimensional coordinates and point features of the first point; determining, according to the three-dimensional coordinates of each first point, a first number of neighborhood points of each first point from the plurality of first points; for each first point, concatenating the point features and the three-dimensional coordinates of the first point to obtain a first feature of the first point; for each neighborhood point of each first point, determining, according to the first feature of the first point and the first feature of the neighborhood point, a relevance between the first point and the neighborhood point; for each first point, weighting and fusing the first features of the neighborhood points of the first point by taking the relevance of each neighborhood point of the first point as a weight to obtain a fused feature corresponding to the first point; obtaining second feature representations of a plurality of second points based on the fused features corresponding to each first point; generating target point cloud data after completion based on the feature representations of the plurality of second points.

2. The method of claim 1, wherein, The relevance between each first point and the neighborhood points of the first point and the fused feature corresponding to the first point are realized by at least one first attention module based on a channel attention mechanism; wherein, when there are at least two first attention modules, the fused feature corresponding to each first point is obtained by the following method: for each first point, taking the first feature of the first point as a query vector and taking the first features of each neighborhood point of the first point as key vectors, respectively, and performing the following operations by each first attention module: for each neighborhood point of the first point, determining a first attention weight between the first point and the neighborhood point based on the first feature of the first point and the first feature of the neighborhood point, and taking the first attention weight as the relevance; weighting and fusing the first features of the respective neighborhood points by taking the first attention weights of the respective neighborhood points as weights to obtain a new feature corresponding to the first point; wherein, the fused feature corresponding to each first point comprises the new features corresponding to each first attention module of the first point; obtaining second feature representations of a plurality of second points based on the fused features corresponding to each first point comprises: taking the new features corresponding to each first attention module of each first point as point features of a second point, respectively, and generating three-dimensional coordinates of each second point based on the point features of the plurality of second points, wherein the feature representation of each second point comprises three-dimensional coordinates and point features of the second point.

3. The method according to claim 1 or 2, characterized in that, It further comprises: for each first point, generating a spatial coordinate feature of the first point according to the three-dimensional coordinates of the first point; for each neighborhood point of each first point, determining the relevance between the first point and the neighborhood point according to the first feature of the first point and the first feature of the neighborhood point comprises: determining the relevance between the first point and the neighborhood point according to the first feature of the first point, the first feature of the neighborhood point and the spatial coordinate feature of the neighborhood point. For each first point, the correlation of each neighborhood point of the first point is taken as a weight, and first features of each neighborhood point of the first point are fused by weighting to obtain a fused feature corresponding to the first point, including: For each neighborhood point of the first point, a second feature of the neighborhood point is obtained by performing feature extraction based on the first feature and the spatial coordinate feature of the neighborhood point; The second features of each neighborhood point of the first point are fused by weighting with the weights corresponding to the neighborhood points to obtain the fused feature corresponding to the first point.

4. The method of claim 1, wherein, The target point cloud data after completion is generated based on the feature representation of the plurality of second points, including: Based on the feature representation of the plurality of second points, a plurality of point features of the plurality of target points are obtained by performing at least one upsampling processing, and three-dimensional coordinates of the plurality of target points are obtained according to the plurality of point features of the plurality of target points, and the target point cloud data includes the three-dimensional coordinates of the plurality of target points.

5. The method of claim 4, wherein, The three-dimensional coordinates of the plurality of target points are obtained by the following way: Based on the three-dimensional coordinates of the plurality of initial points and the three-dimensional coordinates of each second point, an initial point cloud input is determined, and the initial point cloud input includes three-dimensional coordinates of a plurality of input points; The initial point cloud input is taken as a point cloud input of a first time upsampling processing, and the upsampling processing is repeatedly executed until a set number of times is reached, and the three-dimensional coordinates of a plurality of output points obtained by the last time upsampling processing are determined as the three-dimensional coordinates of the plurality of target points; The upsampling processing includes the following steps: For each input point in the point cloud input, distances between the input point and each second point are determined according to the three-dimensional coordinates of the input point and the three-dimensional coordinates of each second point, and a second number of neighborhood points of the input point are determined from each second point according to the distances; For each input point, a point feature of the input point is obtained by fusing point features of each neighborhood point of the input point; Based on the three-dimensional coordinates and the point features of each input point, point features of a plurality of output points are obtained by feature upsampling; Based on the point features of the plurality of output points, three-dimensional coordinates of each output point are obtained, and the three-dimensional coordinates of each output point are taken as a point cloud input of a next time upsampling processing.

6. The method of claim 5, wherein, For each input point, the point feature of the input point is obtained by fusing the point features of each neighborhood point of the input point, including: According to distances between the input point and each neighborhood point of the input point, a weight corresponding to each neighborhood point of the input point is determined, and the weight corresponding to each neighborhood point is negatively correlated with a distance corresponding to the neighborhood point; According to the weights corresponding to each neighborhood point of the input point, the point features of each neighborhood point of the input point are fused by weighting to obtain the point feature of the input point.

7. The method of claim 5, wherein, For each time upsampling processing, the point features of the plurality of output points are obtained by feature upsampling based on the three-dimensional coordinates and the point features of each input point, including: Third features of each input point are obtained by concatenating the point feature and the three-dimensional coordinates of each input point respectively; A third number of neighborhood points of each input point are determined from each input point according to the three-dimensional coordinates of each input point; For each input point, a point feature of at least two output points corresponding to the input point is obtained through feature upsampling according to the third feature of the input point and historical features of each neighbor point of the input point; wherein, for the first upsampling processing, the historical feature of each neighbor point is the third feature of the neighbor point, and for each upsampling processing except the first upsampling processing, the historical feature of each neighbor point is the point feature of the neighbor point as an output point in the last upsampling processing.

8. The method of claim 7, wherein, For each input point, the point feature of at least two output points corresponding to the input point is obtained through feature upsampling according to the third feature of the input point and historical features of each neighbor point of the input point, comprising: Each of the at least two second attention modules respectively performs the following operations: Second attention weights between the third feature of the input point and the historical feature of each neighbor point of the input point are respectively determined; The historical features of each neighbor point of the input point are weighted and fused by using the second attention weights corresponding to each neighbor point of the input point to obtain the point feature of one output point corresponding to the input point.

9. The method of claim 5, wherein, The point feature of each output point is obtained based on the point features of the plurality of output points, comprising: Feature extraction is performed based on the point features of the plurality of output points to obtain a three-dimensional coordinate offset of each output point; The three-dimensional coordinates of each output point are obtained based on the three-dimensional coordinates of each input point and the three-dimensional coordinate offset of each output point.

10. The method according to any one of claims 1 to 2 and 4 to 8, characterized in that, The method is implemented by calling a trained point cloud completion network, the point cloud completion network is obtained by iteratively training a neural network model based on a training data set, and the training data set includes a plurality of training samples, each training sample includes a point cloud data pair, and each point cloud data pair includes sample point cloud data to be completed and corresponding completed sample point cloud data.

11. A point cloud data processing apparatus, characterized by comprising: The device comprises: A to-be-completed data acquisition module configured to acquire original point cloud data to be completed, the original point cloud data comprising three-dimensional coordinates of a plurality of initial points; A data completion module configured to obtain target point cloud data completed by performing a data completion operation, the data completion operation comprising the following steps: Feature extraction is performed on the original point cloud data to obtain feature representations of a plurality of first points, each feature representation of a first point comprising three-dimensional coordinates and a point feature of the first point; A first number of neighbor points of each first point are determined from the plurality of first points according to the three-dimensional coordinates of each first point; For each first point, the point feature and the three-dimensional coordinates of the first point are spliced to obtain a first feature of the first point, and for each neighbor point of each first point, the relevance between the first point and the neighbor point is determined according to the first feature of the first point and the first feature of the neighbor point, and for each first point, the relevance corresponding to each neighbor point of the first point is used as a weight to weightedly fuse the first features of each neighbor point of the first point to obtain a fused feature corresponding to the first point; wherein, the relevance represents the degree of association between the first point and the neighbor point of the first point. The second feature representation of each second point is obtained based on the fusion feature of each first point. The target point cloud data after completion is generated based on the feature representation of the plurality of second points.

12. The apparatus of claim 11, wherein, The correlation between each first point and the neighborhood point of the first point, and the fusion feature corresponding to the first point, are realized by at least one first attention module based on a channel attention mechanism. When there are at least two first attention modules, the fusion feature corresponding to each first point is obtained by the following method: For each first point, the first feature of the first point is taken as a query vector, and the first feature of each neighborhood point of the first point is taken as a key vector, and the following operations are performed by each first attention module: For each neighborhood point of the first point, the first attention weight between the first point and the neighborhood point is determined based on the first feature of the first point and the first feature of the neighborhood point, and the first attention weight is taken as the correlation; The first features of the respective neighborhood points of the first point are weighted and fused by using the first attention weights corresponding to the respective neighborhood points, to obtain the new feature corresponding to the first point. The fusion feature corresponding to each first point includes the new feature corresponding to the first point for each first attention module. When the second feature representation of each second point is obtained based on the fusion feature corresponding to each first point, the data completion module is specifically used for: The new feature of each first point corresponding to each first attention module is taken as the point feature of each second point, and the three-dimensional coordinates of each second point are generated based on the point features of the plurality of second points, wherein the feature representation of each second point includes the three-dimensional coordinates and the point feature of the second point.

13. The apparatus of claim 11 or 12, wherein, The data completion module is further used for: For each first point, the spatial coordinate feature of the first point is generated according to the three-dimensional coordinates of the first point. For each neighborhood point of each first point, when the correlation between the first point and the neighborhood point is determined based on the first feature of the first point and the first feature of the neighborhood point, the data completion module is specifically used for: The correlation between the first point and the neighborhood point is determined based on the first feature of the first point, the first feature of the neighborhood point, and the spatial coordinate feature of the neighborhood point. For each first point, when the fusion feature corresponding to the first point is obtained by weighting and fusing the first features of the respective neighborhood points of the first point by using the correlations corresponding to the respective neighborhood points as weights, the data completion module is specifically used for: For each neighborhood point of the first point, the second feature of the neighborhood point is obtained by feature extraction based on the first feature and the spatial coordinate feature of the neighborhood point. The fusion feature corresponding to the first point is obtained by weighting and fusing the second features of the respective neighborhood points of the first point by using the weights corresponding to the respective neighborhood points.

14. The apparatus of claim 11, wherein, When the target point cloud data after completion is generated based on the feature representation of the plurality of second points, the data completion module is specifically used for: Based on the feature representation of the plurality of second points, a plurality of point features of a plurality of target points are obtained by performing at least one upsampling process, and three-dimensional coordinates of the plurality of target points are obtained according to the point features of the plurality of target points, and the target point cloud data comprises the three-dimensional coordinates of the plurality of target points.

15. The apparatus of claim 14, wherein, The three-dimensional coordinates of the plurality of target points are obtained by the data completion module in the following manner: Based on the three-dimensional coordinates of the plurality of initial points and the three-dimensional coordinates of each second point, an initial point cloud input is determined, and the initial point cloud input comprises three-dimensional coordinates of a plurality of input points; The initial point cloud input is taken as the point cloud input of the first upsampling process, and the upsampling process is repeatedly performed until a set number of times, and the three-dimensional coordinates of a plurality of output points obtained by the last upsampling process are determined as the three-dimensional coordinates of the plurality of target points; The upsampling process comprises the following steps: For each of the input points in the point cloud input, the distance between the input point and each second point is determined according to the three-dimensional coordinates of the input point and the three-dimensional coordinates of each second point, and a second number of neighbor points of the input point are determined from each second point according to the distance; For each of the input points, the point feature of the input point is obtained by fusing the point features of each neighbor point of the input point; Based on the three-dimensional coordinates and the point features of each of the input points, the point features of a plurality of output points are obtained by feature upsampling; Based on the point features of the plurality of output points, the three-dimensional coordinates of each output point are obtained, and the three-dimensional coordinates of each output point are taken as the point cloud input of the next upsampling process.

16. The apparatus of claim 15, wherein, For each of the input points, when the data completion module obtains the point feature of the input point by fusing the point features of each neighbor point of the input point, the data completion module is specifically configured to: determine the weight corresponding to each neighbor point of the input point according to the distance between the input point and each neighbor point of the input point, wherein the weight corresponding to each neighbor point is negatively correlated with the distance corresponding to the neighbor point; weight the point features of each neighbor point of the input point according to the weight corresponding to each neighbor point of the input point to obtain the point feature of the input point.

17. The apparatus of claim 15, wherein, For each upsampling process, when the data completion module obtains the point features of a plurality of output points based on the three-dimensional coordinates and the point features of each of the input points by feature upsampling, the data completion module is specifically configured to: splice the point feature and the three-dimensional coordinates of each input point to obtain a third feature of each input point; determine a third number of neighbor points of each input point from each input point according to the three-dimensional coordinates of each input point; for each input point, obtain the point features of at least two output points corresponding to the input point by feature upsampling according to the third feature of the input point and the historical features of each neighbor point of the input point; wherein, for the first upsampling process, the historical feature of each neighbor point is the third feature of the neighbor point, and for each upsampling process other than the first upsampling process, the historical feature of each neighbor point is the point feature of the neighbor point as an output point in the last upsampling process.

18. The apparatus of claim 17, wherein, For each input point, the data completion module, when obtaining the point features of at least two output points corresponding to the input point according to the third feature of the input point and the historical features of the neighborhood points of the input point through feature upsampling, is specifically configured to: respectively through each attention module in the at least two second attention modules: determine the second attention weight between the third feature of the input point and the historical feature of each neighborhood point of the input point; weight and fuse the historical features of each neighborhood point of the input point according to the second attention weight corresponding to each neighborhood point of the input point to obtain the point feature of one output point corresponding to the input point.

19. The apparatus of claim 15, wherein, When the data completion module obtains the three-dimensional coordinates of each output point based on the point features of the plurality of output points, it is specifically configured to: perform feature extraction based on the point features of the plurality of output points to obtain the three-dimensional coordinate offset of each output point; obtain the three-dimensional coordinates of each output point based on the three-dimensional coordinates of each input point and the three-dimensional coordinate offset of each output point.

20. The apparatus of any one of claims 11-12 and 14-18, wherein, When the data completion module obtains the completed target point cloud data based on the original point cloud data, it is achieved by calling the trained point cloud completion network, the point cloud completion network is obtained by iteratively training a neural network model based on a training data set, and the training data set includes a plurality of training samples, each training sample includes a point cloud data pair, and each point cloud data pair includes sample point cloud data to be completed and corresponding completed sample point cloud data.

21. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1-10 when executing the computer program.

22. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-10.

23. A computer program product, characterised in that, The computer product includes a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-10.

Images