Point cloud data processing method, device, equipment, medium and program product

Abstract

The invention provides a point cloud data processing method and device, equipment, a medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting point cloud data into a trained first network model, mapping the point cloud data to a high-dimensional space, obtaining a point cloud token, and storing the point cloud token in a database; soft weighting processing is carried out on the point cloud tokens through the trained second network model to obtain soft weighting features, the soft weighting features represent semantic association between the point cloud data, hard weighting processing is carried out on the point cloud tokens through a hard weighting module to obtain hard weighting features, and the hard weighting features represent semantic association between the point cloud data; and inputting the soft weighted features and the hard weighted features into a trained third network model for feature fusion to obtain weighted fusion features for intelligent tasks. Through a double-weighted processing mode, semantic features of learning points at different positions are considered, noise points and non-noise points are distinguished, the semantic features of the point cloud data are reflected more accurately, and the accuracy of executing intelligent tasks is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, relates to a point cloud data processing method and device, equipment, medium and program product. BACKGROUND

[0002] With the development of technology, point cloud data has been widely used in many scenarios, such as autonomous driving, virtual visual interaction, three-dimensional reconstruction, etc. However, point cloud data usually has the characteristics of much noise, sparsity and complex relationship between points, which brings great difficulty to point cloud semantic learning.

[0003] Traditional point cloud processing methods often cannot effectively process noise points, resulting in inaccurate semantic extraction and subsequent intelligent task performance. The processing method of point cloud data in related technologies is not robust enough for noise points, and still has limitations in mining the semantic association of each point in the point cloud data, which is difficult to meet the high requirements of point cloud semantic learning accuracy and robustness in actual application scenarios.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present disclosure provides a point cloud data processing method, device, equipment, medium and program product, which at least partially overcomes the problem of low point cloud data processing accuracy and inaccurate processed data in related technologies.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] In a first aspect, the embodiments in the present disclosure provide a point cloud data processing method, which comprises:

[0008] inputting point cloud data into a trained first network model, mapping the point cloud data to a high-dimensional space to obtain point cloud tokens;

[0009] performing soft weighting processing on the point cloud tokens through a trained second network model to obtain soft weighting features; the weight values in the soft weighting features represent the semantics of different point cloud data;

[0010] performing hard weighting processing on the point cloud tokens through a hard weighting module to obtain hard weighting features; the weight values in the hard weighting features represent the association relationship between the point cloud data and intelligent tasks;

[0011] The soft weighting feature and the hard weighting feature are input into a third trained network model for feature fusion to obtain a weighted fusion feature for an intelligent task.

[0012] In a possible embodiment, the method further includes:

[0013] The weighted fusion feature is input into a fourth trained network model for local semantic extraction to obtain a local semantic feature.

[0014] The weighted fusion feature is input into a fifth trained network model for global semantic extraction to obtain a global semantic feature.

[0015] The local semantic feature and the global semantic feature are input into an intelligent task to execute the intelligent task.

[0016] In a possible embodiment, the soft weighting processing of the point cloud token by the second trained network model to obtain a soft weighting feature includes:

[0017] The soft weighting processing of each point cloud token by a weighting matrix in the second network model obtains the soft weighting feature; each soft weighting feature includes weighting information of all point cloud tokens.

[0018] In a possible embodiment, the hard weighting module includes at least one of the following processing methods: Gaussian filtering, mean filtering or Euclidean clustering.

[0019] The hard weighting processing of the point cloud token by the hard weighting module to obtain a hard weighting feature includes:

[0020] The hard weighting feature is obtained by filtering noise data of point cloud data in the point cloud token according to the position of the point in space in the point cloud token by Gaussian filtering, mean filtering or Euclidean clustering; the hard weighting feature includes weighting information of local point cloud tokens.

[0021] In a possible embodiment, the input of the soft weighting feature and the hard weighting feature into a third trained network model for feature fusion to obtain a weighted fusion feature for an intelligent task includes:

[0022] According to the weight value in the hard weighting feature, a first hard weighting feature with a high weight value is identified; the first hard weighting feature represents non-noise point cloud data corresponding to the intelligent task to be executed.

[0023] According to the weight value in the first hard weighting feature and the soft weighting feature, a first soft weighting feature in the soft weighting feature that has a semantic association with the intelligent task to be executed is identified.

[0024] aligning the first hard-weighted feature and the first soft-weighted feature;

[0025] fusing all the hard-weighted features and all the soft-weighted features according to the aligned dimensions to obtain weighted fusion features for the intelligent task.

[0026] In a possible embodiment, comprising: a point cloud data processing model; the point cloud data processing model comprises: a first network model, a weighting module, a third network model, a semantic extraction module; the weighting module comprises: a second network model and a hard weighting module; the semantic extraction module comprises: a fourth network model and a fifth network model.

[0027] In a possible embodiment, the training process of the point cloud data processing model comprises:

[0028] obtaining a to-be-trained data set; the to-be-trained data set is a data set of point cloud data;

[0029] extracting a training sample from the to-be-trained data set and inputting the training sample into the point cloud data processing model for iterative training;

[0030] wherein, one iteration training process comprises:

[0031] inputting the training sample into the first network model, mapping the point cloud data to a high-dimensional space, and obtaining point cloud token training data;

[0032] performing soft weighting processing and hard weighting processing on the point cloud token through the weighting module to obtain soft weighted feature training data and hard weighted feature training data;

[0033] inputting the soft weighted feature and the hard weighted feature into a fusion module for feature fusion to obtain weighted fusion feature training data;

[0034] inputting the weighted fusion feature training data into the semantic extraction module for local semantic extraction and global semantic extraction to obtain local semantic feature training data and global semantic feature training data;

[0035] for different domain training tasks, using the local semantic feature training data and the global semantic feature training data in the training task to determine a loss function value;

[0036] According to the loss function value, adjusting the network parameters of the first network model, the second network model, the third network model, the fourth network model and the fifth network model until the loss function value meets a preset threshold value, obtaining a trained point cloud data processing model.

[0037] In a second aspect, the embodiments in the present disclosure provide a point cloud data processing device, comprising:

[0038] a processing unit, configured to input point cloud data into a trained first network model, map the point cloud data to a high-dimensional space, and obtain point cloud tokens;

[0039] a soft weighting unit, configured to perform soft weighting processing on the point cloud tokens by using a trained second network model, and obtain soft weighting features; weight values in the soft weighting features represent semantics of different point cloud data;

[0040] a hard weighting unit, configured to perform hard weighting processing on the point cloud tokens by using a hard weighting module, and obtain hard weighting features; weight values in the hard weighting features represent an association relationship between the point cloud data and an intelligent task;

[0041] a fusion unit, configured to input the soft weighting features and the hard weighting features into a trained third network model to perform feature fusion, and obtain weighted fusion features for the intelligent task.

[0042] In a possible implementation, the apparatus further includes:

[0043] a local semantic extraction unit, configured to input the weighted fusion features into a trained fourth network model to perform local semantic extraction, and obtain local semantic features;

[0044] a global semantic extraction unit, configured to input the weighted fusion features into a trained fifth network model to perform global semantic extraction, and obtain global semantic features;

[0045] an execution unit, configured to input the local semantic features and the global semantic features into the intelligent task, to execute the intelligent task.

[0046] In a possible implementation, the apparatus further includes:

[0047] a training unit, configured to obtain a to-be-trained data set; the to-be-trained data set is a data set of point cloud data;

[0048] extract training samples from the to-be-trained data set, and input the training samples into the point cloud data processing model to perform iterative training;

[0049] wherein, one iteration training process includes:

[0050] input the training samples into a first network model, map the point cloud data to a high-dimensional space, and obtain point cloud token training data;

[0051] perform soft weighting processing and hard weighting processing on the point cloud tokens by using a weighting module, and obtain soft weighting feature training data and hard weighting feature training data;

[0052] input the soft weighted feature and the hard weighted feature into a fusion module for feature fusion to obtain weighted fusion feature training data;

[0053] input the weighted fusion feature training data into a semantic extraction module for local semantic extraction and global semantic extraction respectively to obtain local semantic feature training data and global semantic feature training data;

[0054] for a training task in different fields, use the local semantic feature training data and the global semantic feature training data in the training task to determine a loss function value;

[0055] adjust network parameters of the first network model, the second network model, the third network model, the fourth network model and the fifth network model according to the loss function value until the loss function value meets a preset threshold value, and obtain a trained point cloud data processing model.

[0056] In a third aspect, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of the first aspect described above by executing the executable instructions.

[0057] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the method of the first aspect described above.

[0058] In a fifth aspect, according to another aspect of the present disclosure, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to perform any of the methods described above.

[0059] The embodiment of the present disclosure provides a point cloud data processing method, device, equipment, medium and program product, relating to the technical field of artificial intelligence. The method comprises: inputting point cloud data into a trained first network model, mapping the point cloud data to a high-dimensional space to obtain point cloud tokens; performing soft weighting processing on the point cloud tokens through a trained second network model to obtain soft weighted features; the weight values in the soft weighted features represent the semantics of different point cloud data; performing hard weighting processing on the point cloud tokens through a hard weighting module to obtain hard weighted features; the weight values in the hard weighted features represent the association between the point cloud data and the intelligent task; and inputting the soft weighted features and the hard weighted features into a trained third network model for feature fusion to obtain weighted fusion features for the intelligent task. Through the double weighting processing mode, the semantic features of the learning points at different positions and the distinction between noise points and non-noise points are taken into account, the semantic features of the point cloud data are more accurately reflected, and the accuracy of executing the intelligent task is improved.

[0060] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0062] Figure 1 A flowchart of a point cloud data processing method in an embodiment of the present disclosure is shown;

[0063] Figure 2 A flowchart of executing an intelligent task in an embodiment of the present disclosure is shown;

[0064] Figure 3 A flowchart of feature fusion in an embodiment of the present disclosure is shown;

[0065] Figure 4 An architectural schematic diagram of a point cloud data processing model in an embodiment of the present disclosure is shown;

[0066] Figure 5 A flowchart of a training process of a point cloud data processing model in an embodiment of the present disclosure is shown;

[0067] Figure 6 A structural schematic diagram of a point cloud data processing device in an embodiment of the present disclosure is shown;

[0068] Figure 7 A structural schematic diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0070] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0071] Traditional point cloud processing methods often struggle to effectively handle noisy points, resulting in inaccurate semantic extraction and subsequent intelligent tasks. Related technologies for processing point cloud data are also insufficiently robust to noisy point clouds and still have limitations in mining the semantic associations between points in point cloud data. This makes it difficult to meet the high accuracy and robustness requirements of point cloud semantic learning in practical applications.

[0072] This exemplary implementation is described in detail below with reference to the accompanying drawings and examples.

[0073] First, a point cloud data processing method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities. In the following process, the electronic device is taken as a server as an example.

[0074] Figure 1 A flow chart of a point cloud data processing method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the point cloud data processing method provided in the embodiment of the present disclosure includes the following steps:

[0075] S102: Input the point cloud data into the trained first network model, map the point cloud data to a high-dimensional space, and obtain a point cloud token.

[0076] In a possible embodiment, the first network model may be constructed based on a deep learning network, for example, based on a CNN network.

[0077] Tokenize the point cloud data input from the outside to obtain point cloud tokens, and the point cloud tokens respectively contain the features of different points in the point cloud data.

[0078] The (x, y, z) in the point cloud data, that is, the Cartesian coordinates of the point in the three-dimensional space, are mapped to a high-dimensional space by the CNN network, and a series of point cloud tokens are divided, and the features in the high-dimensional space are the point cloud tokens corresponding to the point cloud data.

[0079] The input of the first network model is the point cloud data containing the Cartesian coordinates of each point in the three-dimensional space, and the output is the point cloud token after mapping to the high-dimensional space.

[0080] S104: Soft weighting processing is performed on the point cloud token by the trained second network model to obtain a soft weighting feature, and the weight value in the soft weighting feature represents the semantics of different point cloud data.

[0081] In a possible embodiment, the second network model can be composed of a fully connected network, a CNN network, a self-attention mechanism, etc. For example, the second network model is constructed based on a self-attention neural network.

[0082] The second network model constructed by the self-attention mechanism performs soft weighting processing on the point cloud token to establish the semantic relationship between the points in the point cloud data, and the features corresponding to different weight values in the soft weighting feature can represent the semantic relationship between different point cloud data.

[0083] For example, if the point cloud data includes a plurality of semantic data, such as a table, a chair, a cup, a user, and a robot, by performing soft weighting on different point cloud tokens, the table, the chair, the cup, the user, and the robot are respectively assigned different weight values to represent the semantics of different point cloud data, and the semantic relationship between different point cloud data can be established according to the size or proportion relationship of the weight values.

[0084] In a possible embodiment, the soft weighting processing process can include: performing soft weighting processing on each point cloud token by a weighting matrix in the second network model to obtain a soft weighting feature; and each soft weighting feature includes the weighting information of all point cloud tokens.

[0085] The second network model includes a learnable and trainable weighting matrix, and after the weighting matrix is trained through the training process, each point cloud token is weighted by the weighting matrix, and each soft weighting feature includes the weighting information of all point cloud tokens, and the weight values between different point cloud data represent the semantics and semantic relationship of the point cloud data, so that the semantics of the points in the point cloud data at different positions can be learned in the subsequent intelligent task process.

[0086] The input of the second network model is the point cloud token, and the output is a soft weighted feature corresponding to different point cloud tokens.

[0087] The soft weighted process assigns a weight value varying in a continuous range to different point cloud data, allowing the importance of different procedures to be represented and reflecting the association strength between semantics.

[0088] However, the soft weighted process is relatively complex, and compared with conventional point cloud data processing methods such as self-defined weighting, single weighting method or fixed rule point cloud processing, the stability of the single soft weighted process may be relatively low, but it can dynamically capture the semantic relationship of point cloud data and represent the semantics of different point cloud data.

[0089] S106: The point cloud token is processed by the hard weighting module to obtain a hard weighted feature, and the weight value in the hard weighted feature represents the association relationship between the point cloud data and the intelligent task.

[0090] In one possible embodiment, the hard weighting module can include at least one of the following processing methods: Gaussian filtering, mean filtering or Euclidean clustering. For example, the hard weighting module is constructed in the form of Gaussian filtering.

[0091] In one possible embodiment, the hard weighting process can include: filtering noise data of point cloud data in the point cloud token according to the position of the point in the space in the point cloud token by Gaussian filtering, mean filtering or Euclidean clustering to obtain a hard weighted feature; the hard weighted feature includes weighting information of the local point cloud token.

[0092] By processing the point cloud token by Gaussian filtering, the point cloud token of the outlier point is filtered, that is, the noise data in the point cloud token can be identified, and a low weight value is assigned to the noise data or the noise data is directly filtered out, and a high weight value is assigned to the non-noise data in the point cloud token to obtain a hard weighted feature of the local point cloud token.

[0093] The weight value assigned to the point cloud token by the hard weighting module is a high weight value that cannot be learned, and can be used to represent the association relationship between the point cloud data and the intelligent task.

[0094] The input of the hard weighting module is the point cloud token, and the output is a hard weighted feature corresponding to the local point cloud token.

[0095] The hard weighting process assigns a clearly specified weight value to different point cloud data. In the filtering process of the point cloud token, the point cloud token can be selected to assign a high weight value, or the point cloud token that is not selected is discarded in the processing process, has a clear result, is stable, and the processing process is simpler and more direct than the point cloud processing process in the related art, and has strong interpretability.

[0096] S108: input the soft weighted feature and the hard weighted feature into the trained third network model for feature fusion to obtain a weighted fusion feature for the intelligent task.

[0097] In a possible embodiment, the third network model can be constructed by a ResNET50, VGG or the like network.

[0098] Exemplarily, the soft weighted feature and the hard weighted feature can be input into the third network model constructed based on the ResNET50 network for feature fusion, the dimensions of the soft and hard weighted features are aligned, and they are connected.

[0099] The input of the third network model is the soft weighted feature and the hard weighted feature, and the output is the weighted fusion feature.

[0100] Compared with the point cloud processing in the related art, by combining the soft and hard weighting processing procedures, the method in the embodiments of the present disclosure can not only dynamically capture the semantic correlation of each point in the point cloud data, provide accurate guidance for the point cloud encoder, and enable it to effectively learn the semantic features in different positions, but also can utilize the spatial positions of the points in the point cloud data to assign an unlearnable weight to the point cloud token, effectively distinguish the noise points and non-noise points, associate them with the intelligent task, overcome the limitations of the point cloud processing in the related art in terms of semantic feature extraction and noise processing, and improve the accuracy of executing the intelligent task.

[0101] Further, by feature fusion, the weight of the outlier point is reduced by utilizing the hard weighted feature to assist the soft weighting module to learn reasonable weight, the weighted fusion feature is output, more comprehensive and rich feature information is provided for subsequent execution of the intelligent task, the global and local semantic information can be considered at the same time, the real semantic features of the point cloud data can be more accurately reflected, and the accuracy of the intelligent tasks such as point cloud classification and segmentation can be improved.

[0102] Figure 2 A flowchart of executing an intelligent task in the embodiments of the present disclosure is shown, as shown in Figure 2 The flowchart includes the following steps:

[0103] S202: input the weighted fusion feature into a trained fourth network model for local semantic extraction to obtain a local semantic feature.

[0104] In a possible embodiment, the local features of each part in the point cloud are extracted according to the semantics represented by the features in the weighted fusion feature, the semantic relationship between the points in the point cloud is learned according to the weighted fusion feature, the local semantics in the point cloud data are determined, and the local semantic features are extracted.

[0105] In a possible embodiment, the fourth network model can use a linear network to reduce the computational load in view of the complexity of the local semantics. Exemplarily, the fourth network model can be constructed by a linear attention mechanism network.

[0106] The input of the fourth network model is the weighted fusion feature, and the output is the local semantic feature.

[0107] S204: inputting the weighted fusion feature into the trained fifth network model to extract the global semantic feature.

[0108] In a possible embodiment, the global feature of the point cloud is extracted by the fifth network model, the global feature of the point cloud data is learned according to the weighted fusion feature, and the global semantic feature is determined.

[0109] The input of the fifth network model is the weighted fusion feature, and the output is the global semantic feature.

[0110] The fifth network model can be constructed by a CNN, a ViT-B / 16 network structure, etc. Exemplarily, it can be constructed by a CNN network and a pooling layer.

[0111] S206: inputting the local semantic feature and the global semantic feature into an intelligent task to execute the intelligent task.

[0112] In a possible embodiment, the features in the obtained local semantic feature and global semantic feature can represent the semantics of the point cloud data, and the features include the features corresponding to the point cloud data having a correlation with the intelligent task, which are used to execute the intelligent task, and meet the requirements of high precision and high robustness of point cloud semantic learning.

[0113] The intelligent task can include an autonomous driving task, a VR interaction task for identifying objects in the point cloud, or a task of denoising and reconstructing a more accurate three-dimensional model from a noisy point cloud generated by a two-dimensional picture.

[0114] In one possible embodiment, taking autonomous driving as an example, first, point cloud data A in a road environment is acquired by a laser radar and input into a pre-trained network architecture. The point cloud data A is mapped to a high-dimensional space by a first network model to generate a point cloud token containing rich feature information. By a second network model, a learnable weighting matrix is used to dynamically capture the semantic association of each point in the point cloud data, establish the connection between points, and guide the subsequent semantic learning process. At the same time, a hard weighting module weights the point cloud token based on the spatial position of the point by traditional filtering methods such as Gaussian filtering, effectively filtering outliers and reducing the interference of noise points on semantic learning. A feature fusion module aligns and connects the soft weighting features and the hard weighting features. The weighted fusion features after fusion contain both global semantic information and local detailed features. A local semantic extraction module and a global semantic extraction module extract local semantic features (such as shape, texture, etc.) and global semantic features (such as category, position, etc.) of obstacles from the weighted fusion features after fusion, respectively, for final recognition and judgment.

[0115] After the above processing, the autonomous driving system can accurately identify obstacles in the point cloud data, such as vehicles, pedestrians, traffic signs, etc., and classify them. This accurate semantic recognition capability helps autonomous vehicles make reasonable decisions in advance, such as timely avoiding pedestrians, recognizing traffic signs and adjusting the driving route, etc., thereby improving the safety and reliability of autonomous driving.

[0116] For step S108, Figure 3 A flowchart of feature fusion in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the following steps are included:

[0117] S302: According to the weight values in the hard weighting features, a first hard weighting feature with a high weight value is identified, and the first hard weighting feature represents non-noise point cloud data corresponding to the intelligent task to be executed.

[0118] S304: According to the weight values in the first hard weighting feature and the soft weighting feature, a first soft weighting feature in the soft weighting feature that has a semantic association with the intelligent task to be executed is identified.

[0119] S306: The first hard weighting feature and the first soft weighting feature are aligned in feature.

[0120] S308: The entire hard weighting feature and the entire soft weighting feature are fused in feature according to the aligned dimension to obtain weighted fusion features for the intelligent task.

[0121] In a possible embodiment, the first hard-weighted feature corresponding to the non-noise point cloud data is determined by filtering and then hard-weighting the feature, the first soft-weighted feature corresponding to the non-noise data is learned to assist in feature alignment, and finally the fusion of all soft and hard-weighted features is completed to obtain the weighted fusion feature, which not only completes the semantic processing of the point cloud data, but also determines the semantic association and completes the feature alignment. In the output weighted fusion feature, the features that have a correlation relationship with the intelligent task can be accurately identified, and the semantics of the features can be accurately identified.

[0122] For example, taking an automatic driving task as an example, each object in the automatic driving process needs to be identified, and taking identifying a person as the main target as an example, after obtaining the point cloud data, the hard-weighting module assigns a high weight value to the point cloud token corresponding to the person through filtering, and assigns a low weight value or directly filters other noise point cloud data. The soft-weighting processing assigns a certain weight to each object in the point cloud data, and in the feature fusion, the feature alignment and connection fusion are performed. By identifying the point cloud token similar to the hard-weighted feature, the feature representing the semantics of the person in the soft-weighted feature can be learned, and then the feature alignment is performed, and the feature fusion of all soft-weighted features and hard-weighted features is performed according to the aligned dimension. In this way, the feature representing the semantics of the person can be more accurately identified.

[0123] It should be noted that in the hard-weighting processing process, the weight values of hard-weighting of different targets can be set, and then in the feature fusion, the accurate weight values of hard-weighting can be used to assist the soft-weighting to complete the feature fusion of multiple target dimensions, that is, the accuracy of identifying different semantics corresponding to multiple features can be improved.

[0124] Figure 4 An architecture schematic diagram of a point cloud data processing model in an embodiment of the present disclosure is shown, as shown in Figure 4 The point cloud data processing model 40 includes a first network model 410, a weighting module 420, a third network model 430, and a semantic extraction module 440.

[0125] The weighting module 420 includes a second network model 421 and a hard-weighting module 422, and the semantic extraction module 440 includes a fourth network model 441 and a fifth network model 442.

[0126] The first network model 410 is used for tokenizing the point cloud data and outputting the point cloud token.

[0127] The weighting module 420 is used for soft-weighting and hard-weighting, the second network model 421 is used for soft-weighting, and the hard-weighting module 422 is used for hard-weighting.

[0128] The third network model 430 is used for feature fusion.

[0129] The semantic extraction module 440 is configured to perform semantic extraction, the fourth network model 441 is configured to perform local semantic extraction, and the fifth network model 442 is configured to perform global semantic extraction.

[0130] Each network model in the point cloud data processing model described above is trained and then put into an inference process. The training process is as follows, Figure 5 A flowchart of a training process of a point cloud data processing model in an embodiment of the present disclosure is shown in FIG. 5. Figure 5 As shown in FIG. 5, the training process includes the following steps:

[0131] S502: Obtain a training data set.

[0132] S504: Extract a training sample from the training data set and input the training sample into the first network model to map the point cloud data to a high-dimensional space and obtain point cloud token training data.

[0133] S506: Perform soft weighting processing and hard weighting processing on the point cloud token through the weighting module to obtain soft weighted feature training data and hard weighted feature training data.

[0134] S508: Input the soft weighted feature and the hard weighted feature into the fusion module for feature fusion to obtain weighted fusion feature training data.

[0135] S510: Input the weighted fusion feature training data into the semantic extraction module for local semantic extraction and global semantic extraction to obtain local semantic feature training data and global semantic feature training data.

[0136] In a possible embodiment, in the training process, the processing manner of the training sample is similar to the processing manner of the point cloud data in the inference process, which is not described herein.

[0137] S512: For different domain training tasks, use the local semantic feature training data and the global semantic feature training data in the training task to determine a loss function value.

[0138] In a possible embodiment, the structure of the point cloud data processing model in the present embodiment can be applied to intelligent tasks in multiple domains, but for intelligent tasks in different domains, the point cloud data processing model needs to be trained for a training task before use.

[0139] S514: Determine whether the loss function value meets a preset threshold; if yes, perform S516; if no, perform S518.

[0140] S516: Obtain a trained point cloud data processing model.

[0141] S518: Adjust the network parameters of the first network model, the second network model, the third network model, the fourth network model, and the fifth network model, and return to execute S504.

[0142] In a possible embodiment, for any one training task, the global semantic feature and the local semantic feature can be combined and used together in the downstream specific training task for the point cloud to determine the loss function value, and the point cloud data processing model is back propagated for a preset threshold number of times through the loss function value, that is, when the loss function value does not meet the preset threshold, the network parameters of the first network model, the second network model, the third network model, the fourth network model, and the fifth network model are adjusted, new training samples are re-extracted, and finally the training of each network module is realized to obtain the trained point cloud data processing model.

[0143] Through the cooperation of each module in Figure 4 , a complete double-weighted point cloud semantic learning thickening is formed. From the preprocessing (tokenization) of the point cloud data, the weighting processing (soft and hard weighting), the feature fusion to the semantic extraction (local and global), it is different from the single process or simple combination of the prior art, and an efficient point cloud data processing model is given.

[0144] Based on the same inventive concept, the disclosure embodiments also provide a point cloud data processing apparatus, as follows. Since the principles of the problem solved by the apparatus embodiments are similar to those of the above-mentioned method embodiments, the implementation of the apparatus embodiments can be referred to the implementation of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0145] Figure 6 The structure of a point cloud data processing apparatus in the embodiments of the disclosure is shown in a structural schematic diagram as shown in Figure 6 , which includes:

[0146] The processing unit 601 is configured to input the point cloud data into the trained first network model, map the point cloud data to a high-dimensional space, and obtain point cloud tokens; the soft weighting unit 602 is configured to perform soft weighting processing on the point cloud tokens through the trained second network model, and obtain soft weighting features; the weight values in the soft weighting features represent the semantics of different point cloud data; the hard weighting unit 603 is configured to perform hard weighting processing on the point cloud tokens through the hard weighting module, and obtain hard weighting features; the weight values in the hard weighting features represent the association relationship between the point cloud data and the intelligent task; and the fusion unit 604 is configured to input the soft weighting features and the hard weighting features into the trained third network model for feature fusion, and obtain weighted fusion features for the intelligent task.

[0147] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied in a form of entirely hardware, entirely software (including firmware, microcode, etc.), or a combination of hardware and software, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0148] The electronic device 700 according to this embodiment of the present disclosure will be described below with reference to Figure 7 Figure 7 The electronic device 700 is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

[0149] As shown in Figure 7 The electronic device 700 is in the form of a general computing device. The components of the electronic device 700 can include, but are not limited to, the at least one processing unit 710 described above, the at least one storage unit 720 described above, and a bus 730 connecting different system components, including the storage unit 720 and the processing unit 710.

[0150] The storage unit stores program codes which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of the present specification. For example, the processing unit 710 can perform the steps of any one of the method embodiments described above.

[0151] The storage unit 720 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 7201 and / or a cache memory unit 7202, and can further include a read-only memory (ROM) 7203.

[0152] The storage unit 720 can further include a program / utility 7204 having a set of program modules 7205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment.

[0153] The bus 730 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0154] ​The electronic device 700 can also communicate with one or more external devices 740 such as a keyboard or pointing device, a Bluetooth device, or a database, and / or one or more devices that enable a user to interact with the electronic device 700 and / or one or more devices (e.g., a router, a modem, a server, etc.) that enable the electronic device 700 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 750. Still yet, the electronic device 700 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 760. As depicted, the network adapter 760 communicates with the other components of the electronic device 700 via the bus 730. It should be appreciated that the network adapter 760 and / or the bus 730 can be implemented using one or more types of technology, including, but not limited to, Ethernet, Bluetooth, Wi-Fi, and / or any other technology that can be used to enable the electronic device 700 to communicate with one or more other computing devices.

[0155] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0156] In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods in the above embodiments.

[0157] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which can be a readable signal medium or a readable storage medium. A program product capable of implementing the above methods of the present disclosure is stored thereon. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing the terminal device to execute the steps according to various example embodiments of the present disclosure described in the above "example method" section of the specification when the program product is run on the terminal device.

[0158] More specific examples of the computer-readable storage medium in the present disclosure can include but are not limited to an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] In the present disclosure, a computer readable storage medium can include a data signal carrying the readable program code in a baseband or in a carrier wave. Such a propagated data signal can take a wide variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer readable medium can also be any medium that can be read by a computer or a device that can be used to store or transport the program in a functional manner, rather than the medium that the program is presented on.

[0160] Optionally, program code embodied on a computer readable storage medium can be transmitted by any data transmission techniques, including but not limited to radio frequency, wireless, cable, wire, optical fiber cable, or any suitable combination thereof.

[0161] In an implementation, the program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on a user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0162] It should be noted that, although several modules or units for a device to perform actions are mentioned in the above detailed description, the division into the modules or units is not mandatory. In fact, according to an embodiment of the present disclosure, features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0163] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied, nor is it necessary to perform all of the steps shown to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, performed in a different order, broken down into multiple steps, and / or the like.

[0164] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0165] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known use or custom in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A point cloud data processing method, characterized in that: The method comprises: Inputting the point cloud data into the trained first network model, mapping the point cloud data into a high-dimensional space, and obtaining a point cloud token; The point cloud tokens are soft-weighted by the trained second network model to obtain soft-weighted features; the weight values ​​in the soft-weighted features represent the semantics of different point cloud data; The point cloud token is hard-weighted by a hard weighting module to obtain a hard weighted feature; the weight value in the hard weighted feature represents the correlation between the point cloud data and the intelligent task; The soft weighted features and the hard weighted features are input into a trained third network model for feature fusion to obtain weighted fusion features for use in intelligent tasks.

2. The method according to claim 1, characterized in that The method further comprises: Inputting the weighted fusion features into the trained fourth network model to perform local semantic extraction to obtain local semantic features; Inputting the weighted fusion features into the trained fifth network model to perform global semantic extraction to obtain global semantic features; The local semantic features and the global semantic features are input into an intelligent task to execute the intelligent task.

3. The method according to claim 1, characterized in that The point cloud token is soft-weighted by the trained second network model to obtain soft-weighted features, including: Each of the point cloud tokens is soft-weighted by using the weighting matrix in the second network model to obtain the soft-weighted features; each of the soft-weighted features includes weighted information of all the point cloud tokens.

4. The method according to claim 1, wherein The hard weighting module includes at least one of the following processing methods: Gaussian filtering, mean filtering or Euclidean clustering; The hard weighting module performs hard weighting processing on the point cloud token to obtain hard weighted features, including: By Gaussian filtering, mean filtering or Euclidean clustering, the noise data of the point cloud data in the point cloud token is filtered according to the position of the midpoint of the point cloud token in space to obtain a hard weighted feature; the hard weighted feature includes weighted information of the local point cloud token.

5. The method according to claim 1, wherein The step of inputting the soft weighted features and the hard weighted features into a trained third network model for feature fusion to obtain weighted fusion features for use in intelligent tasks includes: identifying a first hard weighted feature with a high weight value based on the weight values ​​in the hard weighted features; wherein the first hard weighted feature represents the non-noise point cloud data corresponding to the intelligent task to be performed; Identifying, based on the weight values ​​of the first hard-weighted features and the soft-weighted features, a first soft-weighted feature in the soft-weighted features that has a semantic association with the intelligent task to be performed; Performing feature alignment on the first hard weighted feature and the first soft weighted feature; All hard-weighted features and all soft-weighted features are fused according to the aligned dimensions to obtain weighted fused features for intelligent tasks.

6. The method according to claim 2, characterized in that include: Point cloud data processing model; The point cloud data processing model includes: a first network model, a weighting module, a third network model, and a semantic extraction module; the weighting module includes: a second network model and a hard weighting module; and the semantic extraction module includes: a fourth network model and a fifth network model.

7. The method according to claim 6, characterized in that The training process of the point cloud data processing model includes: Obtaining a dataset to be trained; the dataset to be trained is a dataset of point cloud data; Extracting training samples from the to-be-trained data set and inputting them into the point cloud data processing model for iterative training; Among them, an iterative training process includes: Inputting the training sample into a first network model, mapping the point cloud data into a high-dimensional space, and obtaining point cloud token training data; Performing soft weighting processing and hard weighting processing on the point cloud tokens through a weighting module to obtain soft weighted feature training data and hard weighted feature training data; Inputting the soft weighted features and the hard weighted features into a fusion module for feature fusion to obtain weighted fusion feature training data; Inputting the weighted fusion feature training data into a semantic extraction module to perform local semantic extraction and global semantic extraction respectively, to obtain local semantic feature training data and global semantic feature training data; For training tasks in different fields, the local semantic feature training data and the global semantic feature training data are used in the training tasks to determine a loss function value; According to the loss function value, the network parameters of the first network model, the second network model, the third network model, the fourth network model and the fifth network model are adjusted until the loss function value meets a preset threshold, thereby obtaining a trained point cloud data processing model.

8. A point cloud data processing device, characterized in that: include: a processing unit, configured to input the point cloud data into the trained first network model, map the point cloud data into a high-dimensional space, and obtain a point cloud token; a soft weighting unit, configured to perform soft weighting processing on the point cloud tokens using a trained second network model to obtain soft weighted features; wherein the weight values ​​in the soft weighted features represent the semantics of different point cloud data; A hard weighting unit, configured to perform hard weighting processing on the point cloud token through a hard weighting module to obtain a hard weighted feature; the weight value in the hard weighted feature represents the association relationship between the point cloud data and the intelligent task; The fusion unit is used to input the soft weighted features and the hard weighted features into a trained third network model for feature fusion to obtain weighted fusion features for use in intelligent tasks.

9. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the method according to any one of claims 1 to 7.

Images