Point cloud sequence recognition model training and recognition method, device, equipment and medium

Abstract

This invention discloses a training and recognition method, apparatus, device, and medium for a point cloud sequence recognition model, comprising: acquiring a point cloud sequence; selecting any two non-adjacent frames of point cloud data from the point cloud sequence to obtain a source point cloud and a target point cloud; inputting the source point cloud and the target point cloud into an auxiliary task model for point cloud reconstruction to obtain a reconstructed point cloud; pre-training the encoder of the auxiliary task model based on the feature similarity error between the reconstructed point cloud and the target point cloud to obtain a trained encoder; acquiring a recognition task decoder; combining the recognition task decoder with the trained encoder to obtain an initialized point cloud sequence recognition model; and inputting the point cloud sequence into the initialized point cloud sequence recognition model for training to obtain a trained point cloud sequence recognition model. The embodiments of this invention can reduce the dependence of the point cloud sequence recognition model on manually labeled data, are applicable to various point cloud recognition tasks, and can be widely applied in the field of artificial intelligence technology.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training and recognition method, apparatus, device and medium for a point cloud sequence recognition model. Background Technology

[0002] With the emergence of dynamic point cloud acquisition technologies and equipment, large amounts of raw point cloud sequences can be easily obtained. Supervised learning methods can then be used to capture spatiotemporal information from these sequences. However, supervised learning methods require a significant investment of accurately labeled data. Since point cloud sequences have a more complex temporal and spatial structure than point clouds or images, it is necessary to reduce the reliance of point cloud sequence recognition systems on manually labeled data. Currently, the supervisory information provided by point cloud sequence recognition models in related technologies is relatively limited and not applicable to a wide range of point cloud recognition tasks. In summary, the technical problems existing in these technologies urgently need to be addressed. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a training and recognition method, apparatus, device and medium for a point cloud sequence recognition model, so as to reduce the dependence of the point cloud sequence recognition model on labeled data and improve its applicability to recognition tasks.

[0004] On one hand, the present invention provides a training method for a point cloud sequence recognition model, comprising:

[0005] Obtain a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain the source point cloud and the target point cloud;

[0006] The source point cloud and the target point cloud are input into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. The encoder of the auxiliary task model is pre-trained based on the feature similarity error between the reconstructed point cloud and the target point cloud to obtain the trained encoder.

[0007] Obtain the recognition task decoder, and combine the recognition task decoder with the trained encoder to obtain an initialized point cloud sequence recognition model;

[0008] The point cloud sequence is input into the initialized point cloud sequence recognition model for training, resulting in a trained point cloud sequence recognition model.

[0009] Optionally, the step of inputting the source point cloud and the target point cloud into the auxiliary task model for point cloud reconstruction to obtain the reconstructed point cloud includes:

[0010] The auxiliary task model includes an encoder, a feature transformer, and an auxiliary task decoder;

[0011] The source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping to obtain source point cloud features and target point cloud features.

[0012] The source point cloud features and the target point cloud features are input into the feature transformer for feature transformation processing to obtain the target point cloud prediction features;

[0013] The predicted features of the target point cloud are input into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud.

[0014] Optionally, the source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping processing to obtain source point cloud features and target point cloud features. The encoder includes four processing layers, and the processing steps of each layer include:

[0015] The input data is selected and processed using the farthest point sampling method to obtain the neighborhood center point;

[0016] Based on the neighborhood center point, a target radius is selected for neighborhood construction to obtain the spatiotemporal neighborhood;

[0017] The data within the spatiotemporal neighborhood is processed by local feature extraction, and the extracted local features are concatenated to obtain the output data.

[0018] The target radius of each processing layer is different.

[0019] Optionally, the step of inputting the source point cloud features and the target point cloud features into the feature transformer for feature transformation processing to obtain the target point cloud prediction features includes:

[0020] The source point cloud features are subjected to mean-variance transformation to obtain the transformed features;

[0021] The transformed feature is added to the feature offset to obtain the additive feature;

[0022] The summation features and the target point cloud features are weighted and summed to obtain the target point cloud prediction features.

[0023] Optionally, the step of inputting the predicted features of the target point cloud into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud includes:

[0024] The auxiliary task decoder includes four identical feature transfer layers and one fully connected layer;

[0025] The target point cloud prediction features are reverse-mapped through the feature transfer layer to obtain low-dimensional features.

[0026] The low-dimensional features are processed by coordinate mapping through the fully connected layer to obtain the reconstructed point cloud.

[0027] Optionally, the step of training the point cloud sequence recognition model by inputting the point cloud sequence into the initialized point cloud sequence recognition model to obtain the trained point cloud sequence recognition model specifically includes:

[0028] The point cloud sequence is input into the initialized point cloud sequence recognition model to obtain the point cloud sequence recognition result;

[0029] The training loss value is determined based on the point cloud sequence recognition results and point cloud sequence labels;

[0030] The parameters of the recognition task decoder are updated based on the loss value to obtain a trained point cloud sequence recognition model.

[0031] On the other hand, embodiments of the present invention also provide a point cloud sequence recognition method, including:

[0032] Obtain the point cloud sequence to be identified;

[0033] The point cloud sequence to be identified is input into the point cloud sequence recognition model obtained by the training method of the point cloud sequence recognition model as described above, and the point cloud sequence recognition result is obtained.

[0034] On the other hand, embodiments of the present invention also provide a point cloud sequence recognition device, comprising:

[0035] The first module is used to acquire a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain a source point cloud and a target point cloud.

[0036] The second module is used to input the source point cloud and the target point cloud into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. Based on the feature similarity error between the reconstructed point cloud and the target point cloud, the encoder of the auxiliary task model is pre-trained to obtain the trained encoder.

[0037] The third module is used to obtain the recognition task decoder, and to combine the recognition task decoder with the trained encoder to obtain the initialized point cloud sequence recognition model.

[0038] The fourth module is used to train the point cloud sequence recognition model by inputting the point cloud sequence into the initialization module, so as to obtain the trained point cloud sequence recognition model.

[0039] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory;

[0040] The memory is used to store programs;

[0041] The processor executes the program to implement the method described above.

[0042] On the other hand, embodiments of the present invention also disclose a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0043] On the other hand, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0044] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention:

[0045] The point cloud sequence recognition model training method provided in this embodiment of the invention involves inputting the source point cloud and the target point cloud into an auxiliary task model for point cloud reconstruction to obtain a reconstructed point cloud. The encoder of the auxiliary task model is then pre-trained based on the feature similarity error between the reconstructed point cloud and the target point cloud to obtain a trained encoder. This method enables training of the encoder using a self-supervised approach, reducing the reliance of the point cloud sequence recognition model on manually labeled data. Furthermore, this embodiment of the invention obtains a recognition task decoder and combines it with the trained encoder to obtain an initialized point cloud sequence recognition model. This allows for the combination of decoders from various point cloud sequence recognition tasks, thereby improving the flexibility and generalization ability of the point cloud sequence recognition model. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a training method for a point cloud sequence recognition model provided in an embodiment of this application;

[0048] Figure 2 This is a system architecture diagram of a point cloud sequence recognition model provided in an embodiment of this application;

[0049] Figure 3 This is a network structure diagram of a feature transformer provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] First, let's analyze some of the terms used in this application:

[0052] Point cloud sequence: A point cloud sequence consists of multiple frames of point clouds. A point cloud sequence can be viewed as a collection of points, containing the positional information of target objects at multiple points in time within the scene.

[0053] Self-supervised learning aims to improve a model's feature extraction capabilities by designing auxiliary tasks to extract the inherent representational characteristics of unlabeled data as supervisory information. The supervisory information obtained here refers not to the original task labels used in self-supervised learning, but rather to the constructed auxiliary task labels. The goal of self-supervised recognition systems is to utilize self-supervised learning theories to design intelligent systems that can automatically process point cloud sequences and apply them to tasks including human action recognition, semantic segmentation, and scene flow estimation.

[0054] Features: A common term in the field of computer vision. Refers to vectors or matrices that can represent data.

[0055] In related technologies, methods for identifying point cloud sequences generally involve applying rotational transformations to the point cloud or computing geometric features. In the rotational transformation approach, the rotation angle is limited, and the resulting changes in the position of points within the point cloud due to different angle values ​​may be negligible, with unclear correspondences. This approach provides limited supervisory information, restricting the types of tasks the point cloud recognition model can be applied to. In the point cloud geometric feature approach, the geometric features used are not stable enough; for example, normals are unstable geometric information and easily affected by the offset of some points. Point cloud recognition models trained in this way are sensitive to changes in the position of points within the point cloud and can only be applied to certain point cloud recognition tasks.

[0056] In view of this, refer to Figure 1 This invention provides a training method for a point cloud sequence recognition model, comprising:

[0057] S101. Obtain a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain the source point cloud and the target point cloud.

[0058] S102. Input the source point cloud and the target point cloud into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. Based on the feature similarity error between the reconstructed point cloud and the target point cloud, pre-train the encoder of the auxiliary task model to obtain the trained encoder.

[0059] S103. Obtain the recognition task decoder, and combine the recognition task decoder with the trained encoder to obtain the initialized point cloud sequence recognition model.

[0060] S104. The point cloud sequence is input into the initialized point cloud sequence recognition model for training to obtain the trained point cloud sequence recognition model.

[0061] In this embodiment of the invention, the self-supervised recognition system based on point cloud sequences introduces a new auxiliary task model, which mines some implicit label information from the point cloud sequence data itself to supervise the training of the point cloud recognition model, thereby greatly reducing the dependence of the point cloud sequence recognition model on manually labeled data.

[0062] This invention divides the training method of the point cloud recognition model into two stages: pre-training and formal training. In the pre-training stage, current-frame-future-frame point cloud reconstruction is introduced as an auxiliary task model. The current-frame-future-frame reconstruction task involves extracting any two frames of point cloud data at intervals from a point cloud sequence to obtain the source point cloud (current frame) and the target point cloud (future frame), and reconstructing the point cloud of the next frame using the features of the previous frame. The auxiliary task model in the pre-training stage includes an encoder, a feature transformer, and an auxiliary task decoder. In the formal training stage, the encoder trained in the auxiliary task model is used, and a recognition task decoder for a specific recognition task is introduced to form the point cloud sequence recognition model. After obtaining the point cloud sequence recognition model, it is trained using a small amount of manually labeled data. Since the encoder is already trained, only a small amount of labeled data is needed to train the point cloud sequence recognition model. By changing different recognition task decoders, it can be applied to different point cloud sequence recognition tasks.

[0063] As a further preferred embodiment, in step S102 above, inputting the source point cloud and the target point cloud into the auxiliary task model to perform point cloud reconstruction to obtain the reconstructed point cloud includes:

[0064] The auxiliary task model includes an encoder, a feature transformer, and an auxiliary task decoder;

[0065] The source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping to obtain source point cloud features and target point cloud features.

[0066] The source point cloud features and the target point cloud features are input into the feature transformer for feature transformation processing to obtain the target point cloud prediction features;

[0067] The predicted features of the target point cloud are input into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud.

[0068] In this embodiment of the invention, to ensure the generalization ability of the algorithm, the self-supervised recognition problem of point cloud sequences is modeled as a representation learning problem, that is, the training of the auxiliary task model is supervised by minimizing the feature similarity error between the reconstructed point cloud and the target point cloud. The core of this auxiliary task model lies in the introduction of a feature transformer for the current-to-future-frame reconstruction task of the point cloud sequence. This feature transformer can effectively aggregate the source point cloud features and the target point cloud features based on feature similarity, thereby predicting the point-by-point coordinates of the reconstructed point cloud based on these features. Since there are different algorithmic requirements for the encoded feature map and the decoded feature map, this embodiment of the invention uses heterogeneous encoded feature extraction networks and decoded feature extraction networks to process the features separately. The encoded feature extraction network and the decoded feature extraction network are referred to as the encoder and the auxiliary task decoder, respectively. (Refer to...) Figure 2 In the pre-training phase, the source point cloud and the target point cloud are input into the auxiliary task model for point cloud reconstruction, resulting in a reconstructed point cloud. The auxiliary task model includes an encoder, a feature transformer, and a decoder. The parameters of the auxiliary task model are updated using the chamfer loss between the target point cloud and the reconstructed point cloud as the loss function, thus obtaining the trained encoder.

[0069] Further, as a preferred embodiment, the source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping processing to obtain source point cloud features and target point cloud features. The encoder includes four processing layers, and the processing steps of each layer include:

[0070] The input data is selected and processed using the farthest point sampling method to obtain the neighborhood center point;

[0071] Based on the neighborhood center point, a target radius is selected for neighborhood construction to obtain the spatiotemporal neighborhood;

[0072] The data within the spatiotemporal neighborhood is processed by local feature extraction, and the extracted local features are concatenated to obtain the output data.

[0073] The target radius of each processing layer is different.

[0074] In this embodiment of the invention, the encoder is used to map an arbitrary frame of input point cloud sequence into a high-dimensional feature representation (feature vector). This invention uses a multi-layered point cloud convolutional neural network as the ground network. The encoder in this embodiment can be based on the MeteorNet network for parsing point cloud sequence information. The encoder uses the first four layers of the MeteorNet network. The encoder has a simple structure. The first layer places multiple frames of point clouds in the same space and selects the neighborhood center point using the farthest point sampling method. A spatiotemporal neighborhood is constructed by taking a certain radius outward from the neighborhood center point. Points in the spatiotemporal neighborhood are combined and local features are extracted using a convolutional neural network. Finally, the local features are concatenated to form global features. The second layer repeats the feature extraction steps of the first layer, but uses a larger radius to combine the spatiotemporal neighborhood. The third and fourth layers follow the same pattern. The four layers are stacked one on top of the other to form the encoder.

[0075] Further, as a preferred embodiment, the step of inputting the source point cloud features and the target point cloud features into the feature transformer for feature transformation processing to obtain the target point cloud prediction features includes:

[0076] The source point cloud features are subjected to mean-variance transformation to obtain the transformed features;

[0077] The transformed feature is added to the feature offset to obtain the additive feature;

[0078] The summation features and the target point cloud features are weighted and summed to obtain the target point cloud prediction features.

[0079] In this embodiment of the invention, the feature transformer is used to infer predicted features of the target point cloud based on the source point cloud features and the target point cloud features, thereby enabling subsequent reconstruction of the target point cloud. (Refer to...) Figure 3 In this embodiment of the invention, the feature transformer takes source point cloud features S and target point cloud features T as inputs and outputs predicted features P of the target point cloud. The feature transformer's processing includes two parts: coarse transformation and fine transformation. In the coarse transformation stage, the source point cloud features S are subjected to mean-variance transformation (…). The mean S and variance of the source point cloud features are unified with the mean and variance of the target point cloud features T to obtain the transformed feature C. In the fine transformation stage, a feature offset W is introduced and added (⊕) to the transformed feature C to obtain the additive feature F. The additive feature F is then input into the attention mechanism, which performs a weighted summation of the input additive feature F and the target point cloud features T to obtain the target point cloud prediction feature P. The feature offset W has the same dimension as the additive feature F and is learned by the neural network. During training, the value of the feature offset W is continuously updated to train the optimal feature offset W value. The mean-variance transformation is performed on the source point cloud features S, and the mean-variance transformation equation is shown below:

[0080] ;

[0081] In the formula, m1 is the mean of the source point cloud features, n1 is the variance of the source point cloud features, m2 is the mean of the target point cloud features, and n2 is the variance of the target point cloud features.

[0082] The input additive features F and the target point cloud features T are subjected to a weighted summation process, and the weighted summation equation is shown below:

[0083] ;

[0084] in and It is learned from a neural network model and continuously adjusted based on the model loss. and The value of .

[0085] Further, as a preferred embodiment, the step of inputting the predicted features of the target point cloud into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud includes:

[0086] The auxiliary task decoder includes four identical feature transfer layers and one fully connected layer;

[0087] The target point cloud prediction features are reverse-mapped through the feature transfer layer to obtain low-dimensional features.

[0088] The low-dimensional features are processed by coordinate mapping through the fully connected layer to obtain the reconstructed point cloud.

[0089] In this embodiment of the invention, the decoder is used to compress the spatial size of the feature map to obtain the feature vector. This embodiment can employ the feature transfer layer of a PointNet network. The decoder consists of four feature transfer layers and one fully connected layer (introduced by a convolutional neural network). Each feature transfer layer reverse-maps the input high-dimensional point cloud features to low-dimensional point cloud features. Finally, the fully connected layer maps the low-dimensional point cloud vector output from the fourth feature transfer layer to point-by-point coordinates.

[0090] Further, as a preferred embodiment, the step of training the point cloud sequence recognition model by inputting the point cloud sequence into the initialized point cloud sequence recognition model to obtain the trained point cloud sequence recognition model specifically includes:

[0091] The point cloud sequence is input into the initialized point cloud sequence recognition model to obtain the point cloud sequence recognition result;

[0092] The training loss value is determined based on the point cloud sequence recognition results and point cloud sequence labels;

[0093] The parameters of the recognition task decoder are updated based on the loss value to obtain a trained point cloud sequence recognition model.

[0094] In this embodiment of the invention, the encoder of the auxiliary task model is pre-trained using a self-supervised learning method to obtain a trained encoder. The trained encoder is then recombined with the recognition task decoder to obtain a point cloud sequence recognition model, wherein the recognition task decoder depends on the point cloud sequence recognition task to be processed. In this embodiment, the task may include a 3D scene flow estimation task and a human behavior recognition task. (Refer to...) Figure 2 In this embodiment of the invention, a point cloud sequence with a small amount of labeled data is input into a point cloud sequence recognition model. The point cloud sequence recognition model includes an encoder and a new decoder. The encoder is a reused encoder trained in the aforementioned auxiliary task model, yielding a predicted scene stream. The target scene stream from the point cloud sequence labels is then compared with the predicted scene stream to calculate the training loss value, i.e., the minimum mean square error. Based on the training loss value, the backpropagation algorithm is used to update the model parameters. After several iterations, a trained point cloud sequence recognition model is obtained.

[0095] It is understood that the input to the 3D scene flow estimation task in this embodiment of the invention is a point cloud sequence containing two frames of point clouds, and the output is the 3D motion vector of the corresponding points between the two frames of point clouds. In the 3D scene flow estimation task, this embodiment of the invention uses the last four layers of the FlowNet3D model as the recognition task decoder. This recognition task decoder includes three upper convolutional layers (introduced by the FlowNet3D network) and one fully connected layer. The upper convolutional layer's function is to pass features from a set of points in the previous layer to a set of points in the next layer. The fully connected layer's function is to map the features in the feature space calculated by the encoder back to the sample label space, thereby identifying the category corresponding to the sample. This embodiment of the invention can also be applied to human behavior recognition tasks. In the human behavior recognition task, the input is a point cloud sequence containing multiple frames of point clouds, and the output is the label corresponding to the entire point cloud sequence, representing the type of human behavior contained in the point cloud sequence. This embodiment of the invention can use a recognition task decoder including two fully connected layers (introduced by a convolutional neural network). The fully connected layer maps the features in the feature space calculated by the encoder back to the sample label space, thereby identifying the category corresponding to the sample.

[0096] On the other hand, embodiments of this application also provide a point cloud sequence recognition method, including:

[0097] Obtain the point cloud sequence to be identified;

[0098] The point cloud sequence to be identified is input into the point cloud sequence recognition model obtained by the training method of the point cloud sequence recognition model as described above, and the point cloud sequence recognition result is obtained.

[0099] In this embodiment, after the model is trained, for the point cloud sequence to be identified, a small amount of annotation can be added to the point cloud sequence, and then the point cloud sequence can be input into the point cloud tower recognition model obtained by the aforementioned training method for the point cloud tower recognition model to obtain the point cloud tower recognition result. This embodiment of the invention can train the point cloud sequence recognition model using a small amount of manually labeled data. Since the encoder is already trained, only a small amount of labeled data is needed to train the model for different point cloud sequence recognition tasks. Extensive experimental verification shows that this model only requires 25% of the original labeled data to achieve the performance of a point cloud sequence recognition model trained with all the original data.

[0100] It is understood that the content of the above-mentioned point cloud sequence recognition model training method embodiment is applicable to this point cloud sequence recognition method embodiment. The specific functions implemented by this point cloud sequence recognition method embodiment are the same as those of the above-mentioned point cloud sequence recognition model training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned point cloud sequence recognition model training method embodiment.

[0101] On the other hand, embodiments of the present invention also provide a point cloud sequence recognition device, comprising:

[0102] The first module is used to acquire a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain a source point cloud and a target point cloud.

[0103] The second module is used to input the source point cloud and the target point cloud into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. Based on the feature similarity error between the reconstructed point cloud and the target point cloud, the encoder of the auxiliary task model is pre-trained to obtain the trained encoder.

[0104] The third module is used to obtain the recognition task decoder, and to combine the recognition task decoder with the trained encoder to obtain the initialized point cloud sequence recognition model.

[0105] The fourth module is used to train the point cloud sequence recognition model by inputting the point cloud sequence into the initialization module, so as to obtain the trained point cloud sequence recognition model.

[0106] It is understood that the content of the above-described training method embodiment for point cloud sequence recognition model is applicable to the point cloud sequence recognition device embodiment. The specific functions implemented by the point cloud sequence recognition device embodiment are the same as those of the above-described training method embodiment for point cloud sequence recognition model, and the beneficial effects achieved are also the same as those achieved by the above-described training method embodiment for point cloud sequence recognition model.

[0107] and Figure 1Corresponding to the method described above, this embodiment of the invention also provides an electronic device, including a processor and a memory; the memory is used to store a program; the processor executes the program to implement the method described above.

[0108] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0109] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0110] In summary, the embodiments of the present invention have the following advantages: The model of the embodiments of the present invention can combine new decoders for various point cloud sequence recognition tasks, thereby being used for various point cloud sequence recognition tasks and improving the flexibility and generalization ability of the point cloud sequence recognition model. Furthermore, the embodiments of the present invention can achieve fully supervised model performance under the supervision of all labeled data while using only 25% of the original point cloud sequence labeled data, greatly reducing the model's dependence on manually labeled data.

[0111] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0112] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0113] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0115] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0116] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0117] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0118] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0119] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A training method for a point cloud sequence recognition model, characterized in that, The method includes: Obtain a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain the source point cloud and the target point cloud; The source point cloud and the target point cloud are input into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. The encoder of the auxiliary task model is pre-trained based on the feature similarity error between the reconstructed point cloud and the target point cloud to obtain the trained encoder. Obtain the recognition task decoder, and combine the recognition task decoder with the trained encoder to obtain an initialized point cloud sequence recognition model; The point cloud sequence is input into the initialized point cloud sequence recognition model for training to obtain the trained point cloud sequence recognition model; the point cloud sequence to be recognized in the point cloud sequence recognition task is obtained, and the point cloud sequence to be recognized is input into the trained point cloud sequence recognition model to obtain the point cloud sequence recognition result. The point cloud sequence recognition task includes a 3D scene flow estimation task and a human behavior recognition task. The step of inputting the source point cloud and the target point cloud into the auxiliary task model for point cloud reconstruction to obtain the reconstructed point cloud includes: The auxiliary task model includes an encoder, a feature transformer, and an auxiliary task decoder; The source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping to obtain source point cloud features and target point cloud features. The source point cloud features and the target point cloud features are input into the feature transformer for feature transformation processing to obtain the target point cloud prediction features; The predicted features of the target point cloud are input into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud.

2. The method according to claim 1, characterized in that, The source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping to obtain source point cloud features and target point cloud features. The encoder includes four processing layers, and the processing steps of each layer include: The input data is selected and processed using the farthest point sampling method to obtain the neighborhood center point; Based on the neighborhood center point, a target radius is selected for neighborhood construction to obtain the spatiotemporal neighborhood; The data within the spatiotemporal neighborhood is processed by local feature extraction, and the extracted local features are concatenated to obtain the output data. The target radius of each processing layer is different.

3. The method according to claim 1, characterized in that, The step of inputting the source point cloud features and the target point cloud features into the feature transformer for feature transformation processing to obtain the target point cloud prediction features includes: The source point cloud features are subjected to mean-variance transformation to obtain the transformed features; The transformed feature is added to the feature offset to obtain the additive feature; The summation features and the target point cloud features are weighted and summed to obtain the target point cloud prediction features.

4. The method according to claim 1, characterized in that, The step of inputting the predicted features of the target point cloud into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud includes: The auxiliary task decoder includes four identical feature transfer layers and one fully connected layer; The target point cloud prediction features are reverse-mapped through the feature transfer layer to obtain low-dimensional features. The low-dimensional features are processed by coordinate mapping through the fully connected layer to obtain the reconstructed point cloud.

5. The method according to claim 1, characterized in that, The step of training the point cloud sequence recognition model by inputting the point cloud sequence into the initialization point cloud sequence recognition model to obtain the trained point cloud sequence recognition model specifically includes: The point cloud sequence is input into the initialized point cloud sequence recognition model to obtain the point cloud sequence recognition result; The training loss value is determined based on the point cloud sequence recognition results and point cloud sequence labels; The parameters of the recognition task decoder are updated based on the loss value to obtain a trained point cloud sequence recognition model.

6. A point cloud sequence recognition method, characterized in that, The method includes: Obtain the point cloud sequence to be identified; The point cloud sequence to be identified is input into the point cloud sequence recognition model obtained by the training method of the point cloud sequence recognition model as described in any one of claims 1-5, and the point cloud sequence recognition result is obtained.

7. A point cloud sequence recognition device, characterized in that, The device includes: The first module is used to acquire a point cloud sequence, and select any two non-adjacent frames of point cloud data from the point cloud sequence to obtain a source point cloud and a target point cloud. The second module is used to input the source point cloud and the target point cloud into the auxiliary task model to reconstruct the point cloud and obtain the reconstructed point cloud. Based on the feature similarity error between the reconstructed point cloud and the target point cloud, the encoder of the auxiliary task model is pre-trained to obtain the trained encoder. The third module is used to obtain the recognition task decoder, and combine the recognition task decoder with the trained encoder to obtain the initialized point cloud sequence recognition model. The fourth module is used to train the point cloud sequence into the initialized point cloud sequence recognition model to obtain the trained point cloud sequence recognition model; to obtain the point cloud sequence to be recognized in the point cloud sequence recognition task, to input the point cloud sequence to be recognized into the trained point cloud sequence recognition model to obtain the point cloud sequence recognition result, wherein the point cloud sequence recognition task includes a three-dimensional scene flow estimation task and a human behavior recognition task. The step of inputting the source point cloud and the target point cloud into the auxiliary task model for point cloud reconstruction to obtain the reconstructed point cloud includes: The auxiliary task model includes an encoder, a feature transformer, and an auxiliary task decoder; The source point cloud and the target point cloud are input into the encoder for high-dimensional feature representation mapping to obtain source point cloud features and target point cloud features. The source point cloud features and the target point cloud features are input into the feature transformer for feature transformation processing to obtain the target point cloud prediction features; The predicted features of the target point cloud are input into the auxiliary task decoder for feature compression processing to obtain the reconstructed point cloud.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store programs; The processor executes the program to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images