Model pre-training and training method, point cloud detection and segmentation method and device

Abstract

The present disclosure provides a model pre-training and training method, as well as a point cloud detection and segmentation method and apparatus, relating to the field of artificial intelligence technology, particularly computer vision, deep learning, and large models, and is intended for use in scenarios such as autonomous driving and intelligent transportation. A specific implementation scheme comprises: determining, from image data, a first image feature that is semantically aligned with a mask voxel; using the first image feature as a mask reconstruction target, and adjusting parameters of a first point cloud model so that the mask reconstruction features obtained by reconstructing the mask voxels using the adjusted first point cloud model in the point cloud voxel space match the mask reconstruction target.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly computer vision, deep learning, and large models, and is used in scenarios such as autonomous driving and intelligent transportation. The present disclosure specifically relates to a point cloud model pre-training and training method, a point cloud detection and segmentation method, device, electronic device, and storage medium. Background Art

[0002] In smart city scenarios, such as intelligent transportation and autonomous driving, vehicles or roadside perception devices acquire point cloud data and image data from the environment, and then use this data to perform operations such as traffic guidance and autonomous driving assistance. The degree of semantic alignment between the point cloud data and its corresponding image data influences the assistance operations in these scenarios. Summary of the Invention

[0003] The present disclosure provides a point cloud model pre-training and training method, a point cloud detection and segmentation method, an apparatus, an electronic device, and a storage medium.

[0004] According to one aspect of the present disclosure, a point cloud model pre-training method is provided, comprising:

[0005] For a mask voxel in a point cloud voxel space, determining, from image data corresponding to the point cloud voxel space, a first image feature that is semantically aligned with the mask voxel;

[0006] Using the first image feature as a mask reconstruction target, adjusting parameters of the first point cloud model so that the mask reconstruction feature matches the mask reconstruction target;

[0007] The mask reconstruction feature is obtained by reconstructing the mask voxels using the point cloud voxel space using the adjusted first point cloud model.

[0008] According to another aspect of the present disclosure, a point cloud model training method is provided, comprising:

[0009] Based on the point cloud samples and the annotation information of the point cloud samples, the second point cloud model is trained to obtain a third point cloud model, wherein the second point cloud model is a model obtained by pre-training according to the point cloud model pre-training method described in any one of the embodiments of the present disclosure.

[0010] According to another aspect of the present disclosure, a point cloud detection method is provided, comprising:

[0011] The target point cloud is detected according to the third point cloud model to obtain a detection result of the target point cloud, wherein the third point cloud model is a model trained according to the point cloud model training method described in any one of the embodiments of the present disclosure.

[0012] According to another aspect of the present disclosure, a point cloud segmentation method is provided, comprising:

[0013] The target point cloud is segmented according to the fourth point cloud model to obtain a segmentation result of the target point cloud, wherein the fourth point cloud model is a model trained according to the point cloud model training method described in any one of the embodiments of the present disclosure.

[0014] According to another aspect of the present disclosure, a point cloud model pre-training device is provided, comprising:

[0015] An image feature determination module is configured to determine, for a mask voxel in a point cloud voxel space, a first image feature semantically aligned with the mask voxel from image data corresponding to the point cloud voxel space;

[0016] a model parameter adjustment module, configured to use the first image feature as a mask reconstruction target and adjust parameters of the first point cloud model so that the mask reconstruction feature matches the mask reconstruction target;

[0017] The mask reconstruction feature is obtained by reconstructing the mask voxels using the point cloud voxel space using the adjusted first point cloud model.

[0018] According to another aspect of the present disclosure, a point cloud model training device is provided, wherein the device is used to:

[0019] Based on the point cloud samples and the annotation information of the point cloud samples, the second point cloud model is trained to obtain a third point cloud model, wherein the second point cloud model is a model obtained by pre-training according to the point cloud model pre-training method described in any one of the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a point cloud detection device is provided, wherein the device is used to detect a target point cloud based on a third point cloud model to obtain a detection result of the target point cloud, wherein the third point cloud model is a model trained by the point cloud model training method described in any one of the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a point cloud segmentation device is provided, wherein the device is used to segment the target point cloud according to a fourth point cloud model to obtain a segmentation result of the target point cloud, wherein the fourth point cloud model is a model trained by the point cloud model training method according to any one of the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any point cloud model pre-training method, any point cloud model training method, any point cloud detection method or any point cloud segmentation method in the embodiments of the present disclosure.

[0026] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any point cloud model pre-training method, any point cloud model training method, any point cloud detection method, or any point cloud segmentation method according to the embodiments of the present disclosure.

[0027] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any point cloud model pre-training method, any point cloud model training method, any point cloud detection method, or any point cloud segmentation method according to the embodiments of the present disclosure.

[0028] According to the technology disclosed in the present invention, for the mask voxels in the point cloud voxel space, a first image feature that is semantically aligned with the mask voxels is determined from the image data corresponding to the point cloud voxel space. Thus, the first image feature that is semantically aligned with the mask voxels is used as the mask reconstruction target, and the parameters of the point cloud model are continuously adjusted until the mask reconstruction features obtained by reconstructing the mask voxels using the point cloud volume space of the adjusted point cloud model match the mask reconstruction target, and the adjustment is stopped, thereby obtaining a pre-trained point cloud model. In this way, the pre-trained point cloud model can achieve alignment of the point cloud features and the corresponding image features in the semantic space when processing the point cloud data. The target point cloud model obtained by subsequent training of the point cloud model can achieve better point cloud data processing effects, such as improving the accuracy of point cloud detection or point cloud segmentation.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0031] Figure 1 is a schematic diagram of an autonomous driving scenario disclosed herein;

[0032] Figure 2 is a flowchart of a point cloud model pre-training method according to an embodiment of the present disclosure;

[0033] Figure 3 is a flowchart of a point cloud model pre-training method according to another embodiment of the present disclosure;

[0034] Figure 4 This is a structural block diagram of a point cloud model pre-training device according to an embodiment of the present disclosure;

[0035] Figure 5 is a structural block diagram of a point cloud model pre-training device according to another embodiment of the present disclosure;

[0036] Figure 6 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] Figure 1 is a schematic diagram of the autonomous driving scenario disclosed herein. Figure 1 As shown, in an autonomous driving scenario, server 11 pre-trains the point cloud model provided by the present disclosure using calibrated and clock-synchronized point cloud data and image data to obtain a pre-trained point cloud model. Server 11 then distributes this point cloud model to various downstream task nodes, such as vehicle 12. Each downstream task node fine-tunes the point cloud model based on a small amount of required point cloud sample data, for example, by specifically training the pre-trained point cloud model to obtain the desired point cloud model. Finally, the downstream task node uses the fine-tuned point cloud model to process the point cloud data. For example, it can identify or segment objects, such as people or objects, within the point cloud data.

[0039] The application of the embodiments of the present disclosure in the above-mentioned autonomous driving scenario is merely an example. The embodiments of the present disclosure can also be applied to other traffic scenarios, such as parking scenarios, which will not be described in detail here.

[0040] Figure 2The flowchart of the point cloud model training method according to one embodiment of the present disclosure is provided. The method can be applied to electronic devices. The electronic device can be, for example, a terminal, a server, or other processing device. The terminal can be a user equipment (UE) such as a desktop computer, a mobile device, a PDA (Personal Digital Assistant), a handheld device, a computing device, an in-vehicle device, or a wearable device. In some implementations, the electronic device can implement the point cloud model pre-training method according to the embodiment of the present disclosure by invoking computer-readable instructions stored in a memory by a processor.

[0041] like Figure 2 As shown, the point cloud model pre-training method may include:

[0042] S210 , for a mask voxel in the point cloud voxel space, determining a first image feature that is semantically aligned with the mask voxel from image data corresponding to the point cloud voxel space;

[0043] S220 , using the first image feature as a mask reconstruction target, and adjusting parameters of the first point cloud model so that the mask reconstruction feature matches the mask reconstruction target.

[0044] The mask reconstruction features are obtained by reconstructing mask voxels using the point cloud voxel space of the adjusted first point cloud model.

[0045] Before the above-mentioned pre-training of the point cloud model, it is necessary to obtain calibrated and clock-synchronized point cloud data and image data. Then, the point cloud data is voxelized to obtain a first voxel space, and some voxels in the first voxel space are masked to obtain a point cloud voxel space. The point cloud voxel space includes one or more voxels, some of which are mask voxels and the other part are non-masked voxels. The correspondence between the point cloud voxel space and the image data refers to the data relationship between the point cloud data and the image data used to generate the point cloud voxel space, which are calibrated and clock-synchronized.

[0046] Point cloud data refers to a collection of vectors in a three-dimensional coordinate system. Data is recorded as points, each of which contains three-dimensional coordinates and can carry additional information about its properties, such as color, reflectivity, and intensity. Point cloud data is typically acquired from real-world scenes using devices such as laser scanners and 3D scanners. It can be used in applications such as 3D modeling, scene reconstruction, robotic navigation, and virtual and augmented reality.

[0047] Image data refers to the grayscale values ​​of each pixel expressed numerically. Image data can be acquired by devices such as cameras.

[0048] It can be understood that the semantic alignment of the mask voxel and the first image feature means that the feature corresponding to the mask voxel is semantically similar to the first image feature.

[0049] For example, a Contrastive Language–Image Pre-training (CLIP) model can be used to extract semantically rich image features from image data, and then a first image feature that is semantically aligned with the mask voxel is determined from the image features.

[0050] It can be understood that the CLIP model is a model generated by pre-training the CLIP model using text samples and image samples, wherein the text samples and image samples have corresponding semantic relationships.

[0051] Exemplarily, the parameters of the first point cloud model are adjusted, and the parameters of the first point cloud model can be adjusted once or multiple times. After each adjustment, the following operations are performed: using the adjusted first point cloud model, the mask voxels are reconstructed based on the point cloud voxel space to obtain mask reconstruction features, and the mask reconstruction features are matched with the mask reconstruction target; if the mask reconstruction features match the mask reconstruction target, the adjustment of the first point cloud model is stopped; if the mask reconstruction features do not match the mask reconstruction target, the process returns to continue adjusting the parameters of the first point cloud model.

[0052] Among them, the process of adjusting the parameters of the first point cloud model may include: using the first point cloud model to reconstruct the mask voxels based on the point cloud voxel space to obtain mask reconstruction features, and adjusting the parameters of the first point cloud model based on the mask reconstruction features and the mask reconstruction target.

[0053] For example, a loss function can be used to calculate the loss value between the mask reconstruction feature and the mask reconstruction target. If the loss value meets the set conditions, for example, the loss value is less than a set threshold, then the mask reconstruction feature can be considered to match the mask reconstruction target. If the loss value does not meet the set conditions, then it can be considered that the mask reconstruction feature does not match the mask reconstruction target, that is, the mask reconstruction feature has not reached the mask reconstruction target, and it is necessary to continue adjusting the parameters of the first point cloud model.

[0054] For example, the Euclidean distance can be used to calculate the similarity between the mask reconstruction feature and the mask reconstruction target. If the similarity meets the set conditions, for example, the similarity is greater than a set threshold, the mask reconstruction feature and the mask reconstruction target can be considered to match. If the similarity does not meet the set conditions, for example, the similarity is less than a set threshold, the mask reconstruction feature and the mask reconstruction target can be considered to not match.

[0055] Exemplarily, the above-mentioned mask reconstruction features and mask reconstruction targets can be used to construct a loss function, and the gradient information in the loss function can be used to adjust the parameters of the first point cloud model.

[0056] It can be understood that the first point cloud model after the above adjustment is used as the target point cloud model generated by pre-training.

[0057] Exemplarily, the above method may further include: for non-masked voxels in the point cloud voxel space, determining a second image feature that is semantically aligned with the non-masked voxel from the image data corresponding to the point cloud voxel space. Thus, using the first image feature as the masked reconstruction target and the second image feature as the non-masked target feature, the first point cloud model is parameter-adjusted so that the masked reconstruction feature matches the masked reconstruction target and the non-masked feature matches the non-masked target feature. The non-masked feature is obtained by encoding the non-masked voxels in the adjusted first point cloud model.

[0058] According to the above embodiment, for a mask voxel in the point cloud voxel space, a first image feature semantically aligned with the mask voxel is determined from the image data corresponding to the point cloud voxel space. Thus, the first image feature semantically aligned with the mask voxel is used as the mask reconstruction target, and the parameters of the point cloud model are continuously adjusted until the mask reconstruction features obtained by reconstructing the mask voxels using the point cloud volume space of the adjusted point cloud model match the mask reconstruction target, thereby obtaining a pre-trained point cloud model. In this way, the pre-trained point cloud model can achieve alignment of point cloud features with corresponding image features in the semantic space when processing point cloud data. The target point cloud model obtained by subsequent training of the point cloud model can achieve better point cloud data processing results.

[0059] The following is an example of generating a point cloud voxel space of the present disclosure, specifically as follows:

[0060] In an exemplary embodiment, the above method may further include: voxelizing the point cloud data to obtain a first voxel space; masking some voxels in the first voxel space to obtain a point cloud voxel space, wherein some voxels in the point cloud voxel space are mask voxels.

[0061] The relationship between the point cloud data and the aforementioned image data is a calibrated and clock-synchronized correspondence.

[0062] It can be understood that the first voxel space includes one or more voxels. The first voxel space is a point cloud voxel space that has not been masked. The voxel format can be a stereo (Voxel) voxel or a voxel that is not voxelized (Pillar) in height. If the voxel format is a voxel format, each voxel is a three-dimensional structure, and the first voxel space can be a three-dimensional structure or a four-dimensional structure. If the voxel format is a pillar format, each voxel is a two-dimensional structure, and the first voxel space can be a two-dimensional or three-dimensional structure. Among them, the two-dimensional structure can be understood as each voxel height of the three-dimensional structure is 1.

[0063] Exemplarily, the first voxel space and the point cloud voxel space may be four-dimensional structures. For example, the voxel space includes a plurality of parallel voxel channels, each voxel channel includes a plurality of continuous voxels, and each voxel is a three-dimensional voxel.

[0064] For example, the voxels in the first voxel space can be divided into two parts, the first part of the voxels are masked, and the second part of the voxels remain unchanged. The voxels in the first voxel space can be divided into two parts randomly, or the voxels in the first voxel space can be divided into two parts according to a set rule, and then the masking operation is performed to improve masking efficiency.

[0065] For example, the first voxel space is divided into a plurality of sequentially arranged voxel units, and a masking operation is performed on the voxel units at odd positions, or on the voxel units at even positions, thereby obtaining a point cloud voxel space. The point cloud voxel space is a masked voxel space.

[0066] According to the above embodiment, the point cloud data is voxelized to obtain a corresponding voxel space, and then a mask operation is performed on some voxels in the voxel space to obtain a masked voxel space, that is, the point cloud voxel space of the embodiment of the present disclosure.

[0067] In an exemplary embodiment, masking a portion of voxels in a first voxel space to obtain a point cloud voxel space includes: when the first voxel space is a four-dimensional structure, converting the first voxel space into a two-dimensional second voxel space, wherein the second voxel space includes at least one voxel channel, and a plurality of voxels in each voxel channel are divided into a first portion of voxels and a second portion of voxels; masking each first portion of voxels in the second voxel space to obtain a third voxel space, wherein the first portion of voxels in the third voxel space are masked voxels and the second portion of voxels in the third voxel space are non-masked voxels; and converting the third voxel space into a point cloud voxel space with a four-dimensional structure.

[0068] Exemplarily, the structure of the first voxel space is H*W*D*C, where H, W, and D represent the height, width, and depth of the voxel space, respectively, and C represents the number of voxel channels included in the first voxel space, and C is a positive integer. If the voxels adopt the Pillar format, H is 1.

[0069] Then, the first voxel space is converted into a second voxel space having a structure of N*C, where N is a positive integer greater than 1 and represents the number of voxels included in each voxel channel. The voxels in each voxel channel are divided into a first portion of voxels and a second portion of voxels, where the number of voxels in the first portion of voxels is N1 and the number of voxels in the second portion of voxels is N2, where N=N1+N2, and N1 and N2 are positive integers.

[0070] The first part of the voxels in each voxel channel in the second voxel space is masked, and the masked second voxel space is used as the third voxel space, each first part of the voxels in the third voxel space is a masked voxel, and each second part of the voxels in the third voxel space is a non-masked voxel.

[0071] Finally, the third voxel space is transformed into a point cloud voxel space with the same structure as the first voxel space.

[0072] According to the above embodiment, the voxel space of the four-dimensional structure is converted into the voxel space of the two-dimensional structure. Then, the first part of the voxels of each voxel channel in the two-dimensional structure is masked, while the second part of the voxels remains unchanged. This can improve the efficiency of the masking. In addition, it is more convenient when locating the masked voxels and the unmasked voxels. Finally, the masked two-dimensional voxel space is converted back to the four-dimensional structure. In this way, the structure of the voxel space remains unchanged after the masking operation, avoiding other problems when the point cloud model processes it.

[0073] The following describes the process of determining the mask reconstruction target according to an embodiment of the present disclosure, with a specific example as follows:

[0074] In an exemplary embodiment, for a mask voxel in a point cloud voxel space, a first image feature semantically aligned with the mask voxel is determined from image data corresponding to the point cloud voxel space, including: inputting the image data corresponding to the point cloud voxel space into a CLIP model to obtain a feature map output by the CLIP model; for the mask voxel in the point cloud voxel space, determining a first image feature semantically aligned with the mask voxel in the feature map.

[0075] It is understood that the CLIP model can be a pre-trained model. The CLIP model is trained using image samples and annotation information with the same textual semantics as the image samples, resulting in a trained CLIP model. The annotation information is image features or feature maps. Thus, the trained CLIP model can extract semantically rich image features or feature maps from the image data. Furthermore, a first image feature that is semantically aligned with the mask voxel can be extracted from the semantically rich image features or feature maps, and this first image feature can be used as a mask reconstruction target for reconstructing the mask voxel.

[0076] Exemplarily, the mask voxel can be projected into the feature map to obtain the projection position of the mask voxel in the feature map, and the first image feature obtained by using the image feature at the projection position and / or the image feature near the projection position is used as the image feature semantically aligned with the mask voxel.

[0077] For example, the projection method may be orthographic projection or projection with a set angle or range.

[0078] Exemplarily, the internal and external parameter matrices may be used to project the mask voxels onto the feature map, thereby obtaining the projection positions of the mask voxels on the feature map.

[0079] According to the above embodiment, the CLIP model can be used to extract semantically rich feature maps from image data corresponding to the point cloud voxel space. Thus, the first image features semantically aligned with the mask voxels can be extracted from the semantically rich image features or feature maps, and used as the mask reconstruction target for reconstructing the mask voxels, thereby enabling the trained point cloud model to have semantic alignment performance.

[0080] In an exemplary embodiment, the above-mentioned determination of the first image features semantically aligned with the mask voxels in the feature map includes: projecting each first point cloud in the mask voxel into the feature map respectively to obtain the projection position of each first point cloud in the feature map; for any first point cloud, based on the projection position of the first point cloud in the feature map, performing interpolation calculation on the corresponding features in the feature map to obtain the image features corresponding to the first point cloud; performing mean calculation on the image features corresponding to each first point cloud to obtain the first image features semantically aligned with the mask voxel.

[0081] It is understandable that the mask voxel includes multiple original first point clouds. During projection, the coordinates of the first point cloud are projected into the coordinate system of the feature map to obtain the projection position of the first point cloud in the feature map. This projection position can be understood as the coordinate in the feature map coordinate system.

[0082] Exemplarily, the projection method can be an orthographic projection onto the feature map or a projection with a certain angle and / or range on the feature map. For example, the projection is performed at an oblique angle of 85 degrees to the plane where the feature map is located, and the projection point is used as the above-mentioned projection position. Alternatively, the projection is performed at an oblique angle of 75 degrees to the plane where the feature map is located, and the projection point is used as the center to enlarge and reach a set range to obtain the above-mentioned projection position.

[0083] Regarding the calculation method of the image features corresponding to the first point cloud, there are actually the following implementation methods:

[0084] In one example, for any first point cloud, the image feature at the projection position corresponding to the first point cloud in the feature map is used as the image feature corresponding to the first point cloud.

[0085] In one example, for any first point cloud, interpolation calculation is performed on image features near a projection position corresponding to the first point cloud in the feature map to obtain image features corresponding to the first point cloud.

[0086] In one example, for any first point cloud, mean calculation is performed on image features near a projection position corresponding to the first point cloud in the feature map to obtain image features corresponding to the first point cloud.

[0087] In one example, for any first point cloud, a median is determined among image features near a projection position corresponding to the first point cloud in the feature map, and the median is used as the image feature of the first point cloud.

[0088] For example, since the mask voxel includes multiple original first point clouds, the image features corresponding to the multiple first point clouds can be averaged, and the calculated result can be used as the first image feature semantically aligned with the mask voxel. Furthermore, the first image feature is used as the target for mask reconstruction of the mask voxel.

[0089] According to the above embodiment, when there are multiple first point clouds in the mask voxel, the image features corresponding to each first point cloud are determined respectively using the projection position of each first point cloud on the feature map, thereby performing mean calculation on the image features corresponding to each first point cloud to obtain representative first image features that can serve as mask reconstruction targets.

[0090] In an exemplary embodiment, based on the projection position of the first point cloud in the feature map, interpolation calculation is performed on the corresponding features in the feature map to obtain image features corresponding to the first point cloud, including: determining a first feature area in the feature map based on the projection position of the first point cloud in the feature map; and interpolation calculation is performed on each feature in the first feature area to obtain image features corresponding to the first point cloud.

[0091] Exemplarily, the distances between the positions of the features in the first feature region and the projection positions of the mask voxels all satisfy the first condition.

[0092] For example, in the feature map, a circle is drawn with the projection position as the center point, and the circle is used as the first feature region. Alternatively, in the feature map, a regular polygon is drawn with the projection position as the center point, and the regular polygon is used as the first feature region. For example, in the feature map, a square is drawn with the projection position as the center point, and the square is used as the first feature region.

[0093] According to the above embodiment, the projection position of the first point cloud in the feature map is used to determine the first feature area including the projection position in the feature map, so as to perform interpolation calculation on each feature in the first feature area, and image features that are semantically aligned with the first point cloud can be accurately obtained.

[0094] In an exemplary embodiment, the first image feature is used as a mask reconstruction target, and the parameters of the first point cloud model are adjusted, including: reconstructing the mask voxels based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features; and adjusting the parameters of the first point cloud model based on the mask reconstruction features and the mask reconstruction target.

[0095] It can be understood that the mask reconstruction feature is to reconstruct the mask voxels to obtain the point cloud feature. The reconstruction process may include an encoding process and a decoding process.

[0096] It can be understood that based on the point cloud voxel space and the first point cloud model, reconstructing the mask voxels to obtain the mask reconstruction features includes: inputting the point cloud voxel space into the first point cloud model so that the first point cloud model uses the non-mask voxels in the point cloud voxel space to reconstruct the mask voxels and outputs the mask reconstruction features, thereby obtaining the mask reconstruction features. Alternatively, the point cloud voxel space is encoded to obtain non-mask features corresponding to the non-mask voxels and mask features corresponding to the mask voxels; then, the non-mask features and the mask features are input into the first point cloud model so that the first point cloud model uses the non-mask features to reconstruct the mask features to obtain the mask reconstruction features, and outputs the mask reconstruction features.

[0097] It is understood that adjusting parameters of the first point cloud model based on the masked reconstruction features and the masked reconstruction target includes adjusting parameters of the first point cloud model using gap information between the masked reconstruction features and the masked reconstruction target. The gap information may be the similarity between the masked reconstruction features and the masked reconstruction target. Alternatively, the gap information may be the degree of loss between the masked reconstruction features and the masked reconstruction target.

[0098] It is understood that the parameters of the first point cloud model can be adjusted once or multiple times until the mask reconstruction features reconstructed by the adjusted first point cloud model match the mask reconstruction target. Otherwise, the process returns to the reconstruction step and parameter adjustment step based on the adjusted first point cloud model. The reconstruction step refers to reconstructing the mask voxels based on the point cloud voxel space and the first point cloud model to obtain the mask reconstruction features. The parameter adjustment step refers to adjusting the parameters of the first point cloud model based on the mask reconstruction features and the mask reconstruction target.

[0099] According to the above embodiment, mask voxels are reconstructed using the first point cloud model and non-masked voxels in the point cloud voxel space to obtain masked reconstruction features. The gap information between the masked reconstruction features and the masked reconstruction target can then be used to accurately adjust the parameters of the first point cloud model so that the point cloud features reconstructed by the first point cloud model approach the masked reconstruction target. Since the masked reconstruction target is an image feature that is semantically aligned with the masked voxel, the point cloud features reconstructed by the first point cloud model can be semantically aligned with the corresponding image features.

[0100] In an exemplary embodiment, the first point cloud model includes a point cloud encoder and a point cloud decoder, and reconstructs the mask voxels based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features, including: encoding the point cloud voxel space using the point cloud encoder to obtain encoding features; extracting the encoding features corresponding to the mask voxels in the point cloud voxel space as mask features from the encoding features, and extracting the encoding features corresponding to the non-mask voxels in the point cloud voxel space as non-mask features; decoding the mask features based on the point cloud decoder and the non-mask features to obtain mask reconstruction features.

[0101] It can be understood that the point cloud voxel space is input into the point cloud encoder to obtain the encoding features output by the point cloud encoding. Then, the mask features corresponding to the mask voxels and non-mask voxels are extracted from the encoding features. Since the point cloud voxel space can mask voxels for the same part (the first part of voxels) in each voxel channel in the aforementioned masking operation, it is only necessary to extract the encoding features corresponding to the first part of the voxels from the encoding features to obtain the mask features, and the remaining part of the encoding features is the non-mask feature. In this way, the efficiency of extracting mask features and non-mask features from the encoding features can be improved.

[0102] Of course, the mask voxels and non-mask voxels in the point cloud voxel space can also be input into the point cloud encoder respectively to obtain the encoding features (mask features) corresponding to the mask voxels and the encoding features (non-mask features) corresponding to the non-mask voxels.

[0103] For example, if the voxel format is voxel, the point cloud encoder uses 3D convolutional layers, and the point cloud decoder also uses 3D convolutional layers. If the voxel format is Pillar, the point cloud encoder uses 2D convolutional layers, and the point cloud decoder also uses 2D convolutional layers. The point cloud encoder and decoder can also use the Transformer structure.

[0104] According to the above embodiment, a point cloud encoder and a point cloud decoder are used to encode and decode mask voxels based on non-mask features, so that the mask reconstruction features can be accurately obtained, and the parameters of the point cloud model with the point cloud encoder and the point cloud decoder as the main structure can be adjusted. Therefore, one or both of the trained point cloud encoders and point cloud decoders can be selected and sent to downstream tasks for point cloud processing, such as point cloud recognition or classification.

[0105] In an exemplary embodiment, decoding the masked features based on the point cloud decoder and the non-masked features to obtain the masked reconstructed features may include: inputting the non-masked features into the point cloud decoder, so that the point cloud decoder decodes the masked features based on the non-masked features to obtain the masked reconstructed features. The point cloud decoder may construct key-value pair information based on the non-masked features, then use the masked features as query information to search for corresponding value information from the key-value pair information, thereby using the encoded features corresponding to the value information as the masked reconstructed features.

[0106] In an exemplary embodiment, the above-mentioned point cloud decoder and non-mask features are used to decode the mask features to obtain the mask reconstruction features, including: constructing the key-value pair information of the point cloud decoder based on the non-mask features; inputting the key-value pair information and the mask features into the point cloud decoder, so that the point cloud decoder uses the key-value pair information to decode the mask features and output the decoded features; and determining the mask reconstruction features based on the decoded features output by the point cloud decoder.

[0107] After obtaining the mask reconstruction feature, a linear layer can be used to adjust the vector length of the mask reconstruction feature so that its vector length is the same as (aligned with) the vector length of the mask reconstruction target. Then, the adjusted mask reconstruction feature and mask reconstruction target are used to adjust the parameters of the first point cloud model.

[0108] According to the above two implementations, non-masked features can be used as key-value pair information. Then, the decoder can use the masked features as the query object (also called key information) to search for the value information corresponding to the query object from the key-value pair information, thereby accurately obtaining the masked reconstruction features.

[0109] In an exemplary embodiment, parameters of the first point cloud model are adjusted based on the mask reconstruction features and the mask reconstruction target, including: determining a target loss function based on the mask reconstruction features and the mask reconstruction target; and adjusting parameters of the first point cloud model based on the target loss function.

[0110] It should be noted that if the vector lengths of the masked reconstruction feature and the masked reconstruction target are inconsistent, the linear layer can be used to adjust the vector length of the masked reconstruction feature so that its vector length is the same as (aligned with) the vector length of the masked reconstruction target.

[0111] It can be understood that a loss function is constructed using the mask reconstruction features and the mask reconstruction target, and the loss function is used as the target loss function. Alternatively, the loss function is fine-tuned to obtain an adjusted loss function, and the adjusted loss function is used as the target loss function.

[0112] For example, the objective loss function can be as follows:

[0113] L all =||f m -f tm ||2

[0114] Among them, f m Characterize the mask reconstruction feature, f tm Characterize the mask reconstruction target. Since the mask reconstruction feature is the reconstructed point cloud feature, and the mask reconstruction target is the image feature, the target loss function can be used to characterize the degree of semantic alignment between the reconstructed point cloud feature and its corresponding image feature. In this case, the value of the target loss function can be considered as the reconstruction loss.

[0115] It is understandable that the parameters of the first point cloud model are adjusted based on the gradient information of the target loss function. For example, the adjustment direction and adjustment step size of the model parameters are determined according to the gradient.

[0116] According to the above embodiment, a target loss function is constructed using mask reconstruction features and mask reconstruction targets. This target loss function reflects the gap information between the mask reconstruction features and the mask reconstruction targets. Thus, the target loss function is used to adjust the parameters of the point cloud model, so that the mask reconstruction features reconstructed by the adjusted point cloud model can be close to the mask reconstruction target.

[0117] In an exemplary embodiment, a target loss function is determined based on the mask reconstruction features and the mask reconstruction target, including: determining a first loss function based on the mask reconstruction features and the mask reconstruction target; for non-masked voxels in the point cloud voxel space, determining a second image feature that is semantically aligned with the non-masked voxels in the feature map; determining a second loss function based on the non-masked features and the second image features; and determining a target loss function based on the first loss function and the second loss function.

[0118] It should be noted that if the vector length of the masked reconstructed feature is inconsistent with the vector length of the masked reconstructed target, a linear layer can be used to adjust the vector length of the masked reconstructed feature so that its vector length is the same as (aligned with) the vector length of the masked reconstructed target. If the vector length of the non-masked feature is inconsistent with the vector length of the second image feature, a linear layer can be used to adjust the vector length of the non-masked feature.

[0119] Exemplarily, the non-masked voxel is projected into the feature map to obtain the projection position of the non-masked voxel in the feature map, and the second image feature obtained by using the image feature at the projection position and / or the image feature near the projection position is used as the image feature semantically aligned with the non-masked voxel.

[0120] For example, the projection method may be orthographic projection or projection with a set angle or range.

[0121] Exemplarily, the non-masked voxels are projected onto the feature map using the internal and external parameter matrices, thereby obtaining the projection positions of the non-masked voxels into the feature map.

[0122] It can be understood that the first loss function is a reconstruction loss describing the difference between the mask reconstruction feature and the mask reconstruction target. The first loss function is used to characterize the degree of semantic alignment between the reconstructed point cloud feature and its corresponding image feature.

[0123] It can be understood that the second loss function describes the distillation loss between the unmasked encoded features (abbreviated as unmasked features) obtained by encoding unmasked voxels and the encoded target features (i.e., the second image features). The encoded target features refer to the image features that are aligned with the unmasked voxels in the feature map. Therefore, the second loss function is used to characterize the degree of semantic alignment between the encoded unmasked features and their corresponding image features.

[0124] It can be understood that the target loss function is obtained by minimizing the sum of the first loss function and the second loss function. Alternatively, the target loss function can be obtained by summing or weighted summing the first loss function and the second loss function. Thus, in this embodiment, the target loss function is used to characterize the degree of semantic alignment between the point cloud features reconstructed by the point cloud model and their corresponding image features, as well as the degree of semantic alignment between the non-masked features encoded by the point cloud model and their corresponding image features.

[0125] For example, the objective loss function can be formulated as follows:

[0126] L all =||f m -f tm ||2+λ||f um -f tum ||2

[0127] Among them, f m Characterize the mask reconstruction feature, f tm Characterize the mask reconstruction target (first image feature), f um Characterize the non-masked features, f tum represents the non-mask target feature (second image feature), and λ represents the weight.

[0128] According to the above embodiment, a first loss function is constructed using masked reconstruction features and masked reconstruction targets, so that the first loss function can characterize the degree of semantic alignment between the point cloud features reconstructed by the point cloud model and their corresponding image features. A second loss function is constructed using non-masked features and second image features, so that the second loss function can characterize the degree of semantic alignment between the non-masked features encoded by the point cloud model and their corresponding image features. Thus, the objective function determined using the first loss function and the second loss function can simultaneously characterize the above two degrees of semantic alignment, that is, simultaneously reflect the reconstruction loss and the distillation loss. In this way, using the gradient information of these two losses to adjust the parameters of the point cloud model is more accurate and more efficient.

[0129] The following is a specific example of obtaining semantically aligned second image features from feature maps for non-masked voxels:

[0130] In an exemplary embodiment, the point cloud voxel space includes multiple non-masked voxels, and the above-mentioned determination of the second image features semantically aligned with the non-masked voxels in the feature map includes: projecting each second point cloud in the non-masked voxel into the feature map respectively to obtain the projection position of each second point cloud in the feature map; for any second point cloud, based on the projection position of the second point cloud in the feature map, interpolating the corresponding features in the feature map to obtain the image features corresponding to the second point cloud; and performing mean calculation on the image features corresponding to each second point cloud to obtain the second image features semantically aligned with the non-masked voxels.

[0131] It is understandable that the non-masked voxels include multiple original second point clouds. During projection, the coordinates of the second point cloud are projected into the coordinate system of the feature map to obtain the projected position of the second point cloud in the feature map. This projected position can be understood as the coordinate in the feature map coordinate system.

[0132] Exemplarily, the projection method can be an orthographic projection onto the feature map or a projection with a certain angle and / or range on the feature map. For example, the projection is performed at an oblique angle of 85 degrees to the plane where the feature map is located, and the projection point is used as the above-mentioned projection position. Alternatively, the projection is performed at an oblique angle of 75 degrees to the plane where the feature map is located, and the projection point is used as the center to enlarge and reach a set range to obtain the above-mentioned projection position.

[0133] Regarding the calculation method of the image features corresponding to the second point cloud, there are actually the following implementation methods:

[0134] In one example, for any second point cloud, the image feature at the projection position corresponding to the second point cloud in the feature map is used as the image feature corresponding to the second point cloud.

[0135] In one example, for any second point cloud, interpolation calculation is performed on image features near a projection position corresponding to the second point cloud in the feature map to obtain image features corresponding to the second point cloud.

[0136] In one example, for any second point cloud, the image features near the projection position corresponding to the second point cloud in the feature map are averaged to obtain the image features corresponding to the second point cloud.

[0137] In one example, for any second point cloud, a median is determined among image features near a projection position corresponding to the second point cloud in the feature map, and the median is used as the image feature corresponding to the non-second point cloud.

[0138] For example, since the non-masked voxel includes multiple original second point clouds, the image features corresponding to the multiple second point clouds can be averaged, and the calculated result is the second image feature. Furthermore, the second image feature is used as the target for encoding the non-masked voxel.

[0139] According to the above embodiment, when there are multiple second point clouds in the non-mask voxel, the image features corresponding to each second point cloud are determined respectively using the projection position of each second point cloud on the feature map, and thus the mean of the image features corresponding to each second point cloud is calculated to obtain a representative encoding target, i.e., the second image feature.

[0140] In an exemplary embodiment, based on the projection position of the second point cloud in the feature map, interpolation calculation is performed on the corresponding features in the feature map to obtain image features corresponding to the second point cloud, including: determining a second feature area in the feature map based on the projection position of the second point cloud in the feature map; and interpolation calculation is performed on each feature in the second feature area to obtain image features corresponding to the second point cloud.

[0141] Exemplarily, the distances between the positions of the features in the second feature region and the projection positions of the non-mask voxels all satisfy the second condition.

[0142] For example, a circle is drawn in the feature map with the projection position of the non-masked voxel in the feature map as the center point, and the circle is used as the second feature region. Alternatively, a regular polygon is drawn in the feature map with the projection position of the non-masked voxel in the feature map as the center point, and the regular polygon is used as the second feature region. For example, a square is drawn in the feature map with the projection position of the non-masked voxel in the feature map as the center point, and the square is used as the second feature region.

[0143] According to the above embodiment, the projection position of the second point cloud in the feature map is used to determine the second feature area including the projection position in the feature map, and then interpolation calculation is performed on each feature in the second feature area, so that image features that are semantically aligned with the second point cloud can be accurately obtained.

[0144] Figure 3 This is a flowchart of a point cloud model pre-training method according to another embodiment of the present disclosure.

[0145] like Figure 3As shown, first, calibrated and clock-synchronized point cloud data and image data are obtained. The point cloud data is input into the voxelization and masking network 301 to voxelize the point cloud data, and some of the voxelized voxels are masked to obtain masked voxels and non-masked voxels. The masked voxels and non-masked voxels are input into the encoder 302 to obtain masked features and non-masked features. The non-masked features and masked voxels are input into the decoder 303. The decoder 303 uses the non-masked features to construct key-value pair information, and then uses the masked features as query information to find the corresponding value information in the key-value pair information, and uses the value information as masked reconstructed features. The decoder 303 outputs the masked reconstructed features and the encoder 302 outputs the non-masked features to the attention mechanism 304, which processes these two features to obtain processed masked reconstructed features and non-masked features.

[0146] At the same time, the image data is input into the pre-trained CLIP model 305 to obtain a feature map. The masked voxels and unmasked voxels are then projected onto the feature map to obtain first image features corresponding to the masked voxels and second image features corresponding to the unmasked voxels. The first image features are used as the masked reconstruction targets, and the second image features are used as the encoded target features.

[0147] Finally, the reconstruction loss is calculated using the masked reconstructed target and the masked reconstructed features processed by the attention mechanism, and the distillation loss is calculated using the encoded target features and the non-masked features processed by the attention mechanism. The total loss determined by the reconstruction loss and the distillation loss is then used to adjust the model parameters of the encoder 302 and decoder 303.

[0148] After pre-training the first point cloud model using the point cloud model training method of any of the above embodiments to obtain a second point cloud model, the second point cloud model is distributed to a downstream task node. The downstream task node fine-tunes the second point cloud model according to its task requirements to obtain a corresponding point cloud model for the next point cloud recognition or point cloud classification task.

[0149] The present disclosure also provides a point cloud model training method, which may include:

[0150] Based on the point cloud samples and the annotation information of the point cloud samples, the second point cloud model is trained to obtain a third point cloud model, wherein the second point cloud model is a model obtained by pre-training through the point cloud model training method provided in any embodiment of the present disclosure.

[0151] The present disclosure also provides a point cloud classification method, comprising:

[0152] The target point cloud is classified according to the third point cloud model to obtain the point cloud category of the target point cloud, wherein the third point cloud model is a model trained by the point cloud model training method of any embodiment of the present disclosure.

[0153] Figure 4 1 is a structural block diagram of a point cloud model pre-training device according to an embodiment of the present disclosure.

[0154] like Figure 4 As shown, the point cloud model pre-training device includes:

[0155] An image feature determination module 410 is configured to determine, for a mask voxel in a point cloud voxel space, from image data corresponding to the point cloud voxel space, a first image feature that is semantically aligned with the mask voxel;

[0156] a model parameter adjustment module 420 for adjusting parameters of the first point cloud model using the first image feature as a mask reconstruction target so that the mask reconstruction feature matches the mask reconstruction target;

[0157] The mask reconstruction feature is obtained by reconstructing the mask voxels using the adjusted first point cloud model and the point cloud voxel space.

[0158] Figure 5 It is a structural block diagram of a point cloud model pre-training device according to another embodiment of the present disclosure.

[0159] like Figure 5 As shown, the image feature determination module 510 and the model parameter adjustment module 520 are respectively Figure 4 The image feature determination module 410 and the model parameter adjustment module 420 have the same structure and function, which will not be described in detail here.

[0160] In an exemplary embodiment, Figure 5 As shown, the above device may further include:

[0161] The point cloud voxelization module 530 is used to voxelize the point cloud data to obtain a first voxel space;

[0162] The mask processing module 540 is configured to mask some voxels in the first voxel space to obtain the point cloud voxel space, wherein the some voxels in the point cloud voxel space are mask voxels.

[0163] In an exemplary embodiment, Figure 5 As shown, the mask processing module 540 includes:

[0164] A first structure conversion unit 541 is configured to convert the first voxel space into a two-dimensional second voxel space when the first voxel space is a four-dimensional structure, wherein the second voxel space includes at least one voxel channel, and a plurality of voxels in each voxel channel are divided into a first portion of voxels and a second portion of voxels;

[0165] a voxel masking unit 542 configured to mask each of the first portion of voxels in the second voxel space to obtain a third voxel space, wherein the first portion of voxels in the third voxel space are masked voxels and the second portion of voxels in the third voxel space are non-masked voxels;

[0166] The second structure conversion unit 543 is configured to convert the third voxel space into the point cloud voxel space with a four-dimensional structure.

[0167] In an exemplary embodiment, the image feature determination module 510 includes:

[0168] A feature map determining unit 511 is configured to input image data corresponding to the point cloud voxel space into a connected text and image model to obtain a feature map output by the connected text and image model;

[0169] The image feature determining unit 512 is configured to determine, for a mask voxel in the point cloud voxel space, in the feature map, the first image feature that is semantically aligned with the mask voxel.

[0170] In an exemplary embodiment, the point cloud voxel space includes a plurality of mask voxels, and the image feature determination unit 512 is specifically configured to:

[0171] Projecting each first point cloud in the mask voxel onto the feature map to obtain a projection position of each first point cloud in the feature map;

[0172] For any of the first point clouds, based on the projection position of the first point cloud in the feature map, perform interpolation calculation on the corresponding features in the feature map to obtain image features corresponding to the first point cloud;

[0173] Mean values ​​of the image features corresponding to the first point clouds are calculated to obtain first image features that are semantically aligned with the mask voxels.

[0174] In an exemplary embodiment, performing interpolation calculation on corresponding features in the feature map based on the projection position of the first point cloud in the feature map to obtain image features corresponding to the first point cloud includes:

[0175] Determining a first feature area in the feature map based on a projection position of the first point cloud in the feature map;

[0176] Interpolation calculation is performed on each feature in the first feature area to obtain image features corresponding to the first point cloud.

[0177] In an exemplary embodiment, the model parameter adjustment module 520 includes:

[0178] a mask feature reconstruction unit 521, configured to reconstruct the mask voxels based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features;

[0179] The model parameter adjustment unit 522 is configured to adjust parameters of the first point cloud model based on the mask reconstruction feature and the mask reconstruction target.

[0180] In an exemplary embodiment, the first point cloud model includes a point cloud encoder and a point cloud decoder, and the mask feature reconstruction unit 521 is specifically configured to:

[0181] Encoding the point cloud voxel space using the point cloud encoder to obtain encoding features;

[0182] Extracting, from the coded features, coded features corresponding to mask voxels in the point cloud voxel space as mask features, and extracting coded features corresponding to non-mask voxels in the point cloud voxel space as non-mask features;

[0183] Based on the point cloud decoder and the non-mask feature, the mask feature is decoded to obtain the mask reconstruction feature.

[0184] In an exemplary embodiment, decoding the mask feature based on the point cloud decoder and the non-mask feature to obtain the mask reconstruction feature includes:

[0185] Constructing key-value pair information of the point cloud decoder based on the non-masked features;

[0186] Inputting the key-value pair information and the mask feature into the point cloud decoder, so that the point cloud decoder decodes the mask feature using the key-value pair information and outputs a decoded feature;

[0187] The mask reconstruction feature is determined based on the decoded feature output by the point cloud decoder.

[0188] In an exemplary embodiment, the model parameter adjustment unit 522 is specifically configured to:

[0189] Determining a target loss function based on the mask reconstruction feature and the mask reconstruction target;

[0190] Based on the target loss function, parameters of the first point cloud model are adjusted.

[0191] In an exemplary embodiment, determining a target loss function based on the mask reconstruction feature and the mask reconstruction target includes:

[0192] Determining a first loss function based on the mask reconstruction feature and the mask reconstruction target;

[0193] For non-masked voxels in the point cloud voxel space, determining, in the feature map, a second image feature that is semantically aligned with the non-masked voxel;

[0194] determining a second loss function based on the non-masked features and the second image features;

[0195] The target loss function is determined based on the first loss function and the second loss function.

[0196] In an exemplary embodiment, the point cloud voxel space includes a plurality of non-masked voxels, and determining, in the feature map, a second image feature semantically aligned with the non-masked voxels includes:

[0197] Projecting each second point cloud in the non-masked voxel onto the feature map to obtain a projection position of each second point cloud in the feature map;

[0198] For any second point cloud, based on the projection position of the second point cloud in the feature map, perform interpolation calculation on the corresponding features in the feature map to obtain image features corresponding to the second point cloud;

[0199] Mean values ​​of the image features corresponding to the respective second point clouds are calculated to obtain second image features that are semantically aligned with the non-masked voxels.

[0200] In an exemplary embodiment, performing interpolation calculation on corresponding features in the feature map based on the projection position of the second point cloud in the feature map to obtain image features corresponding to the second point cloud includes:

[0201] determining a second feature area in the feature map based on a projection position of the second point cloud in the feature map;

[0202] Interpolation calculation is performed on each feature in the second feature area to obtain image features corresponding to the second point cloud.

[0203] The present disclosure also provides a point cloud model training device, wherein the device is used to:

[0204] Based on the point cloud samples and the annotation information of the point cloud samples, a second point cloud model is trained to obtain a second point cloud model, wherein the second point cloud model is a model obtained by pre-training according to the point cloud model pre-training method described in any one of the embodiments of the present disclosure.

[0205] The present disclosure further provides a point cloud detection device, wherein the device is used to:

[0206] The target point cloud is detected according to the third point cloud model to obtain a detection result of the target point cloud, wherein the third point cloud model is a model trained according to the point cloud model training method described in any one of the embodiments of the present disclosure.

[0207] The present disclosure further provides a point cloud segmentation device, wherein the device is used to:

[0208] The target point cloud is detected according to the fourth point cloud model to obtain a detection result of the target point cloud, wherein the fourth point cloud model is a model trained according to the point cloud model training method described in any one of the embodiments of the present disclosure.

[0209] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0210] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0211] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0212] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0213] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0214] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0215] The computing unit 601 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a point cloud model pre-training method, a point cloud model training method, a point cloud detection method, or a point cloud segmentation method. For example, in some embodiments, a point cloud model pre-training method, a point cloud model training method, a point cloud detection method, or a point cloud segmentation method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of a point cloud model pre-training method, a point cloud model training method, a point cloud detection method, or a point cloud segmentation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a point cloud model pre-training method, a point cloud model training method, a point cloud detection method, or a point cloud segmentation method in any other appropriate manner (e.g., via firmware).

[0216] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0217] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0218] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0219] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0220] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0221] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0222] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0223] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A point cloud model pre-training method, comprising: For a mask voxel in a point cloud voxel space, determining a first image feature semantically aligned with the mask voxel from image data corresponding to the point cloud voxel space, including: inputting the image data into a CLIP model to obtain a feature map output by the CLIP model; projecting each first point cloud in the mask voxel onto the feature map to obtain a projection position of each first point cloud on the feature map; determining an image feature corresponding to each first point cloud based on image features at the projection position of each first point cloud on the feature map; and performing mean calculation on the image features corresponding to each first point cloud to obtain the first image feature; Using the first image feature as a mask reconstruction target, adjusting parameters of the first point cloud model so that the mask reconstruction feature matches the mask reconstruction target; The mask reconstruction feature is obtained by reconstructing the mask voxels using the point cloud voxel space using the adjusted first point cloud model.

2. The method according to claim 1, further comprising: voxelize the point cloud data to obtain the first voxel space; Masking is performed on some voxels in the first voxel space to obtain the point cloud voxel space, wherein the some voxels in the point cloud voxel space are mask voxels.

3. The method according to claim 2, wherein: Masking some voxels in the first voxel space to obtain the point cloud voxel space includes: In a case where the first voxel space is a four-dimensional structure, converting the first voxel space into a two-dimensional second voxel space, wherein the second voxel space includes at least one voxel channel, and a plurality of voxels in each voxel channel are divided into a first portion of voxels and a second portion of voxels; Masking each of the first portion of voxels in the second voxel space to obtain a third voxel space, wherein the first portion of voxels in the third voxel space are masked voxels, and the second portion of voxels in the third voxel space are non-masked voxels; The third voxel space is converted into the point cloud voxel space of a four-dimensional structure.

4. The method according to claim 1, wherein The determining, based on the image features at the projection positions of the first point clouds on the feature maps, the image features corresponding to the first point clouds, respectively, includes: For any of the first point clouds, based on the projection position of the first point cloud in the feature map, interpolation calculation is performed on the corresponding features in the feature map to obtain image features corresponding to the first point cloud.

5. The method according to claim 4, wherein The interpolation calculation is performed on the corresponding features in the feature map based on the projection position of the first point cloud in the feature map to obtain the image features corresponding to the first point cloud, including: Determining a first feature area in the feature map based on a projection position of the first point cloud in the feature map; Interpolation calculation is performed on each feature in the first feature area to obtain image features corresponding to the first point cloud.

6. The method according to claim 4 or 5, wherein: The step of using the first image feature as a mask reconstruction target and adjusting parameters of the first point cloud model includes: reconstructing the mask voxels based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features; Based on the mask reconstruction features and the mask reconstruction target, parameters of the first point cloud model are adjusted.

7. The method according to claim 6, wherein: The first point cloud model includes a point cloud encoder and a point cloud decoder, and the mask voxels are reconstructed based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features, including: Encoding the point cloud voxel space using the point cloud encoder to obtain encoding features; Extracting, from the coded features, coded features corresponding to mask voxels in the point cloud voxel space as mask features, and extracting coded features corresponding to non-mask voxels in the point cloud voxel space as non-mask features; Based on the point cloud decoder and the non-mask feature, the mask feature is decoded to obtain the mask reconstruction feature.

8. The method according to claim 7, wherein: The decoding of the mask feature based on the point cloud decoder and the non-mask feature to obtain the mask reconstruction feature includes: Constructing key-value pair information of the point cloud decoder based on the non-masked features; Inputting the key-value pair information and the mask feature into the point cloud decoder, so that the point cloud decoder decodes the mask feature using the key-value pair information and outputs a decoded feature; The mask reconstruction feature is determined based on the decoded feature output by the point cloud decoder.

9. The method according to claim 7, wherein: The step of adjusting parameters of the first point cloud model based on the mask reconstruction feature and the mask reconstruction target includes: Determining a target loss function based on the mask reconstruction feature and the mask reconstruction target; Based on the target loss function, parameters of the first point cloud model are adjusted.

10. The method according to claim 9, wherein: The determining of a target loss function based on the mask reconstruction feature and the mask reconstruction target includes: Determining a first loss function based on the mask reconstruction feature and the mask reconstruction target; For non-masked voxels in the point cloud voxel space, determining, in the feature map, a second image feature that is semantically aligned with the non-masked voxel; determining a second loss function based on the non-masked features and the second image features; The target loss function is determined based on the first loss function and the second loss function.

11. The method according to claim 10, wherein: Determining, in the feature map, a second image feature semantically aligned with the non-masked voxel comprises: Projecting each second point cloud in the non-masked voxel onto the feature map to obtain a projection position of each second point cloud in the feature map; For any second point cloud, based on the projection position of the second point cloud in the feature map, perform interpolation calculation on the corresponding features in the feature map to obtain image features corresponding to the second point cloud; Mean values ​​of the image features corresponding to the respective second point clouds are calculated to obtain second image features that are semantically aligned with the non-masked voxels.

12. The method according to claim 11, wherein The interpolation calculation is performed on the corresponding features in the feature map based on the projection position of the second point cloud in the feature map to obtain the image features corresponding to the second point cloud, including: determining a second feature area in the feature map based on a projection position of the second point cloud in the feature map; Interpolation calculation is performed on each feature in the second feature area to obtain image features corresponding to the second point cloud.

13. A point cloud model training method, comprising: Based on the point cloud samples and the annotation information of the point cloud samples, the second point cloud model is trained to obtain a third point cloud model, wherein the second point cloud model is a model obtained by pre-training according to the point cloud model training method according to any one of claims 1-12.

14. A point cloud detection method, comprising: The target point cloud is detected according to the third point cloud model to obtain a detection result of the target point cloud, wherein the third point cloud model is a model trained according to the point cloud model training method according to claim 13.

15. A point cloud segmentation method, comprising: The target point cloud is segmented according to the fourth point cloud model to obtain a segmentation result of the target point cloud, wherein the fourth point cloud model is a model trained according to the point cloud model training method according to claim 13.

16. A point cloud model pre-training device, comprising: An image feature determination module is configured to determine, for a mask voxel in a point cloud voxel space, a first image feature semantically aligned with the mask voxel from image data corresponding to the point cloud voxel space; a model parameter adjustment module, configured to use the first image feature as a mask reconstruction target and adjust parameters of the first point cloud model so that the mask reconstruction feature matches the mask reconstruction target; The mask reconstruction feature is obtained by reconstructing the mask voxels using the point cloud voxel space of the adjusted first point cloud model; Wherein, the image feature determination module includes: a feature map determining unit, configured to input image data corresponding to the point cloud voxel space into a CLIP model to obtain a feature map output by the CLIP model; An image feature determination unit is configured to project each first point cloud in the mask voxel onto the feature map to obtain a projection position of each first point cloud on the feature map; determine image features corresponding to each first point cloud based on image features at the projection position of each first point cloud on the feature map; and perform mean calculation on the image features corresponding to each first point cloud to obtain the first image features.

17. The apparatus according to claim 16, further comprising: A point cloud voxelization module is used to voxelize the point cloud data to obtain a first voxel space; A mask processing module is used to mask some voxels in the first voxel space to obtain the point cloud voxel space, wherein the some voxels in the point cloud voxel space are mask voxels.

18. The device according to claim 17, wherein The mask processing module includes: a first structure conversion unit, configured to, when the first voxel space is a four-dimensional structure, convert the first voxel space into a two-dimensional second voxel space, wherein the second voxel space includes at least one voxel channel, and a plurality of voxels in each voxel channel are divided into a first portion of voxels and a second portion of voxels; a voxel masking unit, configured to mask each of the first portion of voxels in the second voxel space to obtain a third voxel space, wherein the first portion of voxels in the third voxel space are masked voxels, and the second portion of voxels in the third voxel space are non-masked voxels; The second structure conversion unit is used to convert the third voxel space into the point cloud voxel space with a four-dimensional structure.

19. The device according to claim 16, wherein The determining, based on the image features at the projection positions of the first point clouds on the feature maps, the image features corresponding to the first point clouds, respectively, includes: For any of the first point clouds, based on the projection position of the first point cloud in the feature map, interpolation calculation is performed on the corresponding features in the feature map to obtain image features corresponding to the first point cloud.

20. The device according to claim 19, wherein The interpolation calculation is performed on the corresponding features in the feature map based on the projection position of the first point cloud in the feature map to obtain the image features corresponding to the first point cloud, including: Determining a first feature area in the feature map based on a projection position of the first point cloud in the feature map; Interpolation calculation is performed on each feature in the first feature area to obtain image features corresponding to the first point cloud.

21. The device according to claim 19 or 20, wherein The model parameter adjustment module includes: a mask feature reconstruction unit, configured to reconstruct the mask voxels based on the point cloud voxel space and the first point cloud model to obtain mask reconstruction features; A model parameter adjustment unit is used to adjust the parameters of the first point cloud model based on the mask reconstruction feature and the mask reconstruction target.

22. The device according to claim 21, wherein The first point cloud model includes a point cloud encoder and a point cloud decoder, and the mask feature reconstruction unit is specifically used to: Encoding the point cloud voxel space using the point cloud encoder to obtain encoding features; Extracting, from the coded features, coded features corresponding to mask voxels in the point cloud voxel space as mask features, and extracting coded features corresponding to non-mask voxels in the point cloud voxel space as non-mask features; Based on the point cloud decoder and the non-mask feature, the mask feature is decoded to obtain the mask reconstruction feature.

23. The device according to claim 22, wherein The decoding of the mask feature based on the point cloud decoder and the non-mask feature to obtain the mask reconstruction feature includes: Constructing key-value pair information of the point cloud decoder based on the non-masked features; Inputting the key-value pair information and the mask feature into the point cloud decoder, so that the point cloud decoder decodes the mask feature using the key-value pair information and outputs a decoded feature; The mask reconstruction feature is determined based on the decoded feature output by the point cloud decoder.

24. The device according to claim 23, wherein The model parameter adjustment unit is specifically used to: Determining a target loss function based on the mask reconstruction feature and the mask reconstruction target; Based on the target loss function, parameters of the first point cloud model are adjusted.

25. The apparatus according to claim 24, wherein The determining of a target loss function based on the mask reconstruction feature and the mask reconstruction target includes: Determining a first loss function based on the mask reconstruction feature and the mask reconstruction target; For non-masked voxels in the point cloud voxel space, determining, in the feature map, a second image feature that is semantically aligned with the non-masked voxel; determining a second loss function based on the non-masked features and the second image features; The target loss function is determined based on the first loss function and the second loss function.

26. The device according to claim 25, wherein Determining, in the feature map, a second image feature semantically aligned with the non-masked voxel comprises: Projecting each second point cloud in the non-masked voxel onto the feature map to obtain a projection position of each second point cloud in the feature map; For any second point cloud, based on the projection position of the second point cloud in the feature map, perform interpolation calculation on the corresponding features in the feature map to obtain image features corresponding to the second point cloud; Mean values ​​of the image features corresponding to the respective second point clouds are calculated to obtain second image features that are semantically aligned with the non-masked voxels.

27. The device according to claim 26, wherein The interpolation calculation is performed on the corresponding features in the feature map based on the projection position of the second point cloud in the feature map to obtain the image features corresponding to the second point cloud, including: determining a second feature area in the feature map based on a projection position of the second point cloud in the feature map; Interpolation calculation is performed on each feature in the second feature area to obtain image features corresponding to the second point cloud.

28. A point cloud model training device, wherein: The device is used to: Based on the point cloud samples and the annotation information of the point cloud samples, the second point cloud model is trained to obtain a third point cloud model, wherein the second point cloud model is a model obtained by pre-training according to the point cloud model training method according to any one of claims 1-12.

29. A point cloud detection device, wherein: The device is used to: The target point cloud is detected according to the third point cloud model to obtain a detection result of the target point cloud, wherein the third point cloud model is a model trained according to the point cloud model training method according to claim 13.

30. A point cloud segmentation device, wherein: The device is used to: The target point cloud is segmented according to the fourth point cloud model to obtain a segmentation result of the target point cloud, wherein the fourth point cloud model is a model trained according to the point cloud model training method according to claim 13.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images