Training method for point cloud 3D detection model, point cloud 3D detection method and device

Abstract

This disclosure provides a training method, a point cloud 3D detection method, and an apparatus for a point cloud 3D detection model. This disclosure relates to the field of artificial intelligence technology, particularly to computer vision, deep learning, and large-scale model technologies, and can be applied to scenarios such as intelligent vehicles, autonomous driving, and intelligent robots. The specific solution is as follows: acquire point cloud data; input the point cloud data into the main branch network and bypass branch network of a pre-trained model; train the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to computer vision, deep learning, and large-scale modeling, and can be applied to scenarios such as intelligent vehicles, autonomous driving, and intelligent robots. Specifically, it relates to a training method for a point cloud 3D detection model, a point cloud 3D detection method, and an apparatus. Background Technology

[0002] Point cloud-based 3D object detection typically employs small neural network models. However, these models have limitations in terms of data fit and detection accuracy, resulting in lower accuracy for point cloud 3D detection. Using large models presents two significant drawbacks: first, training large models consumes substantial video memory, placing excessive demands on graphics processing units (GPUs) and making them difficult to implement; second, the large number of parameters in large models also presents a challenge for storage resources. Summary of the Invention

[0003] This disclosure provides a training method for a point cloud 3D detection model, a point cloud 3D detection method, and an apparatus.

[0004] According to a first aspect of this disclosure, a method for training a point cloud 3D detection model is provided, comprising:

[0005] Acquire point cloud data;

[0006] Point cloud data is input into the main branch network and the bypass branch network of the pre-trained model. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model.

[0007] According to a second aspect of this disclosure, a point cloud 3D detection method is provided, comprising:

[0008] Acquire point cloud data;

[0009] Input point cloud data into the point cloud 3D detection model and obtain the 3D detection results output by the point cloud 3D detection model;

[0010] The point cloud 3D detection model is trained based on the point cloud 3D detection model training method provided in the first aspect.

[0011] According to a third aspect of this disclosure, a training apparatus for a point cloud 3D detection model is provided, comprising:

[0012] The first acquisition module is used to acquire point cloud data;

[0013] The training module is used to input point cloud data into the main branch network and the bypass branch network of the pre-trained model. By fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network, the pre-trained model is trained to obtain the point cloud 3D detection model.

[0014] According to a fourth aspect of this disclosure, a point cloud 3D inspection device is provided, comprising:

[0015] The second acquisition module is used to acquire point cloud data;

[0016] The detection module inputs point cloud data into the point cloud 3D detection model and obtains the 3D detection results output by the point cloud 3D detection model.

[0017] The point cloud 3D detection model is trained based on the point cloud 3D detection model training method provided in the first aspect.

[0018] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0019] At least one processor;

[0020] Memory that is communicatively connected to at least one processor;

[0021] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to execute the training method of the point cloud 3D detection model provided in the first aspect and / or the point cloud 3D detection method provided in the second aspect.

[0022] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a training method for a point cloud 3D detection model provided in the first aspect and / or a point cloud 3D detection method provided in the second aspect.

[0023] According to the seventh aspect of this disclosure, a computer program product is provided, including a computer program stored on a storage medium, which, when executed by a processor, implements the training method for the point cloud 3D detection model provided in the first aspect and / or the point cloud 3D detection method provided in the second aspect.

[0024] According to the scheme disclosed herein, compared with the training method of updating the network parameters of the main branch network, by adopting the training method of not updating the network parameters of the main branch network but only updating the network parameters of the bypass branch network, the GPU memory can be significantly saved. This enables the training of a model for point cloud 3D detection under limited GPU memory and storage resources. In this way, the advantages of the object detection capability and 2D pre-training capability of the main branch network can be utilized to extract point cloud features more accurately, thereby improving the accuracy of the 3D detection model.

[0025] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0026] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0027] Figure 1 This is a flowchart illustrating a training method for a point cloud 3D detection model according to an embodiment of the present disclosure.

[0028] Figure 2 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 1 ;

[0029] Figure 3 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 2 ;

[0030] Figure 4 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 3 ;

[0031] Figure 5 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 4 ;

[0032] Figure 6 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 5 ;

[0033] Figure 7 This is a schematic flowchart of a point cloud 3D detection method according to an embodiment of the present disclosure;

[0034] Figure 8 This is a schematic diagram of the detection process based on a point cloud 3D detection model according to an embodiment of the present disclosure;

[0035] Figure 9 This is a schematic diagram of the structure of a training device for a point cloud 3D detection model according to an embodiment of the present disclosure;

[0036] Figure 10 This is a schematic diagram of the structure of a point cloud 3D detection device according to an embodiment of the present disclosure;

[0037] Figure 11This is a scene diagram illustrating a training method for a point cloud 3D detection model according to an embodiment of the present disclosure;

[0038] Figure 12 This is a scene illustration of the point cloud 3D detection method according to an embodiment of the present disclosure;

[0039] Figure 13 This is a schematic diagram of the structure of an electronic device used to implement the point cloud 3D detection model training method and / or point cloud 3D detection method according to the embodiments of this disclosure. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] The terms "first," "second," and "third," etc., used in the embodiments, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0042] In related technologies, deep learning is used for point cloud-based 3D object detection. Inputting laser point clouds, the output includes seven degrees of freedom: 3D position information, target dimensions (length, width, height), and target orientation angle. Currently, mainstream methods employ supervised learning, using radar sensors and camera sensors to collect data, followed by complete manual annotation to supervise the neural network's learning.

[0043] Using small neural network models to implement point cloud 3D detection solutions has the following problems:

[0044] 1. If a small neural network model is used, there are certain limitations in terms of the degree of data fitting and detection accuracy.

[0045] 2. If a large model is used, there are two drawbacks that must be overcome: First, training a large model consumes a lot of video memory, which places excessive demands on the GPU and is not easy to implement; second, the large number of parameters in a large model also poses a challenge to storage resources.

[0046] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, this disclosure proposes a training method for a point cloud 3D detection model. By adopting a training method that does not update the network parameters of the main branch network but only updates the network parameters of the bypass branch network, the GPU memory can be significantly saved, thereby enabling the training of a point cloud 3D detection model under limited GPU memory and storage resources, and thus improving the accuracy of the 3D detection model.

[0047] This disclosure provides a method for training a point cloud 3D detection model. Figure 1 This is a flowchart illustrating a training method for a point cloud 3D detection model according to an embodiment of the present disclosure. This training method can be applied to a training device for a point cloud 3D detection model. The training device is located in an electronic device. This electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, vehicle terminals, laptops, tablets, etc. In some possible implementations, the training method for the point cloud 3D detection model can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method for this point cloud 3D detection model includes:

[0048] S101: Acquire point cloud data;

[0049] S102: Input the point cloud data into the main branch network and the bypass branch network of the pre-trained model. By fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network, the pre-trained model is trained to obtain the point cloud 3D detection model.

[0050] In some implementations, point cloud data refers to a set of vectors in a three-dimensional coordinate system, where each point contains three-dimensional coordinate information, and some points may also contain color information (Red, Green, Blue, RGB) or reflectance intensity information. This data can be acquired by devices such as radar sensor-camera sensor pairs, radar, laser scanners, depth cameras, or stereo cameras.

[0051] In some implementations, this point cloud-based 3D detection technology can be applied to fields such as autonomous driving, intelligent robots, and virtual reality. Specifically, point cloud data for autonomous driving can be acquired through radar sensor-camera sensor pairs, LiDAR, or depth cameras; point cloud data for intelligent robots can be acquired through sensors such as LiDAR or depth cameras; and point cloud data for virtual reality can be acquired through professional 3D scanners or by combining multiple depth cameras.

[0052] In some implementations, this point cloud-based 3D detection technology is applied to the field of autonomous driving, where point cloud data is crucial for the environmental perception and decision-making of autonomous vehicles; for example, it helps vehicles identify roads, obstacles, and other vehicles, thereby enabling autonomous navigation and obstacle avoidance. This point cloud-based 3D detection technology can also be applied to the field of intelligent robotics, enabling object recognition, localization, and grasping operations using point cloud data; for example, in warehousing and logistics, robots can use point cloud data to determine the position and orientation of goods, achieving precise grasping and handling. This point cloud-based 3D detection technology can also be applied to the field of virtual reality, generating realistic 3D models from point cloud data; these models can be used to construct virtual reality scenes, providing users with an immersive experience.

[0053] In some implementations, the pre-trained model is a model consisting of a backbone network of a large model and a network for point cloud 3D detection.

[0054] In some implementations, the pre-trained model includes a main branch network and side branch networks. The main branch network can be the backbone network of a large model. This main branch network consists of multiple Transformer blocks. A Transformer is a neural network architecture based on a self-attention mechanism, and a Transformer block is the basic building block in a Transformer model. A Transformer block can include multiple sub-layers, such as self-attention layers, feedforward neural network layers, etc., used to capture dependencies in the input point cloud data and generate high-dimensional features.

[0055] In some implementations, the bypass branch network can be a bypass network for point cloud 3D detection. This bypass branch network includes multiple fine-tunable blocks. A fine-tunable block can be a simple linear layer or a more complex network structure. Importantly, the fine-tunable block needs sufficient network parameters for fine-tuning, i.e., updating the weights during training. The fine-tunable block is integrated into each Transformer block. Specifically, after the Transformer block has performed self-attention and feedforward neural network operations, the fine-tunable block can be applied to the output features. Adding the features output by the fine-tunable block to the original output features of the Transformer block helps alleviate the vanishing gradient problem during deep neural network training and accelerates the training process. The result with the added residual connection features is fed into the next Transformer block as its input. For example, each Transformer block is connected to a fine-tuning block. The first Transformer block outputs feature A11 based on point cloud data, and the first fine-tuning block learns and outputs additional feature A12 based on point cloud data. Feature A11 and feature A12 are added together to obtain a new feature A1, and the new feature A1 is fed into the second Transformer block.

[0056] In this way, each Transformer block can utilize the fine-tuned feature information of the previous Transformer block, which enhances the model's representational power and helps improve the model's performance.

[0057] In some implementations, this training method, which updates the network parameters of the bypass branch network while fixing the network parameters of the main branch network, includes: during training, the network parameter values ​​of the main branch network remain unchanged; that is, the weights and biases of the main branch network are not updated, and only the network parameter values ​​of the fine-tunable blocks are updated. During training, since the Transformer blocks in the main branch network occupy most of the GPU memory, not updating the parameters of the main branch network can maintain certain specific functions or characteristics of the main branch network, significantly saving GPU memory. Simultaneously, by updating the parameters of the bypass branch network, the bypass branch network learns and adjusts, facilitating feature fine-tuning for each Transformer block. Each Transformer block can utilize the fine-tuned feature information of the previous Transformer block, enhancing the model's representational power and contributing to improved model performance.

[0058] In some implementations, this point cloud 3D detection model refers to a model that uses point cloud data for 3D target detection. This point cloud 3D detection model can be applied to fields such as autonomous driving, intelligent robots, and virtual reality.

[0059] The technical solution of this disclosure involves inputting point cloud data into the main branch network and bypass branch network of a pre-trained model. By fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network, the pre-trained model is trained to obtain a point cloud 3D detection model. Compared to the training method that updates the network parameters of the main branch network, this method, which only updates the network parameters of the bypass branch network without updating the main branch network parameters, significantly saves GPU memory. This allows for training a model for point cloud 3D detection under limited GPU and storage resources. By leveraging the object detection capabilities of the main branch network and its 2D pre-training capabilities, the model can more accurately extract point cloud features, thereby improving the training accuracy of the 3D detection model.

[0060] In this embodiment of the disclosure, the main branch network includes multiple Transformer blocks, and the bypass branch network includes multiple fine-tunable blocks.

[0061] In some implementations, multiple Transformer blocks correspond one-to-one with multiple fine-tunable blocks. For example, the main branch network includes N Transformer blocks, and the bypass branch network includes N fine-tunable blocks. The N Transformer blocks are denoted as 1, 2, ..., N, and the N fine-tunable blocks are denoted as 1, 2, ..., N, where N is a positive integer. For example, the first Transformer block is connected in parallel with the first fine-tunable block, the second Transformer block is connected in parallel with the second fine-tunable block, and the Nth Transformer block is connected in parallel with the Nth fine-tunable block. Here, parallel connection can be understood as follows: the input features can be the same, and the output features can be merged as needed.

[0062] In some implementations, the main branch network can be the main branch network of a large model, specifically the main branch network of a large model such as a computer vision model, an image processing model, or a data processing model.

[0063] In some implementations, the fine-tunable block may include multiple learnable parameters. In a Transformer model, each Transformer block typically performs self-attention mechanisms and feedforward neural network operations to generate and transform features. Configuring a fine-tunable block for each Transformer block, adding the features output by that fine-tunable block to the features output by the previous Transformer block, and then feeding it into the next Transformer block is equivalent to adding a residual connection and fine-tuning mechanism to the original Transformer structure.

[0064] The technical solution of this disclosure, by configuring a fine-tunable block for each Transformer block, allows each Transformer block to utilize the fine-tuned feature information of the previous Transformer block, thereby enhancing the model's representational ability and helping to improve the model's performance.

[0065] In this embodiment of the disclosure, the training method of fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to train the pre-trained model includes: fusing the features of the point cloud data output by the target Transformer block of the main branch network and the features of the point cloud data output by the target fine-tunable block of the bypass branch network, and using the fused features as the input of the next Transformer block of the target Transformer block.

[0066] Here, one or more Transformer blocks from the N Transformer blocks can be used as the target Transformer block, and one or more fine-tunable blocks from the N fine-tunable blocks can be used as the target fine-tunable block.

[0067] Here, the feature fusion method can be one or more of the following fusion methods:

[0068] Additive fusion: Adding different features together to obtain a new fused feature;

[0069] Multiplicative fusion: Multiplying different features together to obtain a new fused feature;

[0070] Splicing and fusion: directly splicing different features together to form new fused features;

[0071] Linear fusion: This method combines different features using a linear combination approach, and obtains new fused features through weight allocation.

[0072] Nonlinear fusion: This method uses nonlinear approaches to fuse different features, such as combining polynomial features.

[0073] Here, additive fusion and multiplicative fusion are particularly suitable for the fusion of numerical features, while splicing fusion is particularly suitable for the fusion of categorical features.

[0074] Thus, the main branch network focuses more on global or high-level feature extraction, while the side branch network focuses more on local or detailed features. Combining the two allows for the capture of more information from point cloud data. The fine-tunable blocks of the side branch network allow for customization and adjustment based on specific tasks or datasets, enabling the model to be flexibly configured to adapt to different application scenarios. By combining the features of the main branch network and the side branch network, the model's performance, flexibility, and robustness in processing point cloud data can be improved, providing more accurate and comprehensive feature representations for subsequent tasks.

[0075] In this embodiment of the disclosure, the training method of updating the network parameters of the bypass branch network while fixing the network parameters of the main branch network to train the pre-trained model further includes: when each target Transformer block corresponds to a target fine-tunable block, determining the i-th Transformer block of the main branch network as the target Transformer block, and determining the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the i-th Transformer block; where i and j are positive integers, 1≤i≤N, 1≤j≤N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, i and j are first target identifiers. Here, the first target identifier can be any one or more values ​​from 1 to N, and the identifier number and quantity of the first target identifier can be set or adjusted according to requirements.

[0076] Here, among the N Transformer blocks, each Transformer block has a different identifier. For example, the identifiers of the 1st to Nth Transformer blocks are 1, 2, 3, ..., N-1, N, respectively. Similarly, among the N fine-tunable blocks, each fine-tunable block has a different identifier. For example, the identifiers of the 1st to Nth fine-tunable blocks are 1, 2, 3, ..., N-1, N, respectively.

[0077] Here, the first target identifier can be any value from 1 to N. For example, if the first target identifiers of i and j are the same, and the first target identifier of j is N, then the first target identifier of i is N. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes fusing the features of the point cloud data output from the Nth fine-tunable block with the features of the point cloud data output from the Nth Transformer block, and feeding the fused features into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features.

[0078] Here, the first target identifier can be any of the values ​​from 1 to N. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the side branch network. This includes: if there is a one-to-one correspondence between the Transformer blocks in the main branch network and the fine-tunable blocks in the side branch network, then the features of the point cloud data output by the i-th fine-tunable block are fused with the features of the point cloud data output by the i-th Transformer block. The fused features are then fed into the (i+1)-th Transformer block, until the features of the point cloud data output by the last fine-tunable block are fused with the features of the point cloud data output by the last Transformer block. The fused features are then fed into the detection head network of the pre-trained model, which is used to obtain 3D detection results based on the input features; i is a positive integer.

[0079] Figure 2 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 1 ,like Figure 2As shown, the main branch network of the pre-trained model includes 4 Transformer blocks (denoted as Transformer block-1, Transformer block-2, Transformer block-3, and Transformer block-4); the side branch network of the pre-trained model includes 4 fine-tunable blocks (denoted as fine-tunable block-1, fine-tunable block-2, fine-tunable block-3, and fine-tunable block-4); each Transformer block corresponds to one fine-tunable block, that is, Transformer block-1 corresponds to fine-tunable block-1, Transformer block-2 corresponds to fine-tunable block-2, Transformer block-3 corresponds to fine-tunable block-3, and Transformer block-4 corresponds to fine-tunable block-4. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. The training process includes: fusing the features of the point cloud data output by fine-tuning block-1 with the features of the point cloud data output by Transformer block-1, and feeding the fused features into Transformer block-2; fusing the features of the point cloud data output by fine-tuning block-2 with the features of the point cloud data output by Transformer block-2, and feeding the fused features into Transformer block-3; fusing the features of the point cloud data output by fine-tuning block-3 with the features of the point cloud data output by Transformer block-3, and feeding the fused features into Transformer block-4; and fusing the features of the point cloud data output by fine-tuning block-4 with the features of the point cloud data output by Transformer block-4, and feeding the fused features into the detection head of the pre-trained model to obtain the 3D point cloud detection results output by the detection head. It should be noted that the number of Transformer blocks and the number of fine-tuning blocks mentioned above are merely illustrative and not restrictive. In practical applications, the number of Transformer blocks and fine-tuning blocks can be increased or decreased according to requirements such as accuracy or speed.

[0080] Thus, by configuring a fine-tunable block in the bypass branch network for each Transformer block in the main branch network, the feature extraction capabilities of each Transformer block in the main branch network can be fully utilized. This enhances the ability of the bypass branch network to adjust and optimize the network parameters of each fine-tunable block during training, thereby improving the accuracy of the entire point cloud 3D detection model. Furthermore, since Transformer blocks consume a significant amount of GPU memory, training without updating the network parameters of the Transformer blocks can greatly save GPU memory.

[0081] In this embodiment of the disclosure, there is a one-to-one correspondence between the Transformer blocks in the main branch network and the fine-tunable blocks in the bypass branch network, and the identifier number of the first target identifier is an odd number. For example, the identifier numbers of the 1st to the Nth Transformer blocks are 1, 2, 3, ..., N-1, N, respectively; the identifier numbers of the 1st to the Nth fine-tunable blocks are 1, 2, 3, ..., N-1, N, respectively; when N is divisible by 2, the first target identifiers of i and j both include 1, 3, 5, ..., N-1; when N is not divisible by 2, the first target identifiers of i and j both include 1, 3, 5, ..., N. The training method involves fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to train the pre-trained model. This includes: when the first target identifier of the fine-tunable block corresponding to the current Transformer block is odd, fusing the features of the point cloud data output by the fine-tunable block corresponding to the current Transformer block with the features of the point cloud data output by the current Transformer block, and feeding the fused features into the next Transformer block; until the features of the point cloud data output by the last Transformer block are fed into the detection head network of the pre-trained model. The detection head network is used to obtain 3D detection results based on the input features.

[0082] Figure 3 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 2 ,like Figure 3As shown, the main branch network of the pre-trained model includes 4 Transformer blocks (denoted as Transformer block-1, Transformer block-2, Transformer block-3, and Transformer block-4); the side branch network of the pre-trained model includes 4 fine-tunable blocks (denoted as fine-tunable block-1, fine-tunable block-2, fine-tunable block-3, and fine-tunable block-4); each Transformer block corresponds to one fine-tunable block, that is, Transformer block-1 corresponds to fine-tunable block-1, Transformer block-2 corresponds to fine-tunable block-2, Transformer block-3 corresponds to fine-tunable block-3, and Transformer block-4 corresponds to fine-tunable block-4. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This process includes: fusing the features of the point cloud data output from fine-tunable block-1 with the features of the point cloud data output from Transformer block-1, and feeding the fused features into Transformer block-2; feeding the features of the point cloud data output from Transformer block-2 into Transformer block-3; fusing the features of the point cloud data output from fine-tunable block-3 with the features of the point cloud data output from Transformer block-3, and feeding the fused features into Transformer block-4; and feeding the features of the point cloud data output from Transformer block-4 into the detection head to obtain the 3D point cloud detection result output by the detection head. It should be noted that the number of Transformer blocks and fine-tunable blocks mentioned above is merely illustrative and not restrictive. In practical applications, the number of Transformer blocks and fine-tunable blocks can be increased or decreased according to requirements such as accuracy or speed.

[0083] In this way, the feature extraction capabilities of each Transformer block in the main branch network can be fully utilized. When the identifier of the fine-tunable block corresponding to the current Transformer block is odd, the features of the point cloud data output by the Transformer block and the features of the point cloud data output by the fine-tunable block corresponding to the current Transformer block are fused together. The fused features are then fed into the next Transformer block, which improves the flexibility of the bypass branch network in adjusting and optimizing network parameters during training, thereby improving the generalization ability of the entire point cloud 3D detection model.

[0084] In this embodiment of the disclosure, there is a one-to-one correspondence between the Transformer blocks in the main branch network and the fine-tunable blocks in the bypass branch network, and the identifier number of the first target identifier is an even number. For example, if the identifier numbers of the 1st to Nth Transformer blocks are 1, 2, 3, ..., N-1, N respectively; and the identifier numbers of the 1st to Nth fine-tunable blocks are 1, 2, 3, ..., N-1, N respectively, then when N is divisible by 2, the first target identifiers of i and j both include 2, 4, 6, ..., N; when N is not divisible by 2, the first target identifiers of i and j both include 2, 4, 6, ..., N-1. The training method involves fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to train the pre-trained model. This includes: if each Transformer block in the main branch network corresponds to a fine-tunable block in the bypass branch network, then if the first target identifier of the fine-tunable block corresponding to the current Transformer block is an even number, the features of the point cloud data output by the fine-tunable block corresponding to the current Transformer block and the features of the point cloud data output by the Transformer network are fused, and the fused features are fed into the next Transformer block; until the features of the point cloud data output by the last Transformer block are fed into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features.

[0085] Figure 4 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 3 ,like Figure 4As shown, the main branch network of the pre-trained model includes 4 Transformer blocks (denoted as Transformer block-1, Transformer block-2, Transformer block-3, and Transformer block-4); the side branch network of the pre-trained model includes 4 fine-tunable blocks (denoted as fine-tunable block-1, fine-tunable block-2, fine-tunable block-3, and fine-tunable block-4); each Transformer block corresponds to one fine-tunable block, that is, Transformer block-1 corresponds to fine-tunable block-1, Transformer block-2 corresponds to fine-tunable block-2, Transformer block-3 corresponds to fine-tunable block-3, and Transformer block-4 corresponds to fine-tunable block-4. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the fine-tuning blocks. This process includes: feeding the features of the point cloud data output from Transformer block-1 into Transformer block-2; fusing the features of the point cloud data output from fine-tuning block-2 with those from Transformer block-2, and feeding the fused features into Transformer block-3; feeding the features of the point cloud data output from Transformer block-3 into Transformer block-4; fusing the features of the point cloud data output from fine-tuning block-4 with those from Transformer block-4, and feeding the fused features into the detection head to obtain the 3D point cloud detection result output by the detection head. It should be noted that the number of Transformer blocks and fine-tuning blocks mentioned above is merely illustrative and not restrictive. In practical applications, the number of Transformer blocks and fine-tuning blocks can be increased or decreased according to requirements such as accuracy or speed.

[0086] In this way, the feature extraction capability of each Transformer block in the main branch network can be fully utilized. When the identifier of the fine-tunable block corresponding to the current Transformer block is an even number, the features of the point cloud data output by the fine-tunable block corresponding to the current Transformer block and the features of the point cloud data output by the current Transformer block are fused together. The fused features are then fed into the next Transformer block. This can improve the flexibility of the bypass branch network in adjusting and optimizing network parameters during training, thereby improving the generalization ability of the entire point cloud 3D detection model.

[0087] In this embodiment of the disclosure, the training method of updating the network parameters of the bypass branch network while fixing the network parameters of the main branch network to train the pre-trained model further includes: when each target Transformer block corresponds to a target fine-tunable block, the i-th Transformer block of the main branch network is determined as the target Transformer block, and the (j+b)-th fine-tunable block of the bypass branch network is determined as the target fine-tunable block corresponding to the i-th Transformer block, where i, j, and b are positive integers, 1≤i≤N, 1≤j≤N, 1≤b<N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, where i and j are first target identifiers. Here, the first target identifier can be any one or more values ​​from 1 to N, and the identifier number and quantity of the first target identifier can be set or adjusted according to requirements.

[0088] The first target identifier can be any value from 1 to N. For example, if the first target identifiers of i and j are both N, the pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes: fusing the features of the point cloud data output by the Nth fine-tunable block with the features of the point cloud data output by the Nbth Transformer block, and feeding the fused features into the N-b+1th Transformer block; until the features of the point cloud data output by the last Transformer block are fed into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features, where i is a positive integer and b is an integer not less than 1.

[0089] The first target identifier can be any of the values ​​from 1 to N. For example, if the first target identifiers of i and j both include 1, 2, 3, ..., Nb, then the features of the 1+b fine-tunable block are fused with the features of the 1st Transformer block, the features of the 2+b fine-tunable block are fused with the features of the 2nd Transformer block, ..., and the features of the Nth fine-tunable block are fused with the features of the Nbth Transformer block. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes: when the 1+b fine-tunable block exists, fusing the features of the point cloud data output by the 1+b fine-tunable block with the features of the point cloud data output by the 1st Transformer block, and feeding the fused features into the 1+1th Transformer block; until the features of the point cloud data output by the last Transformer block are fed into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features, where i is a positive integer and b is an integer not less than 1.

[0090] Figure 5 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 4 ,like Figure 5 As shown, the main branch network of the pre-trained model includes 4 Transformer blocks (denoted as Transformer block-1, Transformer block-2, Transformer block-3, and Transformer block-4); the side branch network of the pre-trained model includes 4 fine-tunable blocks (denoted as fine-tunable block-1, fine-tunable block-2, fine-tunable block-3, and fine-tunable block-4); each Transformer block corresponds to one fine-tunable block, that is, Transformer block-1 corresponds to fine-tunable block-1, Transformer block-2 corresponds to fine-tunable block-2, Transformer block-3 corresponds to fine-tunable block-3, and Transformer block-4 corresponds to fine-tunable block-4. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the fine-tuning block. The training method includes: when b=1 and i=1, fusing the features of the point cloud data output by Transformer block-1 with the features of the point cloud data output by fine-tuning block-2, and feeding the fused features into Transformer block-2; when b=1 and i=2, fusing the features of the point cloud data output by Transformer block-2 with the features of the point cloud data output by fine-tuning block-3, and feeding the fused features into Transformer block-3; when b=1 and i=3, fusing the features of the point cloud data output by Transformer block-3 with the features of the point cloud data output by fine-tuning block-4, and feeding the fused features into Transformer block-4; and feeding the features of the point cloud data output by Transformer block-4 into the detection head to obtain the 3D point cloud detection result output by the detection head. It should be noted that the number of Transformer blocks and the number of fine-tuning blocks mentioned above are merely illustrative and not restrictive. In practical applications, the number of Transformer blocks and fine-tuning blocks can be increased or decreased according to requirements such as accuracy or speed.

[0091] Thus, in the presence of the (i+b)th fine-tunable block, the features of the point cloud data output by the i-th Transformer block are fused with the features of the point cloud data output by the (i+b)th fine-tunable block. The fused features are then fed into the (i+1)th Transformer network. This not only fully utilizes the feature extraction capabilities of each Transformer block in the main branch network but also further enhances the ability of the side branch network to adjust and optimize network parameters during training. Since the features output by the (i+b)th fine-tunable block are richer and more accurate than those output by the i-th fine-tunable block, the accuracy of the entire point cloud 3D detection model is improved.

[0092] In this embodiment, the identifier of the first target identifier is the identifier of the last Transformer block. For example, if the identifiers of the 1st to Nth Transformer blocks are 1, 2, 3, ..., N-1, N respectively; and the identifiers of the 1st to Nth fine-tunable blocks are 1, 2, 3, ..., N-1, N respectively, then the identifier of the first target identifier is N. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes fusing the features output by the last Transformer block with the features of the point cloud data output by the last fine-tunable block, without the features of other Transformer blocks participating in the fusion. The fused features are then fed into the detection head network of the pre-trained model, which is used to obtain 3D detection results based on the input features.

[0093] Figure 6 This is a schematic diagram of the process of training a pre-trained model according to an embodiment of the present disclosure. Figure 5 ,like Figure 6As shown, the main branch network of the pre-trained model includes four Transformer blocks (denoted as Transformer block-1, Transformer block-2, Transformer block-3, and Transformer block-4); the side branch network of the pre-trained model includes four fine-tunable blocks (denoted as fine-tunable block-1, fine-tunable block-2, fine-tunable block-3, and fine-tunable block-4). The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the fine-tunable blocks. This includes fusing the features of the point cloud data output from the last fine-tunable block with the features of the point cloud data output from the last Transformer block, and then feeding the fused features into the detection head. Specifically, the features of the point cloud data output from Transformer block-1 are fed into Transformer block-2; the features of the point cloud data output from Transformer block-2 are fed into Transformer block-3; the features of the point cloud data output from Transformer block-3 are fed into Transformer block-4; the features of the point cloud data output from fine-tuning block-4 are fused with the features of the point cloud data output from Transformer block-4, and the fused features are fed into the detection head to obtain the 3D point cloud detection result output by the detection head. It should be noted that the number of Transformer blocks and fine-tuning blocks mentioned above is merely illustrative and not restrictive. In practical applications, the number of Transformer blocks and fine-tuning blocks can be increased or decreased according to requirements such as accuracy or speed.

[0094] In this way, by fusing the features of the point cloud data output by the last fine-tunable block with the features of the point cloud data output by the last Transformer block, the overall feature extraction capability of multiple Transformer blocks in the main branch network can be fully utilized, the speed at which the network parameters of the bypass branch network are adjusted and optimized during training can be improved, thereby increasing the training speed of the entire point cloud 3D detection model and saving the training time of the point cloud 3D detection model.

[0095] In this embodiment of the disclosure, the training method of updating the network parameters of the bypass branch network while fixing the network parameters of the main branch network to train the pre-trained model further includes: when multiple target Transformer blocks correspond to one target fine-tunable block, determining the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the multiple target Transformer blocks, where j is a positive integer, 1≤j≤N, N is the number of fine-tunable blocks, and j is a second target identifier. Here, the second target identifier can be any one or more values ​​from 1 to N, and the identifier number of the second target identifier can be set or adjusted according to requirements.

[0096] The second target identifier can be any value from 1 to N. For example, if the second target identifier of j is N, and the multiple target Transformer blocks include the 1st Transformer block, the 2nd Transformer block, and the Nth Transformer block, then the pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes: fusing the features of the point cloud data output by the Nth fine-tunable block with the features of the point cloud data output by the 1st, 2nd, and Nth Transformer blocks, respectively; until the features of the point cloud data output by the last Transformer block are fed into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features. It should be noted that the selection of the second target identifier can be set or adjusted according to requirements. The selection of multiple target Transformer blocks can also be set or adjusted according to requirements.

[0097] The second target identifier can be any of the values ​​from 1 to N. For example, if the second target identifier of j includes N-1 and N, the target fine-tunable block corresponding to the 1st, 3rd, 5th, ..., N-1th Transformer blocks is the N-1th fine-tunable block, and the target fine-tunable block corresponding to the 2nd, 4th, 6th, ..., Nth Transformer blocks is the Nth fine-tunable block, then the pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network. This includes: matching the features of the point cloud data output by the N-1th fine-tunable block with the features of the 1st, 3rd, 5th, ..., N-1th Transformer blocks respectively. The features of the point cloud data output from the Transformer block are fused, and the fused features are fed into the next Transformer block. The features of the point cloud data output from the Nth fine-tunable block are then fused with the features of the point cloud data output from the 2nd, 4th, 6th, ..., Nth Transformer blocks, and the fused features are fed into the next Transformer block. This process continues until the features of the point cloud data output from the last Transformer block are fed into the detection head network of the pre-trained model. This detection head network is used to obtain 3D detection results based on the input features. It should be noted that the selection of the second target identifier can be set or adjusted according to requirements. The selection of multiple target Transformer blocks can also be set or adjusted according to requirements.

[0098] In some implementations, when multiple target Transformer blocks correspond to a single target fine-tunable block, the step of determining the target fine-tunable block corresponding to these target Transformer blocks includes:

[0099] Understanding the Task and Structure: First, it's necessary to understand the role of each Transformer block in the model and the specific task or information they handle. Simultaneously, it's crucial to understand the function and capacity of each fine-tunable block, and how they affect the model's output.

[0100] Determining the correspondence: Since multiple target Transformer blocks correspond to one target fine-tunable block, it is necessary to determine the specific meaning of this correspondence. The correspondence includes the possibility that these Transformer blocks share the same fine-tunable block for fine-tuning. If these Transformer blocks handle similar or related tasks, they may share a fine-tunable block to extract common features or perform similar transformations.

[0101] Selecting the target fine-tunable block: Based on the value of j (any value from 1 to N) and the number of fine-tunable blocks N, determine which fine-tunable block will be selected as the target fine-tunable block. For example, a fine-tunable block located in the middle layer of the model may be selected to balance the information before and after, or a block with a larger capacity may be selected to process the output of multiple Transformer blocks.

[0102] Designing Bypass Branch Networks: After determining the target fine-tunable block, design bypass branch networks to connect these Transformer blocks and the fine-tunable block. This may involve adding additional layers, adjusting connection weights, or designing specific fusion strategies to effectively combine the outputs of multiple Transformer blocks and pass them to the target fine-tunable block.

[0103] Because this design involves multiple components and potentially complex interactions, different datasets and tasks are used to test the model, observe performance changes, and adjust and optimize as needed.

[0104] Thus, by allowing multiple target Transformer blocks to correspond to a single target fine-tunable block, the model gains greater structural flexibility. Different Transformer blocks can focus on processing different input features or task information, while the shared fine-tunable block can extract and integrate these features or information, enabling the model to better adapt to different tasks and datasets. By combining the outputs of multiple Transformer blocks, the target fine-tunable block receives richer feature representations, which helps the model capture more contextual information and semantic relationships, thereby improving model performance. Furthermore, the shared fine-tunable block can be fine-tuned on multiple tasks or features, further enhancing its generalization and expressive power. Compared to each Transformer block having its own independent fine-tunable block, sharing a single target fine-tunable block significantly reduces the number of model parameters. This not only helps reduce model complexity and storage requirements but also reduces computational costs, making the model more efficient in practical applications. By corresponding multiple target Transformer blocks to a single target fine-tunable block, the model design and optimization process can be simplified. Designers can focus more on selecting appropriate Transformer blocks and fine-tunable blocks without worrying too much about the complex interactions between them. In addition, this design also helps to reduce the number of hyperparameters and the search space, making model training and tuning more efficient.

[0105] In this embodiment of the disclosure, the training method of the point cloud 3D detection model further includes: for multiple point cloud 3D detection models trained under different scenarios, storing the network parameters corresponding to the bypass branch network of each point cloud 3D detection model in a first storage area, storing the network parameters of the main branch network corresponding to any point cloud 3D detection model in a second storage area, and the network parameters of the main branch networks corresponding to multiple point cloud 3D detection models are the same.

[0106] In this embodiment, the first storage area is used to store network parameters corresponding to the bypass branch network. The second storage area is used to store network parameters of the main branch network.

[0107] In this embodiment of the disclosure, different scenarios can be multiple scenarios in the same field or multiple scenarios in different fields.

[0108] In this embodiment of the disclosure, for multiple point cloud 3D detection models trained in an autonomous driving scenario, the first storage area and the second storage area can be located in an onboard computer or a remote server. For multiple point cloud 3D detection models trained in an intelligent robot scenario, the first storage area and the second storage area can be located in a local computer or a controller. For multiple point cloud 3D detection models trained in a virtual reality scenario, the first storage area and the second storage area can be located in a local computer or a server.

[0109] In contrast to the separate storage of network parameters for the main branch network and side branch network of the model for each training scenario, this disclosure stores the network parameters of the side branch network in the point cloud 3D detection model in different scenarios in the first storage area and the network parameters of the main branch network in the second storage area. For the training weights in different scenarios, only one set of network parameters needs to be stored for the main branch network part, and the network parameters of the side branch network in different scenarios can be stored independently. Since the network parameters of all main branch network parts are shared, it is no longer necessary to store M sets for M models as in the past, which greatly saves storage resources.

[0110] This disclosure provides a point cloud 3D detection method. Figure 7 This is a flowchart illustrating a point cloud 3D detection method according to an embodiment of the present disclosure. This point cloud 3D detection method can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, mobile phones, tablets, and scanning devices. In some possible implementations, the point cloud 3D detection method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 7 As shown, the point cloud 3D detection method includes:

[0111] S701: Acquire point cloud data;

[0112] S702: Input point cloud data into the point cloud 3D detection model and obtain the 3D detection results output by the point cloud 3D detection model.

[0113] The point cloud 3D detection model was trained using the aforementioned point cloud 3D detection model training method.

[0114] In this embodiment of the disclosure, the method of acquiring point cloud data is not limited.

[0115] In this embodiment of the disclosure, the point cloud 3D detection results typically contain a wealth of information that helps in the accurate and comprehensive understanding and analysis of objects and the environment. The point cloud 3D detection results include at least one of the following:

[0116] 3D coordinate information: Each point cloud data point contains its coordinate information (x, y, z) in 3D space, which directly reflects the spatial structure of the object or scene.

[0117] Intensity information: For point cloud data obtained through laser scanning, each point may also contain intensity information of laser reflection. This information can reflect the surface material, roughness, and interaction characteristics of the object with the laser, helping to distinguish between different objects or different parts of the same object.

[0118] Color information: If the point cloud data is obtained through photogrammetry or a combination of laser scanning and photogrammetry, then each point may also contain color information. Color information provides richer visual features to point cloud data, which helps in the identification and classification of objects.

[0119] Target category information: In point cloud 3D detection, algorithms typically assign a category label to each detected object or region. This category information may be based on the object's shape, size, color, or other features, such as vehicles, pedestrians, buildings, etc.

[0120] 3D bounding box information: For each detected object, the point cloud 3D detection result typically includes a 3D bounding box. The bounding box is usually defined by the coordinates of the object's center point, the length, width, and height of a cuboid (or other shape dimensional parameters), and is used to accurately represent the object's position and extent in 3D space.

[0121] Confidence score: For each detected object or feature, the detection result may also include a confidence score, which indicates the certainty or accuracy of the algorithm in the detection result.

[0122] Other attribute information: Depending on the specific application scenario and algorithm design, the point cloud 3D detection results may also include other attribute information, such as the object's posture, velocity, acceleration and other motion state information, or the object's texture, surface roughness and other physical attribute information.

[0123] Through comprehensive analysis and processing of this information, point cloud 3D detection results can provide accurate and reliable environmental perception and target recognition capabilities for fields such as autonomous driving, robot navigation, virtual reality, and 3D reconstruction, thereby promoting the development and application of related technologies.

[0124] Figure 8 A schematic diagram of the detection process based on a point cloud 3D detection model is shown, such as... Figure 8As shown, point cloud data is acquired and input into a 3D point cloud detection model. Specifically: the point cloud data is voxelized to obtain point cloud features; these features are then processed through a main branch network and a bypass branch network to output high-dimensional features; these high-dimensional features are input into the detection head to obtain 3D detection boxes. The main branch network includes multiple Transformer blocks; the bypass branch network includes multiple fine-tunable blocks, each of which includes: a first weight w1, a Rectified Linear Unit (ReLU), and a second weight w2. A fine-tunable block is typically a learning-capable component that can adjust its parameters during training to optimize model performance. w1 is the weight of the first layer, typically used for linearly transforming the input features. The first weight w1 is updated during training using a backpropagation algorithm; the size of the weight matrix depends on the dimension of the input features and the number of neurons in that layer. ReLU is a non-linear activation function used to introduce non-linear characteristics into the model. The mathematical expression for the ReLU function is f(x) = max(0,x), which sets all negative values ​​to 0 while keeping positive values ​​unchanged. In the fine-tunable block, ReLU is typically applied after a linear transformation of the first weight w1 to increase the model's expressive power. w2 is the weight of the second layer, usually used for a further linear transformation of the features processed by the ReLU activation function. The second weight w2 is updated during training via backpropagation, and the size of the weight matrix depends on the output dimension of the previous layer (i.e., the ReLU layer) and the number of neurons in that layer. The fine-tunable block structure allows the model to learn complex feature representations and optimize performance through weight updates. An additional linear transformation or normalization step can be added before adding the features output from the fine-tunable block to the features output from the Transformer block to ensure that the features are additive in terms of dimensionality and numerical range.

[0125] The technical solutions of this disclosure can improve the accuracy of point cloud 3D detection. For example, when point cloud 3D detection is applied to autonomous driving, it can provide autonomous vehicles with more accurate and comprehensive environmental perception capabilities, enabling vehicles to better identify and understand their surroundings and make safer and more reliable decisions, thereby improving the safety of autonomous vehicles. When point cloud 3D detection is applied to intelligent robots, it can provide intelligent robots with more accurate environmental perception and target recognition capabilities, thereby enhancing the performance of robots in navigation, operation, and motion control. When point cloud 3D detection is applied to virtual reality, it can provide virtual reality with more realistic and detailed scene reconstruction, thereby improving the realism of virtual reality.

[0126] This disclosure provides a training device for a point cloud 3D detection model, such as... Figure 9As shown, the training device for the point cloud 3D detection model may include: a first acquisition module 901, used to acquire point cloud data; and a training module 902, used to input the point cloud data into the main branch network and the bypass branch network of the pre-trained model, and to train the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model.

[0127] In some embodiments, the main branch network includes multiple Transformer blocks, and the bypass branch network includes multiple fine-tunable blocks.

[0128] In some embodiments, the training module 902 includes: a fusion submodule, configured to fuse the features of the point cloud data output by the target Transformer block of the main branch network and the features of the point cloud data output by the target fine-tunable block of the bypass branch network, and use the fused features as the input of the next Transformer block of the target Transformer block.

[0129] In some embodiments, the training module 902 includes: a first determining submodule, configured to, when each target Transformer block corresponds to a target fine-tunable block, determine the i-th Transformer block of the main branch network as the target Transformer block, and determine the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the i-th Transformer block; wherein i and j are positive integers, 1≤i≤N, 1≤j≤N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, i and j are first target identifiers.

[0130] In some embodiments, the training module 902 includes: a second determination submodule, configured to, when each target Transformer block corresponds to a target fine-tunable block, determine the i-th Transformer block of the main branch network as the target Transformer block, and determine the j+b-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the i-th Transformer block, wherein i, j, and b are positive integers, 1≤i≤N, 1≤j≤N, 1≤b<N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, i and j are first target identifiers.

[0131] In some embodiments, the first target identifier is any one or more values ​​from 1 to N.

[0132] In some embodiments, the training module 902 includes: a third determination submodule, configured to determine the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the multiple target Transformer blocks when multiple target Transformer blocks correspond to one target fine-tunable block, where j is a positive integer, 1≤j≤N, N is the number of fine-tunable blocks, and j is the second target identifier.

[0133] In some embodiments, the training apparatus for the point cloud 3D detection model further includes: a storage module ( Figure 9 (Not shown in the image) is used to store the network parameters of the bypass branch network of each point cloud 3D detection model trained under different scenarios in the first storage area, and store the network parameters of the main branch network of any point cloud 3D detection model in the second storage area. The network parameters of the main branch networks of multiple point cloud 3D detection models are the same.

[0134] Those skilled in the art should understand that the functions of each processing module in the point cloud 3D detection model training device of the present disclosure embodiments can be understood with reference to the relevant description of the point cloud 3D detection model training method described above. Each processing module in the point cloud 3D detection model training device of the present disclosure embodiments can be implemented by the generation circuit that implements the functions of the present disclosure embodiments, or by the software that executes the functions of the present disclosure embodiments running on an electronic device.

[0135] The point cloud 3D detection model training device of this disclosure adopts a training method that updates only the network parameters of the bypass branch network without updating the network parameters of the main branch network, which can greatly save video memory, thereby enabling the training of the point cloud 3D detection model under limited video memory and storage resources, and thus improving the accuracy of the 3D detection model.

[0136] This disclosure provides a point cloud 3D detection device, such as... Figure 10 As shown, the point cloud 3D detection device may include: a second acquisition module 1001 for acquiring point cloud data; and a detection module 1002 for inputting the point cloud data into the point cloud 3D detection model and acquiring the 3D detection results output by the point cloud 3D detection model.

[0137] The point cloud 3D detection model was trained using the point cloud 3D detection model training method described above.

[0138] Those skilled in the art should understand that the functions of each processing module in the point cloud 3D detection device of this disclosure can be understood with reference to the relevant description of the point cloud 3D detection method described above. Each processing module in the point cloud 3D detection device of this disclosure can be implemented by a generation circuit that implements the functions of this disclosure embodiment, or by running software that executes the functions of this disclosure embodiment on an electronic device.

[0139] The point cloud 3D detection device of this disclosure can improve the accuracy of point cloud 3D detection. When point cloud 3D detection is applied to autonomous driving, it can improve the safety of autonomous vehicles. When point cloud 3D detection is applied to intelligent robots, it can improve the performance of intelligent robots. When point cloud 3D detection is applied to virtual reality, it can improve the realism of virtual reality.

[0140] This disclosure provides a schematic diagram of a training scene for a point cloud 3D detection model, as shown in the embodiment. Figure 11 As shown.

[0141] As previously described, the point cloud 3D detection model training method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as in-vehicle digital assistants, in-vehicle telephones, and other similar computing devices.

[0142] Specifically, the electronic device may perform the following operations:

[0143] Acquire point cloud data;

[0144] Point cloud data is input into the main branch network and the bypass branch network of the pre-trained model. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model.

[0145] The point cloud data can be obtained from a data source. This data source can be various forms of data storage devices, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The data source can also represent various forms of mobile devices, such as in-vehicle digital assistants, in-vehicle phones, and other similar computing devices. Furthermore, the data source can be located either in the vehicle or in the cloud.

[0146] It should be understood that Figure 11 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 11Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0147] This disclosure provides a scene illustration for point cloud 3D detection, such as... Figure 12 As shown.

[0148] As described above, the point cloud 3D detection method provided in this disclosure is applied to electronic devices. Electronic devices are intended to represent various forms of digital computers, such as servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as vehicle-mounted digital assistants, vehicle-mounted telephones, and other similar computing devices.

[0149] Specifically, the electronic device may perform the following operations:

[0150] Acquire point cloud data;

[0151] Input point cloud data into the point cloud 3D detection model and obtain the 3D detection box output by the point cloud 3D detection model;

[0152] The point cloud 3D detection model was trained using the point cloud 3D detection model training method described above.

[0153] The point cloud data can be obtained from a data source. This data source can be various forms of data storage devices, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The data source can also represent various forms of mobile devices, such as in-vehicle digital assistants, in-vehicle phones, and other similar computing devices. Furthermore, the data source can be located either in the vehicle or in the cloud.

[0154] It should be understood that Figure 12 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 12 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0155] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0156] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0157] Figure 13A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0158] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. RAM 1303 may also store various programs and data required for the operation of device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0159] Multiple components in device 1300 are connected to I / O interface 1305, including: input unit 1306, such as keyboard, mouse, etc.; output unit 1307, such as various types of monitors, speakers, etc.; storage unit 1308, such as disk, optical disk, etc.; and communication unit 1309, such as network card, modem, wireless transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0160] The computing unit 1301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the training method / point cloud 3D detection method for a point cloud model. For example, in some embodiments, the training method / point cloud 3D detection method for a point cloud model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by computing unit 1301, one or more steps of the point cloud 3D detection model training method / point cloud 3D detection method described above can be performed. Alternatively, in other embodiments, computing unit 1301 can be configured to execute the point cloud 3D detection model training method / point cloud 3D detection method by any other suitable means (e.g., by means of firmware).

[0161] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0165] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0166] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0167] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0168] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a point cloud 3D detection model, comprising: Acquire point cloud data; The point cloud data is input into the main branch network and the bypass branch network of the pre-trained model. The pre-trained model is trained by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model. The main branch network includes multiple Transformer blocks, and the bypass branch network includes multiple fine-tunable blocks. The training of the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network includes: The features of the point cloud data output by the target Transformer block of the main branch network and the features of the point cloud data output by the target fine-tunable block of the bypass branch network are fused, and the fused features are used as the input of the next Transformer block of the target Transformer block; wherein the target Transformer block is one or more of a plurality of Transformer blocks, and the target fine-tunable block is one or more of a plurality of fine-tunable blocks.

2. The method according to claim 1, wherein, The method of training the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network further includes: In the case where each target Transformer block corresponds to a target fine-tunable block, the i-th Transformer block of the main branch network is determined as the target Transformer block, and the j-th fine-tunable block of the bypass branch network is determined as the target fine-tunable block corresponding to the i-th Transformer block; where i and j are positive integers, 1≤i≤N, 1≤j≤N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, i and j are the first target identifiers.

3. The method according to claim 1, wherein, The method of training the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network further includes: In the case where each target Transformer block corresponds to a target fine-tunable block, the i-th Transformer block of the main branch network is determined as the target Transformer block, and the (j+b)-th fine-tunable block of the bypass branch network is determined as the target fine-tunable block corresponding to the i-th Transformer block, where i, j, and b are positive integers, 1≤i≤N, 1≤j≤N, 1≤b<N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, where i and j are the first target identifiers.

4. The method according to claim 2 or 3, wherein, The first target identifier is any one or more values ​​from 1 to N.

5. The method according to claim 1, wherein, The training method of updating the network parameters of the bypass branch network while fixing the network parameters of the main branch network, and training the pre-trained model, includes: In the case where multiple target Transformer blocks correspond to one target fine-tunable block, the j-th fine-tunable block of the bypass branch network is determined as the target fine-tunable block corresponding to the multiple target Transformer blocks, where j is a positive integer, 1≤j≤N, N is the number of fine-tunable blocks, and j is the second target identifier.

6. The method according to claim 1, further comprising: For multiple point cloud 3D detection models trained under different scenarios, the network parameters corresponding to the bypass branch network of each point cloud 3D detection model are stored in the first storage area, and the network parameters of the main branch network corresponding to any one point cloud 3D detection model are stored in the second storage area. The network parameters of the main branch networks corresponding to the multiple point cloud 3D detection models are the same.

7. A point cloud 3D detection method, comprising: Acquire point cloud data; The point cloud data is input into the point cloud 3D detection model, and the 3D detection results output by the point cloud 3D detection model are obtained. The point cloud 3D detection model is a model obtained after training based on the point cloud 3D detection model training method as described in any one of claims 1-6.

8. A training device for a point cloud 3D detection model, comprising: The first acquisition module is used to acquire point cloud data; The training module is used to input the point cloud data into the main branch network and the bypass branch network of the pre-trained model, and to train the pre-trained model by fixing the network parameters of the main branch network and updating the network parameters of the bypass branch network to obtain the point cloud 3D detection model; wherein, the main branch network includes multiple Transformer blocks, and the bypass branch network includes multiple fine-tunable blocks; The training module includes: The fusion submodule is used to fuse the features of the point cloud data output by the target Transformer block of the main branch network and the features of the point cloud data output by the target fine-tunable block of the bypass branch network, and use the fused features as the input of the next Transformer block of the target Transformer block; wherein the target Transformer block is one or more of a plurality of Transformer blocks, and the target fine-tunable block is one or more of a plurality of fine-tunable blocks.

9. The apparatus according to claim 8, wherein, The training module includes: The first determining submodule is used to determine the i-th Transformer block of the main branch network as the target Transformer block and the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the i-th Transformer block, when each target Transformer block corresponds to a target fine-tunable block; where i and j are positive integers, 1≤i≤N, 1≤j≤N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, i and j are the first target identifiers.

10. The apparatus according to claim 8, wherein, The training module includes: The second determining submodule is used to determine the i-th Transformer block of the main branch network as the target Transformer block and the j+b-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the i-th Transformer block, when each target Transformer block corresponds to a target fine-tunable block, where i, j, and b are positive integers, 1≤i≤N, 1≤j≤N, 1≤b<N, the number of fine-tunable blocks and Transformer blocks is N, and i=j, where i and j are the first target identifiers.

11. The apparatus according to claim 9 or 10, wherein, The first target identifier is any one or more values ​​from 1 to N.

12. The apparatus according to claim 8, wherein, The training module includes: The third determining submodule is used to determine the j-th fine-tunable block of the bypass branch network as the target fine-tunable block corresponding to the multiple target Transformer blocks when multiple target Transformer blocks correspond to one target fine-tunable block, where j is a positive integer, 1≤j≤N, N is the number of fine-tunable blocks, and j is the second target identifier.

13. The apparatus according to claim 8, further comprising: The storage module is used to store the network parameters corresponding to the bypass branch network of each of the multiple point cloud 3D detection models trained under different scenarios in a first storage area, and to store the network parameters of the main branch network corresponding to any one of the point cloud 3D detection models in a second storage area, wherein the network parameters of the main branch networks corresponding to the multiple point cloud 3D detection models are the same.

14. A point cloud 3D inspection device, comprising: The second acquisition module is used to acquire point cloud data; The detection module inputs the point cloud data into the point cloud 3D detection model and obtains the 3D detection results output by the point cloud 3D detection model. The point cloud 3D detection model is a model obtained after training based on the point cloud 3D detection model training method as described in any one of claims 1-6.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program stored on a storage medium, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images