Method and system for using 2D pre-trained model as backbone network for 3D downstream tasks

Abstract

The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method and system for using a 2D pre-trained model as a 3D downstream task backbone network, the method comprising: first, mapping a point cloud into high-dimensional features; then, setting a number of task symbols, splicing the task symbols with the high-dimensional features to obtain spliced ​​data; inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features to obtain updated point cloud features; based on the updated point cloud features, decoding the updated point cloud features to obtain prediction information. The present application can solve the problem in existing 3D pre-training methods that it is difficult to obtain a pre-trained model with rich knowledge due to the small amount of data in the 3D field.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method and system for using a 2D pre-trained model as a backbone network for 3D downstream tasks. Background Art

[0002] At present, the 3D general large model has attracted more and more attention at home and abroad. This technology has great value in the fields of autonomous driving, robotics, etc. The core technology of the general 3D large model is to develop a 3D pre-trained model with rich knowledge. Using this 3D pre-trained model can greatly save the deployment time of downstream tasks and save the development cost of artificial intelligence applications in the industry. Of course, there are two main types of 3D pre-training methods: generation-based methods and contrastive learning-based methods.

[0003] Both of the above methods require large-scale 3D data to obtain a 3D pre-trained model with rich knowledge. However, 3D data is expensive to obtain, 3D data is scarce, and it is extremely difficult to obtain a 3D pre-trained model with rich knowledge. Therefore, due to the small amount of data in the 3D field, it is not enough to train a pre-trained model with rich knowledge, and a pre-trained model needs to be proposed to solve 3D downstream tasks. Summary of the invention

[0004] The embodiments of the present application provide a method and system for using a 2D pre-trained model as a 3D downstream task backbone network, which solves the problem in existing 3D pre-training methods that it is difficult to obtain a pre-trained model with rich knowledge due to the small amount of data in the 3D field.

[0005] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a method for using a 2D pre-trained model as a 3D downstream task backbone network, including: first, mapping the point cloud into high-dimensional features; then, setting a number of task symbols, splicing the task symbols with the high-dimensional features into one, and obtaining spliced ​​data; next, inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features, and obtaining updated point cloud features; based on the updated point cloud features, decoding the updated point cloud features to obtain prediction information.

[0006] In some exemplary embodiments, a point cloud segmenter is used to map the point cloud into high-dimensional features, or a convolutional neural network is used to map the point cloud into high-dimensional features.

[0007] In some exemplary embodiments, the point cloud is mapped to high-dimensional features, including: first, sampling N points in the original point cloud, and aggregating its neighboring points with each point as the center to form N groups; then, mapping the information of each group to the high-dimensional feature space to obtain the high-dimensional features of each group.

[0008] In some exemplary embodiments, a farthest point sampling algorithm is used to sample in the original point cloud.

[0009] In some exemplary embodiments, a multi-layer perceptron network based on deep learning is used to map the information of each group to a high-dimensional feature space.

[0010] In some exemplary embodiments, a downstream task head is used to decode the updated point cloud features.

[0011] In the second aspect, an embodiment of the present application provides a system that uses a 2D pre-trained model as a 3D downstream task backbone network, including a sampling module, a splicing module, a data update module and a decoding module connected in sequence; the sampling module is used to map the point cloud into high-dimensional features; the splicing module includes a plurality of task symbols; the splicing module is used to splice the task symbols with the high-dimensional features into one to obtain spliced ​​data; the data update module includes a 2D pre-trained model, and the data update module is used to update the high-dimensional features using the spliced ​​data as the input of the 2D pre-trained model to obtain updated point cloud features; the decoding module includes a downstream task head, and the downstream task is used to decode the updated point cloud features according to the updated point cloud features to obtain prediction information.

[0012] In some exemplary embodiments, the sampling module includes a point cloud segmenter, which uses the point cloud segmenter to map the point cloud into high-dimensional features; or, the sampling module includes a convolutional neural network, which uses the convolutional neural network to map the point cloud into high-dimensional features.

[0013] In addition, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method of using a 2D pre-trained model as a 3D downstream task backbone network.

[0014] In addition, the present application also provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the method of using the 2D pre-trained model as the backbone network of the 3D downstream task is implemented.

[0015] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0016] The embodiment of the present application provides a method and system for using a 2D pre-trained model as a backbone network for a 3D downstream task, the method comprising: first, mapping a point cloud into high-dimensional features; then, setting a number of task symbols, splicing the task symbols with the high-dimensional features into one, and obtaining spliced ​​data; next, inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features, and obtaining updated point cloud features; based on the updated point cloud features, decoding the updated point cloud features to obtain prediction information. The method of using a 2D pre-trained model as a backbone network for a 3D downstream task provided by the embodiment of the present application uses the task symbol as input and inputs it into the backbone network of the 2D pre-trained model, and the present application can obtain rich knowledge to solve 3D downstream tasks without 3D pre-training. In addition, the method of using a 2D pre-trained model as a backbone network for a 3D downstream task provided by the present application only needs to train a point cloud segmenter, task symbols, and downstream task heads, so as to realize the use of a 2D pre-trained model to solve 3D downstream tasks, saving a lot of pre-training costs and the cost of 3D training data acquisition. At the same time, this method does not require research on 3D pre-training methods, which greatly saves research and development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0018] Figure 1 A flowchart of a method for using a 2D pre-trained model as a 3D downstream task backbone network provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a process framework of a method for using a 2D pre-trained model as a 3D downstream task backbone network provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of a system using a 2D pre-trained model as a 3D downstream task backbone network provided by an embodiment of the present application;

[0021] Figure 4 A flowchart of a system using a 2D pre-trained model as a 3D downstream task backbone network provided by another embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] As can be seen from the background technology, the current method of using 2D pre-trained models as the backbone network for 3D downstream tasks has the problem of difficulty in obtaining pre-trained models with rich knowledge due to the small amount of data in the 3D field.

[0024] Existing 3D pre-training methods mainly include two categories: one is the generation-based method, and the other is the contrastive learning-based method. The generation-based method mainly discards part of the original 3D data and then completes the lost part through a neural network. Or the original point cloud is encoded into high-dimensional features, and then decoded into the original point cloud based on the features. In this process of completion and encoding and decoding, the backbone network is pre-trained. The contrastive learning-based method compares the local and global features within the original 3D data. During the comparison process, the commonality of structural information is used to guide the backbone network to learn. Both of the above methods require large-scale 3D data to obtain a 3D pre-trained model with rich knowledge. However, the acquisition of 3D data is relatively expensive, 3D data is scarce, and it is extremely difficult to obtain a 3D pre-trained model with rich knowledge.

[0025] In order to solve this problem, the present application proposes to use a 2D pre-trained model with rich knowledge to solve 3D downstream tasks. The embodiment of the present application provides a method of using a 2D pre-trained model as the backbone network of a 3D downstream task, including: first, mapping the point cloud into high-dimensional features; then, setting a number of task symbols, splicing the task symbols with the high-dimensional features into one, and obtaining spliced ​​data; next, inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features, and obtaining updated point cloud features; based on the updated point cloud features, decoding the updated point cloud features to obtain prediction information. The present application solves 3D downstream tasks by proposing a 2D pre-trained model with rich knowledge. The method of using a 2D pre-trained model as the backbone network of a 3D downstream task provided in the embodiment of the present application, on the one hand, can obtain rich knowledge without 3D pre-training to solve 3D downstream tasks; on the other hand, it saves a lot of pre-training costs and the cost of 3D training data acquisition. At the same time, the method does not require research on 3D pre-training methods, which greatly saves research and development costs.

[0026] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0027] See also Figure 1, the embodiment of the present application provides a method for using a 2D pre-trained model as a 3D downstream task backbone network, comprising the following steps:

[0028] Step S1: Map the point cloud into high-dimensional features.

[0029] Step S2: set a number of task symbols, and splice the task symbols with high-dimensional features to obtain spliced ​​data.

[0030] Step S3: input the spliced ​​data into the backbone network of the 2D pre-trained model, update the high-dimensional features, and obtain updated point cloud features.

[0031] Step S4: Based on the updated point cloud features, the updated point cloud features are decoded to obtain prediction information.

[0032] The method of using a 2D pre-trained model as a backbone network for 3D downstream tasks provided in an embodiment of the present application can be directly applied to common downstream tasks based on deep learning in 3D point clouds, such as classification, segmentation, target detection, and registration.

[0033] Combine the following Figure 2 The method provided in this application for using a 2D pre-trained model as a 3D downstream task backbone network is specifically described.

[0034] like Figure 2 As shown, given a point cloud P and a downstream task T, the main implementation steps are as follows:

[0035] First, execute step S1 and use a point cloud tokenizer to map the point cloud into high-dimensional features.

[0036] Specifically, the FPS algorithm is first used to sample N points in the original point cloud P, and each point is used as the center to aggregate its neighboring points to form N groups. Then a multi-layer perceptron (MLP) network based on deep learning is used to map the information of each group to a high-dimensional feature space to obtain the high-dimensional features of each group.

[0037] Next, execute step S2: set a number of task tokens, splice the task tokens with high-dimensional features, and obtain spliced ​​data. Since the input data is a 3D point cloud, and the backbone network is pre-trained in 2D image data, in order to reduce the impact of data modality differences, this application splices some learnable task tokens with the N high-dimensional features obtained in step S1, so as to achieve the purpose of dynamically adjusting the point cloud features in subsequent steps.

[0038] Then, step S3 is executed to input the spliced ​​data obtained in step S2 into the backbone network of the 2D pre-trained model to complete the update of the point cloud features.

[0039] Finally, step S4 is executed. After obtaining the updated point cloud features in step S3, different downstream task heads can be used to decode the updated point cloud features into the required prediction information.

[0040] In some embodiments, a point cloud tokenizer is used to map the point cloud to high-dimensional features, or a convolutional neural network is used to map the point cloud to high-dimensional features. That is, in step S1, a point cloud tokenizer can be used, or a convolutional neural network can be used to map the point cloud to high-dimensional features, and the point cloud tokenizer can be replaced by a convolutional neural network.

[0041] In addition, the task token can be replaced by methods such as Prompts and Adapter, and this application does not limit this.

[0042] In some embodiments, as mentioned above, the point cloud is mapped into high-dimensional features in step S1, which specifically includes: first, sampling N points in the original point cloud, and aggregating its neighboring points with each point as the center to form N groups; then, mapping the information of each group to the high-dimensional feature space to obtain the high-dimensional features of each group.

[0043] In some embodiments, the farthest point sampling algorithm is used to sample in the original point cloud. In step S1, the farthest point sampling (FPS) algorithm can be used to sample N points in the original point cloud P, and each point is used as the center to aggregate its neighboring points to form N groups.

[0044] In some embodiments, a multi-layer perceptron network based on deep learning is used to map the information of each group to a high-dimensional feature space. In step S1, after the farthest point sampling algorithm is used to sample in the original point cloud to form N groups, a multi-layer perceptron network (MLP) based on deep learning is used to map the information of each group to a high-dimensional feature space to obtain high-dimensional features of each group.

[0045] In some embodiments, the updated point cloud features are decoded using a downstream task head. After obtaining the updated point cloud features, different downstream task heads (Heads) are used in step S4 to decode the updated point cloud features into the required prediction information.

[0046] See also Figure 3, an embodiment of the present application provides a system that uses a 2D pre-trained model as a 3D downstream task backbone network, including a sampling module 101, a splicing module 102, a data updating module 103 and a decoding module 104 connected in sequence; the sampling module 101 is used to map the point cloud into high-dimensional features; the splicing module 102 includes a plurality of task symbols 1021; the splicing module 102 is used to splice the task symbol 1021 with the high-dimensional features into one to obtain spliced ​​data; the data updating module 103 includes a 2D pre-trained model 1031, and the data updating module 103 is used to update the high-dimensional features using the spliced ​​data as the input of the 2D pre-trained model 1031 to obtain updated point cloud features; the decoding module 104 includes a downstream task head 1041, and the downstream task 1041 is used to decode the updated point cloud features according to the updated point cloud features to obtain prediction information.

[0047] In some embodiments, Figure 3 As shown, the sampling module 101 includes a point cloud segmenter 1011, which is used to map the point cloud into high-dimensional features; or Figure 4 As shown, the sampling module 101 includes a convolutional neural network 1012, which is used to map the point cloud into high-dimensional features.

[0048] The system provided by the embodiment of the present application uses a 2D pre-trained model as the backbone network of a 3D downstream task, including a point cloud segmenter 1011, a task symbol 1021, a 2D pre-trained model 1031 and a downstream task head 1041. The present application proposes a system framework that uses a 2D pre-trained model to solve 3D downstream tasks. It can obtain rich knowledge without 3D pre-training and solve 3D downstream tasks. In addition, the present application proposes a task symbol, and inputs the task symbol as input into the backbone network of the 2D pre-trained model to reduce the impact of data modality differences. The present application splices some learnable task symbols (Task Token) with N high-dimensional features, and can also achieve the purpose of dynamically adjusting point cloud features in subsequent steps.

[0049] The method and system provided in the embodiments of the present application for using a 2D pre-training model as a 3D downstream task backbone network can firstly save a large amount of pre-training costs, secondly save the costs of 3D training data collection, and thirdly greatly save the costs of research and development. Since the method does not require research on 3D pre-training methods, it saves research and development costs.

[0050] In addition, the method of using a 2D pre-trained model as the backbone network for 3D downstream tasks provided in an embodiment of the present application has been experimentally verified, and it has been proved that this method does not require 3D pre-training, and the accuracy of downstream tasks can obtain better performance than the 3D pre-trained model, and can achieve alignment of 3D and 2D modalities.

[0051] refer to Figure 5 Another embodiment of the present application provides an electronic device, comprising: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions executable by the at least one processor 110, and the instructions are executed by the at least one processor 110 so that the at least one processor 110 can execute any of the above method embodiments.

[0052] Among them, the memory 111 and the processor 110 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 110 and the memory 111 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art, so they are not further described in this article. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 110 is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor 110.

[0053] The processor 110 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 111 may be used to store data used by the processor 110 when performing operations.

[0054] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0055] Based on the above technical scheme, the embodiment of the present application provides a method and system for using a 2D pre-trained model as a 3D downstream task backbone network, the method comprising: first, mapping the point cloud into high-dimensional features; then, setting a number of task symbols, splicing the task symbols with the high-dimensional features into one, and obtaining spliced ​​data; next, inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features, and obtaining updated point cloud features; based on the updated point cloud features, decoding the updated point cloud features to obtain prediction information. The method of using a 2D pre-trained model as a 3D downstream task backbone network provided by the embodiment of the present application takes the task symbol as input and inputs it into the backbone network of the 2D pre-trained model, and the present application can obtain rich knowledge to solve 3D downstream tasks without 3D pre-training. In addition, the method of using a 2D pre-trained model as a 3D downstream task backbone network provided by the present application only needs to train a point cloud segmenter, task symbols, and downstream task heads, so as to realize the use of a 2D pre-trained model to solve 3D downstream tasks, saving a lot of pre-training costs and the cost of 3D training data acquisition. At the same time, this method does not require research on 3D pre-training methods, which greatly saves research and development costs.

[0056] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0057] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A method for using a 2D pre-trained model as a 3D downstream task backbone network, characterized in that: Applied to downstream tasks based on deep learning in 3D point clouds, the downstream tasks include classification, segmentation, object detection and registration; the method realizes the alignment of 3D and 2D modalities by training task symbols and downstream task heads; the method includes: Map point clouds into high-dimensional features; Set a number of task symbols, use the task symbols as input, and input them into the backbone network of the 2D pre-trained model; splicing the task symbol and the high-dimensional feature into one to obtain spliced ​​data; Inputting the spliced ​​data into the backbone network of the 2D pre-trained model, updating the high-dimensional features, dynamically adjusting the point cloud features, and obtaining updated point cloud features; Based on the updated point cloud features, the downstream task head is used to decode the updated point cloud features to obtain prediction information; Mapping the point cloud into high-dimensional features includes: The farthest point sampling algorithm is used to sample N points in the original point cloud, and each point is used as the center to aggregate its neighboring points to form N groups; A multi-layer perceptron network based on deep learning is used to map the information of each group to a high-dimensional feature space to obtain the high-dimensional features of each group.

2. The method of using a 2D pre-trained model as a 3D downstream task backbone network according to claim 1, characterized in that: A point cloud segmenter is used to map the point cloud into high-dimensional features, or a convolutional neural network is used to map the point cloud into high-dimensional features.

3. A system using a 2D pre-trained model as a 3D downstream task backbone network, characterized in that: Applied to downstream tasks based on deep learning in 3D point clouds, the downstream tasks include classification, segmentation, object detection and registration; the system realizes the alignment of 3D and 2D modalities by training task symbols and downstream task heads; the system includes a sampling module, a splicing module, a data update module and a decoding module connected in sequence; The sampling module is used to map the point cloud into high-dimensional features; The splicing module includes a plurality of task symbols, which are input into the backbone network of the 2D pre-training model using the task symbols as input; the splicing module is used to splice the task symbols with the high-dimensional features into one to obtain spliced ​​data; The data updating module includes a 2D pre-training model, and the data updating module is used to update the high-dimensional features using the spliced ​​data as input of the 2D pre-training model, dynamically adjust the point cloud features, and obtain updated point cloud features; The decoding module includes a downstream task head, and the downstream task is used to decode the updated point cloud features according to the updated point cloud features to obtain prediction information; Mapping the point cloud into high-dimensional features includes: The farthest point sampling algorithm is used to sample N points in the original point cloud, and each point is used as the center to aggregate its neighboring points to form N groups; A multi-layer perceptron network based on deep learning is used to map the information of each group to a high-dimensional feature space to obtain the high-dimensional features of each group.

4. The system of using a 2D pre-trained model as a 3D downstream task backbone network according to claim 3, characterized in that: The sampling module includes a point cloud segmenter, which maps the point cloud into high-dimensional features; Alternatively, the sampling module includes a convolutional neural network, which uses the convolutional neural network to map the point cloud into high-dimensional features.

5. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of using a 2D pre-trained model as a 3D downstream task backbone network as described in any one of claims 1 to 2.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method of using a 2D pre-trained model as a 3D downstream task backbone network as described in any one of claims 1 to 2 is implemented.

Images