Multi-modal cross-category pseudo sample construction method based on truncation splicing

Abstract

As a data enhancement method, a multi-modal cross-category pseudo sample plays an important role in model pre-training of deep learning, but most existing methods are limited to two-dimensional images and one-dimensional texts, so that wider application of the methods is limited. The invention provides a novel pseudo category generation method taking a three-dimensional point cloud sparse data structure as a reference, the method combines different components of objects belonging to two or more categories to generate pseudo samples, each pseudo category is composed of pseudo samples generated by using a plurality of objects of the same category, and each pseudo category is composed of a plurality of pseudo samples generated by using a plurality of objects of the same category. The pseudo category generated in this way is expected to extend the learning content space of the three-dimensional point cloud. On this basis, a two-dimensional image of a pseudo three-dimensional point cloud sample is generated by performing rendering-splicing on an original three-dimensional point cloud sample, and description information of a pseudo category is generated by using a large language model, and finally a multi-modal cross-category pseudo sample of a three-dimensional point cloud, a two-dimensional image and a one-dimensional text is formed. According to the method, the multi-modal cross-category pseudo samples are used for training a feature extractor under a three-dimensional point cloud small sample class increment task, so that three-dimensional point cloud small sample class increment learning is carried out.

Description

Technical Field

[0001] The present invention belongs to image recognition and image processing (G06V, G06F), and specifically relates to a multimodal cross-category pseudo sample construction method based on truncation and splicing. Background Art

[0002] For edge computing [1,2] and robotics [3] AI models that can adaptively learn like human cognition are essential for many everyday applications such as face recognition. Ideally, these models need to be able to be updated incrementally with minimal data while retaining overall knowledge, such as in facial recognition systems. [4] In scenarios such as 3D video, even updating with as little data as one image is essential.

[0003] However, many existing vision-related AI models are limited to data in the form of one-dimensional text and two-dimensional images. [5,6] , it is impossible to systematically and comprehensively learn new concepts. Due to the need to understand the real world, it is crucial to develop artificial intelligence models that can learn to represent three-dimensional objects. [7] . Humans can naturally grasp 3D concepts with very little data, but training models to simulate this process remains a challenge. To bridge this gap, small-sample incremental learning of 3D point cloud objects is naturally proposed. This method focuses on training models and incrementally learning new concepts from very few samples of 3D point cloud data. In small samples, the number of single-category samples is small, for example, less than a specified number threshold.

[0004] As proposed by Chowdhury et al. in 2022 [8] , 3D point cloud small sample class incremental learning is to divide the training and testing process into different stages. First, in the basic stage, a large amount of training data is allocated to a specific category to pre-train the model, which simulates the learning process of humans in infancy; then, the pre-trained model receives very few samples for each new category in several incremental stages (usually less than five samples per category) to gradually expand its label space, which simulates the process of humans continuing to learn from limited examples as they grow up. In each incremental stage, the data of the previous stage (including the basic stage) cannot be obtained. This requires the model to retain previously learned concepts while absorbing new knowledge.

[0005] Compared with 2D images, the complexity and sparsity of 3D point cloud data makes it challenging to fully capture and utilize the relationships between local features. Specifically, local features in 2D images, such as specific colors, shapes, or textures, can be effectively extracted and utilized, which is very beneficial for incremental learning of new categories. However, 3D point clouds mainly consist of sparse spatial information and lack the rich texture and color details in 2D images. This lack of texture and color information hinders the extraction of universal local features, which are crucial for identifying patterns and relationships between different objects or scenes, making it difficult to incrementally learn small sample classes of 3D point clouds.

[0006] Enhancing forward compatibility has been shown to be a useful approach in incremental learning of few-shot classes of images. [9] , that is, creating pseudo categories or pseudo stages in the basic stage, so that the model can better expand to unseen categories and can effectively improve the model's ability to distinguish common local features. However, traditional methods for creating pseudo categories, such as rotation, color jitter, and aliasing, are not effective for 3D point cloud data. Summary of the invention

[0007] like Figure 2 The present invention proposes a method for generating pseudo-category samples from samples of two or more categories by combining different parts of three-dimensional point cloud data of different categories, thereby forcing the backbone model to capture the common local features between different categories while reserving space for the emergence of new categories. In addition, the present invention also proposes to use other modalities, such as rendered two-dimensional images and descriptive text, to enhance the expansion capability of the three-dimensional backbone model. Therefore, the present invention also proposes a multi-modal cross-category pseudo-sample generation method and a model pre-training method based on contrastive learning and supervised learning in parallel.

[0008] The purpose of this invention is to study effective model pre-training technology under small sample incremental tasks based on the theory and methods of image enhancement and deep learning, which can utilize samples of basic categories and reserve space for possible incremental categories, so that it does not forget old knowledge while learning new knowledge.

[0009] The present invention designs a multimodal cross-category pseudo sample construction method based on truncation and splicing. This technology has made innovations in many aspects, such as pseudo sample generation of three-dimensional point clouds, pseudo sample generation of two-dimensional rendered images, pseudo sample generation of descriptive texts, and combined training of original categories and pseudo categories, thereby solving the pain points of other related technologies. Many experiments have shown that the technology proposed in the present invention has obvious performance advantages.

[0010] The present invention comprises the following steps:

[0011] Step S1: multimodal pseudo sample construction;

[0012] Step S11: constructing a three-dimensional point cloud pseudo sample.

[0013] Step S12: constructing a pseudo sample of a rendered image.

[0014] Step S13: Describe text pseudo sample construction.

[0015] Step S2: Combination of original categories and pseudo categories;

[0016] Step S21: twin object selection.

[0017] Step S22: construct pseudo positive sample pairs.

[0018] Step S3: model training process;

[0019] Step S31: Contrastive learning training based on pseudo sample pairs.

[0020] Step S32: Narrowly supervised training and broadly supervised training.

[0021] Step S33: multi-loss weighted training.

[0022] Based on a full analysis of the model pre-training paradigm in three-dimensional scenes, the present invention locates the sparsity problem of three-dimensional point cloud data format, and proposes a model pre-training technology with broad application value based on the construction of multimodal cross-category pseudo samples and a model pre-training method based on pseudo samples. This technology is used to obtain pre-trained models and perform small sample incremental learning tasks. Experimental results show that the present invention has reached a very advanced level.

[0023] The beneficial effects of the present invention are:

[0024] 1. As Figure 2 As shown, the present invention constructs multimodal cross-category pseudo-category samples based on original category samples and category information, which greatly enriches the sample organization form in the three-dimensional point cloud scene, makes the model pre-training process more sufficient, and has a better capture capability for recognizable fine-grained features.

[0025] 2. The present invention expands multimodal contrastive learning, introduces cross-category pseudo-categories into the framework of contrastive learning, and further enhances the feature distinguishability of the pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the method for constructing cross-category pseudo samples of three-dimensional point clouds in the present invention.

[0027] Figure 2 Schematic diagram of the construction method of multimodal cross-category pseudo samples in the present invention.

[0028] Figure 3 Schematic diagram of the construction method of pseudo-positive sample pairs in the present invention.

[0029] Figure 4 Schematic diagram of the pre-training method in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in combination with specific implementation methods and with reference to the accompanying drawings. It should be understood that the embodiments provided below are only for the purpose of fully and comprehensively disclosing the present invention and fully conveying the technical concept of the present invention to those skilled in the art, but are not intended to limit the scope of the present invention.

[0031] Step S1: multimodal pseudo sample construction;

[0032] Considering the sparsity of the point cloud data structure, the present invention proposes to use two or more samples to generate a pseudo-category sample. Figure 1 This is a schematic diagram of constructing pseudo samples of 3D point clouds, and introduces the cross-category pseudo sample construction method of 3D point clouds. Figure 2 The corresponding multimodal cross-category pseudo sample construction method of images and texts is introduced. The cross-category pseudo samples of point clouds, images and texts together constitute the form of multimodal cross-category pseudo samples.

[0033] Step S11: 3D point cloud pseudo sample construction

[0034] In simple terms, a pseudo sample can be generated by the following formula:

[0035]

[0036] in, is the generation process, X pseudo is a generated pseudo sample, X 0 , X 1 , …, X k is used to generate X pseudo It should be noted that X 0 , X 1 , …, X k Belong to different categories respectively.

[0037] Specifically, for each of the selected k+1 3D point cloud samples, first convert it to a front view. Then select X 0 As the “master” sample, X 1 , …, X k As a "secondary" sample, this means there is only one "primary" sample, and one or more "secondary" samples.

[0038] For the "main" sample, some points are selected from the left to the right; similarly, for the "auxiliary" sample, points are selected from the right to the left. The selection of points is based on the consideration that the edge of a sample is more distinguishable, so most of them should be retained to stimulate the model's ability to mine local relationships between samples. During the sampling process, this method always keeps the ratio of the selected points to the total number of points unchanged, that is, for a "main" sample with a total of N points, N×α points are selected from it; for an "auxiliary" sample with N′ points, since there are k "auxiliary" samples instead of a single sample, N′×(1-α)÷k points are selected, where 0<α<1. Since the three-dimensional point cloud data has spatial invariance, there is no need to consider different projection angles in this method.

[0039] Step S12: Rendering image pseudo sample construction

[0040] and the pseudo 3D point cloud sample X in step S11 pseudo Correspondingly, the present invention also needs to 0 , I 1 ,…,I k Generate fake image I pseudo Similar to the 3D point cloud samples, in order to save time, this method generates the same pseudo-3D point cloud by cutting the pre-rendered images and then splicing them together horizontally.

[0041] The two-dimensional image corresponding to the dimensional point cloud sample is shown in the following formula:

[0042] I pseudo =Concat(I 0 [:cp 0 ], I 1 [cp 1 :],…,I k [cp k :])

[0043] Among them, Concat means horizontal splicing, cp i Represents each image I i The cutting point. i ] means to keep the left half of the image after cutting, [cp i :] means keep the right half of the image after cutting.

[0044] It is worth mentioning that the present invention does not determine the cutting point of the image based on the width of the rendered image, because that may cause a mismatch between the three-dimensional sample and the two-dimensional image. Instead, this paper obtains the cutting point of the image by calculating the boundary of the selected point of the corresponding three-dimensional point cloud in the rendered image. For example, there is a situation where the part of the image where the proportion of pixels in the horizontal axis direction is less than α actually contains more than N×α points in the point cloud, which will cause misalignment between the point cloud and the image. Here, the point cloud is uniformly regarded as the basic data, and the misalignment problem is avoided by aligning the proportion of the image cutting with the point cloud.

[0045] It should be noted that, unlike the spatially invariant 3D point clouds, pseudo images need to be generated from multiple angles to ensure the comprehensiveness of the image modality. It is worth noting that although most images do not conform to the front view, each part is still horizontally spliced ​​in this method. In other words, each part can be regarded as rotated independently rather than simultaneously. If they are rotated simultaneously, different parts can easily overlap each other.

[0046] Step S13: Describe text pseudo sample construction

[0047] For each real sample, its category name can be used to construct a text description through simple prompts. However, for the description of pseudo-category samples, this method alone is not suitable. Therefore, the present invention uses a large language model to generate m different descriptive texts P1, ..., P m Then P is calculated by the following formula j with I pseudo The similarity score between j , where j∈{1,…,m}, represents one of the descriptive texts.

[0048]

[0049] in, and They are the visual encoder and text encoder in the pre-trained visual-language alignment model, and cos represents cosine similarity.

[0050] Then select the text T with the highest score, as shown in the following formula:

[0051] T=P k ,k=argmax([s1,s2,…,s m ])

[0052] Step S2: Combination of original categories and pseudo categories;

[0053] In a 3D point cloud scene, a category consists of a group of point cloud samples with the same label. In step S1, the present invention expands these samples into a multi-modal form, which is different from the single point cloud mode. Correspondingly, the present invention uses To form a single category, X, I and T represent point cloud modality samples, image modality samples and text modality samples respectively. Similarly, a pseudo category composed of multimodal pseudo samples is composed of Composition, of which Yes X i Generated as "main" samples. According to the different categories to which the selected "auxiliary" samples belong, samples of one category can constitute multiple groups of pseudo category samples. The present invention constructs K pseudo categories by randomly selecting K groups of "auxiliary" samples of different categories.

[0054] Step S21: twin pseudo sample selection.

[0055] In the present invention, the pseudo sample generated from a "main" sample is called its twin pseudo sample, that is, generally speaking, for each sample, K pseudo samples are generated using "auxiliary" samples of different categories. In other words, K groups of different auxiliary samples (each group has k auxiliary samples) are used to form K pseudo samples.

[0056] Step S22: construct positive sample pairs and pseudo positive sample pairs.

[0057] Figure 3 The construction method of pseudo-positive sample pairs is shown. Figure 3 The horizontal axis shows the point cloud samples, P1,

[0058] P1-A and P1-B are the original sample and two different twin pseudo samples generated by the original sample P1, respectively. P1…PN represent different original samples. Similarly, I and T on the vertical axis are image samples and descriptive texts, respectively. Naturally, different modalities of the same sample (i.e., P1 and I1, P1 and T1, etc.) constitute positive sample pairs, while different modalities of different samples (i.e., P1 and IN, P1 and TN, etc.) constitute negative sample pairs. Considering that a sample and its twin sample have intrinsic similarities to a certain extent, it is unreasonable to regard them as completely unrelated samples (i.e., negative sample pairs). Therefore, the present invention proposes the concept of pseudo positive sample pairs to solve this problem. All cross-modal sample pairs belonging to the same original sample and its twin sample (i.e., P1 and I1-A, P1 and T1-B, etc.) are regarded as pseudo positive sample pairs. This means that if the true positive sample pairs are ignored, they can exist as positive sample pairs.

[0059] Step S3: model training process;

[0060] In this step, the multimodal cross-category pseudo samples constructed in step S1 and the sample pairs constructed in step S2 are used to perform the model training process. The specific process is as follows: Figure 4 As shown, the present invention uses a point cloud feature extraction model, an image feature extraction model and a text feature extraction model to extract features from the input three-dimensional point cloud, image and text respectively, wherein the point cloud feature extraction model updates parameters while the other two feature extraction models freeze parameters and do not update.

[0061] Step S31: Contrastive learning training based on pseudo sample pairs.

[0062] For sample i, extract features from image, text and 3D point cloud feature extraction models and The contrast loss L(M1, M2) of any two modalities is defined as shown in the following formula, where i and j represent samples in the sample set P, respectively, and constitute a pair of positive sample pairs, τ is the temperature coefficient, and k represents any sample in the sample set P:

[0063]

[0064] The present invention adopts the contrastive learning training loss L1 of point cloud-image and point cloud-text as shown in the following formula:

[0065] L1=L(I,P)+L(T,P)

[0066] Then, the different modes of the original samples regarded as positive sample pairs are masked. At this time, there are no real positive sample pairs. The different modes of the original samples and pseudo samples in the pseudo positive sample pairs constructed by S22 are regarded as positive sample pairs. For each category, there is a contrastive learning training loss L2 based on the original class and the pseudo class as shown in the following formula:

[0067] L2=L(I′,P′)+L(T′,P′)

[0068] Among them, I′, T′, P′ represent the image, text and 3D point cloud modalities after mask processing. Compared with L1, the positive sample pairs are composed of original samples or pseudo samples, while the positive sample pairs in L2 are composed of original samples and twin pseudo samples.

[0069] Step S32: Narrowly supervised training and broadly supervised training.

[0070] The narrow supervised training loss L3 is shown in the following formula, where C all represents the narrow category number of the sample (i.e., the pseudo category and the original category are considered different categories), N represents the number of samples, o represents any sample, and c all represents any narrow category number, is the category label of the sample (unique hot encoding), σ represents the activation function (Softmax), It represents the category prediction of the sample under narrow supervision, and the loss function is the cross entropy loss.

[0071]

[0072] The generalized supervised training loss L4 is similar to the narrow supervised training loss, the difference lies in the granularity of the class labels, as shown in the following formula, where express The corresponding original category (that is, the pseudo-sample constructed from the same "main" sample is also regarded as the same category as the "main" sample, and the pseudo-category is not used for loss calculation), Represents the category prediction of the sample under generalized supervision.

[0073]

[0074] Step S33: Multi-loss weighted training

[0075] The present invention adopts multi-loss weighted training, and the total loss function after weighted summation is shown in the following formula, where α, β, γ, and ω are loss weighting parameters respectively:

[0076] L total =αL1+βL2+γL3+ωL4

[0077] The invention is used for small sample incremental learning tasks in 3D point cloud scenes. After the model is pre-trained on the basic categories (contrastive learning and supervised classification learning), a simple class mean classification method is used for incremental category prediction. After freezing the parameters of the model, for small sample incremental samples, the trained point cloud feature extraction model is used to extract the features of the small sample. To determine the center of each incremental category as an incremental update, and based on cosine similarity Perform incremental classification tests on the test samples.

[0078] In three public datasets ModelNet40

[10] 、ShapeNet

[11] 、CO3D

[12] The method was tested in , and it showed an improvement in recognition accuracy compared with other methods in each round of multiple model updates.

[0079] Table 1 ModelNet experimental results (unit: %)

[0080]

[0081]

[0082] Table 2 ShapeNet experimental results (unit: %)

[0083]

[0084] Table 3 CO3D experimental results (unit: %)

[0085]

[0086] In summary, this method proposes a multimodal cross-category pseudo sample construction method based on truncation and splicing and uses it in the pre-training process of 3D point cloud models. Experiments show that this technology has better overall performance.

[0087] The above is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

[0088] [1] Shi Weisong, Sun Hui, Cao Jie, et al. Edge computing: a new computing model in the era of the Internet of Everything[J]. Journal of Computer Research and Development, 2017, 54(05): 907-924.

[0089] [2] Shi Weisong, Zhang Xingzhou, Wang Yifan, et al. Edge computing: current status and prospects[J]. Journal of Computer Research and Development, 2019, 56(01): 69-89.

[0090] [3] Wang Tianmiao, Tao Yong. Current status of industrial robot technology and industrial development strategy in my country[J]. Journal of Mechanical Engineering, 2014, 50(09): 1-13.

[0091] [4]Li F,Wechsler H.Open set face recognition using transduction[J].IEEE transactions on pattern analysis and machine intelligence,2005,27(11):1686-1697.

[0092] [5]Betker J,Goh G,Jing L,et al.Improving image generation with bettercaptions[J].Computer Science.https: / / cdn.openai.com / papers / dall-e-3.pdf,2023,2(3):8.

[0093] [6]Wang J,Yuan H,Chen D,et al.Modelscope text-to-video technicalreport[J].arXiv preprint arXiv:2308.06571,2023.

[0094] [7]Pontil M,Verri A.Support vector machines for 3D object recognition[J].IEEE transactions on pattern analysis and machine intelligence,1998,20(6):637-646.

[0095] [8]Chowdhury T,Cheraghian A,Ramasinghe S,et al.Few-shot class-incremental learning for 3d point cloud objects[C] / / European Conference onComputer Vision.Cham:Springer Nature Switzerland,2022:204-220.

[0096] [9]Song Z,Zhao Y,Shi Y,et al.Learning with fantasy:Semantic-awarevirtual contrastive constraint for few-shot class-incremental learning[C] / / Proceedings of the IEEE / CVF conference on computer vision and patternrecognition.2023:24183-24192.

[0097]

[10] Wu Z,Song S,Khosla A,et al.3d shapenets:A deep representation forvolumetric shapes[C] / / Proceedings of the IEEE conference on computer visionand pattern recognition.2015:1912-1920.

[0098]

[11] Chang A X,Funkhouser T,Guibas L,et al.Shapenet:An information-rich 3d model repository[J].arXiv preprint arXiv:1512.03012,2015.

[0099]

[12] Reizenstein J,Shapovalov R,Henzler P,et al.Common objects in 3d:Large-scale learning and evaluation of real-life 3d category reconstruction[C] / / Proceedings of the IEEE / CVF international conference on computervision.2021:10901-10911.

Claims

1. A multimodal cross-category pseudo sample construction method based on truncation and splicing and used for model pre-training, characterized in that: The following steps are involved: Step S1: multimodal pseudo sample construction; Step S11: constructing a three-dimensional point cloud pseudo sample; Step S12: constructing a pseudo sample of a rendered image; Step S13: describing the construction of a pseudo text sample; Step S2: Combination of original categories and pseudo categories; Step S21: twin sample selection; Step S22: constructing pseudo positive sample pairs; Step S3: model training process; Step S31: Contrastive learning training based on pseudo sample pairs; Step S32: Narrow supervised training and broad supervised training; Step S33: multi-loss weighted training.

2. The method according to claim 1, characterized in that: In step S11, a pseudo 3D point cloud sample is generated by the following formula, where: is the generation process, X pseudo is the generated pseudo 3D point cloud sample, X 0 ,X 1 ,…,X k is used to generate X pseudo The original 3D point cloud sample, X 0 ,X 1 ,…,X k They belong to different categories, where each pseudo 3D point cloud sample is constructed using k+1 original 3D point cloud samples.

3. The method according to claim 1, characterized in that In step S12, the pre-rendered image of the original 3D point cloud is cut and then the cut images are horizontally spliced ​​together to generate a 2D rendered image corresponding to the pseudo 3D point cloud sample, as shown in the following formula, where Concat represents horizontal splicing, cp i Represents each image I i The cutting point of , where i represents the i-th spliced ​​image, i∈{0,1,…,k}, [:cp i ] means to keep the left half of the image after cutting, [cp i :] means retaining the right half of the cut image, where each pseudo image sample is constructed by stitching k+1 pre-rendered images. I pseudo =Concat(I 0 [:cp 0 ],I 1 [cp 1 :],…,I k [cp k :])。 4. The method according to claim 1, characterized in that: In step S13, a large language model is used to generate m different descriptive texts P1, ..., P m ; Then calculate the jth descriptive text P by the following formula j with I pseudo The similarity score between j ,in, and They are the visual encoder and text encoder in the pre-trained visual language alignment model, cos represents cosine similarity, and then the descriptive text T with the highest score is selected:

5. The method according to claim 1, characterized in that In step S21, the pseudo sample generated from one sample as the main body is called its twin pseudo sample. For each sample, K pseudo samples are generated using different types of "auxiliary" samples.

6. The method according to claim 1, characterized in that In step S22, all cross-modal sample pairs belonging to the same sample, i.e., 3D point cloud-image sample pairs and 3D point cloud-text sample pairs, are regarded as positive sample pairs; while for those cross-modal sample pairs belonging to a sample and its twin sample, they are regarded as pseudo positive sample pairs.

7. The method according to claim 1, characterized in that In step S31, after the different modes of the original samples regarded as positive sample pairs are masked, the different modes of the original samples and pseudo samples in the pseudo positive sample pairs constructed in S22 are regarded as positive sample pairs, and multimodal comparative learning is performed on them.

8. The method according to claim 1, characterized in that In step S32, a training method of parallel narrow supervised training and broad supervised training with different granularity of category labels is adopted. In the narrow supervised training, the pseudo category and the original category are regarded as different categories; while in the broad supervised training, the pseudo sample constructed with the same original sample as the main body is also regarded as the same category as the original sample, and the pseudo category is not used additionally for loss calculation.

9. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images