Multi-modal fusion fine tuning training method and device, electronic equipment and readable storage medium

Through the multimodal fusion fine-tuning training method, combined with point cloud and multi-view feature extraction, the problems of high annotation cost and poor generalization in feature extraction of 3D models are solved, and the retrieval accuracy and model generalization are improved.

CN119169405AActive Publication Date: 2024-12-20粤港澳大湾区(广东)国创中心

Patent Information

Application Number
CN202410795433.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-12-20
Estimated Expiration
2044-06-19

AI Technical Summary

Technical Problem

The existing 3D model feature extraction technology has problems such as high artificial annotation cost, difficulty in transfer learning and weak generalization. The self-supervised + single-modal fine-tuning training technology has insufficient feature signal extraction, which affects the retrieval accuracy.

Method used

By adopting the multimodal fusion fine-tuning training method, by obtaining the point cloud model and multi-view of the 3D model, using self-supervised learning and neural networks such as DGCNN and MVCNN, and extracting feature vectors through DHC loss function and batch center similarity learning mechanism Hierarchical classification and measurement learning fine-tuning to generate 512-dimensional fusion feature vectors.

Benefits of technology

It effectively improves the generalization and retrieval accuracy of the 3D model, reduces the workload of manual calibration, ensures the distance between different subcategories in the feature vector space, and improves the comprehensive feature extraction capability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119169405A_ABST
    Figure CN119169405A_ABST
Patent Text Reader

Abstract

The invention provides a multi-modal fusion fine tuning training method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a point cloud model Pi of a starting 3D model and a plurality of multi-views; obtaining a feature vector Zi of the point cloud model Pi and a multi-view 512-dimensional feature vector hi through self-supervised learning; splicing the feature vector Zi and the feature vector hi to obtain a fusion feature vector fi of 512 dimensions; hierarchical classification fine tuning training of the fusion feature vector fi is carried out through a DHC loss function, and metric learning fine tuning training of the fusion feature vector fi is carried out through a batch center similarity learning mechanism. The method integrates two feature vectors of self-supervised learning, can give consideration to the advantages of different input technical routes, facilitates the improvement of the retrieval precision of the feature vectors of the model, can enable the model to quickly obtain preset classification information based on a specific task through the fine-tuning training of a small amount of classification data, and improves the retrieval efficiency. A metric learning mechanism ensures the distance of different fine classifications in a feature vector space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a multimodal fusion fine-tuning training method, device, electronic device and readable storage medium. Background Art

[0002] 3D intelligent retrieval has important application value in the industrial field, and 3D model feature extraction is the core link of 3D intelligent retrieval. In addition, 3D model feature extraction can also be applied to downstream tasks such as model classification and 3D model local segmentation.

[0003] Since 3D models come in all shapes and sizes, it is very expensive to manually label a large number of 3D models. In addition, existing supervised learning methods for labeled data have problems with transfer learning difficulties and weak generalization.

[0004] At present, although the training model based on self-supervised learning + single-modal fine-tuning can reduce the dependence on labeled data and effectively reduce the generalization problem of the model, there is a problem of insufficient feature signal extraction, which affects the retrieval accuracy of the downstream model retrieval task based on the feature vector. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multimodal fusion fine-tuning training method, device, electronic device and computer-readable storage medium to solve the problem of insufficient information extraction in the existing self-supervision + single-modal fine-tuning training technology route.

[0006] One embodiment of the present invention provides a multimodal fusion fine-tuning training method, comprising the following steps:

[0007] Get the point cloud model P of the starting 3D model i and several multi-views;

[0008] Obtaining the point cloud model P through self-supervised learning i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ;

[0009] For the eigenvector Z i With the feature vector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0010] The fused feature vector f is performed through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

[0011] In this embodiment, by obtaining the point cloud model P of the starting 3D model i and multiple views, and respectively analyze the point cloud model P by using DGCNN and MVCNN neural networks i The feature extraction of multiple views can better extract the feature vector of the 3D model, and the use of massive unlabeled 3D model data through self-supervised learning mechanism can well solve the generalization problem of the model. Fine-tuning training of a small amount of classified data through DHC loss function can enable the model to quickly learn the classification information preset based on specific tasks, reduce the workload of manual calibration, and effectively improve the model fine-tuning effect. The fusion feature vector f is performed by batch center similarity learning mechanism i The metric learning fine-tuning training can ensure the distance between different subcategories in the feature vector space.

[0012] In one embodiment, a point cloud model P of a starting 3D model is obtained. i and several multi-views, including:

[0013] Use open3d open source tool to read 3D CAD model M i , and obtain a 5000×6 point cloud model P i ;

[0014] Use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

[0015] In one embodiment, the point cloud model P is obtained by self-supervised learning. i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ,include:

[0016] From 5000×6 point cloud model P i Randomly sample 2 2048×6 point cloud data;

[0017] Perform random rotation and / or stretching on two 2048×6 point cloud data to obtain a point cloud model and point cloud models

[0018] Use DGCNN to model point cloud and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector

[0019] According to the feature vector With the eigenvector The mean of the point cloud model P iThe eigenvector Z i ;

[0020] Use the MVCNN model to extract features from the multiple views and obtain a 512-dimensional feature vector h i .

[0021] In one embodiment, the feature vector Z i With the feature vector h i After concatenation and transformation, a 512-dimensional fusion feature vector f is obtained. i ,include:

[0022] The feature vector Z i With the feature vector h i After concatenation, a layer of DropOut layer and a layer of full connection layer are transformed to obtain a 512-dimensional fusion feature vector f i .

[0023] In one embodiment, the method further comprises:

[0024] Controlling feature vectors using SimCLR loss function and VICREG loss function With the eigenvector approach;

[0025] Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

[0026] In one embodiment, the fusion feature vector f is performed by using the DHC loss function. i Hierarchical classification fine-tuning training, including:

[0027] Based on the fused feature vector f i , after two layers of full connection layers, we get the first-level class vector distribution C1 and the second-level class vector distribution C2 respectively;

[0028] Constructing a secondary classification framework for three-dimensional model data, extracting several fine-tuning training models from each secondary class, and driving a neural network to learn hierarchical classification information based on the several fine-tuning training models through a DHC loss function;

[0029] Based on the primary class vector distribution C1 and the secondary class vector distribution C2, the multimodal neural network is driven to learn hierarchical classification information through the DHC loss function.

[0030] In one embodiment, the fusion feature vector f is performed by a batch center similarity learning mechanism. i Metric learning fine-tuning training, including:

[0031] Based on the first-level class vector distribution C1 and the second-level class vector distribution C2, calculate the model in the same batch of training data Its corresponding category center vector c t The distance d i , and other categories of models The distance from the center vector c t The minimum distance min(d j );

[0032] Set mind j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

[0033] One embodiment of the present invention further provides a multi-modal fusion fine-tuning training device, comprising:

[0034] The first acquisition module is used to acquire the point cloud model P of the starting 3D model. i and several multi-views;

[0035] The second acquisition module is used to obtain the point cloud model P through self-supervised learning. i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ;

[0036] The splicing module is used to combine the feature vector Z i With the feature vector h i After concatenation and transformation, a 512-dimensional fusion feature vector f is obtained. i ;

[0037] A fine-tuning training module is used to perform the fusion feature vector f through the DHC loss function. i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

[0038] In this embodiment, the advantages and beneficial effects of the multimodal fusion fine-tuning training method have been explained above and will not be repeated here. Moreover, since the multimodal fusion fine-tuning training device uses the multimodal fusion fine-tuning training method, the multimodal fusion fine-tuning training device also has the same advantages and beneficial effects.

[0039] One of the embodiments of the present invention further provides a readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0040] One of the embodiments of the present invention also provides an electronic device, including: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, when the processor executes the computer instructions, the electronic device performs the steps of any one of the multimodal fusion fine-tuning training methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a multimodal fusion fine-tuning training method provided in one embodiment of the present invention;

[0043] Figure 2 A graphical flowchart of a multimodal fusion fine-tuning training method provided in one embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the hierarchical classification and metric learning process in a multimodal fusion fine-tuning training method provided in one embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the working principle of an electronic device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0048] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of the features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] 3D intelligent retrieval has important application value in the industrial field, and 3D model feature extraction is the core link of 3D intelligent retrieval. In addition, 3D model feature extraction can also be applied to downstream tasks such as model classification and 3D model local segmentation.

[0050] Since 3D models come in all shapes and sizes, it is very expensive to manually label a large number of 3D models. In addition, existing supervised learning for labeled data has problems with transfer learning and poor generalization. Moreover, with the emergence of massive 3D models, existing supervised learning mechanisms cannot fully utilize the newly emerging massive unlabeled data and cannot meet the requirements of 3D model feature extraction technology under the new situation.

[0051] Self-supervised learning is a bridge from unsupervised learning to supervised learning. It realizes the transformation from unsupervised learning to supervised learning by automatically building data labels. For example: randomly hiding local blocks in an image and learning to fill in the missing parts of the image.

[0052] At present, self-supervised learning mechanisms have been effectively applied to image processing and natural language processing tasks, while there are relatively few studies on self-supervised mechanisms for 3D model geometric feature extraction tasks, which are mainly based on random transformations of point cloud 3D models to generate corresponding data pairs.

[0053] According to existing research on 3D model feature extraction technology, feature extraction technology based on multi-view input is still ahead of other feature extraction technologies based on voxel and point cloud input. The existing CrossPoint self-supervised learning framework mainly relies on point cloud models for feature vector extraction, and a single image feature extraction branch assists point cloud model feature learning.

[0054] Based on the CrossPoint framework, the present invention proposes to replace the existing single image branch with a multi-view input branch, which can more comprehensively display the geometric features of the 3D model. By utilizing massive unlabeled 3D model data, the generalization problem of the model can be well solved, and the fine-tuning training of a small amount of classified data can allow the model to quickly learn the classification information preset based on specific tasks, which can greatly reduce the workload of manual calibration. Multimodal fusion feature extraction based on point cloud and multi-view input can take into account the advantages of feature extraction of different modalities and improve the information content of the comprehensive feature vector of the model. Based on this type of 3D model feature extraction method, the retrieval accuracy of 3D model retrieval tools can be effectively improved.

[0055] See also Figure 1-Figure 4 One embodiment of the present invention provides a multi-modal fusion fine-tuning training method, comprising the following steps:

[0056] S100, obtaining a point cloud model P of a starting 3D model i and several multi-views;

[0057] In one embodiment, a point cloud model P of a starting 3D model is obtained. i and several multi-views, including:

[0058] S110, use open3d open source tool to read 3D CAD model M i , and obtain a 5000×6 point cloud model P i ;

[0059] S120, use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

[0060] Self-supervised learning framework based on multi-view and point cloud input:

[0061] The present invention adopts Figure 2 The self-supervised learning process framework includes two branches: point cloud feature vector extraction and multi-view feature vector extraction. The SimCLR loss function and VICREG loss function are used to control the point cloud feature vector. and Infinitely close, cross-modal eigenvector Z i With h iInfinitely close. This allows the extraction of geometric feature information from 3D models without classification labels. The details are as follows:

[0062] S200, from 5000×6 point cloud model P i Randomly sample 2 2048×6 point cloud data;

[0063] S210, performing random rotation and / or stretching processing on the two 2048×6 point cloud data to obtain a point cloud model and point cloud models

[0064] S220, using DGCNN for point cloud model and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector

[0065] S230, according to the characteristic vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i ;

[0066] S240, using the MVCNN model, extracting features from the plurality of multi-views to obtain a 512-dimensional feature vector h i .

[0067] In this embodiment, the point cloud feature extraction neural network uses a dynamic graph convolutional neural network (DGCNN): First, the open3d open source tool is used to extract 5000 point data from the starting 3D model, each point contains: 3D coordinates and 3D normal vector information. Then, 2048 point data are randomly sampled from the 5000 point data, and randomly rotated and stretched to form and Two sets of point cloud models. By using DGCNN to extract the features of the point cloud model, the feature vectors are obtained respectively. With the eigenvector Finally, according to the feature vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i .

[0068] S300, the feature vector Z i With the feature vector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0069] In one embodiment, the feature vector Z i With the feature vector h i After concatenation and transformation, a 512-dimensional fusion feature vector f is obtained. i ,include:

[0070] The feature vector Z i With the feature vector h i After concatenation, a layer of DropOut layer and a layer of full connection layer are transformed to obtain a 512-dimensional fusion feature vector f i .

[0071] In this embodiment, the feature vectors generated by the two technical routes of self-supervised learning are fused as the final fused feature vector, which can take into account the advantages of different input technical routes and is conducive to improving the retrieval accuracy based on the model feature vector. In one embodiment, the method also includes:

[0072] Controlling feature vectors using SimCLR loss function and VICREG loss function With the eigenvector approach;

[0073] Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

[0074] This scheme uses both SimCLR loss function and VICREG loss function to control the learning of feature vectors.

[0075] in,

[0076] The SimCLR loss function ensures and Infinitely close, cross-modal eigenvector Z i With h i The infinite approach of Z i Compared with other models in the same batch, Z j With h j Try to stay away.

[0077] In addition, the VICREG loss function can ensure that the variance of the feature vectors of different models in the same learning batch is maximized in the same dimension, preventing the feature vectors of all 3D models from converging and causing model training collapse.

[0078] See also Figure 2 , mainly describes the self-supervised learning process of 3D model feature extraction based on multi-view and point cloud.

[0079] The process framework includes two branches: point cloud feature vector extraction and multi-view feature vector extraction.

[0080] The self-supervised learning framework batches N = 10 3D models.

[0081] The point cloud branch uses open3d open source tools to read 3D CAD models M i Get 5000×6 point cloud model P i , and then randomly sample 2 2048×6 point cloud data from it, and form a point cloud model after random rotation and scaling and The dynamic graph convolutional neural network DGCNN is used to extract the features of the point cloud model and obtain the feature vectors and beg and The mean of the point cloud P is obtained. i The eigenvector Z i .

[0082] The multi-view branch uses open source rendering tools to transform the 3D CAD model into i Rendering generates 20 multi-views (3 latitude slices of 45°, 90°, and 135°, each slice is rendered and sampled at an interval of 60° in azimuth, and one sampling point is set at the upper and lower poles, that is, two poles are obtained). The multiple views are processed by the same VGG11 neural network (pre-trained) to obtain 20 feature vectors, which are then processed by view pooling and fully connected layers to obtain a 512-dimensional feature vector h i . After multiple views are processed by the same pre-trained VGG11 neural network, 20 feature vectors are obtained. Then, these feature vectors are combined into a single feature representation through view pooling. Finally, this pooled feature representation is mapped to a 512-dimensional feature vector through a fully connected layer.

[0083] View pooling is an operation that aggregates feature vectors of multiple views to obtain an overall feature representation. The specific view pooling method can be selected according to specific needs and tasks. For example, common operations such as maxpooling and mean pooling can be used.

[0084] Subsequently, the pooled feature representation is mapped to a 512-dimensional feature vector space through a fully connected layer. A fully connected layer usually includes one or more linear transformations with training parameters, as well as an activation function to introduce nonlinear properties. Such a fully connected layer can further process and extract features to generate higher-level expressive capabilities.

[0085] Sampling SimCLR loss function and VICREG loss function control feature vector and Approaching, the cross-modal eigenvector Z i With h i approach and prevent the characteristic parameters from inflating. Figure 3 , mainly describes the multimodal fusion fine-tuning training process based on metric learning and hierarchical classification.

[0086] The present invention constructs a two-level classification framework of three-dimensional model data based on the real industrial resource library (C1 primary class, C2 secondary class), collects about 20 models for each secondary class for multimodal fusion fine-tuning training.

[0087] The 20 multi-views generated by 3D model rendering are used to extract a 512-dimensional feature vector h through the self-supervised pre-trained multi-view feature extraction neural network. i The point cloud data extracted from the 3D model is used to extract a 512-dimensional feature vector Z through the self-supervised pre-trained DGCNN model. i . Splicing vector h i and Z i , and then converted through a DropOut layer and a fully connected layer to obtain a 512-dimensional fusion feature vector f i The fusion feature vector f is obtained by using the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

[0088] In this embodiment, the hierarchical classification adopts a loss function similar to the deep hierarchical classification model (DHC), which considers both the classification accuracy of the primary and secondary classes and the correlation between the primary and secondary classes, driving the multimodal neural network to learn the manually calibrated hierarchical classification information. The batch center similarity learning mechanism (TripletCenter Loss) adopted by metric learning ensures the Euclidean distance between model vectors of different categories and improves the retrieval accuracy based on the final feature vector.

[0089] In one embodiment, the fusion feature vector f is performed by using the DHC loss function. i Hierarchical classification fine-tuning training, including:

[0090] Based on the fused feature vector f i , after two layers of full connection layers, we get the first-level class vector distribution C1 and the second-level class vector distribution C2 respectively;

[0091] Constructing a secondary classification framework for three-dimensional model data, extracting several fine-tuning training models from each secondary class, and driving a neural network to learn hierarchical classification information based on the several fine-tuning training models through a DHC loss function;

[0092] Based on the primary class vector distribution C1 and the secondary class vector distribution C2, the multimodal neural network is driven to learn hierarchical classification information through the DHC loss function.

[0093] In one embodiment, the fusion feature vector f is performed by a batch center similarity learning mechanism. i Metric learning fine-tuning training, including:

[0094] Based on the first-level class vector distribution C1 and the second-level class vector distribution C2, calculate the model in the same batch of training data Its corresponding category center vector c t The distance d i , and other categories of models The distance from the center vector c t The minimum distance min(d j );

[0095] Set mind j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

[0096] In this embodiment, the min d j )-d i >2, so that the distance between the same type of model and the central vector of this type is less than 2 compared with the distance between other types of models and the central vector, so that different types of models have sufficient distinguishing intervals in the feature vector space.

[0097] The advantages of the present invention are as follows:

[0098] (1) In the image branch, 20 multiple views are used instead of a single image, so that the image branch can represent the 3D model as comprehensively as possible, thereby improving the effectiveness of the 3D model feature vector learned by the overall process framework.

[0099] (2) The VICREG loss function is used as a supplement to the SimCLR loss function, which effectively solves the problem of model parameter expansion caused by the control of the single SimCLR loss function.

[0100] (3) The present invention proposes a fine-tuning training mechanism based on hierarchical classification, which is suitable for classification tasks in industrial contexts with many types of classification labels (>1000 categories) and hierarchical characteristics, and can effectively improve the effect of multimodal fusion fine-tuning training.

[0101] (4) The present invention proposes a fine-tuning training mechanism based on metric learning + hierarchical classification. The metric learning mechanism ensures the distance between different subcategories in the feature vector space. The hierarchical classification mechanism is suitable for classification tasks with many types of classification labels (>1000 categories) and hierarchical characteristics in an industrial context, and effectively improves the effect of multimodal fusion fine-tuning training.

[0102] (5) The present invention proposes to fuse the feature vectors generated by the two technical routes of self-supervised learning as the final feature vector, which can take into account the advantages of different input technical routes and is conducive to improving the retrieval accuracy based on the model feature vector.

[0103] One embodiment of the present invention further provides a multi-modal fusion fine-tuning training device, comprising:

[0104] The first acquisition module is used to acquire the point cloud model P of the starting 3D model. i and several multi-views;

[0105] The second acquisition module is used to obtain the point cloud model P through self-supervised learning. i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ;

[0106] The splicing module is used to combine the feature vector Z i With the feature vector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0107] A fine-tuning training module is used to perform the fusion feature vector f through the DHC loss function. i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

[0108] One of the embodiments of the present invention further provides a readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0109] One of the embodiments of the present invention also provides an electronic device, including: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, when the processor executes the computer instructions, the electronic device performs the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0110] See also Figure 4 , Figure 4A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0111] The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines, or buses, etc., which are not limited in the embodiments of the present invention. It should be understood that in various embodiments of the present invention, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, through various interfaces, transmission lines, buses, etc.

[0112] The processor 21 may be one or more graphics processing units (GPUs). When the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group consisting of multiple GPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present invention.

[0113] The memory 22 can be used to store computer program instructions and various computer program codes including program codes for executing the scheme of the present invention. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0114] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The output device 23 and the input device 24 can be independent devices or an integrated device.

[0115] It is understandable that in the embodiment of the present invention, the memory 22 is not only used to store related instructions, and the embodiment of the present invention does not limit the specific data stored in the memory.

[0116] Understandably, Figure 4Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all video analysis devices that can implement the embodiments of the present invention are within the protection scope of the present invention.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present invention have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program to be sealed. The computer program to be sealed includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0123] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0124] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A multimodal fusion fine-tuning training method, characterized in that: The following steps are involved: Get the point cloud model P of the starting 3D model i and several multi-views; Obtaining point cloud model P through self-supervised learning i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ; For the eigenvector Z i With the feature vector h i Splice and get the 512-dimensional fusion feature vector f i ; The fused feature vector f is performed through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

2. The multimodal fusion fine-tuning training method according to claim 1, characterized in that: Get the point cloud model P of the starting 3D model i and several multi-views, including: Use open3d open source tool to read 3D CAD model M i , and obtain a 5000×6 point cloud model P i ; Use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

3. The multimodal fusion fine-tuning training method according to claim 1, characterized in that: Obtaining point cloud model P through self-supervised learning i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ,include: From 5000×6 point cloud model P i Randomly sample 2 2048×6 point cloud data; Perform random rotation and / or stretching on two 2048×6 point cloud data to obtain a point cloud model and point cloud models Use DGCNN to model point cloud and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector According to the feature vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i ; Construct a model hierarchical classification framework, collect several 3D models for each classification, and form fine-tuning training data with classification labels; Using the MVCNN model, the features of the multiple views are extracted to obtain a 512-dimensional feature vector h i .

4. The multimodal fusion fine-tuning training method according to claim 1, characterized in that: The method further comprises: Controlling feature vectors using SimCLR loss function and VICREG loss function With the eigenvector approach; Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

5. The multimodal fusion fine-tuning training method according to claim 1, characterized in that: For the eigenvector Z i With the feature vector h i Splice and get the 512-dimensional fusion feature vector f i ,include: The feature vector Z i With the feature vector h i After concatenation, a layer of DropOut layer and a layer of full connection layer are transformed to obtain a 512-dimensional fusion feature vector f i .

6. The multimodal fusion fine-tuning training method according to claim 4, characterized in that: The fusion feature vector f is performed by using the DHC loss function i Hierarchical classification fine-tuning training, including: Based on the fused feature vector f i , after two layers of full connection layers, we get the first-level class vector distribution C1 and the second-level class vector distribution C2 respectively; Constructing a secondary classification framework for three-dimensional model data, extracting a number of fine-tuning training models from each secondary class, and driving a neural network to learn hierarchical classification information based on the number of fine-tuning training data through a DHC loss function; Based on the primary class vector distribution C1 and the secondary class vector distribution C2, the multimodal neural network is driven to learn hierarchical classification information through the DHC loss function.

7. The multimodal fusion fine-tuning training method according to claim 4, characterized in that: The fused feature vector f is obtained by batch center similarity learning mechanism. i Metric learning fine-tuning training, including: Based on the first-level class vector distribution C1 and the second-level class vector distribution C2, calculate the model in the same batch of training data Its corresponding category center vector c t The distance d i , and other categories of models The distance from the center vector c t The minimum distance min(d j ); Set min(d j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

8. A multimodal fusion fine-tuning training device, characterized in that: include: The first acquisition module is used to acquire the point cloud model P of the starting 3D model i and several multi-views; The second acquisition module is used to obtain the point cloud model P through self-supervised learning. i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ; The splicing module is used to combine the feature vector Z i With the feature vector h i After concatenation and transformation, a 512-dimensional fusion feature vector f is obtained. i ; A fine-tuning training module is used to perform the fusion feature vector f through the DHC loss function. i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is obtained through the batch center similarity learning mechanism. i Metric learning fine-tuning training.

9. An electronic device, comprising: A processor and a memory, characterized in that the memory is used to store computer program codes, the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the steps of the multimodal fusion fine-tuning training method as described in any one of claims 1-7.

10. A readable storage medium having a computer program stored therein, characterized in that: The computer program includes program instructions, and when the program instructions are executed by a processor of an electronic device, the processor executes the steps of the multimodal fusion fine-tuning training method as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Image classification method for constructing category center

    CN111079790A

  • Apparatus related to metric-learning-based data classification and method thereof

    CN111373417A

  • Three-dimensional model retrieval method based on multi-modal fusion

    CN112270762A

  • Image classification robustness improvement method based on metric learning

    CN114723990A

  • Three-dimensional feature extraction method and device, electronic equipment and readable storage medium

    CN117541810A

Cited By

  • Target retrieval model obtaining method, retrieval method and device of 3D model

    CN120356029A