Multimodal fusion fine-tuning training method, device, electronic device and readable storage medium

Abstract

The present invention provides a multimodal fusion fine-tuning training method, device, electronic device and readable storage medium, the method includes obtaining a point cloud model P of a starting 3D model i and several multi-views; obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ; For the eigenvector Z i With the eigenvector h i Splice and get the 512-dimensional fusion feature vector f i ; Fuse feature vector f through DHC loss function i The hierarchical classification fine-tuning training is performed and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training. The fusion of two self-supervised learning feature vectors leverages the strengths of different input technology approaches, improving the retrieval accuracy of the model's feature vectors. Fine-tuning training on a small amount of classification data allows the model to quickly learn classification information preset for specific tasks. The metric learning mechanism ensures the distance between different subcategories in the feature vector space.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a multimodal fusion fine-tuning training method, device, electronic device and readable storage medium. Background Art

[0002] 3D intelligent retrieval has important application value in the industrial field, and 3D model feature extraction is the core of 3D intelligent retrieval. In addition, 3D model feature extraction can also be applied to downstream tasks such as model classification and 3D model local segmentation.

[0003] Since 3D models come in all shapes and sizes, manually labeling a large number of them is very expensive. In addition, existing supervised learning methods based on labeled data suffer from difficulties in transfer learning and weak generalization.

[0004] At present, although the training model based on self-supervised learning + single-modal fine-tuning can reduce the dependence on labeled data and effectively reduce the generalization problem of the model, there is a problem of insufficient feature signal extraction, which affects the retrieval accuracy of the downstream model retrieval task based on the feature vector. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides a multimodal fusion fine-tuning training method, device, electronic device and computer-readable storage medium to solve the problem of insufficient information extraction in the existing self-supervision + single-modal fine-tuning training technology route.

[0006] One embodiment of the present invention provides a multimodal fusion fine-tuning training method, comprising the following steps:

[0007] Get the point cloud model P of the starting 3D model i and several multi-views;

[0008] Obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ;

[0009] For the eigenvector Z i With the eigenvector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0010] The fused feature vector f is performed by the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training.

[0011] In this embodiment, by obtaining the point cloud model P of the starting 3D model i and multiple views, and respectively analyze the point cloud model P by using DGCNN and MVCNN neural networks i The feature extraction of multiple views can better extract the feature vector of the 3D model, and the use of massive unlabeled 3D model data through self-supervised learning mechanism can well solve the generalization problem of the model. Fine-tuning training of a small amount of classification data through DHC loss function can enable the model to quickly learn the classification information preset based on specific tasks, reduce the workload of manual calibration, and effectively improve the model fine-tuning effect. The fusion feature vector f is performed using the batch center similarity learning mechanism. i The metric learning fine-tuning training can ensure the distance between different subcategories in the feature vector space.

[0012] In one embodiment, a point cloud model P of a starting 3D model is obtained. i and several multi-views, including:

[0013] Use open3d open source tools to read 3D CAD models M i , get a 5000×6 point cloud model P i ;

[0014] Use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

[0015] In one embodiment, the point cloud model P is obtained by self-supervised learning. i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ,include:

[0016] From 5000×6 point cloud model P i Randomly sample 2 2048×6 point cloud data;

[0017] Perform random rotation and / or stretching on two 2048×6 point cloud data to obtain a point cloud model and point cloud models

[0018] Use DGCNN to model point cloud and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector

[0019] According to the feature vector With the eigenvector The mean of the point cloud model P iThe eigenvector Z i ;

[0020] Use the MVCNN model to extract features from the multiple views and obtain a 512-dimensional feature vector h i .

[0021] In one embodiment, the feature vector Z i With the eigenvector h i Splicing and transformation are performed to obtain a 512-dimensional fusion feature vector f i ,include:

[0022] The eigenvector Z i With the eigenvector h i After concatenation, a 512-dimensional fusion feature vector f is obtained through a DropOut layer and a fully connected layer. i .

[0023] In one embodiment, the method further comprises:

[0024] Use SimCLR loss function and VICREG loss function to control the feature vector With the eigenvector approach;

[0025] Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

[0026] In one embodiment, the fusion feature vector f is performed by using the DHC loss function. i Hierarchical classification fine-tuning training, including:

[0027] Based on the fusion feature vector f i , after two layers of full connection, we get the first-level class vector distribution C1 and the second-level class vector distribution C2 respectively;

[0028] Constructing a two-level classification framework for 3D model data, extracting several fine-tuning training models from each second-level class, and driving a neural network to learn hierarchical classification information based on the several fine-tuning training models using a DHC loss function;

[0029] Based on the primary class vector distribution C1 and the secondary class vector distribution C2, a multimodal neural network is driven to learn hierarchical classification information through a DHC loss function.

[0030] In one embodiment, the fusion feature vector f is performed by a batch center similarity learning mechanism. i Metric learning fine-tuning training, including:

[0031] Based on the first-level class vector distribution C1 and the second-level class vector distribution C2, calculate the model in the same batch of training data Its corresponding category center vector c t The distance d i , and other categories of models Distance from the center vector c t The minimum distance min(d j );

[0032] Set mind j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

[0033] One embodiment of the present invention further provides a multimodal fusion fine-tuning training device, comprising:

[0034] The first acquisition module is used to obtain the point cloud model P of the starting 3D model i and several multi-views;

[0035] The second acquisition module is used to obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ;

[0036] The splicing module is used to combine the feature vector Z i With the eigenvector h i Splicing and transformation are performed to obtain a 512-dimensional fusion feature vector f i ;

[0037] Fine-tuning training module for performing the fusion feature vector f through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training.

[0038] In this embodiment, the advantages and beneficial effects of the multimodal fusion fine-tuning training method have been explained above and will not be repeated here. Moreover, since the multimodal fusion fine-tuning training device uses the multimodal fusion fine-tuning training method, the multimodal fusion fine-tuning training device also has the same advantages and beneficial effects.

[0039] One embodiment of the present invention also provides a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0040] One embodiment of the present invention also provides an electronic device, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device performs the steps of any one of the multimodal fusion fine-tuning training methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a multimodal fusion fine-tuning training method according to one embodiment of the present invention is provided;

[0043] Figure 2 A graphical flowchart of a multimodal fusion fine-tuning training method according to one embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the hierarchical classification and metric learning process in the multimodal fusion fine-tuning training method provided in one embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the working principle of an electronic device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0048] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited to "first" and "second" may explicitly or implicitly include at least one of such features. In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or solutions that satisfy both A and B. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] 3D intelligent retrieval has important application value in the industrial field, and 3D model feature extraction is the core of 3D intelligent retrieval. In addition, 3D model feature extraction can also be applied to downstream tasks such as model classification and 3D model local segmentation.

[0050] Due to the diverse shapes of 3D models, manually labeling a large number of them is costly. Furthermore, existing supervised learning methods based on labeled data suffer from difficulties with transfer learning and poor generalization. Furthermore, with the emergence of massive amounts of 3D models, existing supervised learning mechanisms are unable to fully utilize the vast amount of unlabeled data and are unable to meet the requirements of 3D model feature extraction technology in the new era.

[0051] Self-supervised learning is a bridge from unsupervised learning to supervised learning. It automatically constructs data labels, enabling the transition from unsupervised to supervised learning. For example, it can randomly hide local patches in an image and learn to fill in missing parts of the image.

[0052] At present, self-supervised learning mechanisms have been effectively applied to image processing and natural language processing tasks, but there are relatively few studies on self-supervised mechanisms for 3D model geometric feature extraction tasks, which are mainly based on random transformations of point cloud 3D models to generate corresponding data pairs.

[0053] Based on existing research on 3D model feature extraction technology, feature extraction based on multi-view input remains ahead of other feature extraction technologies based on voxel and point cloud input. The existing CrossPoint self-supervised learning framework primarily relies on point cloud models for feature vector extraction, with a single image feature extraction branch assisting in point cloud model feature learning.

[0054] Based on the CrossPoint framework, the present invention proposes to replace the existing single image branch with a multi-view input branch, which can more comprehensively display the geometric features of the 3D model. By utilizing massive unlabeled 3D model data, the generalization problem of the model can be well solved, and the fine-tuning training of a small amount of classification data can enable the model to quickly learn the classification information preset based on specific tasks, which can greatly reduce the workload of manual calibration. Multimodal fusion feature extraction based on point cloud and multi-view input can take into account the advantages of feature extraction of different modalities and improve the information content of the model's comprehensive feature vector. This type of 3D model feature extraction method can effectively improve the retrieval accuracy of 3D model retrieval tools.

[0055] See also Figures 1-4 One embodiment of the present invention provides a multimodal fusion fine-tuning training method, comprising the following steps:

[0056] S100, obtaining a point cloud model P of the starting 3D model i and several multi-views;

[0057] In one embodiment, a point cloud model P of a starting 3D model is obtained. i and several multi-views, including:

[0058] S110, use open3d open source tools to read 3D CAD models M i , get a 5000×6 point cloud model P i ;

[0059] S120, use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

[0060] Self-supervised learning framework based on multi-view and point cloud input:

[0061] The present invention adopts Figure 2 The self-supervised learning process framework includes two branches: point cloud feature vector extraction and multi-view feature vector extraction. The SimCLR loss function and VICREG loss function are used to control the point cloud feature vector. and Infinitely close, cross-modal eigenvector Z i With h iInfinitely close. This allows us to extract geometric feature information from 3D models without any classification labels. The details are as follows:

[0062] S200, from 5000×6 point cloud model P i Randomly sample 2 2048×6 point cloud data;

[0063] S210, randomly rotate and / or stretch the two 2048×6 point cloud data to obtain a point cloud model and point cloud models

[0064] S220, using DGCNN for point cloud model and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector

[0065] S230, according to the characteristic vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i ;

[0066] S240, using the MVCNN model, extracting features from the multiple views to obtain a 512-dimensional feature vector h i .

[0067] In this embodiment, the point cloud feature extraction neural network uses a dynamic graph convolutional neural network (DGCNN): First, the open3d open source tool is used to extract 5000 point data from the starting 3D model. Each point contains: 3D coordinates and 3D normal vector information. Then, 2048 point data are randomly sampled from the 5000 point data, and randomly rotated and stretched to form and Two sets of point cloud models. By using DGCNN to extract the features of the point cloud model, the feature vectors are obtained respectively. With the eigenvector Finally, according to the eigenvector With the eigenvector The mean of the point cloud model P i The eigenvector Z i .

[0068] S300, the feature vector Z i With the eigenvector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0069] In one embodiment, the feature vector Z i With the eigenvector h i Splicing and transformation are performed to obtain a 512-dimensional fusion feature vector f i ,include:

[0070] The eigenvector Z i With the eigenvector h i After concatenation, a 512-dimensional fusion feature vector f is obtained through a DropOut layer and a fully connected layer. i .

[0071] In this embodiment, the feature vectors generated by the two supervised learning technology routes are fused as the final fused feature vector, which can take into account the advantages of different input technology routes and is conducive to improving the retrieval accuracy based on the model feature vector. In one embodiment, the method further includes:

[0072] Use SimCLR loss function and VICREG loss function to control the feature vector With the eigenvector approach;

[0073] Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

[0074] This scheme uses both SimCLR loss function and VICREG loss function to control the learning of feature vectors.

[0075] in,

[0076] The SimCLR loss function ensures and Infinitely close, cross-modal eigenvector Z i With h i Infinitely close, Z i Compared with the Z of other models in the same batch j With h j Try to stay away.

[0077] In addition, the VICREG loss function can ensure that the variance of the feature vectors of different models in the same learning batch is maximized in the same dimension, preventing the feature vectors of all 3D models from converging and causing model training collapse.

[0078] See also Figure 2 , mainly describes the self-supervised learning process of 3D model feature extraction based on multi-view and point cloud.

[0079] The process framework includes two branches: point cloud feature vector extraction and multi-view feature vector extraction.

[0080] The self-supervised learning framework batch processes N=10 3D models.

[0081] The point cloud branch uses open3d open source tools to read 3D CAD models i Get 5000×6 point cloud model P i , and then randomly sample two 2048×6 point cloud data from it, and form a point cloud model after random rotation and scaling and The dynamic graph convolutional neural network DGCNN is used to extract the features of the point cloud model, and the feature vectors are obtained respectively. and beg and The mean of the representative point cloud P i The eigenvector Z i .

[0082] The multi-view branch uses open source rendering tools to extract the 3D CAD model from the M i Rendering generates 20 multi-views (3 latitude slices of 45°, 90°, and 135°, each slice is rendered and sampled at 60° azimuth intervals, with one sampling point set at the upper and lower poles, that is, two poles are obtained). The multiple views are processed by the same VGG11 neural network (pre-trained) to obtain 20 feature vectors, which are then processed by view pooling and fully connected layers to obtain a 512-dimensional feature vector h i Multiple views are processed separately using the same pre-trained VGG11 neural network, yielding 20 feature vectors. These feature vectors are then combined into a single feature representation through view pooling. Finally, this pooled feature representation is mapped to a 512-dimensional feature vector through a fully connected layer.

[0083] View pooling aggregates the feature vectors of multiple views to obtain an overall feature representation. The specific view pooling method can be selected based on specific needs and tasks. For example, common operations such as maxpooling and mean pooling can be used.

[0084] The aggregated feature representations are then mapped to a 512-dimensional feature vector space through a fully connected layer. A fully connected layer typically includes one or more linear transformations with trainable parameters, as well as an activation function to introduce nonlinear properties. Such a fully connected layer can further process and extract features to generate higher-level representations.

[0085] Sampling SimCLR loss function and VICREG loss function control feature vector and Approaching, cross-modal eigenvector Z i With h i Approximate and prevent the characteristic parameters from inflating. Figure 3 , mainly describes the multimodal fusion fine-tuning training process based on metric learning and hierarchical classification.

[0086] The present invention constructs a two-level classification framework for three-dimensional model data based on the real industrial resource library (C1 primary class, C2 secondary class), and collects about 20 models for each secondary class for multimodal fusion fine-tuning training.

[0087] The 20 multi-views generated by 3D model rendering are used to extract the 512-dimensional feature vector h through the self-supervised pre-trained multi-view feature extraction neural network. i The point cloud data extracted from the 3D model is used to extract the 512-dimensional feature vector Z through the self-supervised pre-trained DGCNN model. i . Splicing vector h i and Z i , and then converted through a DropOut layer and a fully connected layer to obtain a 512-dimensional fusion feature vector f i The fusion feature vector f is performed through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training.

[0088] In this embodiment, hierarchical classification uses a loss function similar to the Deep Hierarchical Classification (DHC) model, which considers both the accuracy of the primary and secondary classes, as well as the correlation between the primary and secondary classes, driving the multimodal neural network to learn manually calibrated hierarchical classification information. Metric learning uses a batch center similarity learning mechanism (Triplet Center Loss) to ensure the Euclidean distance between model vectors of different categories, improving the retrieval accuracy based on the final feature vector.

[0089] In one embodiment, the fusion feature vector f is performed by using the DHC loss function. i Hierarchical classification fine-tuning training, including:

[0090] Based on the fusion feature vector f i , after two layers of full connection, we get the first-level class vector distribution C1 and the second-level class vector distribution C2 respectively;

[0091] Constructing a two-level classification framework for 3D model data, extracting several fine-tuning training models from each second-level class, and driving a neural network to learn hierarchical classification information based on the several fine-tuning training models using a DHC loss function;

[0092] Based on the primary class vector distribution C1 and the secondary class vector distribution C2, a multimodal neural network is driven to learn hierarchical classification information through a DHC loss function.

[0093] In one embodiment, the fusion feature vector f is performed by a batch center similarity learning mechanism. i Metric learning fine-tuning training, including:

[0094] Based on the first-level class vector distribution C1 and the second-level class vector distribution C2, calculate the model in the same batch of training data Its corresponding category center vector c t The distance d i , and other categories of models Distance from the center vector c t The minimum distance min(d j );

[0095] Set mind j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

[0096] In this embodiment, set min d j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector, so that different types of models have sufficient distinction intervals in the feature vector space.

[0097] The advantages of the present invention are as follows:

[0098] (1) In the image branch, 20 multiple views are used instead of a single image, so that the image branch can represent the 3D model as comprehensively as possible, thereby improving the effectiveness of the 3D model feature vector learned by the overall process framework.

[0099] (2) The VICREG loss function is used as a supplement to the SimCLR loss function, which effectively solves the problem of model parameter expansion caused by the control of the single SimCLR loss function.

[0100] (3) The present invention proposes a fine-tuning training mechanism based on hierarchical classification, which is suitable for classification tasks in an industrial context with a large number of classification labels (>1000 categories) and hierarchical characteristics, and can effectively improve the effect of multimodal fusion fine-tuning training.

[0101] (4) This paper proposes a fine-tuning training mechanism based on metric learning and hierarchical classification. The metric learning mechanism ensures the distance between different sub-categories in the feature vector space. The hierarchical classification mechanism is suitable for industrial classification tasks with a large number of classification labels (>1000 categories) and hierarchical characteristics, effectively improving the effect of multimodal fusion fine-tuning training.

[0102] (5) The present invention proposes to fuse the feature vectors generated by the two technical routes of self-supervised learning as the final feature vector, which can take into account the advantages of different input technical routes and is conducive to improving the retrieval accuracy based on the model feature vector.

[0103] One embodiment of the present invention further provides a multimodal fusion fine-tuning training device, comprising:

[0104] The first acquisition module is used to obtain the point cloud model P of the starting 3D model i and several multi-views;

[0105] The second acquisition module is used to obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i ;

[0106] The splicing module is used to combine the feature vector Z i With the eigenvector h i Splice and get the 512-dimensional fusion feature vector f i ;

[0107] Fine-tuning training module for performing the fusion feature vector f through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training.

[0108] One embodiment of the present invention also provides a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0109] One embodiment of the present invention also provides an electronic device, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device performs the steps of any one of the multimodal fusion fine-tuning training methods described above.

[0110] See also Figure 4 , Figure 4A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0111] The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled via a connector, which may include various interfaces, transmission lines, or buses, etc., although this is not limited in the present embodiment. It should be understood that in various embodiments of the present invention, coupling refers to interconnection in a specific manner, including direct connection or indirect connection through other devices, such as various interfaces, transmission lines, buses, etc.

[0112] The processor 21 may be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Alternatively, the processor 21 may be a processor group consisting of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Alternatively, the processor may be another type of processor, and the embodiments of the present invention are not limited thereto.

[0113] The memory 22 can be used to store computer program instructions and various computer program codes, including program codes for executing the embodiments of the present invention. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0114] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The output device 23 and the input device 24 can be independent devices or an integrated device.

[0115] It is understandable that in the embodiment of the present invention, the memory 22 is not only used to store relevant instructions, and the embodiment of the present invention does not limit the specific data stored in the memory.

[0116] It is understandable that Figure 4Only a simplified design of an electronic device is shown. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all video analysis devices that can implement the embodiments of the present invention are within the scope of protection of the present invention.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0118] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of the present invention have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in one embodiment, reference can be made to the descriptions of other embodiments.

[0119] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program to be sealed. The computer program to be sealed includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by a computer program instructing related hardware to perform the processes. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformation made by using the contents of the present description and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A multimodal fusion fine-tuning training method, characterized in that: The following steps are involved: Get the point cloud model P of the starting 3D model i and several multi-views; Obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the 512-dimensional feature vector h of the multi-view i ,include: From 5000×6 point cloud model P i Randomly sample two 2048×6 point cloud data; randomly rotate and / or stretch the two 2048×6 point cloud data to obtain a point cloud model and point cloud models Use DGCNN to model point cloud and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector According to the feature vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i ; Build a model hierarchical classification framework, collect several 3D models for each classification, and form fine-tuning training data with classification labels; use the MVCNN model to extract features from the multiple views and obtain a 512-dimensional feature vector h i ; For the eigenvector Z i With the eigenvector h i Splice and get the 512-dimensional fusion feature vector f i ,include: The eigenvector Z i With the eigenvector h i After concatenation, a 512-dimensional fusion feature vector f is obtained through a DropOut layer and a full-link layer. i ; The fused feature vector f is performed through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i Metric learning fine-tuning training, including: Based on the fusion feature vector f i , after two layers of fully connected layers, the first-level class vector distribution C1 and the second-level class vector distribution C2 are obtained respectively; a two-level classification framework for three-dimensional model data is constructed, and several fine-tuning training models are extracted from each second-level class. Through the DHC loss function, the neural network is driven to learn hierarchical classification information based on the several fine-tuning training data; based on the first-level class vector distribution C1 and the second-level class vector distribution C2, the multimodal neural network is driven to learn hierarchical classification information through the DHC loss function; based on the first-level class vector distribution C1 and the second-level class vector distribution C2, the model in the same batch of training data is calculated. Its corresponding category center vector c t The distance d i , and other categories of models The minimum distance min(d j ); set min(d j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

2. The multimodal fusion fine-tuning training method according to claim 1, wherein: Get the point cloud model P of the starting 3D model i and several multi-views, including: Use open3d open source tools to read 3D CAD models M i , get the 5000×6 point cloud model Pi; Use open source rendering tools to render 3D CAD models i Render and generate 20 spherical perspective screenshots.

3. The multimodal fusion fine-tuning training method according to claim 1, wherein: The method further comprises: Use SimCLR loss function and VICREG loss function to control the feature vector With the eigenvector approach; Use SimCLR loss function and VICREG loss function to control the feature vector Z i With the eigenvector h i Approaching.

4. A multimodal fusion fine-tuning training device, characterized in that: include: The first acquisition module is used to obtain the point cloud model P of the starting 3D model i and several multi-views; The second acquisition module is used to obtain the point cloud model P through self-supervised learning i The eigenvector Z i , and the multi-view 512-dimensional feature vector h i , including: from 5000×6 point cloud model P i Randomly sample two 2048×6 point cloud data; randomly rotate and / or stretch the two 2048×6 point cloud data to obtain a point cloud model and point cloud models Use DGCNN to model point cloud and point cloud models Perform feature extraction and obtain feature vectors With the eigenvector According to the feature vector With the eigenvector The mean of the point cloud model P i The eigenvector Z i ; Build a model hierarchical classification framework, collect several 3D models for each classification, and form fine-tuning training data with classification labels; use the MVCNN model to extract features from the multiple views and obtain a 512-dimensional feature vector h i ; The splicing module is used to combine the feature vector Z i With the eigenvector h i Splicing and transformation are performed to obtain a 512-dimensional fusion feature vector f i , including: combining the eigenvector Zi with the eigenvector h i After concatenation, a 512-dimensional fusion feature vector f is obtained through a DropOut layer and a full-link layer. i ; Fine-tuning training module for performing the fusion feature vector f through the DHC loss function i The hierarchical classification fine-tuning training is performed, and the fusion feature vector f is performed through the batch center similarity learning mechanism i The metric learning fine-tuning training includes: based on the fusion feature vector f i , after two layers of fully connected layers, the first-level class vector distribution C1 and the second-level class vector distribution C2 are obtained respectively; a two-level classification framework for three-dimensional model data is constructed, and several fine-tuning training models are extracted from each second-level class. Through the DHC loss function, the neural network is driven to learn hierarchical classification information based on the several fine-tuning training data; based on the first-level class vector distribution C1 and the second-level class vector distribution C2, the multimodal neural network is driven to learn hierarchical classification information through the DHC loss function; based on the first-level class vector distribution C1 and the second-level class vector distribution C2, the model in the same batch of training data is calculated. Its corresponding category center vector c t The distance d i , and other categories of models Distance from the center vector c t The minimum distance min(d j ); set min(d j )-d i >2, so that the distance between the same type of model and the center vector of this type is less than 2 compared with the distance between other types of models and the center vector.

5. An electronic device comprising: A processor and a memory, characterized in that the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device performs the steps of the multimodal fusion fine-tuning training method as described in any one of claims 1-3.

6. A readable storage medium having a computer program stored therein, characterized in that: The computer program includes program instructions, which, when executed by a processor of an electronic device, cause the processor to perform the steps of the multimodal fusion fine-tuning training method as described in any one of claims 1 to 3.

Images