Multi-modal feature fusion target classification method and system based on double-tower structure

Abstract

The invention provides a multi-modal feature fusion target classification method and system based on a double-tower structure, and relates to the technical field of target recognition, and the method comprises the steps: constructing a multi-modal unsupervised data set and a supervised data set; performing data feature extraction of different modes on the multi-mode unsupervised data set and the supervised data set; performing unsupervised training and supervised training on the double-tower multimode feature fusion model; a double-tower multi-mode feature fusion model is adopted to carry out multi-mode feature query on a target object, and multi-mode target classification is realized based on similarity retrieval. According to the method, the data features of the multi-modal data are dynamically fused by constructing the double-tower multi-modal feature fusion model, so that complementation among different modal features is realized, and the problem of model incompatibility caused by insufficient single-modal data expression ability and modal deficiency is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a multimodal feature fusion target classification method and system based on a double-tower structure. Background Art

[0002] Image target classification refers to automatically determining the predetermined category of the original image through a computer program. Traditional image classification technology mainly relies on manually designed features and classic machine learning algorithms. In recent years, the introduction of deep learning models, especially convolutional neural networks, has greatly promoted the development of image classification technology. Deep learning models can automatically learn useful feature representations from original images, greatly improving the accuracy of classification. In some specific fields, target classification is also a crucial task, which involves identifying and distinguishing different types of targets for effective supervision, decision-making and action. However, due to the complex and changeable target environment, it is difficult to obtain complete information of a single modality for some target objects. For example, only local images or partial descriptions of the target objects can be obtained, and local target information is easily confused, making it difficult to express the global information of the target object. Summary of the invention

[0003] The present invention provides a multimodal feature fusion target classification method and system based on a dual-tower structure, which solves the problem of model incompatibility caused by insufficient single-modal expression ability of the target object and modality loss in the existing target classification technology.

[0004] In a first aspect, an embodiment of the present invention provides a multimodal feature fusion target classification method based on a dual-tower structure, the method comprising the following process: Acquire multimodal data and construct multimodal unsupervised and supervised datasets based on the multimodal data; Extract data features of different modalities from multimodal unsupervised data sets and supervised data sets to construct unsupervised feature training sets and supervised feature training sets; Based on the unsupervised feature training set and the supervised feature training set, the dual-tower multi-mode feature fusion model is trained unsupervised and supervised; The dual-tower multimodal feature fusion model after unsupervised training and supervised training is used to perform multimodal feature query on the target object, and multimodal target classification is achieved based on similarity retrieval.

[0005] In the above embodiment, the present invention constructs a dual-tower multi-modal feature fusion model to dynamically fuse the data features of multi-modal data to achieve complementarity between different modal features, thereby solving the problem of insufficient expression ability of single modal data and model incompatibility caused by modality missing.

[0006] As some optional implementation modes of the present application, the multimodal data includes text data, image data and signal data.

[0007] In the above-mentioned embodiments, the multimodal data involved in the present invention has stronger expression ability and can achieve complementarity between different modal data.

[0008] As some optional implementations of the present application, the process of constructing a multimodal unsupervised dataset and a supervised dataset based on multimodal data is as follows: Acquire original data of multimodal data; wherein the original data of the multimodal data includes original data without category labels and original data with category labels; For the original data without category labels, the enhanced data is constructed by deleting part of the original data information, and a data pair is constructed based on the original data and the enhanced data to obtain a multimodal unsupervised dataset; For the original data with category labels, data pairs are constructed based on the original data of two target objects to obtain a multimodal supervised dataset.

[0009] In the above embodiment, the present invention constructs an unsupervised data set and a supervised data set, so that the dual-tower multi-mode feature fusion model can effectively realize unsupervised learning and supervised learning.

[0010] As some optional implementations of the present application, the process of extracting data features of different modalities for multimodal unsupervised data sets and supervised data sets is as follows: If the modality of the data is image data, a pre-trained image feature extraction model is used to extract image features from the image data to obtain image features; If the modality of the data is text data, a pre-trained text feature extraction model is used to extract text features to obtain text features; If the mode of the data is signal data, a pre-trained residual neural network is used to extract signal features from the signal data to obtain signal features.

[0011] As some optional implementation modes of the present application, the dual-tower multi-mode feature fusion model includes a feature input layer, a feature mapping layer, a feature normalization layer, a feature splicing layer, a feature fusion layer and a similarity calculation layer.

[0012] In the above embodiment, the present invention constructs a dual-tower multimodal feature fusion model to input the data feature pairs of multimodal data into the dual towers. The model is trained and judged based on the similarity between the features of the two objects, which solves the problem that the existing model has poor adaptability to data of different scales.

[0013] As some optional implementation methods of the present application, the process of unsupervised training and supervised training of the dual-tower multi-mode feature fusion model based on the unsupervised feature training set and the supervised feature training set is as follows: Based on the data features of the unsupervised feature training set, the dual-tower multi-mode feature fusion model is unsupervisedly trained; Based on the data features of the supervised feature training set, the dual-tower multi-mode feature fusion model after unsupervised training is supervised trained; The data features include image features, text features and signal features.

[0014] In the above embodiment, the present invention constructs a dual-tower multi-modal feature fusion model based on contrastive learning that is compatible with unsupervised learning and supervised learning, and implements model training on unlabeled data and labeled data through contrastive learning, thereby solving the problem of difficulty in obtaining labeled data.

[0015] As some optional implementation methods of the present application, the process of performing unsupervised training on the dual-tower multi-mode feature fusion model based on the data features of the unsupervised feature training set is as follows: Input the data features of the unsupervised training set into the dual-tower multi-mode feature fusion model; The data features are mapped, normalized, and fused by the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object. The dual-tower multimodal feature fusion model is used to calculate the similarity of the multimodal features of the target objects to obtain the multimodal feature similarity between the target objects, and a cross-entropy loss function is constructed based on the multimodal feature similarity. The unsupervised training of the dual-tower multimodal feature fusion model is achieved through iterative training of the cross-entropy loss function.

[0016] In the above embodiments, the present invention uses unsupervised learning to enhance the robustness of the model to missing information, and uses a dynamic feature extraction method to deal with the missing modality, which effectively improves the generalization ability of the model in different scenarios.

[0017] As some optional implementation methods of the present application, the process of supervised training of the dual-tower multi-mode feature fusion model after unsupervised training based on the data features of the supervised feature training set is as follows: Input the data features of the supervised training set into the dual-tower multi-mode feature fusion model after unsupervised training; The data features are mapped, normalized, and fused by the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object. The dual-tower multimodal feature fusion model is used to calculate the similarity of the multimodal features of the target objects to obtain the multimodal feature similarity between the target objects, and a cross-entropy loss function is constructed based on the multimodal feature similarity. The supervised training of the dual-tower multimodal feature fusion model is achieved through iterative training of the cross-entropy loss function.

[0018] In the above embodiment, the present invention distinguishes target categories based on the feature similarity between target objects, and reduces the impact of missing details and feature differences of similar targets by combining a dual-tower structure with a supervised learning reinforcement model's perception of related features of similar target features and sensitivity to feature differences of different categories of targets.

[0019] As some optional implementation methods of the present application, the dual-tower multi-modal feature fusion model after unsupervised training and supervised training is used to perform multi-modal feature query on the target object, and the process of implementing multi-modal target classification based on similarity retrieval is as follows: Acquire multimodal data of the target object to be classified, and extract data features of different modes from the multimodal data to obtain data features of the target object to be classified; The data features of the target object to be classified are input into the dual-tower multi-mode feature fusion model after unsupervised training and supervised training, and the data features are subjected to feature mapping, normalization, feature splicing and fusion processing through the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object to be classified; A feature query is performed based on the multimodal features of the target object to obtain multimodal features of several known categories, and the category corresponding to the multimodal feature with the highest similarity is selected as the multimodal target classification result through similarity calculation.

[0020] In the above embodiment, the present invention can realize feature learning in supervised and unsupervised modes at the same time by constructing a dual-tower architecture model, and improves the basic learning ability of the model for domain data based on a large amount of unlabeled data, and improves the model's ability to perceive feature differences of different categories with a small amount of labeled data, thereby breaking through the problem of difficulty in obtaining data for certain specific application scenarios and achieving accurate target classification.

[0021] In a second aspect, the present invention provides a multimodal feature fusion target classification system based on a dual-tower structure, the system comprising: An original data acquisition unit, the original data acquisition unit is used to acquire multimodal data and construct a multimodal unsupervised data set and a supervised data set based on the multimodal data; A data feature construction unit, wherein the data feature construction unit is used to extract data features of different modalities from a multimodal unsupervised data set and a supervised data set to construct an unsupervised feature training set and a supervised feature training set; A model joint training unit, wherein the model joint training unit performs unsupervised training and supervised training on the dual-tower multi-mode feature fusion model based on an unsupervised feature training set and a supervised feature training set; A multimodal target classification unit uses a dual-tower multimodal feature fusion model after unsupervised training and supervised training to perform multimodal feature query on the target object, and realizes multimodal target classification based on similarity retrieval.

[0022] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multimodal feature fusion target classification method based on a dual-tower structure when executing the computer program.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal feature fusion target classification method based on a dual-tower structure is implemented.

[0024] The beneficial effects of the present invention are as follows: 1. The present invention constructs a dual-tower multi-modal feature fusion model that uses unsupervised learning to enhance the robustness of the model to missing information, and uses a dynamic feature extraction method to deal with the missing modality, effectively improving the generalization ability of the model in different scenarios.

[0025] 2. The present invention distinguishes target categories based on the feature similarity between target objects. It reduces the impact of missing details and feature differences of similar targets by combining a dual-tower structure with a supervised learning reinforcement model's ability to perceive related features of similar target features and its sensitivity to feature differences of different categories of targets.

[0026] 3. By constructing a dual-tower multi-mode feature fusion model, the present invention can simultaneously realize feature learning in supervised and unsupervised modes, and based on a large amount of unlabeled data, improve the model's basic learning ability for domain data, and combine a small amount of labeled data to improve the model's ability to perceive feature differences of different categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1is a schematic diagram of the computer device structure of the hardware operating environment described in the embodiment of the present invention; Figure 2 is a flow chart of a multimodal feature fusion target classification method according to an embodiment of the present invention; Figure 3 It is a structural diagram of the dual-tower multi-mode feature fusion model described in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] In order to solve the problem of model incompatibility caused by insufficient single modal expression ability and modality loss of existing target classification technology, the embodiment of the present invention provides a multimodal feature fusion target classification method and system based on a dual-tower structure. Before introducing the specific technical solution of this application, the hardware operating environment involved in the embodiment of this application is first introduced.

[0031] See also Figure 1 , Figure 1 The figure is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present invention.

[0032] like Figure 1 As shown, the computer device may include: a processor, such as a central processing unit (CPU), a communication bus, a user interface, a network interface, and a memory. Among them, the communication bus is used to realize the connection and communication between these components. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally, the user interface may also include a standard wired interface and a wireless interface. The network interface may be a wired interface or a wireless interface (such as a wireless fidelity interface). The memory may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory may optionally be a storage device independent of the aforementioned processor.

[0033] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0034] like Figure 1 As shown, a memory as a storage medium may include an operating system, a network communication module, a user interface module, and an electronic program module.

[0035] exist Figure 1 In the computer device shown, the network interface is mainly used for data communication with the network server; the user interface is mainly used for data interaction with the user; the processor and memory in the computer device of the present application can be set in the computer device, and the computer device calls the electronic program stored in the data storage module through the processor, and executes the multimodal feature fusion target classification method based on the dual-tower structure provided in the embodiment of the present application.

[0036] Based on the hardware environment of the aforementioned embodiment, the embodiment of the present application provides a multi-modal feature fusion target classification method based on a dual-tower structure, please refer to Figure 2 , Figure 2 The flowchart of the multimodal feature fusion target classification method is as follows: (1) Acquire multimodal data and construct multimodal unsupervised and supervised datasets based on the multimodal data.

[0037] In the embodiment of the present invention, the multimodal data includes but is not limited to text data, image data, and signal data.

[0038] In an embodiment of the present invention, the process of constructing a multimodal unsupervised dataset and a supervised dataset based on multimodal data is as follows: (1.1) Obtaining original data of multimodal data; wherein the original data of the multimodal data includes original data without category labels and original data with category labels.

[0039] (1.2) For the original data without category labels, the enhanced data is constructed by deleting part of the original data information, and a data pair is constructed based on the original data and the enhanced data to obtain a multimodal unsupervised data set; for example, the enhanced data is constructed by data enhancement methods such as deleting some pixels of the image, deleting some text description fragments, and a data pair is constructed based on the original data and the enhanced data; wherein, the sample representation of the unsupervised data set is as follows: ;

[0040] in, Represents the original data of target object a mode 1~n, Indicates the enhanced data of modality 1~n of the target object b. If the target object lacks data of a certain modality, NULL is used to replace the data of that modality.

[0041] (1.3) For the original data with category labels, data pairs are constructed based on the original data of the two target objects to obtain a multimodal supervised dataset; the sample representation of the supervised dataset is as follows: ;

[0042] in, Represents the original data of target object a mode 1~n, Represents the raw data of target object b mode 1~n.

[0043] Specifically, for the sample data of the multimodal supervised data set, the label of the sample data is marked when the sample data is stored. If the target object a and the target object b belong to the same category, the label of the sample data is marked as True; otherwise, if the target object a and the target object b do not belong to the same category, the label of the sample data is marked as False.

[0044] (2) Extract data features of different modalities from multimodal unsupervised datasets and supervised datasets to construct unsupervised feature training sets and supervised feature training sets.

[0045] In the embodiment of the present invention, the process of extracting data features of different modalities is as follows: (2.1) If the mode of the data is For image data, a pre-trained image feature extraction model is used ( ) performs image feature extraction on the image data to obtain image features; specifically, the image feature extraction model can also be a residual neural network ( )、Convolutional Neural Network( )wait.

[0046] Among them, the pre-trained image feature extraction model ( ) The extracted image features are expressed as: ;

[0047] ;

[0048] (2.2) If the mode of the data is If the data is text data, a pre-trained text feature extraction model (ERNIE) is used to extract text features to obtain text features; specifically, the text feature extraction model can also be a BERT model, etc.

[0049] Among them, the pre-trained text feature extraction model (ERNIE) is used to extract text features as follows: ;

[0050] ;

[0051] (2.3) If the mode of the data is If it is signal data, the pre-trained residual neural network ( ) Perform signal feature extraction on signal data to obtain signal features.

[0052] Among them, the signal feature of the pre-trained residual neural network is expressed as: In the embodiment of the present invention, for other modal data, the general feature extraction model under the modality can be used to extract relevant features. If the modal data corresponding to the object is missing, that is, NULL, then the corresponding feature vector is set to a zero vector.

[0053] (3) Based on the unsupervised feature training set and the supervised feature training set, the dual-tower multi-mode feature fusion model is trained unsupervised and supervised.

[0054] In the embodiment of the present invention, the dual-tower multi-mode feature fusion model includes a feature input layer, a feature mapping layer, a feature normalization layer, a feature concatenation layer, a feature fusion layer and a similarity calculation layer, see Figure 3 , Figure 3 It is a structural schematic diagram of the dual-tower multi-mode feature fusion model.

[0055] In an embodiment of the present invention, the process of performing unsupervised training and supervised training on the dual-tower multi-mode feature fusion model based on the unsupervised feature training set and the supervised feature training set is as follows: (3.1) Performing unsupervised training on a dual-tower multi-modal feature fusion model based on data features of an unsupervised feature training set, wherein the data features include image features, text features, and signal features.

[0056] Specifically, the process of unsupervised training is as follows: (3.11) Inputting the data features of the unsupervised training set into the dual-tower multi-mode feature fusion model; that is, accessing the data features through the feature input layer of the dual-tower multi-mode feature fusion model.

[0057] (3.12) Perform feature mapping, normalization, and feature concatenation and fusion processing on the data features through the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object; Specifically, since the dimensions of data features of different modalities are often different, a single-layer feedforward neural network (FFN) is used as the feature mapping layer to map data features of different modalities to a vector space of the same dimension, that is, the data features of the original data and enhanced data of the samples of the unsupervised dataset are regarded as data features originating from different target objects. For data features of the same modality of different target objects, a single-layer feedforward neural network is used to implement feature mapping: ;

[0058] ;

[0059] Specifically, in order to avoid the influence of features that are too large or too small on fusion during the fusion process, the feature normalization layer is used to adjust the feature scale of the data features to ensure the consistency of the feature vector scale between different modalities: ;

[0060] ;

[0061] Specifically, a feature concatenation layer is used to concatenate data features of the same modality, and a multi-layer perceptron (MLP) is used as a feature fusion layer to fuse the concatenated data features to construct multi-modal features of the target object. , : ;

[0062] ;

[0063] (3.13) The multimodal features of the target objects are similarly calculated through the dual-tower multimodal feature fusion model to obtain the multimodal feature similarity between the target objects, that is, the cosine similarity algorithm of the similarity calculation layer is used to calculate a single sample of the unsupervised data set. The multimodal feature similarity between two target objects: ;

[0064] Specifically, after obtaining the multimodal feature similarity, a cross entropy loss function is constructed according to the multimodal feature similarity: ;

[0065] Specifically, through iterative training of the cross entropy loss function, the unsupervised training of the dual-tower multi-modal feature fusion model is realized, so that the dual-tower multi-modal feature fusion model can still judge the category according to the similarity of existing features when some features of the target object are lost, thereby reducing the sensitivity of the dual-tower multi-modal feature fusion model to feature missing and improving the robustness of the dual-tower multi-modal feature fusion model.

[0066] (3.2) Performing supervised training on the dual-tower multi-modal feature fusion model after unsupervised training based on data features of the supervised feature training set, wherein the data features include image features, text features, and signal features.

[0067] Specifically, the process of supervised training is as follows: (3.21) The data features of the supervised training set are input into the dual-tower multi-modal feature fusion model after unsupervised training.

[0068] (3.22) The data features are mapped, normalized, and concatenated through the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object.

[0069] Specifically, the process of feature mapping, normalization, and feature concatenation and fusion processing for supervised training is the same as that for unsupervised training. For details, please refer to steps (2.11) to (2.12).

[0070] (3.23) The multi-modal features of the target object are similarly calculated through the dual-tower multi-modal feature fusion model to obtain a single sample of the supervised data set The multimodal feature similarity between the two target objects is calculated, and the cross entropy loss function is constructed based on the multimodal feature similarity: ;

[0071] Among them, when the sample When marked as True, ; When marked as False, Through iterative training based on the loss function, the mapping features of objects in the same category are made as similar as possible, and objects in different categories are made as different as possible, so as to achieve supervised training of the dual-tower multi-modal feature fusion model.

[0072] (4) The dual-tower multimodal feature fusion model after unsupervised training and supervised training is used to perform multimodal feature queries on the target objects, and multimodal target classification is achieved based on similarity retrieval.

[0073] Specifically, in the actual implementation process, a dual-tower multi-mode feature fusion model is used to construct multi-mode features for the target object of the query category. , retrieve the feature vector of the known category object based on the multi-modal feature, and select the object feature with the highest correlation with the target object feature of the query category , the class of the object The category of the target object to be queried.

[0074] ; ;

[0075] In summary, the present invention constructs a dual-tower multimodal feature fusion model, and combines multimodal data to perform unsupervised learning and supervised learning on the model. Based on a large amount of unlabeled data, the basic learning ability of the model for domain data is improved, and a small amount of labeled data is combined to improve the model's ability to perceive feature differences of different categories, thus breaking through the problem of difficulty in obtaining data for certain specific application scenarios.

[0076] In addition, in one embodiment, based on the same inventive concept as the above embodiment, the embodiment of the present invention provides a multimodal feature fusion target classification system based on a dual-tower structure, and the system corresponds one-to-one to the method of embodiment 1, and the system includes: An original data acquisition unit, the original data acquisition unit is used to acquire multimodal data and construct a multimodal unsupervised data set and a supervised data set based on the multimodal data; A data feature construction unit, wherein the data feature construction unit is used to extract data features of different modalities from a multimodal unsupervised data set and a supervised data set to construct an unsupervised feature training set and a supervised feature training set; A model joint training unit, wherein the model joint training unit performs unsupervised training and supervised training on the dual-tower multi-mode feature fusion model based on an unsupervised feature training set and a supervised feature training set; A multimodal target classification unit uses a dual-tower multimodal feature fusion model after unsupervised training and supervised training to perform multimodal feature query on the target object, and realizes multimodal target classification based on similarity retrieval.

[0077] It should be noted that each unit in the multimodal feature fusion target classification system based on a dual-tower structure in this embodiment corresponds one-to-one to each step in the multimodal feature fusion target classification method based on a dual-tower structure in the aforementioned embodiment. Therefore, the specific implementation method and technical effects achieved by this embodiment can refer to the implementation method of the aforementioned multimodal feature fusion target classification method based on a dual-tower structure, which will not be repeated here.

[0078] In addition, in one embodiment, the present application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the method in the aforementioned embodiment when executed by the processor.

[0079] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored, and the computer program implements the method in the aforementioned embodiment when executed by a processor.

[0080] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0081] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0082] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0083] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0084] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0085] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0087] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multimodal feature fusion target classification method based on a dual-tower structure, characterized in that: The method comprises the following steps: Acquire multimodal data and construct multimodal unsupervised and supervised datasets based on the multimodal data; Extract data features of different modalities from multimodal unsupervised data sets and supervised data sets to construct unsupervised feature training sets and supervised feature training sets; Based on the unsupervised feature training set and the supervised feature training set, the dual-tower multi-mode feature fusion model is trained unsupervised and supervised; The dual-tower multimodal feature fusion model after unsupervised training and supervised training is used to perform multimodal feature query on the target object, and multimodal target classification is achieved based on similarity retrieval.

2. According to the multimodal feature fusion target classification method based on the double-tower structure of claim 1, it is characterized in that: The multimodal data includes text data, image data and signal data.

3. According to the multimodal feature fusion target classification method based on the double-tower structure of claim 2, it is characterized in that: The process of constructing multimodal unsupervised and supervised datasets based on multimodal data is as follows: Acquire original data of multimodal data; wherein the original data of the multimodal data includes original data without category labels and original data with category labels; For the original data without category labels, the enhanced data is constructed by deleting part of the original data information, and a data pair is constructed based on the original data and the enhanced data to obtain a multimodal unsupervised dataset; For the original data with category labels, data pairs are constructed based on the original data of two target objects to obtain a multimodal supervised dataset.

4. According to the multimodal feature fusion target classification method based on the double-tower structure of claim 2, it is characterized in that: The process of extracting data features of different modalities for multimodal unsupervised data sets and supervised data sets is as follows: If the modality of the data is image data, a pre-trained image feature extraction model is used to extract image features from the image data to obtain image features; If the modality of the data is text data, a pre-trained text feature extraction model is used to extract text features to obtain text features; If the mode of the data is signal data, a pre-trained residual neural network is used to extract signal features from the signal data to obtain signal features.

5. According to the multimodal feature fusion target classification method based on the double-tower structure of claim 1, it is characterized in that: The dual-tower multi-mode feature fusion model includes a feature input layer, a feature mapping layer, a feature normalization layer, a feature splicing layer, a feature fusion layer and a similarity calculation layer.

6. According to the multimodal feature fusion target classification method based on the double-tower structure of claim 4, it is characterized in that: The process of unsupervised training and supervised training of the dual-tower multi-mode feature fusion model based on the unsupervised feature training set and the supervised feature training set is as follows: Based on the data features of the unsupervised feature training set, the dual-tower multi-mode feature fusion model is unsupervisedly trained; Based on the data features of the supervised feature training set, the dual-tower multi-mode feature fusion model after unsupervised training is supervised trained; The data features include image features, text features and signal features.

7. The multimodal feature fusion target classification method based on a dual-tower structure according to claim 6 is characterized in that: The process of unsupervised training of the dual-tower multi-mode feature fusion model based on the data features of the unsupervised feature training set is as follows: Input the data features of the unsupervised training set into the dual-tower multi-mode feature fusion model; The data features are mapped, normalized, and fused by the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object. The dual-tower multimodal feature fusion model is used to calculate the similarity of the multimodal features of the target objects to obtain the multimodal feature similarity between the target objects, and a cross-entropy loss function is constructed based on the multimodal feature similarity. The unsupervised training of the dual-tower multimodal feature fusion model is achieved through iterative training of the cross-entropy loss function.

8. The multimodal feature fusion target classification method based on a dual-tower structure according to claim 6 is characterized in that: The process of supervised training of the dual-tower multi-mode feature fusion model after unsupervised training based on the data features of the supervised feature training set is as follows: Input the data features of the supervised training set into the dual-tower multi-mode feature fusion model after unsupervised training; The data features are mapped, normalized, and fused by the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object. The dual-tower multimodal feature fusion model is used to calculate the similarity of the multimodal features of the target objects to obtain the multimodal feature similarity between the target objects, and a cross-entropy loss function is constructed based on the multimodal feature similarity. The supervised training of the dual-tower multimodal feature fusion model is achieved through iterative training of the cross-entropy loss function.

9. The multimodal feature fusion target classification method based on a dual-tower structure according to claim 1 is characterized in that: The dual-tower multimodal feature fusion model after unsupervised training and supervised training is used to perform multimodal feature query on the target object, and the process of multimodal target classification based on similarity retrieval is as follows: Acquire multimodal data of the target object to be classified, and extract data features of different modes from the multimodal data to obtain data features of the target object to be classified; The data features of the target object to be classified are input into the dual-tower multi-mode feature fusion model after unsupervised training and supervised training, and the data features are subjected to feature mapping, normalization, feature splicing and fusion processing through the dual-tower multi-mode feature fusion model to obtain the multi-mode features of the target object to be classified; A feature query is performed based on the multimodal features of the target object to obtain multimodal features of several known categories, and the category corresponding to the multimodal feature with the highest similarity is selected as the multimodal target classification result through similarity calculation.

10. A multimodal feature fusion target classification system based on a dual-tower structure, characterized in that: The system comprises: An original data acquisition unit, the original data acquisition unit is used to acquire multimodal data and construct a multimodal unsupervised data set and a supervised data set based on the multimodal data; A data feature construction unit, wherein the data feature construction unit is used to extract data features of different modalities from a multimodal unsupervised data set and a supervised data set to construct an unsupervised feature training set and a supervised feature training set; A model joint training unit, wherein the model joint training unit performs unsupervised training and supervised training on the dual-tower multi-mode feature fusion model based on an unsupervised feature training set and a supervised feature training set; A multimodal target classification unit uses a dual-tower multimodal feature fusion model after unsupervised training and supervised training to perform multimodal feature query on the target object, and realizes multimodal target classification based on similarity retrieval.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the multimodal feature fusion target classification method based on the dual-tower structure as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for multimodal feature fusion target classification based on a dual-tower structure as described in any one of claims 1 to 9 is implemented.

Images