Multimodal resource balancing method and system based on associative memory neural network
By constructing a deep bidirectional associative memory feature matching network (BAM), missing modal information is generated, which solves the problems of modal imbalance and missingness in multimodal data, realizes resource balance and adaptive adjustment of multimodal data, and improves the performance of the model.
Patent Information
- Application Number
- CN202211391363.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-08
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2042-11-08
AI Technical Summary
Existing technologies fail to effectively utilize multimodal information resources, resulting in inconsistent modal information strength, modal missingness, and inconsistent modal alignment associations, affecting the performance of multimodal data. In addition, existing methods do not fully consider the feature consistency of heterogeneous modalities, resulting in data redundancy or noise in the model.
A deep bidirectional associative memory feature matching network (BAM) is constructed to complete the missing modal information and use associative memory neural network for modal mutual generation to achieve adaptive adjustment of multimodal knowledge representation model in the case of modal missing and imbalance.
By generating missing modal information, the modal imbalance problem is solved, resource balance of multimodal data is achieved, the heterogeneous differences and redundant noise between heterogeneous modal features are reduced, and the adaptability and performance of the model are improved.
Smart Images
Figure CN115761436B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multimodal resource balancing method and system based on an associative memory neural network. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute the prior art that has been known to those skilled in the art.
[0003] Real-world information is typically transmitted to people through multiple modalities, allowing them to perceive things through various means (such as vision, hearing, and touch). Different modalities expressing the same thing have consistent semantics, and this rich information facilitates more accurate judgment. However, multimodal information resources in reality are often unbalanced, often manifested in: inconsistent modal information strength, modal omissions, and inconsistent modal alignment. This unbalanced multimodal data can lead to performance degradation, even falling short of single-modal performance.
[0004] Existing methods propose using the strong modality as soft supervision to supplement the weak modality, and consider weighting and other operations to eliminate interference from inconsistent samples. While the strong modality can assist the weak modality, the high cost of collecting the ideal strong modality makes it challenging to fully utilize the existing imbalanced information. Furthermore, existing technologies fail to fully consider the feature consistency of heterogeneous modalities. Training non-aligned features can lead to data redundancy or noise in the model. Therefore, how to improve the imbalance of multimodal data resources remains an unresolved issue. Summary of the Invention
[0005] In order to solve the above problems, the present invention provides a multimodal resource balancing method and system based on associative memory neural network. By constructing a deep bidirectional associative memory feature matching network (Bidirectional associative memory, BAM) for modal mutual generation, the corresponding information of the missing modality is generated from the non-missing modal information to complete the missing modality, so as to realize the adaptive adjustment of the multimodal knowledge representation model in the case of modal missing and imbalance.
[0006] In order to achieve the above object, the present invention mainly includes the following aspects:
[0007] In a first aspect, an embodiment of the present invention provides a multimodal resource balancing method based on an associative memory neural network, comprising:
[0008] Acquiring multimodal data, wherein the multimodal data includes text modal data and image modal data;
[0009] Extracting original text features from the text modality data and extracting original image features from the image modality data; inputting the original text features into a pre-trained bidirectional associative memory neural network model to generate an image structure representation corresponding to the original text features; and inputting the original image features into the bidirectional associative memory neural network model to generate a text structure representation corresponding to the original image features;
[0010] The original text features are spliced with the generated text structure representation to obtain joint text features, and the original image features are spliced with the generated image structure representation to obtain joint image features, so as to achieve resource balance between different modal data.
[0011] In a possible implementation, after obtaining the joint text features and the joint image features, it also includes: performing typical correlation analysis on the joint text features and the joint image features, extracting consistent feature representations between each modality, and performing fusion analysis on multimodal data.
[0012] In one possible implementation, the joint text features and the joint image features are used as input, and a set of canonical correlation variables is generated using deep canonical correlation analysis; wherein the canonical correlation variables are obtained by multiplying the original input with the canonical correlation coefficient, and the canonical correlation coefficient is used to characterize the weight of the fine-grained features.
[0013] In a possible implementation, the original text features of the text modality data are extracted using a BERT pre-trained model.
[0014] In one possible implementation, a pre-trained vgg16 model is used to extract original image features of the image modality data.
[0015] In one possible embodiment, the training method of the bidirectional associative memory neural network model includes: taking image-text pairs with consistent semantics and expressions as sample data sets; extracting text features and image features from the image-text pairs, and inputting them into the bidirectional associative memory neural network model for training, thereby obtaining a bidirectional associative memory neural network model with associative memory function.
[0016] In a second aspect, an embodiment of the present invention further provides a multimodal resource balancing system based on an associative memory neural network, comprising:
[0017] A data acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text modal data and image modal data;
[0018] a feature extraction module for extracting original text features from the text modality data and original image features from the image modality data; inputting the original text features into a pre-trained bidirectional associative memory neural network model to obtain an image structure representation corresponding to the original text features; and inputting the original image features into the bidirectional associative memory neural network model to obtain a text structure representation corresponding to the original image features;
[0019] The resource balancing module is used to splice the original text features with the generated text structure representation to obtain joint text features, and to splice the original image features with the generated image structure representation to obtain joint image features, thereby achieving resource balancing between different modal data.
[0020] In a possible implementation, the method further includes:
[0021] The canonical correlation analysis module is used to perform canonical correlation analysis on the joint text features and the joint image features, extract consistent feature representations between the modalities, and perform fusion analysis on the multimodal data.
[0022] In a third aspect, an embodiment of the present invention provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the multimodal resource balancing method based on the associative memory neural network as described in the first aspect and any possible implementation scheme of the first aspect are performed.
[0023] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal resource balancing method based on an associative memory neural network as described in the first aspect and any possible implementation scheme of the first aspect are executed.
[0024] Based on the above technical solution, the present invention has the following beneficial effects:
[0025] 1. By acquiring multimodal data, extracting raw text features from text modal data and raw image features from image modal data, inputting the raw text features into a pre-trained bidirectional associative memory neural network model to obtain the corresponding image structure representation, and inputting the raw image features into a bidirectional associative memory neural network model to obtain the corresponding text structure representation, the generated image and text structure representations compensate for the existing modal imbalance. The raw text features are then concatenated with the text structure representation to obtain joint text features, and the raw image features are concatenated with the image structure representation to obtain joint image features. This approach can address resource imbalance issues such as imbalance between strong and weak modalities or modality loss, and enable adaptive adjustment of the multimodal knowledge representation model in the presence of modal loss and imbalance.
[0026] 2. By expressing the semantic consistency of multimodal data based on canonical correlation analysis, the heterogeneous differences between heterogeneous modal features can be reduced, and inconsistent information and redundant noise between modalities can be eliminated. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0028] Figure 1 1 is a flow chart of a multimodal resource balancing method based on an associative memory neural network provided by an embodiment of the present invention;
[0029] Figure 2 Schematic diagram of the structure of a bidirectional associative memory neural network provided by an embodiment of the present invention;
[0030] Figure 3 1 is a flow chart of a resource balancing method with signal loss provided by an embodiment of the present invention;
[0031] Figure 4 This is a diagram of a graphic and text modality resource balancing architecture provided by an embodiment of the present invention;
[0032] Figure 5 Schematic diagram of the structure of a multimodal resource balancing system based on an associative memory neural network provided by an embodiment of the present invention;
[0033] Figure 6 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0035] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.
[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.
[0037] Example 1
[0038] See also Figure 1 , Figure 1 is a flow chart of a multimodal resource balancing method based on an associative memory neural network provided by an embodiment of the present invention, such as Figure 1 As shown, the method specifically includes the following steps:
[0039] S101: Acquire multimodal data, where the multimodal data includes text modal data and image modal data.
[0040] In practice, information is typically transmitted to people through multiple modalities, enabling them to perceive things through various means (such as vision, hearing, and touch). First, multimodal data must be acquired. Multimodal data includes multiple modal data, such as text and image data, and other single-modal data in different modalities. For example, medical images and medical records that can reflect disease conditions are often unbalanced. These multimodal information resources are often manifested in: inconsistent modal information strength, modal missingness, and inconsistent modal alignment.
[0041] S102: Extracting original text features of the text modal data and extracting original image features of the image modal data; inputting the original text features into a pre-trained bidirectional associative memory neural network model to generate an image structure representation corresponding to the original text features; inputting the original image features into the bidirectional associative memory neural network model to generate a text structure representation corresponding to the original image features.
[0042] In the specific implementation, the unimodal feature extraction module is used to extract features from text modal data and image modal data, respectively, to obtain raw text features and raw image features. The unimodal feature extraction module directly uses the corresponding structure and weights by using a pre-trained model previously trained on a large dataset. The pre-trained model is already well-trained, eliminating the need to modify excessive weights in a short period of time. By adopting the concept of transfer learning, fine-tuning is performed on the annotated training data for specific downstream tasks to complete the construction of the feature extraction module.
[0043] As an optional implementation, the original text features of the text modality data are extracted using a BERT pre-trained model, and the original image features of the image modality data are extracted using a pre-trained VGG16 model.
[0044] In the specific implementation, (1) for the extraction of image features:
[0045] Input the image I into the VGG16 network and extract the image features V = {v1,...,v k},v i ∈R D , each image feature v i Encode a region, k represents the number of regions;
[0046] (2) Extraction of text features:
[0047] Given a text T, represent it with a series of text features E = {e1, ..., e n}, e i ∈R D , each word feature e i Encode a word in a sentence, where n represents the number of words in the sentence. Given the i-th word in a sentence, embed the i-th word into a d-dimensional vector (d=300) using an embedding matrix, which is randomly initialized as follows:
[0048] B i =W t e i , i∈[1,n] (1)
[0049] Among them, e i Represents the one-hot encoding in the sentence.
[0050] The word embedding is initialized using the pre-trained Bert model. After initialization, it is input into the Bert model after passing through two special symbols [CLS] and [SEP].
[0051] F={[CLS]e1,e2,...,e n ,[SEP]} (2)
[0052] B i =Bert(F i ), i∈[1,n] (3)
[0053] The associative memory network has properties similar to those of the human brain. It can recall complete and clear images from incomplete or blurred images. To this end, we proposed a multimodal resource balancing method based on a deep associative memory feature matching network and modal correlation analysis. We use the associative memory network to recall missing information from images (texts) and supplement it with image and text features. The canonical correlation variables obtained by canonical correlation analysis of the spliced image (text) features are used to obtain multimodal features for resource balancing.
[0054] Associative memory networks differ from how humans memorize information. They learn a sample only once, whereas human memory is often an iterative process, requiring repeated learning of missed samples. After repeated learning, they can memorize all samples and distinguish between similar samples. To this end, we use a bidirectional associative memory (BAM) network, modeled on the human brain's iterative learning behavior, to enable it to fully retain key information.
[0055] The following describes the extraction of missing information for images and text:
[0056] The original text features are input into a pre-trained bidirectional associative memory neural network model (Bidirectional Association Memory, BAM) to obtain an image structure representation that contains text semantics. The original image features are input into the bidirectional associative memory neural network model to obtain a text structure representation that contains image semantics. Among them, the associative memory neural network simulates the memory and association of the human brain. In the memory stage, some sample patterns are linked in the form of memory pairs and stored in the weights. In the association stage, the noisy data is calculated with the previously learned weights to obtain the original pattern itself, realizing associative recall. Hetero-associative memory refers to the associative memory from the input pattern to other corresponding patterns, from itself to elsewhere.
[0057] This embodiment uses a bidirectional associative memory neural network (BAM) to implement heterogeneous associative memory functions between different modalities. Figure 2As shown in , the BAM network is a two-layer bidirectional network. When an input signal is added to one layer, the other layer can obtain an output. Through the weight matrix, the two modal data are generated from each other, and the weights are trained until a stable state is reached. For a fully trained weight matrix, when an incomplete stored pattern is input to one side of the BAM network, the network can not only achieve correct heteroassociation on the other side after a limited number of runs, but also reconstruct the complete input pattern on the input side. In other words, from the incomplete input pattern, BAM can not only achieve heteroassociation, but also reconstruct the complete input pattern. After training BAM with different modal data, BAM has the function of storing memory pattern pairs and performing bidirectional association.
[0058] In the specific implementation, the image and text features are first input into BAM, and the image features are matched with all the word features. If the correlation between them is high, the text information is weighted as follows:
[0059]
[0060] Among them, S(v j , e n ) is the image v j With text e n Correlation, ω j ∈[0, 1] is the attention coefficient for each relevant text.
[0061] After all images are matched with all texts, the weighted text information is put into a collection to generate image features Fv_t containing text semantics, which are then concatenated with the original image features to generate a joint feature representation Ft_con of a semantically rich text structure.
[0062] Then match the text features with all the image features. If the correlation between them is high, the image information is weighted as follows:
[0063]
[0064] Among them, S(e i , v k ) is the text e i With image v k Correlation, ω i ∈[0, 1] is the attention coefficient for each related image. After all texts are matched with all images, the weighted image information is put into the set to generate text features Fv_t that contain image semantics, and then concatenated with the original text features to generate a joint feature representation Fv_con of the semantically rich image structure. Figure 3As shown in , taking image modal data as an example, the image modal data passes through the image feature extractor and outputs the image feature Fv. Through the trained BAM model, the image feature Fv is used to generate the missing text feature Ft_v, which is used to complete the missing original text modal data.
[0065] The missing modal information is supplemented by a bidirectional associative memory network. The missing information in the image is extracted from the text, and the missing information in the text is extracted from the image. The unbalanced modalities are coordinated to achieve the balance of heterogeneous modal resources.
[0066] As an optional implementation, the training method of the bidirectional associative memory neural network model includes: using image-text pairs with consistent semantics and expressions (such as the wukong dataset proposed by Huawei Noah's Ark Laboratory, which contains 100 million sets of image-text pairs) as sample datasets; extracting text features and image features from the image-text pairs, and inputting them into the bidirectional associative memory neural network model for training, thereby obtaining a bidirectional associative memory neural network model with associative memory function.
[0067] S103: splicing the original text features with the generated text structure representation to obtain joint text features, and splicing the original image features with the generated image structure representation to obtain joint image features, so as to achieve resource balance between different modal data.
[0068] In a specific implementation, the original text features are spliced with the text structure representation to obtain the joint text features of the semantically rich text structure, and the original image features are spliced with the text structure representation to obtain the joint image features of the semantically rich image structure, so as to optimize the imbalance problem of different modal data or complete the missing modality.
[0069] As an optional implementation, after obtaining the joint text features and the joint image features, it also includes: performing typical correlation analysis on the joint text features and the joint image features, extracting consistent feature representations between each modality, and performing fusion analysis on multimodal data.
[0070] In the specific implementation, the idea of canonical correlation analysis (CCA) is used for feature-level fusion, and the correlation features between the two sets of features are used as discriminant information, which not only achieves the purpose of information fusion, but also effectively eliminates redundant information, providing a new idea for the fusion of two sets of features for classification. Specifically, canonical correlation analysis is performed on the joint text features and joint image features, and consistent feature representations between the modalities are extracted to perform fusion analysis on multimodal data. In this way, the application scope of CCA is theoretically extended, and the extracted canonical correlation features can not only achieve compression of the original feature dimension, but also extract consistent features between different feature representations, providing a more robust feature representation for subsequent tasks.
[0071] As an optional implementation, the joint text features and the joint image features are used as input, and a set of canonical correlation variables is generated using deep canonical correlation analysis; wherein the canonical correlation variables are obtained by multiplying the original input with the canonical correlation coefficient, and the canonical correlation coefficient is used to characterize the weight of the fine-grained features.
[0072] like Figure 4 As shown in , the overall framework execution process is as follows:
[0073] 4.1 Input the image and text data into BERT and VGG16 respectively to extract the original text features and image features;
[0074] 4.2 Input the original text features into BAM to generate image features Fv_t containing text semantics, and concatenate them with the original image features to generate a joint feature representation Ft_con of the semantically rich text structure; input the original image features into BAM to generate text features Fv_t containing image semantics, and concatenate them with the original text features to generate a joint feature representation Fv_con of the semantically rich image structure.
[0075] 4.3 Perform canonical correlation analysis on the resource-balanced Ft_con and Fv_con features to generate the corresponding text canonical correlation variables Ft_cca and image canonical correlation variables Fv_cca, further match the data of the two modalities, balance resources, and provide input for subsequent downstream tasks.
[0076] In this way, by constructing a bidirectional associative memory neural network for modal mutual generation, the imbalance problem of multimodal data resources is effectively improved; and, through the typical correlation analysis module, the features of each modality are further weighted and optimized to extract consistent feature representations between modalities, while achieving the effect of dimensionality reduction and improving the operating efficiency of the model.
[0077] Example 2
[0078] See also Figure 5 , Figure 5 : is a structural diagram of a multimodal resource balancing system based on an associative memory neural network provided by an embodiment of the present invention, such as Figure 5 As shown, the multimodal resource balancing system 500 includes:
[0079] A data acquisition module 510 is configured to acquire multimodal data, wherein the multimodal data includes text modal data and image modal data;
[0080] A feature extraction module 520 is configured to extract original text features from the text modality data and original image features from the image modality data; input the original text features into a pre-trained bidirectional associative memory neural network model to obtain an image structure representation; and input the original image features into the bidirectional associative memory neural network model to obtain a text structure representation;
[0081] The resource balancing module 530 is used to splice the original text features with the text structure representation to obtain a joint text feature, and to splice the original image features with the image structure representation to obtain a joint image feature, thereby achieving resource balancing between different modal data.
[0082] As an optional implementation, the invention further includes:
[0083] The canonical correlation analysis module is used to perform canonical correlation analysis on the joint text features and the joint image features, extract consistent feature representations between the modalities, and perform fusion analysis on the multimodal data.
[0084] This embodiment provides a multimodal resource balancing system based on an associative memory neural network for implementing the aforementioned multimodal resource balancing method based on an associative memory neural network. Therefore, the specific implementation method of the multimodal resource balancing system based on an associative memory neural network can be seen in the embodiment section of the multimodal resource balancing method based on an associative memory neural network in the previous text, and will not be repeated here.
[0085] Example 3
[0086] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 6 As shown, the computer device 600 includes a processor 610 , a memory 620 and a bus 630 .
[0087] The memory 620 stores machine-readable instructions executable by the processor 610. When the computer device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 1 and Figure 4 The steps of the multimodal resource balancing method based on the associative memory neural network in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.
[0088] Example 4
[0089] Based on the same inventive concept, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal resource balancing method based on associative memory neural network described in the above method embodiment are executed.
[0090] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0091] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.
Claims
1. A multimodal resource balancing method based on associative memory neural network, characterized in that: include: Acquiring multimodal data, wherein the multimodal data includes text modal data and image modal data; Extracting original text features from the text modality data and extracting original image features from the image modality data; inputting the original text features into a pre-trained bidirectional associative memory neural network model to generate an image structure representation corresponding to the original text features; and inputting the original image features into the bidirectional associative memory neural network model to generate a text structure representation corresponding to the original image features; The original text features are spliced with the generated text structure representation to obtain joint text features, and the original image features are spliced with the generated image structure representation to obtain joint image features, so as to achieve resource balance between different modal data.
2. The multimodal resource balancing method based on associative memory neural network according to claim 1, characterized in that: After obtaining the joint text features and the joint image features, the method further includes: performing canonical correlation analysis on the joint text features and the joint image features, extracting consistent feature representations between the modalities, and performing fusion analysis on the multimodal data.
3. The multimodal resource balancing method based on associative memory neural network according to claim 2, characterized in that: The joint text features and joint image features are taken as input, and a set of canonical correlation variables are generated using deep canonical correlation analysis; wherein the canonical correlation variables are obtained by multiplying the original input with the canonical correlation coefficient, and the canonical correlation coefficient is used to characterize the weight of fine-grained features.
4. The multimodal resource balancing method based on associative memory neural network according to claim 1, characterized in that: The original text features of the text modality data are extracted using the BERT pre-trained model.
5. The multimodal resource balancing method based on associative memory neural network according to claim 1, characterized in that: The pre-trained vgg16 model is used to extract the original image features of the image modality data.
6. The multimodal resource balancing method based on associative memory neural network according to claim 1, characterized in that: The training method of the bidirectional associative memory neural network model includes: taking image-text pairs with consistent semantics and expressions as sample data sets; extracting text features and image features from the image-text pairs, and inputting them into the bidirectional associative memory neural network model for training, thereby obtaining a bidirectional associative memory neural network model with associative memory function.
7. A multimodal resource balancing system based on associative memory neural network, characterized in that: include: A data acquisition module, configured to acquire multimodal data, wherein the multimodal data includes text modal data and image modal data; a feature extraction module for extracting original text features from the text modality data and original image features from the image modality data; inputting the original text features into a pre-trained bidirectional associative memory neural network model to generate an image structure representation corresponding to the original text features; and inputting the original image features into the bidirectional associative memory neural network model to generate a text structure representation corresponding to the original image features; The resource balancing module is used to splice the original text features with the generated text structure representation to obtain joint text features, and to splice the original image features with the generated image structure representation to obtain joint image features, thereby achieving resource balancing between different modal data.
8. The multimodal resource balancing system based on associative memory neural network according to claim 7, characterized in that: Also includes: The canonical correlation analysis module is used to perform canonical correlation analysis on the joint text features and the joint image features, extract consistent feature representations between the modalities, and perform fusion analysis on the multimodal data.
9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the multimodal resource balancing method based on the associative memory neural network as described in any one of claims 1 to 6 are performed.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the multimodal resource balancing method based on associative memory neural network according to any one of claims 1 to 6 are executed.