Multi-modal feature hierarchical alignment method and system based on adaptive temperature coefficient
By using an adaptive temperature coefficient multimodal feature hierarchical alignment method, the problem that a fixed temperature coefficient cannot adapt to different levels of features is solved, and fine alignment of multimodal features is achieved, improving the accuracy and stability of cross-modal retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-07
- Publication Date
- 2026-04-14
AI Technical Summary
In existing multimodal alignment methods, the fixed temperature coefficient cannot adapt to the characteristics of features at different levels, resulting in unstable training or blurred alignment, which affects the performance of downstream tasks.
A hierarchical alignment method for multimodal features with adaptive temperature coefficient is adopted. Hierarchical coding networks for text and images are constructed respectively, and a contrast loss with adaptive temperature coefficient is introduced in each layer. The temperature coefficient is adjusted through a feature attribute-aware dynamic attenuation strategy to achieve hierarchical alignment of multimodal features.
It improves the accuracy and stability of multimodal semantic feature alignment, enhances the response accuracy and robustness of cross-modal retrieval, and optimizes training efficiency.
Smart Images
Figure CN121859019A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal semantic feature alignment technology, and particularly relates to a multimodal feature hierarchical alignment method and system based on adaptive temperature coefficient. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] In recent years, with the rapid development of artificial intelligence technology, multimodal processing has become a key driving force for advancing machine perception and cognition capabilities. Multimodal semantic alignment, as a core foundational task, aims to map features from different modalities (such as text, images, and speech) into a shared semantic space, making semantically similar samples closer together within this space, thereby enabling advanced applications such as cross-modal retrieval, reasoning, and generation.
[0004] Currently, deep learning-based multimodal alignment methods primarily rely on the contrastive learning paradigm. In this paradigm, the model learns joint representations between modalities by optimizing a contrastive loss function (such as InfoNCE loss). The core idea of this loss function is to narrow the representational distance between positive sample pairs (such as an image and its corresponding text description) while widening the representational distance between negative sample pairs (such as an image and a randomly mismatched text description). In this process, the temperature parameter, as a key hyperparameter in contrastive learning, plays the role of a "regulator." It controls the model's focus on difficult negative samples. The smaller the temperature parameter value, the more the model focuses on distinguishing difficult negative samples that are highly similar to positive samples, thus learning more discriminative feature boundaries; conversely, the larger the temperature parameter value, the less sensitive the loss function is to differences in similarity, resulting in a smoother feature distribution.
[0005] However, most existing methods, when dealing with multimodal alignment problems, typically set a globally fixed temperature coefficient for the entire model or the entire contrastive loss function. This approach has significant limitations, as it ignores a fundamental characteristic of deep neural networks: the hierarchical nature of feature representations. Features at different levels have different inherent semantic distributions and discrimination difficulties. A fixed temperature coefficient cannot simultaneously adapt to the characteristics of features at all levels: if the temperature coefficient is set too small, it may lead to training instability for shallow features with unclear semantic distributions and high noise levels; if the temperature coefficient is set too large, it cannot effectively widen the inter-class distance for deep semantic features that should have high discriminative power, resulting in alignment ambiguity and affecting the performance of downstream tasks. Summary of the Invention
[0006] To address at least one of the technical problems mentioned above, this invention provides a hierarchical alignment method and system for multimodal features based on an adaptive temperature coefficient. This method constructs hierarchical coding networks for both text and image modalities, and introduces a contrast loss based on an adaptive temperature coefficient between different layers of the deep neural network to achieve hierarchical alignment of multimodal features, thereby improving the accuracy of multimodal semantic feature alignment.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a multimodal feature hierarchical alignment method based on adaptive temperature coefficient, comprising the following steps: Acquire multimodal data; The multimodal feature hierarchical alignment model is trained based on the acquired multimodal data to obtain the trained multimodal feature hierarchical alignment model; the training process of the multimodal feature hierarchical alignment model includes: Construct a coding network for each modality of data, deconstruct the coding network of each modality into multiple layers, and extract the feature representation of each layer during the forward propagation of the coding network; A temperature coefficient is set for each layer, and the contrast loss between different modal features of that layer is calculated based on the temperature coefficient of each layer. By combining the feature representations of each mode in each layer, and based on the preset feature attribute perception temperature coefficient dynamic attenuation strategy, the specific modulation factor of each layer is calculated, and the temperature coefficient of the corresponding layer is adjusted based on the specific modulation factor of each layer. The overall objective loss function is obtained by fusing the contrastive losses of all levels, and the overall objective loss function is used to perform end-to-end joint training of the encoding networks of each modality. Cross-modal retrieval is performed based on a trained multimodal feature hierarchical alignment model.
[0008] Furthermore, within each layer, the contrastive loss between different modal features of that layer is expressed as: , , in, For any layer Contrast loss between text and image modal features It is the first Text features of the layer It is the first Image features of the layer, Indicates the first in the batch One sample, For the first Temperature coefficient of the layer yes, yes, It is the number of text-image pairs. Representing text features and image features Cosine similarity between them.
[0009] Furthermore, the step of combining the feature representations of each mode in each layer and calculating the specific modulation factor of each layer according to the preset feature attribute-aware temperature coefficient dynamic attenuation strategy includes: Calculate the statistical entropy of the activation values of the text features and image features in each layer, and then calculate the total entropy value of the layer based on the statistical entropy of the activation values of the text features and image features. The difference in the distribution of text and image features at each layer is measured using distribution differences based on cosine similarity. The modulation factor for each layer is calculated based on the summation entropy value of each layer and the difference in the distribution of text and image features.
[0010] Furthermore, the formula for calculating the modulation factor of each layer is as follows: , , , in, and It is a hyperparameter that controls the degree of influence of entropy and divergence; and The threshold parameter is used; the sigmoid function is used to convert the difference into a smooth value between 0 and 1. For the first The overall entropy of the layer For text features in the first The entropy of the layer, For image features in the first The entropy of the layer, No. The divergence of text and image feature distributions. Furthermore, the encoding network structure for each modality consists of an input layer, several intermediate hidden layers, and a final output layer.
[0011] Furthermore, the text encoding network adopts a deep neural network model based on the Transformer architecture; the image encoding network uses a deep convolutional neural network model.
[0012] Furthermore, by utilizing the total loss after fusion, the text encoding network and the image encoding network are jointly trained and optimized end-to-end through the backpropagation algorithm.
[0013] A second aspect of the present invention provides a multimodal feature hierarchical alignment system based on an adaptive temperature coefficient, comprising: A multimodal data acquisition module, which is used to acquire multimodal data; The hierarchical alignment module is used to train a hierarchical alignment model of multimodal features based on the acquired multimodal data, resulting in a trained hierarchical alignment model of multimodal features. It constructs an encoding network for each modality, deconstructing each modality's encoding network into multiple layers, and extracts the feature representation of each layer during the forward propagation of the encoding network. A temperature coefficient is set for each layer, and the contrast loss between different modal features of that layer is calculated based on the temperature coefficient. Combining the feature representations of each modality in each layer, and according to a preset feature attribute-aware temperature coefficient dynamic attenuation strategy, a specific modulation factor for each layer is calculated, and the temperature coefficient of the corresponding layer is adjusted based on the specific modulation factor. The overall objective loss function is obtained by fusing the contrastive losses of all levels, and the overall objective loss function is used to perform end-to-end joint training of the encoding networks of each modality. The cross-modal retrieval module is used for cross-modal retrieval based on a trained multimodal feature hierarchical alignment model.
[0014] A third aspect of the present invention provides a computer-readable storage medium.
[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described above.
[0016] A fourth aspect of the present invention provides a computer device.
[0017] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the multimodal feature hierarchical alignment based on adaptive temperature coefficient as described above.
[0018] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a method for hierarchical feature extraction of text and image modalities using deep coding networks, and introduces an alignment loss based on adaptive temperature coefficient contrastive learning at each level to achieve fine-grained semantic alignment from local to global. Instead of simply learning the final semantic match, modal feature alignment learning is performed at each step of feature abstraction, achieving fine-grained alignment of semantic features at different levels, from raw semantics to abstract semantics. Compared to existing methods that mostly rely on top-level features for global matching, this invention can capture finer-grained semantic associations, significantly improving the response accuracy and robustness to complex queries in cross-modal retrieval. Simultaneously, an adaptive adjustment mechanism optimizes training efficiency, ultimately achieving more accurate and stable retrieval results.
[0019] The adaptive temperature coefficient mechanism of this invention ensures that the alignment learning of each layer matches the semantic complexity of its features. Shallow layers use a higher temperature coefficient to avoid overfitting to noisy features; deeper layers use a lower temperature coefficient, forcing the model to learn more discriminative semantic boundaries. This effectively enhances the personalization of the model's learning of features at different levels.
[0020] The method described in this invention does not depend on a specific network structure or modality combination, and its core idea can be easily transferred to other cross-modal learning scenarios that require multi-level modal feature alignment.
[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0023] Figure 1 This is a schematic diagram of the overall process of the multimodal feature hierarchical alignment method based on adaptive temperature coefficient provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of data flow for the multimodal feature hierarchical alignment method based on adaptive temperature coefficient provided in an embodiment of the present invention; Figure 3 This is a block diagram of a multimodal feature hierarchical alignment system based on adaptive temperature coefficient provided in an embodiment of the present invention; Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0026] Most existing methods, when dealing with multimodal alignment problems, usually set a globally fixed temperature coefficient for the entire model or the entire contrastive loss function. This approach has obvious limitations, as it ignores a fundamental characteristic of deep neural networks: the hierarchical nature of feature representation. In a typical deep encoder (such as Transformer, ResNet), feature formation is a gradual process from concrete to abstract, from local to global. (1) Input layer and shallow network: The extracted features are more biased towards the low-level, modally strongly related raw information. For example, for images, it may be edges, colors, textures, etc.; for text, it may be parts of speech, local grammatical structures, etc. (2) Intermediate network: Gradually integrate and abstract the low-level information to form local concepts with more semantic representation capabilities. Such as object parts in images, phrase semantics in text; (3) Output layer and deep network: Gather global information to form high-level semantic representations or decision information closely related to the task objective. Such as the overall scene semantics of images, paragraph themes of text. This hierarchical difference in features means that the inherent semantic distribution and discrimination difficulty of features at different levels are different. For a fixed temperature coefficient, it is difficult to adapt to all levels of feature characteristics at the same time: if the temperature coefficient is set too small, it may lead to unstable training for shallow features with unclear semantic distribution and a lot of noise; if the temperature coefficient is set too large, it cannot effectively widen the inter-class distance for deep semantic features that should have high discriminativeness, resulting in blurred alignment and affecting the performance of downstream tasks.
[0027] This invention proposes a hierarchical alignment method and system for multimodal features based on an adaptive temperature coefficient. This method constructs hierarchical coding networks for multimodal data such as text and images, and introduces a contrastive loss based on an adaptive temperature coefficient between different layers of the deep neural network to achieve hierarchical alignment of multimodal features. It can adapt to the characteristics of semantic features at different levels, improving the accuracy of multimodal semantic feature alignment.
[0028] Example 1 like Figure 1 and Figure 2 As shown, this embodiment provides a multimodal feature hierarchical alignment method based on adaptive temperature coefficient, including the following steps: Step 1: Acquire multimodal data, construct the encoding network for each modality, deconstruct the encoding network of each modality into multiple layers, and extract the feature representation of each layer during the forward propagation of the encoding network; Specifically, the steps include the following: Step 101: Acquire multimodal data; In this embodiment, the acquired multimodal data includes image data and text data. Of course, the multimodal combination used in this invention is not limited to this and is also applicable to other multimodal combinations.
[0029] Step 102: Construct the encoding network for each modality of data. The encoding network for each modality is deconstructed into multiple layers, and the feature representation of each layer is extracted during the forward propagation of the encoding network. In this embodiment, a text encoding network and an image encoding network are constructed separately. The text encoding network adopts a deep neural network model based on the Transformer architecture; the image encoding network uses a deep convolutional neural network model. These networks are explicitly deconstructed into an input layer, several intermediate hidden layers, and a final output layer. During the forward propagation of the encoding network, the feature representations of the text and image at each preset network layer are recorded and output simultaneously. For a pair of input text and image, a text feature sequence and an image feature sequence can be obtained, which fully reflects the process of feature evolution from raw, local information to abstract, global semantics layer by layer.
[0030] Specifically, a text encoding network is constructed, which employs a pre-trained Transformer-based model (such as BERT). Let the text input be... After passing through a text encoding network, its first... Layer feature representation ,in , This represents the total number of layers (including input, intermediate, and output layers). Typically, Indicates the input layer. Indicates the output layer.
[0031] Construct an image coding network, which can use a deep convolutional neural network (such as ResNet) or a visual Transformer (ViT).
[0032] Let the image input be After passing through an image coding network, its first... Layer feature representation ; The text feature sequence is obtained through forward propagation. and image feature sequences .
[0033] Step 2: Set a temperature coefficient for each layer, and calculate the contrast loss between different modal features of the layer based on the temperature coefficient of each layer; This invention independently calculates a contrast loss for each extracted network layer feature to achieve semantic alignment between text features and image features at that layer. The contrast loss function for each layer introduces an adjustable temperature coefficient specific to that layer.
[0034] The core calculation logic of this loss function is as follows: for each text-image pair in a batch, its similarity score as a positive sample pair is calculated, and simultaneously compared with the similarity scores of all other unpaired images (i.e., negative samples) within the batch. The temperature coefficient acts as a "regulator" in this process, controlling the model's sensitivity to differences in similarity. A lower temperature coefficient amplifies small differences in similarity, driving the model to focus more on distinguishing those difficult-to-distinguish negative samples; while a higher temperature coefficient makes the loss function's perception of similarity differences smoother.
[0035] Specifically, it includes: For any layer Using the InfoNCE loss function, the contrast loss between text and image modal features is calculated. The loss function adopts the form of InfoNCE loss and introduces a layer-specific temperature coefficient. .
[0036] Suppose a batch has Text-image pairs For the first Layer, text features are Image features are ,in Indicates the first in the batch One sample.
[0037] No. The contrastive loss function for each layer is defined as follows: , in, Indicates the first Layer Text features of each sample express, Indicates the first Layer Image features of each sample in, It is the first Layer Text features of each sample It is the first Layer Image features of each sample.
[0038] in, It is the first The temperature coefficient of the layer is a real number greater than 0; the summation in the denominator is the summation of the similarity of all images (including positive and negative samples) within the batch after exponential scaling. Representing text features and image features The cosine similarity between them is calculated using the following formula: , This loss function encourages positive sample pairs (text and image of the same sample pair) to have higher similarity scores than negative sample pairs (text and different images). Temperature coefficient Used to adjust the distribution of similarity scores, when When the similarity is small, the loss function is more sensitive to differences in similarity, and the model will pay more attention to negative samples that are difficult to distinguish; when When the similarity is large, the loss function penalizes the difference in similarity more smoothly.
[0039] Step 3: Combining the feature representations of each modality in each layer, and based on the preset feature attribute-aware temperature coefficient dynamic attenuation strategy, calculate the specific modulation factor of each layer, and adjust the temperature coefficient of the corresponding layer based on the specific modulation factor of each layer. Because shallow features are rich in detail but have low semantic abstraction, they require smoother loss (higher temperature coefficient) for stable training; deep features have highly abstract semantics, requiring more sensitive loss (lower temperature coefficient) to form a highly discriminative decision boundary. Therefore, this invention, based on a pre-defined exponential decay framework, innovatively proposes a feature attribute-aware decision maker. The core of this scheme is that the final value of the temperature coefficient is no longer determined solely by network depth, but rather by a "base decay value" and a "layer-specific modulation factor." This modulation factor is dynamically generated by a lightweight, parameter-free decision maker based on the feature attributes of the current training batch at a specific network layer. The decision maker automatically fine-tunes the temperature coefficient by analyzing key indicators such as feature entropy (reflecting the degree of feature disorder and uncertainty) and cross-modal feature distribution divergence (reflecting the alignment difficulty of the layer): for layers with high entropy and high alignment difficulty, the temperature coefficient is appropriately increased; for layers with concentrated features and high discriminative power, the temperature is decreased to enhance discriminative ability. This mechanism enables the temperature coefficient allocation to intelligently respond to the distribution characteristics of the actual data, significantly improving the robustness and convergence efficiency of the training process, while avoiding the computational overhead caused by introducing additional trainable parameters.
[0040] Detailed description of temperature coefficient The adaptive adjustment strategy uses a feature attribute-aware decision-maker to dynamically determine... Specifically, it includes the following steps: Step 301: Define a baseline temperature coefficient decay framework. Baseline temperature coefficient Set according to exponential decay: , in, It is the base temperature coefficient value of the input layer; yes The attenuation factor of the layer is between 0 and 1.
[0041] Step 302: Introduce a feature attribute-aware decision-maker to calculate the specific modulation factor for each layer. ; The input to this decision-maker is the current batch in the [number]th [period]. The layer's characteristic attributes include feature information entropy and cross-modal feature distribution divergence.
[0042] Feature information entropy estimation: For the th The layer calculates the statistical entropy of the activation values of the text and image features in that layer. Taking text features as an example, the batch features are first processed... Normalize it (e.g., convert it to a probability distribution using a softmax function), and then calculate its entropy. Specifically, for the Layered text features, calculated as follows: , in, It is the feature dimension; It is the first The text feature vector at the th ... The normalized value on the dimension can be calculated using the softmax function; For text features in the first The entropy value of a layer. Similarly, the entropy value of image features can be calculated. The estimated comprehensive entropy of this layer is: .
[0043] Cross-modal feature distribution divergence estimation: using distribution difference based on cosine similarity to measure the 1st... Differences in the distribution of features between layered text and images.
[0044] Specifically, the cosine similarity matrix between text features and image features within a batch is calculated, and then the difference between the two distributions is calculated.
[0045] In this embodiment, the method used is to calculate the mean cosine similarity between text features and image features, and then take the negative value as the divergence estimate: , in: No. The divergence of the layer's text and image feature distributions; the larger the value, the greater the difficulty of cross-modal alignment for that layer.
[0046] The comprehensive entropy calculated based on the above process and distribution divergence Calculate the modulation factor : , in: and It is a hyperparameter that controls the degree of influence of entropy and divergence; and The threshold parameter is used; the sigmoid function is used to convert the difference into a smooth value between 0 and 1.
[0047] Step 303: Calculate the first... Temperature coefficient of layer , represented as: , After the above steps, the temperature coefficient not only decays with network depth, but can also be dynamically adjusted based on the entropy of multimodal features and cross-modal divergence.
[0048] Step 4: Combine the contrastive losses of all layers to obtain the total objective loss function, and use the total objective loss function to perform end-to-end joint training of the encoding networks of each modality; Alignment signals from all levels are integrated to guide the optimization direction of the model. The calculated contrastive losses from all levels are summed to form the overall objective loss function for model training. Subsequently, using the fused overall loss, the text encoding network and image encoding network are jointly trained and optimized end-to-end through the backpropagation algorithm.
[0049] In this embodiment, when fusing the contrast loss of all layers, the loss of each layer can be directly summed, or a weighted summation can be performed.
[0050] After obtaining the total loss function Then, using the fused total loss, the text encoding network and image encoding network are jointly trained and optimized end-to-end through the backpropagation algorithm, specifically including: Step 5: After training is complete, for downstream tasks, use the features of the output layer, or fuse the features of multiple layers and use them for cross-modal retrieval. During the model deployment phase, a pre-trained hierarchical coding network is first used to extract complete feature sequences from shallow to deep layers for all images and text in the retrieval database. For a single query (whether text or image), the network maps to the same multi-layered feature space. Shallow features capture detailed information such as object edges, color textures, etc., suitable for retrieval with high requirements for specific attributes; mid-level features contain more complex local combinations and relationships; while deep features carry the global semantics and core intent of the image or text.
[0051] During retrieval and matching, a single level of features is not used consistently. Instead, the similarity calculation strategy is dynamically adjusted based on the inherent attributes of the query. For queries with rich details and complex structures, shallow and mid-level features are given higher weights, utilizing their high information entropy to match subtle features. For queries with abstract concepts or clear overall semantics, deeper features and their lower temperature coefficients create a more sensitive decision boundary. The final comprehensive similarity score is a weighted fusion of similarities at each level, ensuring that the retrieval process captures both macroscopic semantics and microscopic details.
[0052] After returning the search results sorted by fusion similarity, the weights of different levels of features in the fusion are further fine-tuned based on user interaction feedback (such as clicks and dwell time) to achieve self-optimization of cross-modal retrieval.
[0053] Example 2 like Figure 3 As shown, this embodiment provides a multimodal feature hierarchical alignment system based on an adaptive temperature coefficient, including: A multimodal data acquisition module, which is used to acquire multimodal data; The hierarchical alignment module is used to train a hierarchical alignment model of multimodal features based on the acquired multimodal data, resulting in a trained hierarchical alignment model of multimodal features. It constructs an encoding network for each modality, deconstructing each modality's encoding network into multiple layers, and extracts the feature representation of each layer during the forward propagation of the encoding network. A temperature coefficient is set for each layer, and the contrast loss between different modal features of that layer is calculated based on the temperature coefficient. Combining the feature representations of each modality in each layer, and according to a preset feature attribute-aware temperature coefficient dynamic attenuation strategy, a specific modulation factor for each layer is calculated, and the temperature coefficient of the corresponding layer is adjusted based on the specific modulation factor. The overall objective loss function is obtained by fusing the contrastive losses of all levels, and the overall objective loss function is used to perform end-to-end joint training of the encoding networks of each modality. The cross-modal retrieval module is used for cross-modal retrieval based on a trained multimodal feature hierarchical alignment model.
[0054] It should be noted that the specific implementation of the multimodal feature hierarchical alignment system based on adaptive temperature coefficient in this embodiment of the invention is similar to the specific implementation of the multimodal feature hierarchical alignment method based on adaptive temperature coefficient in this embodiment of the invention. For details, please refer to the description in the method section. To reduce redundancy, it will not be repeated here.
[0055] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described above.
[0056] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described above.
[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0061] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multimodal feature hierarchical alignment method based on adaptive temperature coefficient, characterized in that, Includes the following steps: Acquire multimodal data; The multimodal feature hierarchical alignment model is trained based on the acquired multimodal data to obtain the trained multimodal feature hierarchical alignment model; the training process of the multimodal feature hierarchical alignment model includes: Construct a coding network for each modality of data, deconstruct the coding network of each modality into multiple layers, and extract the feature representation of each layer during the forward propagation of the coding network; A temperature coefficient is set for each layer, and the contrast loss between different modal features of that layer is calculated based on the temperature coefficient of each layer. By combining the feature representations of each mode in each layer, and based on the preset feature attribute perception temperature coefficient dynamic attenuation strategy, the specific modulation factor of each layer is calculated, and the temperature coefficient of the corresponding layer is adjusted based on the specific modulation factor of each layer. The overall objective loss function is obtained by fusing the contrastive losses of all levels, and the overall objective loss function is used to perform end-to-end joint training of the encoding networks of each modality. Cross-modal retrieval is performed based on a trained multimodal feature hierarchical alignment model.
2. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, In each layer, the contrastive loss between different modal features of that layer is expressed as: ， ， in, For any layer Contrast loss between text and image modal features It is the first Text features of the layer It is the first Image features of the layer, Indicates the first in the batch One sample, For the first Temperature coefficient of the layer yes, yes, It is the number of text-image pairs. Representing text features and image features Cosine similarity between them.
3. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, The process of combining the feature representations of each mode in each layer and calculating the specific modulation factor for each layer based on a preset temperature coefficient dynamic attenuation strategy that senses feature attributes includes: Calculate the statistical entropy of the activation values of text features and image features for each layer, and then calculate the comprehensive entropy value of the layer based on the statistical entropy of the activation values of text features and image features. The difference in the distribution of text and image features at each layer is measured using distribution differences based on cosine similarity. The modulation factor for each layer is calculated based on the summation entropy value of each layer and the difference in the distribution of text and image features.
4. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, The formula for calculating the modulation factor of each layer is: ， ， ， in, and It is a hyperparameter that controls the degree of influence of entropy and divergence; and The threshold parameter is used; the sigmoid function is used to convert the difference into a smooth value between 0 and 1. For the first The overall entropy of the layer For text features in the first The entropy of the layer, For image features in the first The entropy of the layer, No. Divergence of layered text and image feature distributions.
5. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, The coding network for each modality is deconstructed into an input layer, several intermediate hidden layers, and the final output layer.
6. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, The text encoding network uses a deep neural network model based on the Transformer architecture; the image encoding network uses a deep convolutional neural network model.
7. The multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in claim 1, characterized in that, Using the total loss after fusion, the text encoding network and the image encoding network are jointly trained and optimized end-to-end through the backpropagation algorithm.
8. A multimodal feature hierarchical alignment system based on adaptive temperature coefficient, characterized in that, include: A multimodal data acquisition module, which is used to acquire multimodal data; The hierarchical alignment module is used to train the hierarchical alignment model of multimodal features based on the acquired multimodal data, and obtain the trained hierarchical alignment model of multimodal features; it constructs the encoding network for each modality of data, deconstructs the encoding network of each modality into multiple layers, and extracts the feature representation of each layer during the forward propagation of the encoding network; A temperature coefficient is set for each layer, and the contrast loss between different modal features of the layer is calculated based on the temperature coefficient of each layer. Combining the feature representation of each modality of each layer, a specific modulation factor of each layer is calculated according to the preset feature attribute-aware temperature coefficient dynamic attenuation strategy, and the temperature coefficient of the corresponding layer is adjusted based on the specific modulation factor of each layer. The overall objective loss function is obtained by fusing the contrastive losses of all levels, and the overall objective loss function is used to perform end-to-end joint training of the encoding networks of each modality. The cross-modal retrieval module is used for cross-modal retrieval based on a trained multimodal feature hierarchical alignment model.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in any one of claims 1-7.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal feature hierarchical alignment method based on adaptive temperature coefficient as described in any one of claims 1-7.