Sequence level alignment method and multi-modal retrieval method thereof
By integrating sequence-level alignment methods and audio modalities, the problems of lack of audio modalities and neglect of details in global alignment in multimodal action retrieval are solved, achieving higher-precision retrieval and natural interaction, which is applicable to virtual reality, games and other fields.
Patent Information
- Application Number
- CN202510802341.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-16
- Publication Date
- 2025-10-31
AI Technical Summary
Existing multimodal action retrieval methods lack audio modality integration, rely on text or visual modality, resulting in unnatural user interaction, lack of dataset diversity, and global alignment methods that ignore key details, affecting retrieval performance.
A sequence-level alignment method is adopted to achieve fine-grained alignment between modalities by calculating the similarity matrix and weights between modalities. Combined with an audio modal augmentation dataset, multimodal retrieval is performed.
It significantly improves retrieval accuracy, enables more natural user interaction, enhances dataset diversity and retrieval performance, and performs exceptionally well in audio-driven motion retrieval tasks.
Smart Images
Figure CN120873233A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a sequence-level alignment method and its multimodal retrieval method. Background Technology
[0002] With the rapid development of multimedia technology, human motion has been widely used in animation, games, film production, robotics, and virtual / augmented reality. Generally, there are two main ways to acquire human motion: one is to retrieve motion sequences from existing datasets, and the other is to generate new motion sequences through algorithms.
[0003] Action generation technology has made significant progress in recent years; however, due to the "black box" nature of generative models, the generation process is difficult to control. Models often produce unpredictable and difficult-to-modify outputs, especially in complex real-world scenarios, where generated motions often fail to meet expected requirements. In contrast, action retrieval technology overcomes the limitations of generative models by selecting specific sequences of human actions from large-scale datasets. This approach is particularly important in industries requiring high accuracy, controllability, and editability.
[0004] Multimodal learning refers to improving model performance by integrating information from multiple modalities (such as text, images, and videos). However, although existing multimodal action retrieval techniques have made some progress in text and visual modalities, they still have some limitations:
[0005] First, there is a lack of integration of audio modalities. Existing methods mainly rely on text or visual modalities, ignoring the potential of audio modalities and limiting their application in real-world scenarios. Audio, as a more intuitive and user-friendly interaction method, can significantly enhance user immersion and convenience.
[0006] Secondly, the datasets lack diversity. Existing motion retrieval datasets mainly contain text and motion sequences, lacking paired data for audio modalities. This limits the training and evaluation of models in audio-driven motion retrieval tasks.
[0007] Third, there are limitations to global alignment. Existing multimodal action retrieval methods typically construct a joint embedding space through global alignment, which involves compressing the sequence features of each modality into a single vector and aligning them through contrastive learning. However, many modalities (such as video, text, and audio) are inherently serial, and key information is usually concentrated in specific segments. Global alignment methods often ignore these key details, resulting in poor alignment and consequently affecting retrieval performance. Summary of the Invention
[0008] To address the shortcomings of global alignment methods, which often overlook key details, resulting in poor alignment and consequently affecting retrieval performance, this invention proposes a sequence-level alignment method.
[0009] The technical solution adopted in this invention is a sequence-level alignment method, comprising:
[0010] S10. Obtain a first sequence set and a second sequence set, each consisting of multiple sequences, from the first modality and the second modality, respectively. The sequences are composed of word units.
[0011] S20. Calculate the similarity between word elements in the first sequence set and word elements in the second sequence set, and select the maximum similarity of each word element in the first sequence set in the second sequence set as the similarity measure of the sequences in the second sequence set.
[0012] S30. Calculate the first similarity matrix of the first sequence set relative to the second sequence set based on the similarity between the word elements in the first sequence set and the word elements in the second sequence set, as well as the similarity measure of the sequences in the second sequence set.
[0013] S40. Obtain the first weight by the difference in information content between the words in the first sequence set and the second sequence set, and generate the first intermodal similarity matrix between the first sequence set and the second sequence set based on the first weight and the first similarity matrix.
[0014] S50. Align the first mode and the second mode at the sequence level based on the similarity matrix between the first modes.
[0015] In some embodiments, step S20 further includes: calculating the similarity between word elements in the first sequence set and word elements in the second sequence set, and selecting the maximum similarity of each word element in the second sequence set in the first sequence set as a similarity measure of sequences in the first sequence set;
[0016] Step S30 further includes: calculating a second similarity matrix of the second sequence set relative to the first sequence set based on the similarity between the lexical units in the first sequence set and the lexical units in the second sequence set, as well as the similarity measure of the sequences in the first sequence set;
[0017] Step S40 further includes: obtaining a second weight by comparing the information content difference between the words in the second sequence set and the first sequence set, and generating a second intermodal similarity matrix between the second sequence set and the first sequence set based on the second weight and the second similarity matrix;
[0018] Step S50 includes: performing sequence-level bidirectional alignment of the first mode and the second mode based on the first mode similarity matrix and the second mode similarity matrix.
[0019] In some embodiments, step S40 further includes generating the first inter-modal similarity matrix and the second inter-modal similarity matrix into a matrix of the same type;
[0020] Step S50 includes: after adding the similarity matrix between the first mode and the similarity matrix between the second mode, performing sequence-level bidirectional alignment of the first mode and the second mode based on the calculation result.
[0021] In some embodiments, step S30 includes:
[0022] The first similarity matrix has only three dimensions: the sequence of the first sequence set, the word of the first sequence set, and the sequence of the second sequence set.
[0023] In some embodiments, step S40 includes:
[0024] The first weight contains the weight parameters of each word. The first weight is learned, and the weight parameters of each word in the first weight are adjusted.
[0025] To address the shortcomings of existing methods that primarily rely on text or visual modalities, neglecting the potential of audio modalities and lacking audio modal integration, this invention proposes a multimodal retrieval method.
[0026] The technical solution adopted in this invention is a multimodal retrieval method, comprising:
[0027] S100. Place the content to be retrieved and multiple data samples from the known dataset into a joint embedding space;
[0028] S200. Within the joint embedding space, the data sample and the content to be retrieved are aligned using the sequence-level alignment method of any one of claims 1 to 5 to obtain the alignment result;
[0029] S300. Calculate the similarity between the data sample and the content to be retrieved based on the alignment results;
[0030] S400. Obtain the best data sample and / or the best combination of data samples based on the degree of similarity.
[0031] In some embodiments, step S100 further includes:
[0032] Comparative loss is calculated for the alignment results in the joint embedding space.
[0033] In some embodiments, step S100 includes:
[0034] The data samples include audio, text, and video information.
[0035] In some embodiments, step S100 includes:
[0036] The dataset includes audio data, which is then augmented with colloquial language to obtain spoken language data.
[0037] The spoken data is stored in the dataset and then updated.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] This application discloses a sequence-level alignment method that, through a sequence-level contrastive learning mechanism, can effectively capture the local alignment relationship between modalities and obtain key cross-modal information, thereby significantly improving alignment accuracy. This amplifies the importance of fine-grained alignment in improving tasks such as retrieval, matching, and recognition, and significantly improves the accuracy when performing these tasks.
[0040] Compared with existing technologies, the sequence-level alignment method disclosed in this application can achieve the goal of focusing on key details and improving alignment results.
[0041] This application also discloses a multimodal retrieval method that introduces the audio modality into a multimodal motion retrieval framework, combining text, video, and motion modalities to achieve a more natural and intuitive user interaction. By extending existing text-action datasets, the audio modality is expanded, providing data support for audio-driven motion retrieval tasks. Furthermore, spoken language text and audio are specifically generated. This not only fills a gap in the data but also lays a data foundation for future research and applications.
[0042] Compared with existing technologies, the multimodal retrieval method disclosed in this application can avoid relying mainly on text or visual modalities and achieve the goal of integrating audio modalities with other modalities. Attached Figure Description
[0043] The present invention will now be described in detail with reference to the embodiments and accompanying drawings, wherein:
[0044] Figure 1 A schematic diagram of a sequence-level alignment method provided according to an embodiment of the present invention is shown;
[0045] Figure 2 The following diagram illustrates the application effect of a multimodal retrieval method provided according to an embodiment of the present invention.
[0046] Figure 3 A flowchart illustrating a multimodal retrieval method provided according to an embodiment of the present invention is shown;
[0047] Figure 4 A comparison chart of recall rates for TMR, LAVIMO, and our model when used for action retrieval is shown.
[0048] Figure 5 A comparison graph showing recall with and without using sequence-level alignment methods when used for action retrieval is shown;
[0049] Figure 6 A graph comparing recall rates when using spoken expressions versus raw expressions for action retrieval is shown. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Examples of embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0051] This invention discloses a sequence-level alignment method by employing a sequence-level contrastive learning mechanism to calculate the similarity between each token in each modality, thereby capturing key details across modalities. Figure 1 ,include:
[0052] S10. Obtain a first sequence set and a second sequence set, each consisting of multiple sequences, from the first modality and the second modality, respectively. The sequences are composed of word units.
[0053] S20. Calculate the similarity between word elements in the first sequence set and word elements in the second sequence set, and select the maximum similarity of each word element in the first sequence set in the second sequence set as the similarity measure of the sequences in the second sequence set.
[0054] S30. Calculate the first similarity matrix of the first sequence set relative to the second sequence set based on the similarity between the word elements in the first sequence set and the word elements in the second sequence set, as well as the similarity measure of the sequences in the second sequence set.
[0055] S40. Obtain the first weight by the difference in information content between the words in the first sequence set and the second sequence set, and generate the first intermodal similarity matrix between the first sequence set and the second sequence set based on the first weight and the first similarity matrix.
[0056] S50. Align the first mode and the second mode at the sequence level based on the similarity matrix between the first modes.
[0057] Specifically, for each word in modality 1, its similarity to all words in modality 2 is calculated, and the maximum similarity is selected as its similarity to the sequence in modality 2. This method can effectively capture the local alignment relationships between modalities. To account for the differences in information content among different words, learnable weights are introduced to weight the similarity matrix, generating the final inter-modal similarity matrix.
[0058] More specifically, for a given word in modality 1 with shape B1×L1, its similarity to a word in modality 2 with shape B2×L2 is calculated, and the maximum similarity is selected as the similarity measure between it and the sequence in modality 2. The calculation formula for each word in modality 1 and the sequence in modality 2 is as follows:
[0059]
[0060] Where e x Represents a given word in modality 1. Let represent any word in modality 2, and (·,·) denote the dot product between two vectors. This indicates the result of selecting the largest dot product among L2 tokens. Each token in modality 1 is processed as described above to obtain a similarity sequence of length B2. This process ultimately generates a similarity matrix of shape B1×L1×B2.
[0061] To fully account for the differences in information content among different lexical units, we introduce learnable weights, ultimately generating an inter-modal similarity matrix with dimensions B1×B2. Here, a shape of B1×L1 refers to having B1 sequences, each containing L1 lexical units; a shape of B2×L2 refers to having B2 sequences, each containing L2 lexical units.
[0062] In this application, the information content of each lexical unit is not explicitly calculated. Instead, learnable weight parameters are introduced to automatically learn the importance of each lexical unit in cross-modal alignment during training. Since the distribution and expressive power of lexical units vary in different samples, this approach can adaptively adjust the contribution of each lexical unit in similarity calculation, thereby improving the robustness and accuracy of inter-modal matching.
[0063] Fine-grained alignment significantly improves retrieval accuracy: Through a sequence-level contrastive learning mechanism, this application captures key details across modalities, significantly enhancing retrieval accuracy. Experimental results show that on the HumanML3D dataset, the R@10 for text-to-motion retrieval is improved by 10.16%, and the R@1 for video-to-motion retrieval is improved by 25.43%. This high-precision retrieval capability has significant application value in industries requiring high-precision motion data (such as filmmaking and robot control).
[0064] In this context, a lexical unit refers to a meaningful unit separated from a modality. The first modality and the second modality can be the same modality or different modalities.
[0065] Similarity measurement is a quantitative indicator used to measure the degree of similarity between two objects. In this application, the maximum similarity of each word in the first sequence set in the second sequence set is selected as the similarity measure of the sequences in the second sequence set because the maximum similarity can focus on the extreme values and abnormal positions of the sequences in the second sequence set, thereby reflecting the key features of the second sequence.
[0066] It should be noted that, in order to fully consider the information content difference between the first sequence set and the second sequence set, and to accurately capture and quantify the local detail information between different sequence sets without affecting the overall performance of the model, this application further generates an inter-modal similarity matrix between the first sequence set and the second sequence set based on the information content difference and similarity matrix between the first and second sequence sets. The information content difference between the words in the first sequence set and the second sequence set, as well as the first similarity matrix, can be multiplied to obtain the first inter-modal similarity matrix between the first and second sequence sets.
[0067] The first and second modes are aligned sequentially based on the similarity matrix between the first modes. Specifically, the optimal alignment path can be obtained from the similarity matrix between the first modes, and the sequence alignment is performed according to the optimal alignment path.
[0068] Furthermore, the intermodal similarity matrix obtained in this embodiment is relative to the second sequence set and is used to represent how the lexical units in the first sequence set are mapped to the lexical units in the second sequence set. It can be applied to one-way or two-way alignment tasks and can control the amount of computation in some scenarios where the amount of computation is required.
[0069] In some embodiments, please refer to Figure 1 Step S20 further includes: calculating the similarity between word elements in the first sequence set and word elements in the second sequence set, and selecting the maximum similarity of each word element in the second sequence set in the first sequence set as the similarity measure of the sequences in the first sequence set;
[0070] Step S30 further includes: calculating a second similarity matrix of the second sequence set relative to the first sequence set based on the similarity between the lexical units in the first sequence set and the lexical units in the second sequence set, as well as the similarity measure of the sequences in the first sequence set;
[0071] Step S40 further includes: obtaining a second weight by comparing the information content difference between the words in the second sequence set and the first sequence set, and generating a second intermodal similarity matrix between the second sequence set and the first sequence set based on the second weight and the second similarity matrix;
[0072] Step S50 includes: performing sequence-level bidirectional alignment of the first mode and the second mode based on the first mode similarity matrix and the second mode similarity matrix.
[0073] Specifically, the sequence-level alignment method disclosed in this application generates not only a first inter-modal similarity matrix of the first sequence set relative to the second sequence set, but also a second inter-modal similarity matrix of the second sequence set relative to the first sequence set. Both matrices can be used separately or together to obtain the alignment result. In actual operation, the steps of obtaining the two inter-modal similarity matrices can be performed simultaneously, thereby reducing processing time.
[0074] In some embodiments, step S40 further includes generating the first inter-modal similarity matrix and the second inter-modal similarity matrix into a matrix of the same type;
[0075] Step S50 includes: after adding the similarity matrix between the first mode and the similarity matrix between the second mode, performing sequence-level bidirectional alignment of the first mode and the second mode based on the calculation result.
[0076] Specifically, identical matrix refers to two modality similarity matrices having the same dimensions (i.e., the number of rows and columns). This facilitates the processing of the two modality similarity matrices, and after addition, a combined modality similarity matrix is obtained. Then, bidirectional sequence-level alignment is performed based on the combined modality similarity matrix.
[0077] In some embodiments, step S30 includes:
[0078] The first similarity matrix has only three dimensions: the sequence of the first sequence set, the word of the first sequence set, and the sequence of the second sequence set.
[0079] The three-dimensional setup offers several advantages. First, it enables finer-grained alignment modeling: compared to directly calculating the overall similarity between two samples, this three-dimensional matrix can capture the differences at various positions within Modality 1 at the lexical level, thus achieving more refined cross-modal alignment. Second, it preserves local information: certain lexical units may have specific correspondences with samples in Modality 2 (such as local actions and descriptions, local images and speech segments). This structure can retain this important local matching information without being overwhelmed by overall averaging or weighting processes.
[0080] In some embodiments, step S40 includes:
[0081] The first weight contains the weight parameters of each word. The first weight is learned, and the weight parameters of each word in the first weight are adjusted.
[0082] The first set of weights are learnable weights. These weights are obtained by inputting the feature vectors of each modality into independent linear transformation layers, and then normalizing them using softmax to obtain the normalized weight distribution. Formally, this is represented as:
[0083] w x =softmax(w x ·X),w y =softmax(w y ·Y)
[0084] Where w x and w y It is a learnable linear transformation matrix, where X and Y are the tokens corresponding to the first and second modes, i.e., the corresponding modal features, represented as a vector.
[0085] The weights are optimized together with the main loss function through backpropagation, without the need for explicit manual annotation or design. During training, the model can automatically learn and adjust the weight parameters of each word, making them more conducive to optimizing inter-modal alignment. After introducing weights, the original inter-modal word similarity matrix of shape B1×L1×B2 will be weighted and summed along the L1 dimension to obtain a global modal similarity matrix of shape B1×B2.
[0086] The weights are also represented in matrix form, with the same dimension as the first sequence set. The information content difference reflects the difference in information content between the first and second sequence sets. Since the focus is on the characteristics of the first sequence set, using weights with the same dimension as the first sequence set is most appropriate, resulting in a more accurate representation of the inter-modal similarity matrix. Furthermore, these weights are learnable, automatically adjusting parameters based on data during model training. By learning these weights, the model can reflect the impact of anomalous features or key details on its output, thus allowing for targeted parameter adjustments.
[0087] In other embodiments, differences in information content can be expressed by indicators such as entropy value, mutual information, or basic quantization parameters such as bits and bytes.
[0088] To address the shortcomings of existing methods that primarily rely on text or visual modalities, neglecting the potential of audio modalities and lacking audio modal integration, this invention proposes a multimodal retrieval method.
[0089] Existing datasets and model designs primarily focus on text and visual modalities, lacking systematic research and integration of audio modalities. The technical solution adopted in this invention is a multimodal retrieval method; please refer to [reference needed]. Figure 2 and Figure 3 ,include:
[0090] S100. Place the content to be retrieved and multiple data samples from the known dataset into a joint embedding space;
[0091] S200. Within the joint embedding space, the data sample and the content to be retrieved are aligned using the sequence-level alignment method of any one of claims 1 to 5 to obtain the alignment result;
[0092] S300. Calculate the similarity between the data sample and the content to be retrieved based on the alignment results;
[0093] S400. Obtain the best data sample and / or the best combination of data samples based on the degree of similarity.
[0094] The modality of the content to be retrieved is action information, while the modality used for retrieval can be multimodal information, including text, speech, and video. Regarding the choice of pre-trained model for encoding multimodal data, BERT, DistilBERT, or other audio encoders can be used instead of WavLM to achieve similar multimodal encoding results.
[0095] After text, audio, and video inputs are encoded using a pre-trained model, a fine-grained contrastive loss method is employed to align them within a joint embedding space. Following sequence-level alignment, the alignment results are used to capture detailed information. This detailed information is then placed within the joint embedding space to calculate similarity. The joint embedding space maps data from different modalities to the same space, and similarity is calculated within this space. Finally, data is retrieved from a known dataset based on the similarity scores.
[0096] Multimodal integration enhances user experience: This application introduces audio modality into a multimodal motion retrieval framework for the first time, combining text, video, and motion modality to achieve more natural and intuitive user interaction. Audio, as a low-barrier interaction method, can significantly enhance user immersion and convenience, and is particularly suitable for fields requiring high interactivity such as virtual reality, games, and animation. This multimodal integration not only enhances the user experience but also provides technical support for product differentiation.
[0097] The four-modal framework of this application outperforms existing two-modal or three-modal methods in multiple retrieval tasks, demonstrating the comprehensiveness and leading nature of the technology.
[0098] Figure 2 This diagram illustrates the application of a multimodal retrieval method. It encodes text, video, or audio descriptions and calculates their similarity within a shared joint embedding space, thereby retrieving the most relevant content and ranking the top three results by similarity. The sample with the highest similarity is the most similar to the content being retrieved; this sample is considered the optimal data sample. Conversely, in some cases where operators need to analyze samples with low similarity, the least similar sample becomes the optimal data sample. It's important to note that the optimal data sample combination consists of multiple samples with similar levels of similarity, facilitating analysis by the operator.
[0099] Figure 3 This is a flowchart illustrating a multimodal retrieval method, showcasing the model's training process and involving four input modalities: video, text, action, and audio. First, features are extracted from video, text, action, and audio using their respective encoders. Next, a sequence-level alignment mechanism is used to align the features of different modalities at a fine-grained level to capture key details. Finally, a contrastive loss is calculated in the joint embedding space to ensure that the similarity between modalities is effectively learned.
[0100] In some embodiments, step S100 further includes:
[0101] Contrastive loss is calculated for the alignment results in the joint embedding space. Specifically, the alignment result is a modality similarity matrix between different modalities. Furthermore, the semantic pairings of samples between different modalities are one-to-one, so cross-entropy can be used to measure the difference between the alignment result and the true pairing.
[0102] In addition to calculating the similarity in the joint embedding space, it is also necessary to calculate the contrast loss for detailed information to ensure that the similarity between modalities is effectively learned.
[0103] Please refer to Figure 4 TMR, LAVIMO, and our model are algorithmic models used for action retrieval. Retrieval performance is evaluated using standard metrics, including recall at different rankings (e.g., R@1, R@3) to measure text-to-motion retrieval accuracy. A higher R-value indicates higher retrieval accuracy. Figure 4 The accuracy of text-action and action-text retrieval for several models is demonstrated. Our quadmodal version outperforms existing methods and our trimodal version, proving the effectiveness of our fine-grained alignment-based multimodal framework. Introducing the audio modality significantly improves retrieval accuracy.
[0104] Please refer to Figure 5Compared to methods that do not use sequence-level alignment, our fine-grained contrastive learning mechanism improves retrieval performance by enabling the model to capture richer sequence information.
[0105] Existing dataset construction methods primarily focus on text and visual modalities, failing to adequately consider the diversity and practicality of audio modalities. In some embodiments, step S100 includes:
[0106] The data samples include audio, text, and video information.
[0107] To address the shortcomings of existing dataset construction methods that primarily focus on text and visual modalities and fail to adequately consider the diversity and practicality of audio modalities, in some embodiments, step S100 includes:
[0108] The dataset includes audio data, which is then augmented with colloquial language to obtain spoken language data.
[0109] The spoken data is stored in the dataset and then updated.
[0110] Expanding the multimodal dataset provides a foundation for future research: By extending the existing text-action dataset, an audio modality was added, providing data support for audio-driven motion retrieval tasks. Furthermore, spoken language text and audio were specifically generated. This not only fills a gap in the dataset but also lays the data foundation for future research and applications.
[0111] Conventional training methods rely on existing text-action datasets and lack systematic support for audio modalities, failing to meet the needs of audio-driven motion retrieval. This application, however, extends existing text-motion datasets using ChatGPT and Tortoise to construct a multimodal motion retrieval dataset that includes audio modalities. This dataset construction method is an innovation not addressed in existing technologies. Experimental results show that models trained using our constructed dataset perform excellently in audio-to-motion retrieval tasks, demonstrating the uniqueness and necessity of our dataset construction method.
[0112] Please refer to Figure 6 When the model was trained on the original dataset, its performance dropped significantly, which we attributed to the stylistic differences between conversational and formal expressions. This highlights the necessity of building a dedicated spoken audio dataset for real-world applications.
[0113] To address the shortcomings of global alignment devices, which often overlook key details, resulting in poor alignment and consequently affecting retrieval performance, this invention proposes a computer device.
[0114] The technical solution adopted in this invention is a computer device, comprising: a processor and a memory, wherein the memory is used to store computer program code, the computer program code including computer instructions, and the computer device executes the above-described method when the processor executes the computer instructions.
[0115] The computer device includes a processor and memory. Optionally, the computer device also includes an input device and an output device. The processor, memory, input device, and output device are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this application embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.
[0116] The processor may include one or more processors, such as one or more central processing units (CPUs). If the processor is a CPU, it can be a single-core CPU or a multi-core CPU. Optionally, the processor may be a processor group consisting of multiple CPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of this application.
[0117] The memory can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.
[0118] Input devices are used to input data and / or signals, and output devices are used to output data and / or signals. Input and output devices can be independent devices or an integrated device.
[0119] It is understood that in this embodiment of the application, the memory can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.
[0120] In practical applications, computer devices may also include other necessary components, including but not limited to any number of input / output devices, processors, memory, etc., and all computer devices that can implement the embodiments of this application are within the protection scope of this application.
[0121] In the description of this specification, the use of terms such as "Embodiment 1," "this embodiment," or "in one embodiment" indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example; moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in one or more embodiments or examples.
[0122] In the description of this specification, the terms "connection," "installation," "fixing," "setting," and "having" are interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0123] In the description of this specification, relational terms such as “first” and “second” are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0124] The above description of the embodiments is intended to enable those skilled in the art to understand and apply the technology of this invention. Those skilled in the art can easily make various modifications to these examples and apply the general principles described herein to other embodiments without creative effort. Therefore, this invention is not limited to the above embodiments. Modifications in the following situations should be within the scope of protection of this invention: ① New technical solutions implemented based on the technical solution of this invention and combined with existing common knowledge, where the technical effects of the new technical solution do not exceed the technical effects of this invention; ② Equivalent substitutions of some features of the technical solution of this invention using known technology, resulting in the same technical effects as those of this invention; ③ Extendable technical solutions based on the technical solution of this invention, where the substantive content of the extended technical solution does not exceed the technical solution of this invention; ④ Equivalent transformations made using the content of this specification and drawings, directly or indirectly applied to other related technical fields.
Claims
1. A sequence-level alignment method, characterized in that, include: S10. Obtain a first sequence set and a second sequence set, each consisting of multiple sequences, from the first modality and the second modality, respectively. The sequences are composed of word units. S20. Calculate the similarity between word units in the first sequence set and word units in the second sequence set, and select the maximum similarity of each word unit in the first sequence set in the second sequence set as the similarity measure of the sequences in the second sequence set; S30. Calculate the first similarity matrix of the first sequence set relative to the second sequence set based on the similarity between the word units in the first sequence set and the word units in the second sequence set, as well as the similarity measure of the sequences in the second sequence set. S40. Obtain a first weight by the information content difference between the words in the first sequence set and the second sequence set, and generate a first intermodal similarity matrix between the first sequence set and the second sequence set based on the first weight and the first similarity matrix. S50. Based on the first modality similarity matrix, perform sequence-level alignment between the first modality and the second modality.
2. The sequence-level alignment method according to claim 1, characterized in that, Step S20 further includes: calculating the similarity between word elements in the first sequence set and word elements in the second sequence set, and selecting the maximum similarity of each word element in the second sequence set in the first sequence set as the similarity measure of the sequences in the first sequence set; Step S30 further includes: calculating a second similarity matrix of the second sequence set relative to the first sequence set based on the similarity between the words in the first sequence set and the words in the second sequence set and the similarity measure of the sequences in the first sequence set; Step S40 further includes: obtaining a second weight by the information content difference between the second sequence set and the first sequence set, and generating a second intermodal similarity matrix between the second sequence set and the first sequence set based on the second weight and the second similarity matrix; Step S50 includes: performing sequence-level bidirectional alignment of the first mode and the second mode based on the first inter-modal similarity matrix and the second inter-modal similarity matrix.
3. The sequence-level alignment method according to claim 2, characterized in that, Step S40 further includes generating the first inter-modal similarity matrix and the second inter-modal similarity matrix into a matrix of the same type; Step S50 includes: after adding the first modality similarity matrix and the second modality similarity matrix, performing sequence-level bidirectional alignment of the first modality and the second modality based on the calculation result.
4. The sequence-level alignment method according to claim 1, characterized in that, Step S30 includes: The first similarity matrix has only three dimensions: the sequence of the first sequence set, the word of the first sequence set, and the sequence of the second sequence set.
5. The sequence-level alignment method according to claim 1, characterized in that, Step S40 includes: The first weight has weight parameters for each word element. The first weight is learned, and the weight parameters for each word element in the first weight are adjusted.
6. A multimodal retrieval method, characterized in that, include: S100. Place the content to be retrieved and multiple data samples from the known dataset into a joint embedding space; S200. Within the joint embedding space, the data sample and the content to be retrieved are aligned using the sequence-level alignment method of any one of claims 1 to 5 to obtain an alignment result; S300: Calculate the similarity between the data sample and the content to be retrieved based on the alignment result; S400. Obtain the optimal data sample and / or the optimal combination of data samples based on the degree of similarity.
7. The multimodal retrieval method according to claim 6, characterized in that, Step S100 further includes: The contrast loss is calculated for the alignment result in the joint embedding space.
8. The multimodal retrieval method according to claim 6, characterized in that, Step S100 includes: The data samples include audio information, text information, and video information.
9. The multimodal retrieval method according to claim 6, characterized in that, Step S100 includes: The dataset includes audio data, which is then augmented with colloquial language to obtain colloquial data. The spoken language data is stored in the dataset, and the dataset is updated.