Method and system for dynamic attention multi-modal semantic alignment and vector duplication checking

Abstract

The present application relates to the technical field of data processing, in particular to a method and system for multi-modal semantic alignment and vector duplication checking based on dynamic attention. The present application first extracts and serializes features of multi-modal media content such as images, texts and videos; then, a dynamic attention calculation module of a lightweight gated routing network is introduced to generate dynamic gating values for different cross-modal interaction pairs, and the dynamic gating values are fused with standard attention weights to form dynamic attention weights that can adaptively enhance key semantic interactions and suppress noise; on this basis, multi-level semantic alignment is performed locally and globally, and a unified and semantic-rich multi-modal semantic vector is generated through end-to-end joint training; and efficient and accurate cross-modal content duplication checking is realized through a vector database. The present application effectively solves the technical defects of traditional methods such as coarse duplication checking granularity, weak cross-modal capability and shallow semantic understanding, and significantly improves the intelligent level and efficiency of media content management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a method and system for multi-modal semantic alignment and vector duplication checking based on dynamic attention. BACKGROUND

[0002] With the rapid development of the Internet and multimedia technology, the media content production industry has witnessed an explosive growth in data volume. News agencies, content platforms and self-media creators need to process massive amounts of multi-modal data every day, including but not limited to news videos, interview audios, news pictures and text manuscripts, etc. Under this background, how to efficiently manage and retrieve these materials, and effectively avoid duplicate publication and copyright infringement, has become a core challenge faced by the industry.

[0003] At present, the existing media content management system usually adopts the following technical solutions for content retrieval and duplication checking:

[0004] Keyword-based retrieval method: This method relies on manual annotation or automated tools (such as object recognition, automatic speech recognition ASR) to extract keywords or tags of the content. When checking for duplication, the similarity is determined by comparing the keyword sets of the contents. However, this method is heavily dependent on the quality and completeness of the annotations, and cannot understand semantics. For different descriptions of the same event (such as synonyms, paraphrases) or deep semantic content not reflected in the keywords, this method is prone to miss.

[0005] Hash value or low-level feature-based matching method: This method calculates the hash value (such as MD5, SHA-1) or low-level visual features (such as color histogram, texture) of the media file for accurate matching. The disadvantage is that the granularity of duplication checking is extremely rough, and any minor modification to the content (such as video editing, image scaling, brightness adjustment, audio noise reduction) will cause a significant change in the hash value or feature, making it impossible to detect substantial duplicate content.

[0006] General semantic vector-based retrieval method: This method uses pre-trained deep learning models (such as BERT for text, ResNet for images) to map media content into a semantic vector space, and determines the content relevance by calculating the similarity (such as cosine similarity) between vectors. Although this method achieves semantic-level retrieval to some extent, the model is usually trained in a general domain, and when dealing with cross-modal content (such as determining whether a video and a text manuscript describe the same event), it is difficult to achieve fine-grained semantic alignment, resulting in low recall and precision.

[0007] Therefore, there is an urgent need in the art for a new technical solution that can deeply understand the semantics of multi-modal content, achieve accurate cross-modal alignment, and balance retrieval efficiency and duplication accuracy. SUMMARY

[0008] The present application aims to overcome the shortcomings of the prior art, and provides a method and system for dynamic attention multi-modal semantic alignment and vector duplication detection, which aims to achieve fine-grained semantic alignment and vector generation of multi-modal media content through a dynamic attention mechanism, thereby improving duplication detection precision, cross-modal capability and retrieval efficiency.

[0009] To achieve the above-mentioned purpose, the present application provides a system for dynamic attention multi-modal semantic alignment and vector duplication detection, which is used for efficient duplication detection of query content, and the query content is media content, which includes:

[0010] A feature extraction and serialization module is configured to extract image features, text features and video features from the media content, and convert each feature into an embedding vector sequence; wherein the media content includes at least one of image data, text data and video data;

[0011] A dynamic attention calculation module is connected to the feature extraction and serialization module, and is configured to calculate the dynamic attention weight of the cross-modal interaction pair based on a dynamic attention gate module, so as to adaptively enhance the key interaction and suppress the non-key interaction; wherein the cross-modal interaction pair is determined based on the matching of the embedding vector sequence of each feature;

[0012] A semantic alignment and vector fusion module is connected to the dynamic attention calculation module, and is configured to perform local and global semantic alignment using the dynamic attention weight, and generate a unified multi-modal semantic vector;

[0013] A duplication execution module is connected to the semantic alignment and vector fusion module, and includes a vector database and a similarity calculation unit; wherein the vector database is used to store the multi-modal semantic vector, and the similarity calculation unit is used to determine the content duplication by calculating the cosine similarity.

[0014] As a further solution, the feature extraction and serialization module includes:

[0015] An image encoding unit is configured to convert image features into image block embedding vector sequences using a CNN or ViT model;

[0016] A text encoding unit is configured to convert text features into word embedding vector sequences using a BERT model;

[0017] A video processing unit is configured to extract video key frames using an FFMPEG tool, and convert the key frame features into image block embedding vector sequences using a CNN or ViT model.

[0018] As a further solution, the dynamic attention calculation module includes:

[0019] a standard attention calculation unit configured to calculate a standard attention weight of the cross-modal interaction pair;

[0020] a dynamic gating unit configured to calculate a dynamic gating value of each cross-modal interaction pair by a light-weight gating routing network;

[0021] a fusion unit configured to fuse the standard attention weight and the dynamic gating value to obtain the dynamic attention weight.

[0022] As a further solution, the light-weight gating routing network in the dynamic gating unit comprises a first fully connected layer, a ReLU activation function, a second fully connected layer and a Sigmoid activation function connected in sequence;

[0023] The light-weight gating routing network passes the input feature through the first fully connected layer and the ReLU activation function to obtain an intermediate feature, and then passes the intermediate feature through the second fully connected layer and the Sigmoid activation function to output the dynamic gating value.

[0024] Wherein, the input is the context feature of the cross-modal interaction pair, and the output is a scalar gating value between 0 and 1.

[0025] As a further solution, the semantic alignment and vector fusion module comprises:

[0026] a local alignment unit configured to calculate a text-guided image representation and an image-guided text representation using the dynamic attention weight, and to close the paired local features by a local alignment loss function;

[0027] a global alignment unit configured to aggregate the global image representation and the text representation enhanced by the dynamic attention, and to constrain by a global alignment loss function;

[0028] a vector generation unit configured to concatenate, sum or process by a fusion network the embedding vector sequence after global alignment to generate the unified multi-modal semantic vector.

[0029] As a further solution, a training module is further included and connected with the semantic alignment and vector fusion module, configured to perform joint training in an end-to-end multi-task learning manner, so that the dynamic gating unit can allocate the most effective attention resources for different inputs; wherein, the total loss function is set as a weighted sum of the local alignment loss, the global alignment loss and the task loss.

[0030] As a further solution, the vector database in the duplicate detection execution module is established with high efficiency index to support efficient duplicate detection based on the multi-modal semantic vector; wherein,

[0031] When there is media content to be checked, the multi-modal semantic vector of the media content is calculated and compared with the cosine similarity of the multi-modal semantic vectors in the vector database;

[0032] If the similarity exceeds the preset threshold, the media content is determined to be duplicate content or highly similar content, and an alarm and a similar content link are outputted;

[0033] If the similarity does not exceed the preset threshold, the media content is marked as new content, the multi-modal semantic vector of the media content is calculated and added to the vector database.

[0034] In another aspect, the application also provides a method for dynamic attention multi-modal semantic alignment and vector duplication checking, which is applied to the system as described in any one of the above, and includes the following steps:

[0035] Step 1: multi-modal feature extraction and serialization; wherein image features, text features and video features are extracted from the media content, and each feature is converted into an embedded vector sequence;

[0036] Step 2: dynamic cross-modal attention calculation; wherein based on a dynamic attention gate module, the dynamic attention weight of the cross-modal interaction pair is calculated to adaptively enhance the key interaction and suppress the non-key interaction;

[0037] Step 3: multi-level semantic alignment and vector fusion; wherein the dynamic attention weight is used for local and global semantic alignment to generate a unified multi-modal semantic vector;

[0038] Step 4: efficient duplication checking based on multi-modal semantic vector; wherein the multi-modal semantic vector is stored in the vector database, and the content duplication is determined by calculating the cosine similarity.

[0039] As a further solution, the multi-modal semantic vector is generated by the following steps:

[0040] Obtain image features and text features, and determine cross-modal interaction pairs based on embedded vector sequence matching of each feature;

[0041] Calculate the standard attention weight of the cross-modal interaction pair, and calculate the dynamic gate value of each cross-modal interaction pair through a lightweight gate routing network;

[0042] Fuse the standard attention weight and the dynamic gate value to obtain the dynamic attention weight;

[0043] Performing local fine-grained alignment: calculating the text-guided image representation and the image-guided text representation using dynamic attention weight, and narrowing the local features through a local alignment loss function to obtain a local alignment loss and provide a fine-grained supervision signal;

[0044] Performing global fine-grained alignment: aggregating the global image representation and the text representation enhanced by dynamic attention, and constraining them through a global alignment loss function to obtain a global alignment loss; wherein the global fine-grained alignment process is supervised by the fine-grained supervision signal;

[0045] Performing loss fusion: weighting and summing the global alignment loss and the local alignment loss, and performing training convergence judgment; wherein if convergence is achieved, the corresponding multi-modal semantic vector is output; otherwise, the generation step of the multi-modal semantic vector is re-executed.

[0046] As a further solution, the standard attention weight is calculated by the following steps:

[0047] Taking the image features and the text features as inputs, a query matrix Q, a key matrix K and a value matrix V are obtained through linear transformation,

[0048] The attention score matrix is obtained by multiplying the transpose of the query matrix Q and the key matrix K;

[0049] The attention score matrix is scaled; wherein Scaled Scores = Scores / √(d_k), (d_k) is the dimension of the key vector, Scaled Scores is the scaled score, Scores is the attention score matrix, and √ represents the square root;

[0050] The scaled attention score matrix is normalized by Softmax to obtain the original attention weight;

[0051] The value matrix V is weighted and summed based on the original attention weight, and the weighted result is taken as the standard attention weight.

[0052] Compared with the related art, the method and system for multi-modal semantic alignment and vector duplication detection based on dynamic attention provided by the present application have the following advantages:

[0053] 1. The present application realizes deep and fine-grained semantic understanding through a dynamic attention mechanism, effectively identifies substantially repeated content that has been subjected to complex processing such as editing and restating, and overcomes the defect of coarse granularity of traditional duplication detection methods;

[0054] 2、The application constructs a unified multi-modal semantic vector space, so that the system can accurately perform cross-modal content duplication checking such as "video-text", "audio-picture", and solves the problem of weak cross-modal semantic alignment capability in the prior art.

[0055] 3、The application converts complex semantic comparison into near neighbor search in the vector database, realizes millisecond-level response to the massive media library, and fundamentally solves the industry pain point of time-consuming traversal comparison. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the accompanying drawings needed to be used in the embodiments or related art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor.

[0058] Figure 1 A system structure diagram based on dynamic attention multi-modal semantic alignment and vector duplication checking is provided for the application.

[0059] Figure 2 A method step diagram based on dynamic attention multi-modal semantic alignment and vector duplication checking is provided for the application.

[0060] Figure 3 A multi-modal semantic vector generation flowchart is provided for the application.

[0061] Figure 4 A standard attention weight calculation flowchart is provided for the application.

[0062] Figure 5 A lightweight gate routing network processing flowchart is provided for the application.

[0063] The purpose of the application, functional characteristics and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0065] Embodiment 1

[0066] Referring to Figure 1 The embodiment provides a system based on dynamic attention multi-modal semantic alignment and vector duplication checking, which is used for efficient duplication checking of to-be-queried content, the to-be-queried content being media content, comprising:

[0067] A feature extraction and serialization module is configured to extract image features, text features and video features from the media content, and convert the features into embedded vector sequences; wherein the media content comprises at least one of image data, text data and video data;

[0068] A dynamic attention calculation module is connected with the feature extraction and serialization module, and is configured to calculate dynamic attention weights of cross-modal interaction pairs based on a dynamic attention gate module, so as to adaptively enhance key interactions and suppress non-key interactions; wherein the cross-modal interaction pairs are determined based on matching of the embedded vector sequences of the features;

[0069] A semantic alignment and vector fusion module is connected with the dynamic attention calculation module, and is configured to perform local and global semantic alignment using the dynamic attention weights, and generate a unified multi-modal semantic vector;

[0070] A duplication checking execution module is connected with the semantic alignment and vector fusion module, and comprises a vector database and a similarity calculation unit; wherein the vector database is configured to store the multi-modal semantic vector, and the similarity calculation unit is configured to determine content duplication by calculating cosine similarity.

[0071] It should be noted that: the core of the present application is to fundamentally improve the accuracy, efficiency and cross-modal retrieval capability of media content duplication checking by introducing a dynamic attention mechanism and a multi-level semantic alignment strategy.

[0072] Specifically, the dynamic attention gate module designed in the present application can adaptively evaluate and enhance the interaction of key semantic units between different modalities, while suppressing non-key or noise interference. This mechanism enables the model to no longer treat all information equally when performing semantic alignment, but can dynamically focus on the core elements of the content, such as event key frames, core characters, main points, etc., just like a human.

[0073] Therefore, the system can accurately identify the substantially duplicated content after complex editing such as cutting, scaling, dubbing or text restatement, and significantly overcome the defects of traditional methods based on hash value or keyword matching, etc. in terms of rough granularity.

[0074] Furthermore, by constructing a unified multimodal semantic vector space, this invention achieves true cross-modal semantic-level plagiarism detection. Whether it's news videos and text articles, or audio clips and news images, as long as they describe the same core event or theme, the generated semantic vectors will be sufficiently close in distance within the vector space. This enables the system to efficiently and accurately complete complex tasks such as "searching for text by image" and "searching for articles by video," solving the industry pain point of weak cross-modal plagiarism detection capabilities in existing systems.

[0075] In terms of efficiency, this invention transforms the complex semantic understanding process into similarity calculation of high-dimensional vectors. Once media content is transformed into a unified multimodal semantic vector, subsequent massive database retrieval can be completed using an efficient vector database and nearest neighbor search algorithm, achieving millisecond-level response speeds. This design cleverly balances retrieval accuracy and efficiency, making real-time, fine-grained deduplication of hundreds of millions of media materials possible, effectively supporting the practical application needs of media organizations in content production and distribution.

[0076] Finally, through an end-to-end joint training paradigm, the model can specifically learn semantic patterns unique to the media domain, such as the five elements of news, interpersonal relationships, and opinion biases, thus possessing stronger domain adaptability. Compared to general pre-trained models, the semantic vectors generated by this invention provide a more discriminative representation of media content, further ensuring the professionalism and reliability of the plagiarism detection results.

[0077] Furthermore, the feature extraction and serialization module includes:

[0078] An image coding unit is configured to use a CNN or ViT model to convert image features into a sequence of image patch embedding vectors.

[0079] The text encoding unit is configured to use the BERT model to convert text features into a sequence of word embedding vectors.

[0080] The video processing unit is configured to extract video keyframes using the FFMPEG tool and convert the keyframe features into a sequence of image patch embedding vectors using a CNN or ViT model.

[0081] The image features are converted into a sequence of image patch embedding vectors V={v1,v2,...,v...} m}; Convert text features into a word embedding vector sequence T={t1,t2,...,t n For video features, keyframes can be extracted and converted into image patch embedding vector sequences. In addition, if there is audio text in the video, it can be converted into corresponding word embedding vector sequences after audio recognition.

[0082] Further, the dynamic attention calculation module comprises:

[0083] a standard attention calculation unit configured to calculate a standard attention weight of each cross-modal interaction pair;

[0084] a dynamic gating unit configured to calculate a dynamic gating value of each cross-modal interaction pair by a lightweight gating routing network;

[0085] a fusion unit configured to fuse the standard attention weight and the dynamic gating value to obtain the dynamic attention weight.

[0086] Specifically, the dynamic attention calculation module is a core module; wherein, for each cross-modal interaction pair (such as an image block v i and a text word t j ), not only a standard attention weight A ij is calculated, but also a dynamic gating value G ij is calculated; wherein,

[0087] Calculation of the dynamic gating value G ij : the value is calculated by a lightweight gating routing network, and an input of the lightweight gating routing network is a context feature of the current interaction pair, for example, [v i ,t j ,v i ⊙t j ][v i ,t j ,v i ⊙t j ] (wherein, ⊙ represents element-wise multiplication or splicing), and an output of the lightweight gating routing network is a scalar between 0 and 1. The gating value represents the “importance” or “correlation strength” of the current cross-modal interaction pair.

[0088] Dynamic weight fusion: the standard attention weight A ij is fused with the dynamic gating value G ij to form a final dynamic attention weight A ij =Softmax(A ij * G ij ). This design enables the model to dynamically suppress unimportant interactions and enhance key interactions, thereby realizing adaptive selection of attention paths.

[0089] Further, the lightweight gating routing network in the dynamic gating unit comprises a first fully connected layer, a ReLU activation function, a second fully connected layer and a Sigmoid activation function connected in sequence;

[0090] The lightweight gated routing network passes the input features through a first fully connected layer and a ReLU activation function to obtain intermediate features, and then passes them through a second fully connected layer and a Sigmoid activation function to output dynamic gate values.

[0091] The input consists of contextual features of cross-modal interaction pairs, and the output is a scalar gating value between 0 and 1.

[0092] Specifically, such as Figure 5 The lightweight gated routing network shown in this embodiment transforms a complex decision problem of "how to allocate attention" into a differentiable neural network forward computation process through an ingenious process of "input concatenation → feature transformation (FC1+ReLU) → decision output (FC2+Sigmoid)".

[0093] Step 1: Preparing Input – Generating Collaborative Context Features

[0094] Operation: Take a feature vector v from an image modality i (For example, representing a region in an image), a feature vector t from a text modality. j (For example, representing a word), and the interaction feature v between the two. i ⊙t j (Usually, this is done by element-wise multiplication or concatenation) to form a comprehensive co-context feature vector.

[0095] Explanation: The purpose of this step is to provide the gating network with a "comprehensive intelligence" report on the current "image-text" combination. This includes not only the individual pieces of information (v...) i and t j It also includes preliminary interaction information between them (v) i ⊙t j This allows gating networks to make decisions based on rich contextual information, rather than looking at two features in isolation.

[0096] Step 2: First-level transformation and nonlinear activation

[0097] Operation: Input the collaborative context feature vector obtained in the previous step into the first fully connected layer (FC1), the weight matrix of which is W1∈R. 3d×hidden_dim The bias is set to b1. Then, the ReLU activation function is used to perform a non-linear transformation on the output;

[0098] The calculation can be expressed as: h1 = ReLU(W1·input + b1).

[0099] Explanation: The role of the fully connected layer (FC1) is to linearly combine and reduce the dimension of high-dimensional input features, extracting more abstract and higher-level interaction patterns.

[0100] We can understand the gating network in Figure 5 as a miniature "decision-making brain", and hidden_dim determines the "working memory capacity" of this brain when thinking.

[0101] Position in the structure:

[0102] Input layer: The dimension is 3d (because it is composed of v i ,t j ,v i ⊙t j ).

[0103] First fully connected layer (FC1): Map 3d input to hidden_dim dimensional space.

[0104] Second fully connected layer (FC2): Map hidden_dim dimensional intermediate features to 1-dimensional output.

[0105] Function:

[0106] Feature learning and compression: The role of the first fully connected layer (FC1) is to learn and extract the most refined information that can represent the current cross-modal interaction relationship from high-dimensional (3d) input features. The size of hidden_dim directly determines the expression ability of this intermediate representation.

[0107] Balance model capability and efficiency:

[0108] If hidden_dim is set too small (e.g. 8 or 16), the network's "working memory" is insufficient, and it may not be able to learn complex enough interaction patterns, leading to inaccurate gating decisions and affecting overall performance. This is called "underfitting".

[0109] If hidden_dim is set too large (e.g. 1024 or more), the network's "working memory" is very strong, although the learning ability is high, but will lead to two problems: one is the increase of computational complexity, which is contrary to the original design of "lightweight"; two is easy to pay too much attention to the noise in the training data rather than the general pattern, leading to "overfitting".

[0110] The key to "lightweight": In your invention of the lightweight gated routing network, hidden_dim is usually set to a relatively small but sufficient value (e.g. 64, 128 or 256), much smaller than the input dimension 3d (if d = 768, then 3d = 2304). This makes the module bring huge performance improvement with little increase in computational overhead.

[0111] The next ReLU activation function introduces nonlinearity, enabling the model to learn and fit more complex decision functions, which is the key to the powerful expression of neural networks.

[0112] Step three: second layer transformation and gate value generation

[0113] Operation: input the ReLU-activated intermediate features h1 into the second fully connected layer (FC2), whose weight matrix is W2 ∈ R hidden_dim×1 , and the bias is b2. FC2 maps the features to a one-dimensional scalar. Finally, the scalar (original output logits) is processed by the Sigmoid activation function to obtain the dynamic gating value g ij .

[0114] The calculation can be expressed as: g ij = σ(W2·h1+b2).

[0115] Explanation: The role of the FC2 layer is to summarize the learned high-level interaction patterns into a rough "importance score". The Sigmoid activation function plays the role of a "soft switch", which compresses the score into the range of 0 to 1. The output g ij is the dynamic gating value, which quantifies the relevance strength of the interaction between the image region v i and the text word t j . The closer the value is to 1, the more important the interaction is; the closer to 0, the less important or belongs to noise.

[0116] Step four: apply the gating value to the attention weight

[0117] Operation: fuse the calculated dynamic gating value g ij with the original attention weight A ij calculated by the standard attention mechanism, usually by element-wise multiplication, i.e. = Softmax(A ij * G ij ), to generate the final dynamic attention weight .

[0118] Explanation: This is the ultimate mission of the gating value. It serves as an adaptive weight coefficient to "modulate" the standard attention. For important interactions (gij ≈1), whose attention weight is reserved or even enhanced; for unimportant interactions (g ij ≈0), whose attention weight is significantly suppressed. This process realizes the adaptive selection of attention paths, enabling the model to dynamically focus computational resources on the most critical cross-modal semantic associations, thus achieving fine-grained and precise semantic alignment.

[0119] Further, the semantic alignment and vector fusion module comprises:

[0120] a local alignment unit for calculating text-guided image representation and image-guided text representation using the dynamic attention weight, and narrowing the pair of local features through a local alignment loss function;

[0121] a global alignment unit for aggregating the global image representation and text representation enhanced by dynamic attention, and constraining through a global alignment loss function;

[0122] a vector generation unit for concatenating, summing or processing through a fusion network on the embedded vector sequence after global alignment, to generate the unified multi-modal semantic vector.

[0123] Specifically, local alignment: using the generated dynamic attention weight, calculate the text-guided image representation and image-guided text representation. Through a local alignment loss function (such as a similarity-based alignment loss), explicitly narrow the pair of local features (such as the word "golden hair" and the vector of the golden retriever region in the image).

[0124] Global alignment: aggregate the global image representation and text representation enhanced by dynamic attention layer, and constrain through a global alignment loss function (such as a contrastive learning loss).

[0125] Further, it also includes a training module connected with the semantic alignment and vector fusion module, for joint training in an end-to-end multi-task learning manner, so that the dynamic gating unit can allocate the most effective attention resources for different inputs; wherein the total loss function is set as the weighted sum of local alignment loss, global alignment loss and task loss.

[0126] The model is trained in a multi-task learning manner, and the total loss function is: L_total = λ1 * L_global_align + λ2 * L_local_align + λ3 * L_task; wherein, L_local_align represents the local alignment loss, L_global_align represents the global alignment loss, L_task represents the task loss, λ1, λ2, λ3 represent the corresponding weight coefficients respectively;

[0127] Through joint optimization, the dynamic attention gate module learns how to allocate the most effective attention resources for different inputs.

[0128] Further, the vector database in the duplicate detection execution module is established with efficient index to support efficient duplicate detection based on multi-modal semantic vectors.

[0129] When there is media content to be detected, the multi-modal semantic vector of the media content is calculated and compared with the cosine similarity of the multi-modal semantic vector in the vector database.

[0130] If the similarity exceeds the preset threshold, the media content is determined to be duplicate content or highly similar content, and an alarm and a similar content link are output.

[0131] If the similarity does not exceed the preset threshold, the media content is marked as new content, the multi-modal semantic vector of the media content is calculated and added to the vector database.

[0132] Embodiment 2

[0133] Please refer to Figure 2 , the embodiment based on the dynamic attention multi-modal semantic alignment and vector duplicate detection method provided in embodiment 1 further includes the following steps:

[0134] Step 1: multi-modal feature extraction and serialization; wherein image features, text features and video features are extracted from the media content, and each feature is converted into an embedded vector sequence.

[0135] Step 2: dynamic cross-modal attention calculation; wherein based on the dynamic attention gate module, the dynamic attention weight of the cross-modal interaction pair is calculated to adaptively enhance the key interaction and suppress the non-key interaction.

[0136] Step 3: multi-level semantic alignment and vector fusion; wherein the dynamic attention weight is used for local and global semantic alignment to generate a unified multi-modal semantic vector.

[0137] Step 4: efficient duplicate detection based on multi-modal semantic vectors; wherein the multi-modal semantic vector is stored in the vector database, and the content duplication is determined by calculating the cosine similarity.

[0138] It should be noted that: the application takes the dynamic attention mechanism as a bridge to connect the multi-modal feature input and the deep semantic understanding; a complete and closed-loop technical path from "original features" to "standardized attention", then to "dynamically modulated" attention through the gating network, and finally to the generation of "unified semantic vector" driven by multi-task loss is shown. This process effectively solves the core problems of rough duplication detection granularity, weak cross-modal ability and shallow semantic understanding mentioned in the background art.

[0139] Specifically, as shown in the figure, Figure 3 The multi-modal semantic vector is generated by the following steps:

[0140] Obtain image features and text features, and determine cross-modal interaction pairs based on embedding vector sequences of the features;

[0141] Calculate the standard attention weight of the cross-modal interaction pairs, and calculate the dynamic gating value of each pair of cross-modal interaction pairs through a lightweight gating routing network;

[0142] Fuse the standard attention weight and the dynamic gating value to obtain the dynamic attention weight;

[0143] Local fine-grained alignment: calculate the text-guided image representation and the image-guided text representation using the dynamic attention weight, and use the local alignment loss function to narrow the local features of the pair to obtain the local alignment loss and provide a fine-grained supervision signal;

[0144] Global fine-grained alignment: aggregate the global image representation and the text representation enhanced by the dynamic attention, and constrain them through the global alignment loss function to obtain the global alignment loss; wherein the global fine-grained alignment process is supervised by the fine-grained supervision signal;

[0145] Loss fusion: weight the sum of the global alignment loss and the local alignment loss, and perform training convergence judgment; wherein if convergence is reached, the corresponding multi-modal semantic vector is output; otherwise, the generation step of the multi-modal semantic vector is re-executed.

[0146] As shown in the figure, Figure 4 The standard attention weight is calculated by the following steps:

[0147] Take the image features and the text features as inputs, and obtain the query matrix Q, the key matrix K and the value matrix V through linear transformation; wherein, Figure 4 The parameter X represents the input feature, and W_Q, W_K and W_V represent the corresponding linear transformation templates respectively;

[0148] The attention score matrix is obtained by multiplying the transpose of the query matrix Q and the key matrix K;

[0149] scaling the attention score matrix; wherein, Scaled Scores = Scores / √(d_k), (d_k) is the dimension of the key vector, Scaled Scores is the scaled score, Scores is the attention score matrix, and √ represents the square root;

[0150] performing Softmax normalization on the scaled attention score matrix to obtain the original attention weight;

[0151] performing weighted summation on the value matrix V based on the original attention weight, and taking the weighted result as the standard attention weight.

[0152] It should be noted that: Figure 4 The standard attention calculation shown is one of the cornerstones of the present application; it provides a comprehensive, but unfiltered, cross-modal interaction view. However, its disadvantage is that it can also assign a certain weight to some irrelevant or secondary interactions. This is precisely the reason for the introduction of the dynamic gating routing network shown. Figure 5

[0153] As Figure 3 shown in the flow, Figure 4 the output (standard attention weight) will be fused with the output (dynamic gating value) of Figure 5 . Through this design, the model realizes "precise focusing" on the basis of the "broad-spectrum" correlation provided by the standard attention, thereby achieving more intelligent and efficient semantic alignment.

[0154] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the contents of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.​

Claims

1. A system based on dynamic attention multimodal semantic alignment and vector deduplication, used for efficient deduplication of query content, wherein the query content is media content, characterized in that, include: The feature extraction and serialization module is used to extract image features, text features, and video features from media content and convert each feature into an embedded vector sequence; wherein the media content includes at least one of image data, text data, and video data; A dynamic attention calculation module, connected to the feature extraction and serialization module, is used to calculate the dynamic attention weights of cross-modal interaction pairs based on the dynamic attention gating module, so as to adaptively enhance key interactions and suppress non-key interactions; wherein, the cross-modal interaction pairs are determined based on the matching of the embedding vector sequences of each feature; The semantic alignment and vector fusion module is connected to the dynamic attention calculation module and is used to perform local and global semantic alignment using the dynamic attention weights and generate a unified multimodal semantic vector. The plagiarism detection module, connected to the semantic alignment and vector fusion module, includes a vector database and a similarity calculation unit; wherein, the vector database is used to store the multimodal semantic vectors, and the similarity calculation unit is used to determine content duplication by calculating cosine similarity; The dynamic attention calculation module includes: Standard attention computation unit, used to compute standard attention weights for cross-modal interaction pairs; A dynamic gating unit is used to calculate the dynamic gating value for each pair of cross-modal interaction pairs through a lightweight gating routing network; A fusion unit is used to fuse the standard attention weights with the dynamic gating value to obtain the dynamic attention weights; The lightweight gated routing network in the dynamic gating unit includes a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function connected in sequence. The lightweight gated routing network passes the input features through a first fully connected layer and a ReLU activation function to obtain intermediate features, and then passes them through a second fully connected layer and a Sigmoid activation function to output dynamic gate values. The input consists of contextual features of cross-modal interaction pairs, and the output is a scalar gating value between 0 and 1. The calculated dynamic gating value g ij The original attention weights A calculated using the standard attention mechanism ij To achieve fusion, element-wise multiplication is used. =Softmax(A ij * G ij This generates the final dynamic attention weights. .

2. The system for dynamic attention-based multimodal semantic alignment and vector deduplication according to claim 1, characterized in that, The feature extraction and serialization module includes: An image coding unit is configured to use a CNN or ViT model to convert image features into a sequence of image patch embedding vectors. The text encoding unit is configured to use the BERT model to convert text features into a sequence of word embedding vectors. The video processing unit is configured to extract video keyframes using the FFMPEG tool and convert the keyframe features into a sequence of image patch embedding vectors using a CNN or ViT model.

3. The system for dynamic attention-based multimodal semantic alignment and vector deduplication according to claim 1, characterized in that, The semantic alignment and vector fusion module includes: The local alignment unit is used to calculate the text-guided image representation and the image-guided text representation using the dynamic attention weights, and to bring pairs of local features closer together using the local alignment loss function; The global alignment unit is used to aggregate the global image representation and text representation after dynamic attention enhancement, and is constrained by the global alignment loss function; The vector generation unit is used to concatenate, sum, or process the globally aligned embedded vector sequence through a fusion network to generate the unified multimodal semantic vector.

4. The system for dynamic attention-based multimodal semantic alignment and vector deduplication according to claim 3, characterized in that, It also includes a training module connected to the semantic alignment and vector fusion module for joint training in an end-to-end multi-task learning manner, so that the dynamic gating unit can allocate the most effective attention resources to different inputs; wherein, the total loss function is set as a weighted sum of local alignment loss and global alignment loss.

5. The system for dynamic attention-based multimodal semantic alignment and vector deduplication according to claim 1, characterized in that, The deduplication execution module has a vector database with efficient indexes to support efficient deduplication based on multimodal semantic vectors; among which, When there is media content to be checked for plagiarism, the multimodal semantic vector of the media content is calculated and compared with the cosine similarity of the multimodal semantic vectors in the vector database. If the similarity exceeds a preset threshold, the media content is determined to be duplicate content or highly similar content, and an alarm and a link to similar content are output. If the similarity does not exceed a preset threshold, the media content is marked as new content, the multimodal semantic vector of the media content is calculated, and it is added to the vector database.

6. A method for dynamic attention-based multimodal semantic alignment and vector deduplication, applied to a system for dynamic attention-based multimodal semantic alignment and vector deduplication as described in any one of claims 1 to 5, characterized in that, Includes the following steps: Step 1: Perform multimodal feature extraction and serialization; this involves extracting image features, text features, and video features from media content, and converting each feature into an embedding vector sequence. Step 2: Perform dynamic cross-modal attention calculation; wherein, based on the dynamic attention gating module, calculate the dynamic attention weights of cross-modal interaction pairs to adaptively enhance key interactions and suppress non-key interactions; Step 3: Perform multi-level semantic alignment and vector fusion; wherein, dynamic attention weights are used to perform local and global semantic alignment to generate a unified multimodal semantic vector; Step 4: Perform efficient deduplication based on multimodal semantic vectors; wherein, the multimodal semantic vectors are stored in a vector database, and content duplication is determined by calculating cosine similarity.

7. The method for multimodal semantic alignment and vector deduplication based on dynamic attention as described in claim 6, characterized in that, The multimodal semantic vector is generated through the following steps: Image and text features are acquired, and cross-modal interaction pairs are determined based on the sequence matching of the embedding vectors of each feature. Calculate the standard attention weights for cross-modal interaction pairs, and compute the dynamic gating value for each cross-modal interaction pair via a lightweight gated routing network; The standard attention weights are fused with the dynamic gating values ​​to obtain the dynamic attention weights; Perform local fine-grained alignment: Calculate text-guided image representation and image-guided text representation using dynamic attention weights, and use the local alignment loss function to bring pairs of local features closer together to obtain the local alignment loss and provide fine-grained supervision signals; Perform global fine-grained alignment: Aggregate the global image representation and text representation after dynamic attention enhancement, and constrain them through the global alignment loss function to obtain the global alignment loss; wherein, the global fine-grained alignment process is supervised by a fine-grained supervision signal. Loss fusion is performed: the global alignment loss and the local alignment loss are weighted and summed, and the training convergence is judged; if convergence is achieved, the corresponding multimodal semantic vector is output; otherwise, the multimodal semantic vector generation step is repeated.

8. The method for multimodal semantic alignment and vector deduplication based on dynamic attention as described in claim 7, characterized in that, The standard attention weights are calculated through the following steps: Using image and text features as input, a linear transformation is performed to obtain the query matrix Q, the key matrix K, and the value matrix V. The attention score matrix is ​​obtained by multiplying the query matrix Q by the transpose of the key matrix K; The attention score matrix is ​​scaled; where Scaled Scores = Scores / √(d_k), (d_k) is the dimension of the key vector, Scaled Scores is the scaling score, Scores is the attention score matrix, and √ represents the square root. The scaled attention score matrix is ​​subjected to Softmax normalization to obtain the original attention weights. The value matrix V is weighted and summed based on the original attention weights, and the weighted result is used as the standard attention weights.

Images