Video comment and picture accurate matching method and system driven by multi-modal large model

Abstract

The invention discloses a multi-modal large model driven video comment and picture accurate matching method and system, and the method comprises the steps: S1, respectively sending a video and a comment into a convolutional neural network and a recurrent neural network, and extracting a video key frame picture and a comment text keyword; s2, sending the video key frame picture into a visual coding module for processing, extracting visual features from the video frame picture, and converting original visual data into a high-dimensional feature vector; s3, sending the comment text keywords into a text coding module for processing, extracting semantic features of the text, and converting the semantic features into semantic feature vectors compatible with the visual feature vectors; s4, performing weighted fusion on the high-dimensional feature vector and the semantic feature vector through a gating mechanism; s5, the fused feature vectors are sent to a Transform encoder, and deep feature extraction and conversion are carried out; s6, fusing the converted feature vector and the high-dimensional feature vector again; and S7, sending an output result of the second fusion into an evaluator, performing matching degree scoring in the evaluator, and outputting a matching degree matrix. According to the invention, an innovative video comment and picture matching identification technology is realized, and the matching degree between the specific comment and the key frame in the video can be accurately identified.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and computer vision, and in particular to a method and system for accurately matching video comments and pictures driven by a multimodal large model. Background Art

[0002] With the explosive growth of digital media and social platforms, user-generated content has become one of the most important sources of information on the Internet. Especially on video sharing platforms, users not only upload video content, but also leave a large number of comments below the video. These comments often contain reactions, explanations or supplementary information to the video content, which is of great significance for understanding the popularity of the video, the emotional tendencies of users, and the potential value of the video.

[0003] Since it was first introduced by the Vaswani team in 2017, the Transformer model has become a core deep learning framework in natural language processing and sequence transformation tasks. The core innovation of this model is that its self-attention mechanism greatly enhances the ability to process sequence data.

[0004] The OpenAI team proposed in CLIP (Contrastive Language-Image Pre-Training) to build a large-scale image-text pairing dataset to train a model that can map images and text to a shared embedding space, and implement it through this dataset. In this space, the model maximizes the similarity between matching image-text pairs and minimizes the similarity between unmatched pairs by contrastive learning. The CLIP model consists of two parts: an image encoder and a text encoder. The image encoder can be a convolutional neural network (such as ResNet) or a Transformer model (such as VIT), while the text encoder is usually a Transformer model. In this way, the CLIP model can achieve cross-modal semantic understanding, that is, it can perform zero-shot learning on a variety of tasks, such as image classification, image-text matching, etc., without additional training data. The innovation of this method is that it directly uses natural language descriptions as regulatory signals without relying on category labels in traditional regulatory learning methods, thereby making the model more widely applicable and transferable.

[0005] At present, the technical challenges of video comment and picture matching mainly come from the following aspects: (1) Video pictures and text comments are two completely different data modalities, which contain visual and language information respectively. How to effectively extract features from these two modalities and compare and match them in a unified feature space is the key to achieving video comment and picture matching. (2) Video content and comments usually contain rich contextual information, which is crucial for understanding the relevance of comments and pictures. However, existing methods often lack in-depth understanding of contextual information, resulting in limited accuracy of matching results. (3) It is also necessary to consider that due to the diversity of video content and comment styles, the model needs to have good generalization ability, be able to adapt to different video content and comment styles, and handle various noises and abnormal situations. (4) With the development of globalization, the model needs to be able to handle comments in multiple languages, which poses challenges to language processing and cross-language feature extraction. Summary of the invention

[0006] The present invention provides a method and system for accurately matching video comments and pictures driven by a multimodal large model, which can accurately identify the matching degree between specific comments and key frames in the video, and identify the semantic association between key frames in the video content and related comments through deep learning and natural language processing technology, thereby realizing rapid inspection and matching of key focus parts of the video.

[0007] In order to achieve the above objectives, the present invention provides the following technical solutions:

[0008] The present invention provides a multi-modal large model driven video comment and picture accurate matching method, comprising:

[0009] S1, send the video and comments to the convolutional neural network and recurrent neural network respectively, extract the video key frame images and comment text keywords;

[0010] S2, sending the video key frame image obtained in step S1 to the visual encoding module for processing, extracting visual features from the video frame image, and converting the original visual data into a high-dimensional feature vector;

[0011] S3, sending the review text keywords obtained in step S1 to the text encoding module for processing, extracting the semantic features of the text, and converting them into a semantic feature vector compatible with the visual feature vector;

[0012] S4, performing weighted fusion on the high-dimensional feature vector obtained in step S2 and the semantic feature vector obtained in step S3 through a gating mechanism to obtain a fused feature vector;

[0013] S5, sending the fused feature vector obtained in step S4 to the Transformer encoder for deep feature extraction and conversion to obtain a converted feature vector;

[0014] S6, fusing the converted feature vector obtained in step S5 and the high-dimensional feature vector obtained in step S2 again;

[0015] S7. Send the output result of the second fusion to the evaluator, perform matching score in the evaluator, and output the matching matrix.

[0016] In a preferred solution, in step S2, the visual encoding module uses VIT (Vison-Transformer) to extract visual features from video key frames and convert the original visual data into a high-dimensional feature vector for subsequent semantic matching and analysis.

[0017] Specifically, in the visual encoding module VIT, the feature vector E is defined patch =W patch ·P+b patch , and add location information E to each patch pos =PE(pos), add a class token to capture global information E cls =W cls ·C+b cls ;

[0018] Among them, W patch and b patch are learnable weights and biases; P is the image patch; PE is the position encoding function; pos is the position index of the patch; C is the class token; W cls and b cls are learnable weights and biases;

[0019] Extract visual features V = {v1, v2, ..., v n}, where v i ∈H image Represents a high-dimensional feature vector.

[0020] In a preferred solution, in step S3, the text encoding module uses BERT (Bidirectional-Encoder Representations from Transformers) to process the comment text, extract the semantic features of the text, and convert it into a semantic feature vector compatible with the visual feature vector to achieve cross-modal feature matching.

[0021] Specifically, in the text encoder Bert, define E token =W token ·T k +b token , E seg =W seg·S+b seg , extract the semantic features of the text T = {t1,t,...,t n};

[0022] Among them, E token is the embedding vector; E seg Add embeddings to different text segments; k is the token sequence; S is the text segment identifier; W,b are learnable weights and biases, t i ∈H text Represents a semantic feature vector that is compatible with the visual feature vector.

[0023] In a preferred solution, in step S4, after the visual encoding module and the text encoding module, a gated aggregation module is introduced, which performs weighted fusion of visual and text features through a gating mechanism, so that the model can dynamically adjust the fusion strategy according to the importance of the features, thereby enhancing the flexibility and adaptability of feature representation;

[0024] In the gated aggregation module, the first layer is specified as a concatenation operation, and for each piece of text the concatenation operation is shown as:

[0025] in Indicates the connection of feature vectors. Concat and || are both connection functions;

[0026] The concatenated features are then projected into a unified embedding space through a linear mapping layer and the ELU() function to obtain complementary embeddings, namely:

[0027] The fusion adjusts the contribution of the feature vector through the gating mechanism to generate the gating graph G i : Finally get the fused embedding from the gated aggregator

[0028] Where σ represents the Sigmoid activation function; GA represents element-wise multiplication.

[0029] In the preferred solution, in step S5, the Transformer encoder uses the encoding part of the Transformer architecture, replaces the traditional self-attention mechanism with the orthogonal random feature method (FAVOR+), and combines it with the feedforward neural network to further extract and transform the fused feature vector in depth, providing a richer and more abstract feature representation for the final opinion expression recognition;

[0030] The regular bidirectional dot product note has the following form:

[0031] D=diag(A1 L ),

[0032] Where Q is the query matrix; K is the key matrix; V is the value matrix; A is the attention weight matrix; d is the dimension of key K, which is used to scale the dot product to prevent gradient disappearance; D is the diagonal matrix; X is the input feature matrix, T is the transpose, and diag is the diagonal matrix function;

[0033] The FAVOR+ mechanism is used to find a generalized kernel to map Q / K to Q' / K', so that the attention weight matrix A is approximately the product of Q' and K':

[0034] The generalized form of the mapping function φ(x) is:

[0035]

[0036] The effective approximate attention is expressed as:

[0037]

[0038] Where h(x) represents the transformation function of x; m is the number of functions used in the mapping function; and is the weight vector of the mapping function; g1, g k is the activation function; is the diagonal matrix in the approximate attention mechanism; 1 L represents a vector of all 1s of length L.

[0039] By modifying the multi-head attention mechanism of the Transformer encoder to FAVOR+, an orthogonal random feature method, it can effectively capture long-range dependencies. The Transformer encoding-only architecture is adopted, and the multi-head FAVOR+ mechanism is used as the model backbone; it approximates the attention weights through random projection, thereby reducing the computational complexity while maintaining the richness and accuracy of feature representation.

[0040] In a preferred solution, in step S5, the Transformer encoder includes N=6 identical layers, each layer including a sublayer of a multi-head FAVOR+ mechanism and a sublayer of a positional fully connected feedforward network, and residual connections and layer normalization are used around each sublayer.

[0041] In a preferred solution, in step S7, the evaluator includes a feature normalization layer, a cosine similarity layer, and a sigmoid layer;

[0042] The feature normalization layer is responsible for normalizing the feature vectors, the cosine similarity layer calculates the similarity between the normalized feature vectors, and the Sigmoid layer converts these similarities into probability outputs to provide a basis for the final classification decision.

[0043] Specifically, the feature normalization layer first processes the feature vectors output from the Transformer encoder module to ensure that the norm of each feature vector is unit length; this step is crucial because it eliminates the potential bias between different feature vectors due to different dimensions, making the subsequent similarity calculation more fair and accurate;

[0044] Next, the cosine similarity layer calculates the similarity between the normalized feature vectors. Cosine similarity is a widely used metric that evaluates the similarity of two vectors by calculating the cosine of the angle between them. This method is particularly effective for data in high-dimensional space because it focuses on the direction of the vector rather than its size, thus more accurately reflecting the semantic relationship between features.

[0045] Finally, the Sigmoid layer converts the cosine similarity into a probability output; the Sigmoid function is a nonlinear activation function that maps the input value to between 0 and 1, which can be interpreted as the probability of existing in a certain category; this probability value provides a direct basis for the final classification decision, allowing the model to output its prediction results in the form of probability;

[0046] Through the collaborative work of these three levels, the evaluator module is able to convert a high-dimensional feature vector that combines visual and textual information into an explicit probabilistic output, thus providing an accurate, efficient and interpretable solution for the classification task of video comment and picture matching.

[0047] The present invention also provides a multimodal large model driven video comment and picture accurate matching system, including a preprocessing module, a visual encoding module, a text encoding module, a gated aggregation module, a Transformer encoder module and an evaluator module.

[0048] Furthermore, the evaluator module includes a feature normalization layer, a cosine similarity layer, and a sigmoid layer, which work together to ensure the accuracy and reliability of the classification process. The use of multiple modules can better calculate the image-text matching degree.

[0049] The present invention extracts video key frame images and text keywords through convolutional neural networks and recurrent neural networks, then extracts visual features and semantic features through visual encoders and text encoders, and preliminarily fuses these features through a gated aggregation module, and then further extracts dependent features through a multi-layer Transformer encoder. After being fused with image embedding again, the matching degree is calculated through feature normalization, cosine similarity calculation and Sigmoid activated evaluator module, thereby realizing accurate pairing recognition between image and text.

[0050] The present invention proposes an innovative image-text matching technology, which improves the efficiency and accuracy of video content analysis while providing a powerful tool for the precise dissemination of Chinese culture. By accurately matching video images with text comments, it is possible to better understand the audience's response, optimize content creation, and achieve personalized recommendations, thereby promoting the unique charm and profound connotation of Chinese culture on a global scale.

[0051] The present invention has the following characteristics:

[0052] 1. The network structure of the present invention adopts a two-stage fusion strategy. First, the image encoding and text encoding are first fused in the gated aggregation module. This step enables the model to capture the initial correlation between cross-modalities. Subsequently, the fused feature representation is sent to the Transformer encoding layer for deep feature extraction and processing, which enhances the semantic richness and expression ability of the features.

[0053] 2. Through the above-mentioned two-stage feature fusion and deep-level feature extraction, the network structure of the present invention can perform more accurate classification prediction. In particular, when processing complex multimodal data, this structure can significantly improve the accuracy of classification.

[0054] 3. The network structure of the present invention is flexible in design and can adapt to different types and lengths of pictures and text data. This adaptability makes the present invention widely applicable to various multimodal classification tasks.

[0055] These characteristics and advantages make the present invention have significant competitiveness and application potential in the fields of multimodal data classification, sentiment analysis, content recommendation, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a module diagram of a multi-modal large model driven video comment and picture accurate matching method of the present invention;

[0057] Figure 2 This is a process diagram of a multi-modal large model driven video comment and picture accurate matching method of the present invention;

[0058] Figure 3This is an example result diagram of the present invention;

[0059] Figure 4 It is a schematic diagram of the model of the present invention. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0061] A multi-modal large model driven video comment and picture accurate matching method, comprising:

[0062] Videos and comments are fed into convolutional neural networks and recurrent neural networks respectively to extract video frame images and text keywords;

[0063] Images and text are processed by visual and text encoders. The visual encoding module uses VIT to extract visual features from video key frames and converts raw visual data into image embeddings for subsequent semantic matching and analysis; the text encoding module uses BERT to process comment text, extract semantic features of text, and convert them into text embeddings compatible with visual feature vectors to achieve cross-modal feature matching;

[0064] The feature vectors of the two modalities extracted from the visual encoding module and the text encoding module are sent to the gated aggregation module; here, the feature vector of each modality is adjusted by a gating map to adjust its contribution in the fusion process. The size of the gating map is the same as the feature map, which is generated by the convolution operation of the learnable weights and the feature map, and is normalized by the Sigmoid function to generate a fused feature vector that integrates visual and text information;

[0065] After successfully fusing the features of the two modalities, they are fed into an N-layer Transformer encoder. The encoder’s output feature vector and the image embedding are then fused again through a gated aggregation module to obtain more features.

[0066] The output of the second gated aggregation module is fed into the evaluator to obtain the matching matrix.

[0067] Example 1

[0068] like Figure 1 As shown, a multimodal large model-driven video comment and picture accurate matching method of the present invention mainly integrates the following six core components. This model significantly improves the semantic understanding and matching accuracy of multimodal data, and achieves more accurate cross-modal information processing and opinion expression recognition by deeply integrating the feature representation of video content and text comments.

[0069] Preprocessing module: This module extracts video key frame images and text keywords through convolutional neural networks and recurrent neural networks respectively.

[0070] Visual encoding module: This module first divides the video keyframe image into a series of patches, and then converts each patch into a vector representation as an input sequence. In this way, the visual encoding module converts the original visual data into a high-dimensional feature vector, providing rich visual information for subsequent semantic matching and analysis. These feature vectors not only contain the local details of the image, but also capture the global context information through the self-attention mechanism, allowing the model to understand and represent the image content more comprehensively.

[0071] Text encoding module: This module uses the BERT pre-trained language model, which is a deep learning model based on the Transformer architecture and is specifically designed for natural language processing tasks. The text is encoded through the BERT model, and then the text encoding module further processes the semantic representation output by BERT to generate a semantic feature vector that is compatible with the visual feature vector.

[0072] Gated aggregation module: This module is a key component used to integrate visual and text features in the present invention. After the visual encoding module and the text encoding module, the present invention introduces this module, which performs weighted fusion of visual features from video keyframe images and semantic features of comment text through a gating mechanism. This gating mechanism allows the model to dynamically adjust the weights of different modal features, thereby flexibly optimizing the feature fusion strategy according to the importance of the features and the contextual environment. The design of the gated aggregation module takes into account the complementarity and mutual relationship between features, and enhances the model's understanding and representation of the complex relationship between video content and user comments by intelligently selecting and emphasizing relevant information. The introduction of the gated aggregation module not only improves the efficiency and accuracy of feature fusion, but also enhances the model's ability to understand contextual information. This design enables the model to better capture the subtle connections between video content and comments, thereby understanding and identifying users' opinion expressions at a deep level. In addition, the gated aggregation module also helps to improve the adaptability of the model, enabling it to handle video content and comments of different styles and fields, and improves the applicability and generalization ability of the present invention.

[0073] Transformer encoder module: This module adopts the encoding part of the Transformer architecture, replaces the traditional self-attention mechanism with FAVOR+, and combines it with a feedforward neural network to perform further deep feature extraction and transformation on the fused feature vector. FAVOR+ is an efficient attention mechanism that approximates the attention weights through random projections to reduce computational complexity while maintaining the richness and accuracy of feature representation. This approach effectively reduces the variance of the estimate and improves the average precision of a single estimate by introducing orthogonal random features. While maintaining linear spatial and temporal complexity, this method also ensures good accuracy and can be applied to independent softmax operations. This provides a richer and more abstract feature representation for the final opinion expression recognition, and enhances the model's ability to understand and recognize the complex relationship between video content and user comments.

[0074] Evaluator module: This module consists of three main parts including feature normalization layer, cosine similarity layer and sigmoid layer, which work together to ensure the accuracy and reliability of the classification process.

[0075] The feature normalization layer first processes the feature vectors output from the Transformer encoder module to ensure that the norm of each feature vector is unit length. This step is crucial because it eliminates potential biases between different feature vectors due to different dimensions, making subsequent similarity calculations more fair and accurate.

[0076] Next, the cosine similarity layer calculates the similarity between the normalized feature vectors. Cosine similarity is a widely used metric that evaluates the similarity between two vectors by calculating the cosine of the angle between them. This method is particularly effective for data in high-dimensional space because it focuses on the direction of the vector rather than its magnitude, thus more accurately reflecting the semantic relationship between features.

[0077] Finally, the Sigmoid layer converts the cosine similarity into a probability output. The Sigmoid function is a nonlinear activation function that maps the input value to between 0 and 1, which can be interpreted as the probability of existing in a certain category. This probability value provides a direct basis for the final classification decision, allowing the model to output its prediction results in the form of probability.

[0078] Through the collaborative work of these three levels, the evaluator module is able to convert a high-dimensional feature vector that combines visual and textual information into an explicit probabilistic output, thus providing an accurate, efficient and interpretable solution for the classification task of video comment and picture matching.

[0079] Example 2

[0080] like Figure 2 As shown, a multimodal large model driven video comment and picture accurate matching method of the present invention includes 7 steps in total. The first step is to extract video key frame pictures and text keywords; the second step is to send the pictures to the visual encoding module, divide the input image into patches of fixed size, and then add position information to each patch to capture the spatial relationship; the third step is to send the text to the text encoder, extract the deep semantic features of the text, and convert it into a semantic feature vector compatible with the visual feature vector; the fourth step is to send the feature vectors of the two modes to the gated aggregation module to obtain a fused feature vector; the fifth step is to send the fused feature vector to the transformer encoder; the sixth step is to embed the picture and fuse it with the output features of the transformer encoder again through the gated aggregation module; the seventh step is to send it to the evaluator module to obtain the matching degree.

[0081] Step 1: Send the video content and text comments into the convolutional neural network and recurrent neural network respectively to extract video key frame images and text keywords;

[0082] Step 2: Feed the video content into the visual encoder; define E patch =W patch ·P+b patch , and add location information E to each patch pos =PE(pos), add a class token to capture global information E cls =W cls ·C+b cls ; Extract visual features V = {v1, v2, ..., v n}, where v i ∈H image Represents a high-dimensional feature vector. First, the input image is divided into patches of fixed size, and then position information is added to each patch to capture spatial relationships;

[0083] Step 3: Feed the text content into the text encoder. Definition E token =W token ·T+b token , add embedding E for different text segments seg =W seg ·S+b seg ;

[0084] Extract the semantic features of the text T = {t1, t, ..., t n}, where t i ∈H textRepresenting semantic feature vectors that are compatible with visual feature vectors to achieve cross-modal feature matching;

[0085] Step 4: The feature vectors of the two modalities extracted from the visual encoding module and the text encoding module are fed into the gated aggregation module;

[0086] 4.1: First, for each text t i Its feature vector and the image feature vector are initially fused through a concatenation operation:

[0087] 4.2: Concatenated feature vector Through a linear mapping layer, followed by nonlinear activation through the ELU function, to enhance the expressive power of the model:

[0088] 4.3: Adjust the contribution of the feature vector through the gating mechanism to generate the gating graph G i : Where σ is the Sigmoid function, which is used to normalize the gate value to between 0 and 1

[0089] 4.4: Finally, use the gated graph G i The activated feature vector Perform weighted fusion embedding:

[0090] Step 5: This fused embedding is further fed into the Transformer encoder, using the Transformer encoding-only architecture and the multi-head FAVOR+ mechanism as the model backbone; the encoder consists of N = 6 identical layers, each of which includes multiple sub-layers of the FAVOR+ mechanism and sub-layers of the positional fully connected feed-forward network, using residual connections and layer normalization around each sub-layer;

[0091] The regular bidirectional dot product note has the following form:

[0092] D=diag(A1 L ),

[0093] Where Q = W Q X,K=W K X,V=W V X;

[0094] The FAVOR+ mechanism is used to find a generalized kernel to map Q / K to Q' / K', so that the attention weight matrix A is approximately the product of Q' and K':

[0095] The generalized form of the mapping function is:

[0096]

[0097] The effective approximate attention is expressed as:

[0098]

[0099] Step 6: The image embedding from the visual encoding module and the output feature vector of the Transformer encoder are fed back into the gated aggregation module;

[0100] The feature vector E output by the Transformer encoder transformer and image embed v i As input, they are represented as and

[0101] Where N is the number of features output by Transformer, M is the number of features embedded in the image, and d is the dimension of the feature;

[0102] For each feature vector, perform a concatenation operation C = [E transformer ;v i ], the dimension is (N+M)×d, where C is the feature function after connection;

[0103] The concatenated feature vector is projected into a unified embedding space A = ELU (W·C+b) through a linear mapping layer, where W and b are learnable weights and biases, and A is the activated feature vector;

[0104] The final fused feature vector E after the gating layer fused Denoted as σ ( W g ·A+b g )⊙A, where σ is the Sigmoid function, W g and b g are the learnable weights and biases of the gated layer, ⊙ denotes element-wise multiplication;

[0105] This step aims to further optimize the feature representation and ensure that the image information is fully reflected in the final feature vector;

[0106] Step 7: The feature vector after secondary fusion is sent to the evaluator module. Here, the feature normalization and unification of scale accelerates convergence and facilitates the calculation of cosine similarity, which is specifically expressed as:

[0107]

[0108] where ||E fused ||2 is E fusedThe L2 norm of ;

[0109] Calculate the normalized fusion feature vector E norm And the feature vector v of each frame i The cosine similarity between them is used to measure their similarity:

[0110]

[0111] For each text keyword, we calculate its corresponding text feature vector t i And the feature vector v of each frame i The cosine similarity between them gives a matching score E match .

[0112] A threshold value θ is set to determine the degree of match between the text keyword and the image. If the match score is higher than the threshold, the text keyword is considered to match the frame image.

[0113] For each text keyword, output the matching result with each frame of the picture to form a matching matrix M, where M i,j Indicates the matching status between the i-th text keyword and the j-th frame image.

[0114] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multi-modal large model driven video comment and picture accurate matching method, characterized in that: include: S1, send the video and comments to the convolutional neural network and recurrent neural network respectively, extract the video key frame images and comment text keywords; S2, sending the video key frame image obtained in step S1 to the visual encoding module for processing, extracting visual features from the video frame image, and converting the original visual data into a high-dimensional feature vector; S3, sending the review text keywords obtained in step S1 to the text encoding module for processing, extracting the semantic features of the text, and converting them into a semantic feature vector compatible with the visual feature vector; S4, performing weighted fusion on the high-dimensional feature vector obtained in step S2 and the semantic feature vector obtained in step S3 through a gating mechanism to obtain a fused feature vector; S5, sending the fused feature vector obtained in step S4 to the Transformer encoder for deep feature extraction and conversion to obtain a converted feature vector; S6, fusing the converted feature vector obtained in step S5 and the high-dimensional feature vector obtained in step S2 again; S7. Send the output result of the second fusion to the evaluator, perform matching score in the evaluator, and output the matching matrix.

2. According to claim 1, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: In step S2, the visual encoding module uses VIT to extract visual features from video key frames and converts the original visual data into high-dimensional feature vectors for subsequent semantic matching and analysis; In the visual encoding module VIT, the feature vector E is defined patch =W patch ·P+E patch , and add location information E to each patch pos =PE(pos), add a class token to capture global information E cls =W cls ·C+b cls ; Among them, W patch and b patch are learnable weights and biases; P is the image patch; PE is the position encoding function; pos is the position index of the patch; C is the class token; W cls and b cls are learnable weights and biases; Extract visual features V = {v1, v2, ..., v n }, where v i ∈H image Represents a high-dimensional feature vector.

3. According to claim 1, a multimodal large model driven video comment and picture accurate matching method is characterized in that: In step S3, the text encoding module uses BERT to process the comment text, extract the semantic features of the text, and convert it into a semantic feature vector compatible with the visual feature vector to achieve cross-modal feature matching; In the text encoder Bert, define E token =W token ·T k +b token , E seg =W seg ·S+b seg , extract the semantic features of the text T = {t1,t,...,t n }; Among them, E token is the embedding vector; E seg Add embeddings to different text segments; k is the token sequence; S is the text segment identifier; W,b are learnable weights and biases, t i ∈H text Represents a semantic feature vector that is compatible with the visual feature vector.

4. According to claim 1, a multimodal large model driven video comment and picture accurate matching method is characterized in that: In step S4, after the visual encoding module and the text encoding module, a gated aggregation module is introduced. The module performs weighted fusion of visual and text features through a gating mechanism, so that the model can dynamically adjust the fusion strategy according to the importance of the features, thereby enhancing the flexibility and adaptability of feature representation; In the gated aggregation module, the first layer is specified as a concatenation operation, and for each piece of text the concatenation operation is shown as: in Indicates the connection of feature vectors. Concat and || are both connection functions; The concatenated features are then projected into a unified embedding space through a linear mapping layer and the ELU() function to obtain complementary embeddings, namely: The fusion adjusts the contribution of the feature vector through the gating mechanism to generate the gating graph G i : Finally get the fused embedding from the gated aggregator Where σ represents the Sigmoid activation function; GA represents element-wise multiplication.

5. According to claim 1, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: In step S5, the Transformer encoder uses the encoding part of the Transformer architecture, replaces the traditional self-attention mechanism with the orthogonal random feature method (FAVOR+), and combines it with the feedforward neural network to further extract and transform the fused feature vector in depth, providing a richer and more abstract feature representation for the final opinion expression recognition; The regular bidirectional dot product note has the following form: Where Q is the query matrix; K is the key matrix; V is the value matrix; A is the attention weight matrix; d is the dimension of key K, which is used to scale the dot product to prevent gradient disappearance; D is the diagonal matrix; X is the input feature matrix, T is the transpose, and diag is the diagonal matrix function; The FAVOR+ mechanism is used to find a generalized kernel to map Q / K to Q' / K', so that the attention weight matrix A is approximately the product of Q' and K': The generalized form of the mapping function φ(x) is: The effective approximate attention is expressed as: Where h(x) represents the transformation function of x; m is the number of functions used in the mapping function; and is the weight vector of the mapping function; g1, g k is the activation function; is the diagonal matrix in the approximate attention mechanism; 1 L represents a vector of all 1s of length L.

6. According to claim 5, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: By modifying the multi-head attention mechanism of the Transformer encoder to FAVOR+, an orthogonal random feature method, it can effectively capture long-range dependencies. The Transformer encoding-only architecture is adopted, and the multi-head FAVOR+ mechanism is used as the model backbone; it approximates the attention weights through random projection, thereby reducing the computational complexity while maintaining the richness and accuracy of feature representation.

7. According to claim 5, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: In step S5, the Transformer encoder consists of N = 6 identical layers, each of which includes a sublayer of the multi-head FAVOR+ mechanism and a sublayer of the positional fully connected feed-forward network, with residual connections and layer normalization around each sublayer.

8. According to claim 1, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: In step S6, the image embedding from the visual encoding module and the output feature vector of the Transformer encoder are fed into the gated aggregation module again; The feature vector E output by the Transformer encoder transformer and image embed v i As input, they are represented as and Where N is the number of features output by Transformer, M is the number of features embedded in the image, and d is the dimension of the feature; For each feature vector, perform a concatenation operation C = [E transformer ;v i ], the dimension is (N+M)×d, where C is the feature function after connection; The concatenated feature vector is projected into a unified embedding space A = ELU (W·C+b) through a linear mapping layer, where W and b are learnable weights and biases, and A is the activated feature vector; The final fused feature vector E after the gating layer fused Expressed as σ(W g ·A+b g )⊙A, where σ is the Sigmoid function, W g and b g are the learnable weights and biases of the gated layer, ⊙ represents element-wise multiplication; This step aims to further optimize the feature representation and ensure that the image information is fully reflected in the final feature vector.

9. According to claim 1, a multi-modal large model driven video comment and picture accurate matching method is characterized in that: In step S7, the evaluator includes a feature normalization layer, a cosine similarity layer, and a sigmoid layer; The feature normalization layer is responsible for normalizing the feature vectors, the cosine similarity layer calculates the similarity between the normalized feature vectors, and the Sigmoid layer converts these similarities into probability outputs to provide a basis for the final classification decision.

10. A multi-modal large model driven video comment and picture accurate matching system, characterized in that: It includes a preprocessing module, a visual encoding module, a text encoding module, a gated aggregation module, a Transformer encoder module and an evaluator module.

Images