Coreference resolution and word sense disambiguation method and system based on joint modeling of vision and text

Abstract

The present invention belongs to the technical field of multimodal joint modeling for dialog intention processing. A method and system for coreference resolution and word sense disambiguation based on visual-text joint modeling is provided. Based on text enhancement features and video enhancement features, a cross-attention feature network is used to adjust features to obtain adjusted features. The adjusted features are coordinated using a distribution similarity metric, and coreference resolution prediction is performed on the coordinated features to obtain coreference resolution results corresponding to the input text. The present invention fuses text features and visual features through a cross-modal attention mechanism to generate an accurate joint representation, capture the deep semantic relationship between text and video content, and thus construct an efficient word sense disambiguation model, overcoming the problem of poor disambiguation performance of existing methods when facing complex and changeable natural language query statements from users.

Description

Technical Field

[0001] The present invention relates to the technical field of dialogue intention processing of multimodal joint modeling, and in particular to a coreference resolution and word sense disambiguation method and system based on visual-text joint modeling. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] Word reference and word sense disambiguation in natural language queries can be divided into two sub-problems, namely coreference resolution and word sense disambiguation. The former is mainly to identify different expressions referring to the same entity in the text, while the latter is used to determine the correct meaning of a word in a specific context.

[0004] Coreference resolution is a classic problem in the field of natural language processing. Early coreference resolution work was mainly based on statistical rules and handwriting principles. The process was complicated and the resolution effect depended entirely on the richness of the rules. In recent years, a large number of machine learning-based studies have been proposed, but the current research work is still in the preliminary verification stage. How to effectively align large language models and how to solve the coreference resolution task from a multimodal perspective still need further research.

[0005] Word sense disambiguation tasks can be divided into "text disambiguation" and "visual-text disambiguation" according to the different modalities used. For text disambiguation, word sense disambiguation in a given sentence is generally performed based on pure text modality, such as the Lesk algorithm that uses the overlap ratio between dictionary definitions. In recent years, deep learning methods have also been widely used in this field, but the effectiveness of existing deep learning methods is highly dependent on the quality of external databases; in addition, due to the ambiguity of language itself, defining ambiguous words based solely on text may itself bring new ambiguities. Therefore, some recent research works have turned to the visual-text disambiguation task, which is a research direction based on the joint modeling of visual information and text information to eliminate text ambiguity, but existing research works still lack the effective application of existing widely used pre-trained models, and ignore the complex interaction and fusion of features between modalities.

[0006] In summary, in real application scenarios, due to the complexity and ambiguity of natural language, words and phrases in query statements often have multiple meanings, which can easily lead to ambiguous and inaccurate query results. Therefore, disambiguation processing for natural language query statements is an important research topic to improve the accuracy of video positioning technology and user satisfaction. Although existing work has made progress in coreference resolution and word sense disambiguation of language queries, there are still some key challenges, as follows:

[0007] (1) In the real field of conversational robots, users' input sentences usually contain multiple multimodal references, while many previous works are based on unimodal coreference resolution. If we want to solve the multimodal coreference resolution task, we need to design cross-modal feature enhancement and fusion methods to resolve the problem of unclear reference in sentences and improve the model's ability to accurately understand pronouns; (2) Social development is changing with each passing day, which leads to the same word having multiple meanings in a short period of time. This requires modern ambiguity resolution models to effectively utilize the word features of multimodal external knowledge bases to enhance the model and improve the comprehensive representation ability of words that may contain ambiguity in query sentences. Summary of the invention

[0008] In order to address the deficiencies in the prior art, the present invention provides a coreference resolution and word sense disambiguation method and system based on visual-text joint modeling. Through a cross-modal attention mechanism, text features and visual features are fused to generate an accurate joint representation, capture the deep semantic relationship between the input text and the video content, thereby constructing an efficient word sense disambiguation model, overcoming the problem that the prior methods have poor disambiguation performance when faced with complex and changeable natural language query statements from users.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides a coreference resolution method based on visual-text joint modeling.

[0011] A coreference resolution method based on visual-text joint modeling includes the following steps:

[0012] Obtain input text and reference video, use a text encoder to convert the input text into a text embedding vector, and use a video encoder to convert the reference video into a video embedding vector;

[0013] A text semantic tree is constructed based on the text embedding vector, and a video semantic tree is constructed based on the video embedding vector. In the text semantic tree and the video semantic tree, graph convolution operations are performed to perform feature fusion enhancement based on the distance between nodes to obtain text enhancement features and video enhancement features.

[0014] Based on the text enhancement features and the video enhancement features, a cross-attention feature network is used to adjust the features to obtain the adjusted features, the adjusted features are coordinated, and coreference resolution prediction is performed on the coordinated features to obtain the coreference resolution result corresponding to the input text.

[0015] As a further limitation of the first aspect of the present invention, feature adjustment is performed using a cross-attention feature network, comprising:

[0016] The similarity between the text enhancement feature and the video enhancement feature is obtained, and the text enhancement feature and the video enhancement feature are regularized according to the similarity: , ,in, is a regularization parameter based on the similarity, and Respectively The text enhancement features and indivual text enhancement features, and Respectively video enhancement features and Video enhancement features, and They are respectively the regularized i text features and the regularized i Video features;

[0017] According to the regularized text features and the regularized video features, a cross-modal fusion network is used to obtain multimodal adjustment features: ,in, , is the scaling factor, , and are parameter matrices, is the normalized exponential function, and T stands for transpose.

[0018] As a further limitation of the first aspect of the present invention, coordinating the multimodal adjustment features includes:

[0019] Obtain annotations for coreference resolution from real reference information;

[0020] Use a text encoder and a video encoder to convert the acquired annotations into text embedding vectors and video embedding vectors respectively, and convert the text embedding vectors and video embedding vectors into multimodal annotation features;

[0021] Based on the multimodal adjustment features and multimodal annotation features, the distribution similarity is calculated and training is iteratively performed to make the adjustment features closer to the true distribution until the distribution similarity is greater than the set threshold or the iteration round is reached.

[0022] As a further limitation of the first aspect of the present invention, performing coreference resolution prediction on the coordinated features to obtain a coreference resolution result corresponding to the input text includes:

[0023] The coordinated features are converted into action confidence scores using a fully connected network. When the action confidence score is greater than a set threshold, it is determined that the content of the input text exists in the reference video.

[0024] In a second aspect, the present invention provides a word sense disambiguation method based on visual-text joint modeling.

[0025] A word sense disambiguation method based on visual-text joint modeling, which uses the coreference resolution method based on visual-text joint modeling described in the first aspect of the present invention to perform coreference resolution;

[0026] Obtain ambiguous words, and generate corresponding multi-ambiguous images based on the text graph model guided by external knowledge and guided by external knowledge updated in real time;

[0027] According to the multiple ambiguous images, the input text and the ambiguous vocabulary, a pre-trained ambiguous word feature enhancement model based on visual-language modeling is used to obtain an enhanced ambiguous vocabulary embedding expression.

[0028] As a further limitation of the second aspect of the present invention, a pre-trained ambiguous word feature enhancement model based on visual-language modeling is used to obtain an enhanced ambiguous word embedding expression, including:

[0029] Calculate the cross similarity of ambiguous images, ambiguous words and original text information to obtain a similarity matrix;

[0030] A fully connected ambiguous decision maker is applied to determine the real corresponding image corresponding to the ambiguous words based on the similarity matrix, and the embedding vector of the real corresponding image is fused with the embedding vector of the ambiguous words to obtain the enhanced embedding expression of the ambiguous words.

[0031] As a further limitation of the second aspect of the present invention, based on the external knowledge-guided text graph model, using the external knowledge guided by real-time updates, a corresponding multi-ambiguous image is generated, including:

[0032] Extracting multiple extended interpretations of the text embedding vector through an external knowledge base;

[0033] Obtaining ambiguous words according to the multiple extended interpretations, and aligning the ambiguous words with the corresponding multiple extended interpretations in a vector space to obtain an aligned feature vector;

[0034] Using the aligned feature vectors as a guide, corresponding images are sampled in the image space to construct multi-ambiguous images.

[0035] In a third aspect, the present invention provides a coreference resolution system based on visual-text joint modeling.

[0036] A coreference resolution system based on visual-text joint modeling includes the following processes:

[0037] The embedding vector generating unit is configured to: obtain input text and a reference video, convert the input text into a text embedding vector by using a text encoder, and convert the reference video into a video embedding vector by using a video encoder;

[0038] The feature enhancement unit is configured to: construct a text semantic tree according to the text embedding vector, construct a video semantic tree according to the video embedding vector, and perform feature fusion enhancement based on the distance between nodes in the text semantic tree and the video semantic tree through graph convolution operation to obtain text enhancement features and video enhancement features;

[0039] The coreference resolution unit is configured to: perform feature adjustment using a cross-attention feature network based on text enhancement features and video enhancement features to obtain adjusted features, coordinate the adjusted features using a distribution similarity metric, perform coreference resolution prediction on the coordinated features, and obtain a coreference resolution result corresponding to the input text.

[0040] As a further limitation of the third aspect of the present invention, in the coreference resolution unit, feature adjustment is performed using a cross-attention feature network, including:

[0041] The similarity between the text enhancement feature and the video enhancement feature is obtained, and the text enhancement feature and the video enhancement feature are regularized according to the similarity: , ,in, is a regularization parameter based on the similarity, and Respectively The text enhancement features and indivual text enhancement features, and Respectively video enhancement features and Video enhancement features, and They are respectively the regularized i The text features and the regularized i Video features;

[0042] According to the regularized text features and the regularized video features, a cross-modal fusion network is used to obtain multimodal adjustment features: ,in, , is the scaling factor, , and are parameter matrices, is the normalized exponential function, and T stands for transpose.

[0043] In a fourth aspect, the present invention provides a word sense disambiguation system based on visual-text joint modeling, which uses the coreference resolution method based on visual-text joint modeling described in the first aspect of the present invention to perform coreference resolution, including:

[0044] The multi-ambiguous image generation unit is configured to: obtain ambiguous words, and based on the text graph model guided by external knowledge, generate corresponding multi-ambiguous images by using external knowledge guidance updated in real time;

[0045] The word sense disambiguation unit is configured to: obtain an enhanced embedding expression of the ambiguous vocabulary by using a pre-trained ambiguous word feature enhancement model based on visual-language modeling according to the multiple ambiguous images, the input text and the ambiguous vocabulary.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The present invention takes into account multiple key factors, including the reference complexity of multimodal coreference resolution, the dynamic variability of polysemy of ambiguous words, and high-quality text representation, which greatly improves the accuracy of coreference resolution and word sense disambiguation.

[0048] 2. Faced with users' complex and changeable natural language query statements, the present invention designs a cross-modal attention mechanism to capture the deep semantic relationship between text and video content and improve the accuracy of coreference resolution.

[0049] 3. The ambiguous word feature enhancement model of the present invention is based on visual-language modeling, which fuses text features and visual features to generate an accurate joint representation, constructs an efficient word sense disambiguation model, and improves the accuracy of disambiguation.

[0050] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0052] Figure 1 A comprehensive schematic diagram of coreference resolution and word sense disambiguation provided by the present invention;

[0053] Figure 2 A schematic diagram of a coreference resolution method based on visual-text joint modeling provided by the present invention;

[0054] Figure 3 A schematic diagram of the word sense disambiguation method based on visual-text joint modeling provided by the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0057] like Figure 1 As shown in the figure, assuming that the input text is a query sentence, namely "Zhang San put the assessment next to the book and then took it away", the "it" has an unclear reference problem (i.e. took the "apple" or took the "book"), and the "apple" has a confusion in meaning (i.e. the "apple" in fruit or the "apple" in mobile phone). Therefore, according to the query sentence and the encoding of the reference video, coreference resolution and word sense disambiguation are performed respectively.

[0058] Specifically, this implementation method first proposes a coreference resolution method based on visual-text joint modeling, which includes the following processes:

[0059] S1.1: Input text T Encode to get the text embedding vector.

[0060] Enter text T The initial encoding process is to separate the independent words of the given input text and use the text encoder Convert to text embedding vector.

[0061] S1.2: Encode the reference video to obtain an initial encoding.

[0062] The initial encoding process of the reference video is as follows: a given video clip is cut into a sequence of video frames, adjacent video frames in the sequence are subtracted, and the video frames with larger difference are taken as key frames of independent scenes, and then the image encoder is used to encode the video clip into a sequence of video frames. The features of the key frames of these independent scenes are extracted to obtain the video embedding vectors.

[0063] S1.3: Based on the converted text embedding vector and video embedding vector, the cross-modal interactive graph convolution structure is used to perform feature fusion enhancement, including:

[0064] S1.3.1: Construct a text semantic tree by converting the converted text embedding vectors according to their order in the original sentence , for the text semantic tree Each node in the text integrates features based on the distance of the text in the original sentence to obtain text enhancement features. More specifically, in the text semantic tree Through graph convolution operations, adjacent, sub-adjacent and other node features in the semantic tree are integrated based on distance to obtain text enhancement features.

[0065] S1.3.2: Construct a video semantic tree by converting the video embedding vectors of the converted video independent scenes according to their time order in the reference video , for the video semantic tree Each scene node in is integrated into a video enhancement feature based on the distance of the scene in the reference video. More specifically, in the video semantic tree In the method, graph convolution operation is performed to integrate adjacent, sub-adjacent and other node features in the semantic tree based on distance to obtain video enhancement features.

[0066] S1.3.3: Based on the obtained video enhancement features and text enhancement features, use the cross-attention feature network to perform feature adjustment, specifically including:

[0067] The similarity between the text enhancement feature and the video enhancement feature is obtained. According to the similarity, regularization processing is performed based on the text enhancement feature and the video enhancement feature respectively (the video enhancement feature uses a fully connected network to correct the dimension and scene features before regularization), and the regularized text feature and the regularized video feature are obtained:

[0068] (1);

[0069] (2);

[0070] in, is a regularization parameter based on similarity, Enter the text i text enhancement features, Enter the text j text enhancement features, For reference video i Scene features, For reference video j Scene features, and is the regularized i The text features and i Video features.

[0071] Based on the obtained regularized features, a cross-modal fusion network is used to obtain multi-modal adjustment features. :

[0072] (3);

[0073] in, , is the scaling factor, , and All are parameter matrices, and T represents transpose.

[0074] S1.3.4: Use distribution similarity measures to adjust features for multimodality Coordinate to form features for predicting coreference resolution, including:

[0075] Get the annotations for coreference resolution from the real reference information (i.e., gound tuth annotations, which are used to optimize parameters during training);

[0076] Use the text encoder and image encoder to convert the acquired annotations into text annotation embedding vectors and video annotation embedding vectors respectively, reuse the process of S1.3.1-S1.3.3, and convert the text annotation embedding vectors and video annotation embedding vectors into multimodal annotation features ;

[0077] Based on multi-modal adjustment features and multimodal annotation features , calculate the distribution similarity, iterate the training, make the multimodal adjustment feature closer to the real distribution, until the distribution similarity is greater than the set threshold or reaches the iteration round, and obtain the final multimodal adjustment feature .

[0078] S1.4: Predict the coreference resolution of the coordinated features, including:

[0079] For a given input text , calculate the final multimodal adjustment features (i.e., coordinated features), the final multimodal adjusted features It includes both interactive multimodal features and contextual features carried by the graph convolutional network, which are output as action confidence scores through an MLP. , action confidence score Represents whether the model believes that the query sentence corresponding to the input text exists in the reference video. When it is greater than the set threshold, the input text is judged The corresponding query statement exists in the reference video, otherwise, the input text is determined The corresponding query sentence is not in the reference video.

[0080] Specifically, Figure 2The figure shows an illustration. Assuming a query sentence is input, i.e., "Zhang San took the evaluation", and a reference video is input at the same time, the video enhancement feature and text feature are obtained through the cross-modal interactive convolution structure (the acquisition of text features needs to be obtained through a fully connected layer), and then the multi-modal adjustment feature is obtained, and the output "action confidence score" is obtained by combining the context information; among them, the edge types in the figure include: blue is video-video, green is video-text, and orange is text-text.

[0081] This implementation also provides a word sense disambiguation method based on visual text joint modeling, based on the text graph model guided by external knowledge, according to the ambiguous vocabulary, using the external knowledge guidance updated in real time, to generate corresponding multiple ambiguous images to assist in enhancing vocabulary representation; obtain the generated multiple ambiguous images, input text and ambiguous vocabulary, and enhance the comprehensive understanding of ambiguous vocabulary through overall semantics. Specifically, it includes the following processes:

[0082] S2.1: Based on the input text , which is converted into a text embedding vector through a text encoder.

[0083] In a specific implementation, the initial text encoding process is: Separate the words into separate words and use the text encoder Convert it into a text embedding vector;

[0084] S2.2: Extract multiple extended interpretations of text embedding vectors through external knowledge base and embed the input text T Align with multiple extended interpretations in vector space to obtain aligned feature vectors;

[0085] In a specific implementation, a large model trained on a large amount of data is used as an external knowledge base, and a prompt learning method is used to allow the external knowledge base to predict the input text. T Extended interpretation of ambiguous words contained in :

[0086] (4);

[0087] in, is the original text, L is the length of the judgment statement of ambiguous words, including whether there are ambiguous words in the input text T and the extended interpretation of ambiguous words. It is the text content predicted by the external knowledge base. Each predicted text content is based on all the previous text contents. Get, are the model parameters of the external knowledge base.

[0088] S2.3: Using the aligned feature vector as a guide, sample corresponding picture information in the image space to form multiple ambiguous images; wherein the image space refers to the overall sampling space of images contained in the external knowledge base, and samples pictures corresponding to the aligned feature vectors in this space based on the aligned feature vectors.

[0089] In a specific implementation method, the extended interpretation is input, and the image sampling can be performed using a text-guided stable diffusion process. The specific process is:

[0090] S2.3.1: Using the diffusion process, given an initial data distribution, noise is gradually added to the data to generate a series of noisy images. Each step can be expressed as:

[0091] (5);

[0092] in, Represents the data at a given time step t-1 , at time step The distribution of pictures on represents a Gaussian distribution, Represents the mean term of the forward process in the diffusion model, representing the data point at the current time step is based on the data point at the previous time step A scaled version of represents the variance of the noise added at the current time step, Determines the size of the noise, I is the unit matrix;

[0093] S2.3.2: Then, a back diffusion process is used to gradually denoise and restore the original distribution based on the aligned feature vectors starting from random noise. This process uses a parameterized model with the conditional probability:

[0094] (6);

[0095] in, Indicates that in the reverse diffusion process, given the current time step data In the case of The distribution of In the reverse process, the mean at time step t-1 is inferred from time step t. It is defined as the variance of the inference at time step t to time step t-1 in the backward process.

[0096] S2.3.3: The whole process is optimized by variational lower bound:

[0097] (7);

[0098] in, Refers to the mean between different time steps (i.e. 0-T steps), It is the KL divergence (Kullback-Leibler Divergence), which is used to measure the and The difference between:

[0099] (8).

[0100] S2.4: The ambiguous word feature enhancement model of visual-language modeling receives multiple ambiguous images and ambiguous words, performs modal feature enhancement, and obtains the enhanced ambiguous word embedding expression. The specific steps include:

[0101] S2.4.1: Compute the feature embedding vectors of the multi-ambiguous images generated by the ambiguous vocabulary and the feature embedding vectors corresponding to the original video information, which are obtained by a visual encoder with shared parameters:

[0102] (9);

[0103] in, is the acquired feature embedding vector, It is a visual encoder that obtains the feature embedding vectors corresponding to the multiple ambiguous images and the feature embedding vectors corresponding to the original video information according to the different input information V.

[0104] S2.4.2: Obtain the cross-similarity of the feature embedding vectors corresponding to the multiple ambiguous images and the feature embedding vectors corresponding to the original video information to form an ambiguous image-reference video similarity matrix. The cross-similarity metric can be obtained by formula (10):

[0105] (10);

[0106] in, Represent the features corresponding to the multi-ambiguous images and the features corresponding to the original video information, respectively. i Representative i Features, n Represents the number of features.

[0107] S2.4.3: Apply a fully connected ambiguous decision maker to determine the real corresponding image corresponding to the ambiguous word based on the ambiguous image-reference video similarity matrix, fuse the embedding vector of the real corresponding image with the embedding vector of the ambiguous word, and obtain the enhanced embedding expression of the ambiguous word. , the cross-modal fusion method is shown in formula (11):

[0108] (11);

[0109] in, , is the scaling factor, and Represent the embedding vector of the ambiguous word and the embedding vector of the real corresponding image respectively, K T represent K The transpose of .

[0110] like Figure 3 As shown, the present implementation provides a specific schematic diagram, in which the ambiguous word is "apple". After the text graph model, the "fruit apple image" and the "mobile phone apple image" are obtained, which are input into the visual encoder together with the reference video to obtain the ambiguous visual encoding and the reference video encoding respectively, and the similarity between the ambiguous visual encoding and the reference video encoding is calculated to obtain the ambiguous image-reference video similarity matrix, determine the real corresponding image corresponding to the ambiguous word, and obtain the embedding vector of the real corresponding image; at the same time, the ambiguous word is encoded by the text encoder (that is, the embedding vector of the ambiguous word is obtained), and the embedding vector of the ambiguous word and the embedding vector of the real corresponding image are fused cross-modally to obtain the final enhanced embedding expression of the ambiguous word.

[0111] This implementation proposes a coreference resolution system based on visual-text joint modeling, including:

[0112] The embedding vector generating unit is configured to: obtain input text and a reference video, convert the input text into a text embedding vector by using a text encoder, and convert the reference video into a video embedding vector by using a video encoder;

[0113] The feature enhancement unit is configured to: construct a text semantic tree according to the text embedding vector, construct a video semantic tree according to the video embedding vector, and perform feature fusion enhancement based on the distance between nodes in the text semantic tree and the video semantic tree through graph convolution operation to obtain text enhancement features and video enhancement features;

[0114] The coreference resolution unit is configured to: perform feature adjustment using a cross-attention feature network based on text enhancement features and video enhancement features to obtain adjusted features, coordinate the adjusted features using a distribution similarity metric, perform coreference resolution prediction on the coordinated features, and obtain a coreference resolution result corresponding to the input text.

[0115] Specifically, the working process of the embedding vector generation unit, the feature enhancement unit and the coreference resolution unit can refer to the process of steps S1.1-S1.4, which will not be repeated here.

[0116] It can be understood that the present implementation also provides a word sense disambiguation system based on visual-text joint modeling, including:

[0117] The multi-ambiguous image generation unit is configured to: obtain ambiguous words, and based on the text graph model guided by external knowledge, generate corresponding multi-ambiguous images by using external knowledge guidance updated in real time;

[0118] The word sense disambiguation unit is configured to: obtain an enhanced embedding expression of the ambiguous vocabulary by using a pre-trained ambiguous word feature enhancement model based on visual-language modeling according to the multiple ambiguous images, the input text and the ambiguous vocabulary.

[0119] Specifically, the working process of the multi-ambiguous image generation unit and the word sense disambiguation unit can refer to the process of steps S2.1-S2.4, which will not be repeated here.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A coreference resolution method based on visual-text joint modeling, characterized in that: The process includes: Obtain input text and reference video, use a text encoder to convert the input text into a text embedding vector, and use a video encoder to convert the reference video into a video embedding vector; A text semantic tree is constructed based on the text embedding vector, and a video semantic tree is constructed based on the video embedding vector. In the text semantic tree and the video semantic tree, graph convolution operations are performed to perform feature fusion enhancement based on the distance between nodes to obtain text enhancement features and video enhancement features. Based on the text enhancement features and video enhancement features, a cross-attention feature network is used to adjust the features to obtain the adjusted features, and the distribution similarity measurement is used to coordinate the adjusted features. The coreference resolution prediction is performed on the coordinated features to obtain the coreference resolution result corresponding to the input text; Based on the video enhancement features and text enhancement features, a cross-attention feature network is used to adjust the features, including: The similarity between the text enhancement feature and the video enhancement feature is obtained. According to the similarity, regularization processing is performed based on the text enhancement feature and the video enhancement feature respectively. Before regularization, the video enhancement feature uses a fully connected network to correct the dimension and scene features, and obtains the regularized text feature and the regularized video feature: ； ； in, is a regularization parameter based on similarity, Enter the text i text enhancement features, Enter the text j text enhancement features, For reference video i Scene features, For reference video j Scene features, and is the regularized i The text features and i Video features; Based on the obtained regularized features, a cross-modal fusion network is used to obtain multi-modal adjustment features. : ； in, , is the scaling factor, , and are all parameter matrices, T represents transpose, is the normalized exponential function.

2. The coreference resolution method based on visual-text joint modeling according to claim 1, characterized in that: The multimodal adjustment features are coordinated using distribution similarity measures, including: Obtain annotations for coreference resolution from real reference information; Use a text encoder and a video encoder to convert the acquired annotations into text embedding vectors and video embedding vectors respectively, and convert the text embedding vectors and the video embedding vectors into multimodal annotation features; Based on the multimodal adjustment features and multimodal annotation features, the distribution similarity is calculated and training is iteratively performed to make the adjustment features closer to the true distribution until the distribution similarity is greater than the set threshold or the iteration round is reached.

3. The coreference resolution method based on visual-text joint modeling according to claim 1, characterized in that: The coreference resolution prediction is performed on the coordinated features to obtain the coreference resolution result corresponding to the input text, including: The coordinated features are converted into action confidence scores using a fully connected network. When the action confidence score is greater than a set threshold, it is determined that the content of the input text exists in the reference video.

4. A word sense disambiguation method based on visual-text joint modeling, characterized in that: Using the coreference resolution method based on visual-text joint modeling as described in any one of claims 1 to 3 to perform coreference resolution; Obtain ambiguous words, and generate corresponding multi-ambiguous images based on the text graph model guided by external knowledge and guided by external knowledge updated in real time; According to the multiple ambiguous images, the input text and the ambiguous vocabulary, a pre-trained ambiguous word feature enhancement model based on visual-language modeling is used to obtain an enhanced ambiguous vocabulary embedding expression.

5. The word sense disambiguation method based on visual-text joint modeling as claimed in claim 4, characterized in that: The pre-trained ambiguous word feature enhancement model based on visual-language modeling is used to obtain the enhanced ambiguous word embedding expression, including: Calculate the cross similarity of ambiguous images, ambiguous words and original text information to obtain a similarity matrix; A fully connected ambiguous decision maker is applied to determine the real corresponding image corresponding to the ambiguous words based on the similarity matrix, and the embedding vector of the real corresponding image is fused with the embedding vector of the ambiguous words to obtain the enhanced embedding expression of the ambiguous words.

6. The word sense disambiguation method based on visual-text joint modeling as claimed in claim 4, characterized in that: Based on the external knowledge-guided text graph model, the corresponding multi-ambiguous images are generated using real-time updated external knowledge guidance, including: Extracting multiple extended interpretations of the text embedding vector through an external knowledge base; Obtaining ambiguous words according to the multiple extended interpretations, and aligning the ambiguous words with the corresponding multiple extended interpretations in a vector space to obtain an aligned feature vector; Using the aligned feature vectors as a guide, corresponding images are sampled in the image space to construct multi-ambiguous images.

7. A coreference resolution system based on visual-text joint modeling, characterized in that: The process includes: The embedding vector generating unit is configured to: obtain input text and a reference video, convert the input text into a text embedding vector by using a text encoder, and convert the reference video into a video embedding vector by using a video encoder; The feature enhancement unit is configured to: construct a text semantic tree according to the text embedding vector, construct a video semantic tree according to the video embedding vector, and perform feature fusion enhancement based on the distance between nodes in the text semantic tree and the video semantic tree through graph convolution operation to obtain text enhancement features and video enhancement features; A coreference resolution unit is configured to: perform feature adjustment using a cross-attention feature network based on the text enhancement feature and the video enhancement feature to obtain adjusted features, coordinate the adjusted features using a distribution similarity metric, perform coreference resolution prediction on the coordinated features, and obtain a coreference resolution result corresponding to the input text; Based on the video enhancement features and text enhancement features, a cross-attention feature network is used to adjust the features, including: The similarity between the text enhancement feature and the video enhancement feature is obtained. According to the similarity, regularization processing is performed based on the text enhancement feature and the video enhancement feature respectively. Before regularization, the video enhancement feature uses a fully connected network to correct the dimension and scene features, and obtains the regularized text feature and the regularized video feature: ； ； in, is a regularization parameter based on similarity, Enter the text i text enhancement features, Enter the text j text enhancement features, For reference video i Scene features, For reference video j Scene features, and is the regularized i The text features and i Video features; Based on the obtained regularized features, a cross-modal fusion network is used to obtain multi-modal adjustment features. : ； in, , is the scaling factor, , and are all parameter matrices, T represents transpose, is the normalized exponential function.

8. A word sense disambiguation system based on visual-text joint modeling, characterized in that: Using the coreference resolution method based on visual-text joint modeling according to any one of claims 1 to 3 to perform coreference resolution, comprising: The multi-ambiguous image generation unit is configured to: obtain ambiguous words, and based on the text graph model guided by external knowledge, generate corresponding multi-ambiguous images by using external knowledge guidance updated in real time; The word sense disambiguation unit is configured to: obtain an enhanced embedding expression of the ambiguous vocabulary by using a pre-trained ambiguous word feature enhancement model based on visual-language modeling according to the multiple ambiguous images, the input text and the ambiguous vocabulary.

Images