Enhanced entity perception sports video subtitle generation method combining dominant knowledge and implicit knowledge

Abstract

The invention discloses an entity perception sports video subtitle generation method combining dominant knowledge and implicit knowledge enhancement, and belongs to the field of video analysis and understanding. An existing traditional video subtitle generation method is difficult to directly generate text description with player identities and specific scene categories according to visual contents of videos. The list of players participating in the competition is used as dominant knowledge to help the model to obtain pre-competition knowledge. Video features related to a scene are adaptively captured under the guidance of knowledge by using learnable vectors. Based on the attention mechanism, the relationship between the player and the video content is modeled, and entity-related video features are generated. And a decoder based on the principle of'scene first and entity second 'is provided, so that the accuracy and continuity of generating subtitles are enhanced. Video spatio-temporal dynamic information is modeled to capture timing and spatial relationships between video frames. The validity of the method is verified on basketball data sets VCNBA2022 and NSVA and a football data set Goal.

Description

Technical Field

[0001] Based on deep learning technology, a method for generating entity-aware sports video subtitles that is enhanced by combining explicit knowledge and implicit knowledge is studied. First, according to the game information, all players in the corresponding matches of each video clip are taken as a list of candidate players, that is, explicit game knowledge. The visual features of various scene video data are averaged according to various scene labels to obtain implicit visual scene knowledge of various scenes; then, learnable vectors are used to adaptively capture scene-related video features under the guidance of knowledge. Based on the attention mechanism, the relationship between players and video content is modeled to generate entity-related video features. Then, a decoder based on the principle of "scene first, entity later" is proposed to enhance the accuracy and coherence of generated subtitles. Finally, the spatiotemporal dynamic information of the video is modeled to capture the temporal and spatial relationship between video frames. It belongs to the field of computer vision, specifically involving deep learning, video analysis and understanding, video subtitle generation and other technologies. Background Art

[0002] The explosive growth of video data has become a prominent feature of the current information age. In order to effectively understand and analyze these massive video contents and extract useful information from them, various video content description technologies have emerged. Video subtitle generation refers to the process of automatically analyzing and processing video data through computer vision, deep learning and other technologies, extracting key information from the video, and classifying, identifying, understanding and interpreting it. This technology can help people better understand and analyze video content and improve the dissemination effect and viewing quality of the video. In addition, video content description tasks have a wide range of applications in sports events, education and training, medical care, and human-computer interaction. Although various video content understanding technologies have made great progress and their performance has been significantly improved. However, in the actual application scenario of text live broadcast of sports events, what is needed behind it is a deep semantic understanding of the video content. Event commentators not only need to accurately identify players and various scenes with various pre-match information and professional knowledge, but also need to integrate background knowledge to bring a professional and vivid interpretation of the game to the audience. However, the parameter knowledge of the model itself is limited, and there is a lack of relevant knowledge in a specific field, which limits the performance of the model. Therefore, knowledge-enhanced video subtitle generation is very necessary for applications in the sports field.

[0003] Sports video caption generation is an important research topic in computer vision. Current research can be divided into traditional methods and methods based on external explicit knowledge. These methods have their own limitations. Among them, traditional methods aggregate a set of video frame features into a video representation through a visual encoder, and use a language decoder on the video representation to learn visual-text alignment to generate text descriptions. Traditional methods cannot directly generate text descriptions with player identity information and fine-grained scene categories based on visual content. Methods based on external explicit knowledge use visual details detected by external detectors or knowledge obtained from external knowledge bases to enhance visual content, helping the model to generate text descriptions with player identity and fine-grained specific scene categories. However, the text descriptions of such methods are still not accurate enough. They only focus on external explicit knowledge and ignore the complex underlying patterns and deeper relationships hidden inside the visual data. In fact, different scenes with similar visual appearance are often mistaken for the same, which may cause the model to confuse the scene. In addition, different videos in the same scene category may share certain paradigms or subtle associations of the scene. This implicit information cannot be obtained from the encoder or external explicit sources. Therefore, mining implicit knowledge from data and designing a framework to effectively integrate explicit and implicit knowledge are crucial for entity-aware sports video captioning. Summary of the invention

[0004] In order to effectively address the limitations of existing traditional methods and methods based on external explicit knowledge, a deep learning-based entity-aware sports video caption generation method that combines explicit knowledge and implicit knowledge enhancement is proposed. Sports video clips can be used as input to generate text descriptions with player identities and fine-grained scene categories based on explicit knowledge and implicit knowledge.

[0005] According to the principle of analogical reasoning in cognitive psychology, people can compare the source video with other videos with similar visual scenes to achieve a more accurate understanding. Inspired by this, in order to more accurately generate descriptions of specific visual scenes, implicit visual scene knowledge is mined from training videos. All video features within the same scene category are averaged to obtain the visual feature center. Each center serves as a representative description of each scene category. It is a highly abstract, implicit scene knowledge representation that defines the specific paradigm of a scene and also demarcates the boundaries with other scenes. This knowledge can guide the model to distinguish different visual scenes. In addition, a list of players participating in each game is used as explicit game knowledge to assist the model in generating text descriptions with entity names.

[0006] In order to effectively utilize and aggregate both kinds of knowledge, a joint explicit knowledge and implicit knowledge-enhanced network is designed for entity-aware sports video caption generation. Firstly, a learnable query vector is used to adaptively capture scene-related video features under the guidance of implicit scene knowledge to enhance the model's depth of scene understanding. Then, an attention mechanism is used to deeply model the interaction between players and scenes to generate entity-related video features. In order to fully utilize both scene-related and entity-related video features, a decoder with a "scene-to-entity" approach is designed, which first decodes scene-related features and then decodes subtitles containing player names in the scene. It improves the accuracy and coherence of generated subtitles by providing contextual understanding, enabling them to more accurately match scene-related entities. Finally, spatiotemporal dynamic information is encoded by capturing the temporal and spatial relationships between video frames.

[0007] The combined use of explicit knowledge and implicit knowledge has been well applied in the task of entity-aware sports video subtitle generation, which has a good effect on improving the performance of subsequent tasks. The specific steps are as follows:

[0008] 1) Extraction of scene knowledge

[0009] When people watch a video, they usually compare it with other videos with the most similar scenes to gain a more accurate understanding, and use similar videos to compensate for visual details that may be missing in the source video. However, retrieving relevant information from external databases online to enhance video features increases computational costs. This method mines hidden scene knowledge in videos in a simple way. This implicit knowledge can not only provide complementary visual information, but also enhance the model's ability to discern and understand visual scenes. Specifically, the visual encoder is used to extract global features of all video clips in the training set. N t represents the number of video clips in the training set, Represents the feature dimension. Video features are grouped according to the labels annotated for each video in the training set. The central features of each label scene are obtained by averaging the video features of the same label.

[0010]

[0011] Among them, V r g is a video collection with label r, and the global feature of the jth video in the collection is represented by v j K is the number of video features of label r. C ris the central feature of each label. Finally, we get the central feature set C = {C1, C2, C3, ..., C9} of 9 labels. These 9 central features are defined as implicit scene knowledge, which helps the model distinguish different visual scenes and make up for the lack of information in the source video.

[0012] 2) Extraction of competition knowledge

[0013] In live sports broadcasts, commentators are provided with information about the game in advance, such as the teams and the identities of all players on each team. If the model has such explicit knowledge about the game, it will be better able to generate text descriptions with player identities. Therefore, the game name, team, and all players corresponding to each video clip are obtained from the basketball knowledge graph proposed by Xi et al. All players in a game are aggregated to form a candidate player list, so that the model can obtain knowledge about the game. The global image features of all players in the candidate list are extracted using the visual encoder and the text encoder, respectively. and the global name text feature Among them, N e Represents the number of players in the candidate player list. Add the image features and name text features according to the corresponding positions to obtain the multimodal player features E m .

[0014] E m =E p +E n W1, (2)

[0015] Among them, W1 is a randomly initialized learnable matrix that maps video features into text space.

[0016] 3) Spatiotemporal dynamic information modeling

[0017] Videos not only contain spatial information from static frames, but also embody behaviors and events that evolve over time, which are expressed through spatiotemporal features. Extracting these spatiotemporal dynamics is crucial for accurately recognizing objects, actions, and the overall meaning of scenes in videos.

[0018] First, the visual encoder encodes 18 frames of images into visual features Then convert it into visual features Along the spatial dimension, the video features are averaged to obtain the spatial features Similarly, along the temporal dimension, the video features are averaged to obtain the temporal features The spatial features and temporal features are concatenated to obtain the spatiotemporal features V of a given video. st .

[0019]

[0020] Among them, [,] represents the concatenation function.

[0021] 4) Video-knowledge information interaction

[0022] Since there is a lot of redundant information in the video, directly interacting with the video knowledge will introduce a lot of noise information. Therefore, a learnable method is used to adaptively capture key information. Specifically, 18 learnable query vectors are randomly initialized. Then the learnable query vector interacts with the scene knowledge C and the spatiotemporal video features in turn to obtain the video features related to the scene. The specific interaction process is shown in the following formula:

[0023]

[0024] in, and is a randomly initialized learnable matrix, and i is the number of attention heads. i ′ is the interactive feature obtained by the attention mechanism between the query vector and the scene knowledge. δ(·) represents the Softmax normalization function. The concatenated output features of i attention heads [V1′, V2′, …, V i ′] through the fully connected layer Integrate and obtain the intermediate feature V′. i ″ is the feature obtained by the attention mechanism between the intermediate feature and the spatiotemporal feature. The concatenated output feature [V1″, V2″, …, V i ″] Through the fully connected layer Integrate to obtain scene-related features V i ″. FFN(·) represents the feed-forward layer. A total of 4 VKIM layers are stacked.

[0025] 5) Entity-Video Information Interaction (EVIM)

[0026] Learning entity-related video features based on attention mechanism The scaled dot product attention function is used to associate entities with scenes, and the residual connection operation is used to enhance the feature representation.

[0027]

[0028] in, and is a randomly initialized learnable matrix, V scene is the scene-related feature V″ obtained by formula (7). m is the multimodal player feature obtained by formula (2), and δ(·) represents the Softmax normalization function.

[0029] 6) Scene-Entity Decoder

[0030] The designed scene-to-entity decoder can first decode scene-related text features, and then decode text features with entity information based on this, making full use of scene and entity information. Scene cross attention is used to generate scene-related features

[0031]

[0032] in, Indicates layer normalization, l represents the current decoding layer, and l-1 represents the previous decoding layer. scene is the scene-related feature V″ obtained by formula (7). t is the current decoding time, 1:t is the time from the first decoding time to the current decoding time of the autoregressive decoding operation, is the previous self-attention layer M self (·) Sentence features output at decoding time t. M S-Att (·) indicates the scene cross attention layer. The first layer embedding representation As shown in formula (11):

[0033]

[0034] Among them, the token ω is the result obtained by the word segmenter after performing word segmentation on the input sentence. 0:t-1 represents the token at time t-1. ε(·) represents the token embedding representation extractor. τ pe (·) represents triangular position embedding, which enables the model to distinguish different positions in the sequence and thus better understand the sequence structure.

[0035] In order to obtain entity-aware text description, we use entity cross-attention M E-Att (·)Entity information V entity Scene-related features Interact so that entity information can be integrated into text features with scene context. Residual connection operations and feedforward networks are used to enhance feature representation and output entity-related features.

[0036]

[0037] in, Denotes layer normalization. FFN(·) denotes the feed-forward layer. The final decoder output is obtained by stacking L layers of decoding layers. The value of L is 3. The decoder combines the probability and the word dictionary to decode the corresponding word. The output probability O is obtained through the multi-layer perception layer MLP(·) and the Softmax function δ(·). p .

[0038]

[0039] Compared with the existing technology, it has the following advantages:

[0040] Mining implicit scene knowledge from training video data in a simple and effective way helps the model distinguish different scene types. Designing an entity-aware sports video subtitle generation framework that jointly enhances explicit knowledge and implicit knowledge. In it, learnable query vectors are used to adaptively learn scene-related video features. Using the attention mechanism, entities and video content are associated to learn entity-related video features. The feature representation of the video is enhanced by extracting the spatiotemporal dynamic features of the video. Designing a "scene first, then entity" decoding method to provide contextual understanding to improve the accuracy and coherence of generated subtitles, so that they can more accurately match scene-related entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart. DETAILED DESCRIPTION

[0042] A method for generating entity-aware sports video subtitles based on deep learning and joint explicit knowledge and implicit knowledge enhancement is proposed. The specific implementation steps of the invention are as follows:

[0043] Step 1: In a simple and effective way, the training video features are grouped according to the labels, and the central features of each group of features are averaged to obtain the central features of the scene, thereby mining the visual scene knowledge hidden in the data. Specifically, the visual encoder is used to extract the global features of all video clips in the training set. N t represents the number of video clips in the training set, Represents the feature dimension. Video features are grouped according to the labels annotated for each video in the training set. The central features of each label scene are obtained by averaging the video features of the same label.

[0044]

[0045] Among them, V r g is a video collection with label r, and the global feature of the jth video in the collection is represented by v j K is the number of video features of label r. C r is the central feature of each label. Finally, we get the central feature set C = {C1, C2, C3, ..., C9} of 9 labels. These 9 central features are defined as implicit scene knowledge, which helps the model distinguish different visual scenes and make up for the lack of information in the source video.

[0046] Step 2: To simulate the fact that the commentator knows all the player information of both teams before the game, the name of the game, the team, and all the player information corresponding to each video clip are obtained from the basketball knowledge graph proposed by Xi et al. All the players in a game are aggregated to form a candidate player list, so that the model can obtain game-related knowledge. The global image features of all the players in the candidate list are extracted using the visual encoder and the text encoder respectively. and the global name text feature Among them, N e Represents the number of players in the candidate player list. Add the image features and name text features according to the corresponding positions to obtain the multimodal player features E m .

[0047] E m =E p +E n W1, (2)

[0048] Among them, W1 is a learnable matrix that maps video features into text space.

[0049] Step 3: Videos contain not only spatial information from static frames, but also behaviors and events that evolve over time, expressed through spatiotemporal features. Extracting these spatiotemporal dynamics is critical to accurately identify objects, actions, and the overall meaning of scenes in videos.

[0050] First, the visual encoder encodes 18 frames of images into visual features Then convert it into visual features Along the spatial dimension, the video features are averaged to obtain the spatial features Similarly, along the temporal dimension, the video features are averaged to obtain the temporal features The spatial features and temporal features are concatenated to obtain the spatiotemporal features V of a given video. st .

[0051]

[0052] Among them, [,] represents the concatenation function.

[0053] Step 4: Since there is a lot of redundant information in the video, directly interacting with the video knowledge will introduce a lot of noise information. Therefore, a learnable method is used to adaptively capture key information. Specifically, first randomly initialize 18 learnable query vectors Then the learnable query vector interacts with the scene knowledge C and the spatiotemporal video features in turn to obtain the video features related to the scene. The specific interaction process is shown in the following formula:

[0054]

[0055] V″=FFN(F c 2 (V1″,V2″,…,V i ″)) (7)

[0056] in, and is a randomly initialized learnable matrix, and i is the number of attention heads. i ′ is the interactive feature obtained by the attention mechanism between the query vector and the scene knowledge. δ(·) represents the Softmax normalization function. The concatenated output features of i attention heads [V1′, V2′, …, V i ′] through the fully connected layer F c 1 (·) is integrated to obtain the intermediate feature V′. i ″ is the feature obtained by the attention mechanism between the intermediate feature and the spatiotemporal feature. The concatenated output feature [V1″, V2″, …, V i ″] Through the fully connected layer Integrate to obtain scene-related features V i ″. FFN(·) represents the feed-forward layer. A total of 4 VKIM layers are stacked.

[0057] Step 5: Learn entity-related video features based on the attention mechanism The scaled dot product attention function is used to associate entities with scenes, and the residual connection operation is used to enhance the feature representation.

[0058]

[0059] in, and is a randomly initialized learnable matrix, V scene is the scene-related feature V″ obtained by formula (7).

[0060] Step 6: The designed scene-to-entity decoder can first decode the scene-related text features, and then decode the text features with entity information based on this, making full use of the scene and entity information. Scene cross attention is used to generate scene-related features

[0061]

[0062] in, Indicates layer normalization, l represents the current decoding layer, l-1 represents the previous decoding layer. t is the current decoding time, 1:t is the time from the first decoding time of the autoregressive decoding operation to the current decoding time, Vscene is the scene-related feature V″ obtained by formula (7). is the previous self-attention layer M self (·) Sentence features output at decoding time t. M S-Att (·) indicates the scene cross attention layer. The first layer embedding representation As shown in formula (11):

[0063]

[0064] Among them, the token ω is the result obtained by the word segmenter after performing word segmentation on the input sentence. 0:t-1 represents the token at time t-1, ε(·) represents the token embedding representation extractor, τ pe (·) represents triangular position embedding, which enables the model to distinguish different positions in the sequence and thus better understand the sequence structure.

[0065] In order to obtain entity-aware text description, we use entity cross-attention M E-Att (·)Entity information V entity Scene-related features Interact so that entity information can be integrated into text features with scene context. Residual connection operations and feedforward networks are used to enhance feature representation and output entity-related features.

[0066]

[0067] in, Denotes layer normalization. FFN(·) denotes the feed-forward layer. The final decoder output is obtained by stacking L layers of decoding layers. The value of L is 3. The decoder combines the probability and the word dictionary to decode the corresponding word. The output probability O is obtained through the multi-layer perception layer MLP(·) and the Softmax function δ(·). p .

[0068]

[0069] In order to verify the effectiveness of the algorithm, the performance comparison experiment was carried out on the entity perception sports datasets VC_NBA_2022, Goal and NSVA. First, the performance of the algorithm was tested on the VC_NBA_2022 dataset. As shown in Table 1, it achieved better performance comparison results compared with the current best method, the event knowledge guidance method "A Simple yet Effective KnowledgeGuided Method for Entity-aware Video Captioning" proposed by Wu Lifang's team and the universal video understanding model "OmniViD: A Generative Framework for Universal Video Understanding" proposed by Wu Zuxuan's team. Among them, the general model* and the general model** represent explicit knowledge enhancement and joint explicit knowledge and implicit knowledge enhancement, respectively. And knowledge guidance* means that implicit knowledge is equipped on the basis of explicit knowledge enhancement.

[0070] Table 1: Performance comparison on the VC_NBA_2022 dataset

[0071] CIDEr METEOR Rouge-L BLEU-1 BLEU-2 BLEU-3 BLEU-4 General Model 71.2 27.5 50.7 49.7 42.4 36.5 29.2 General Model* 125.2 28.6 53.3 52.5 45.2 38.7 30.5 General Model 130.8 29.0 56.0 53.7 46.6 39.7 32.6 Knowledge guidance 138.5 28.0 54.9 53.1 46.4 38.8 32.4 Knowledge Guidance* 144.6 28.7 56.1 54.5 47.6 40.2 34.4 This method 140.7 29.5 56.8 57.1 50.3 43.1 36.7

[0072] Then, on the football match commentary dataset Goal, the algorithm was experimentally compared with the best method at present, namely the event knowledge guidance method "A Simple yet Effective Knowledge Guided Method for Entity-aware Video Captioning" proposed by Wu Lifang's team and the universal video understanding model "OmniViD: A Generative Framework for Universal Video Understanding" proposed by Wu Zuxuan's team. As shown in Table 2, the algorithm achieved better performance comparison results. Among them, the universal model* and the universal model** respectively represent explicit knowledge enhancement and joint explicit knowledge and implicit knowledge enhancement. And knowledge guidance* means that implicit knowledge is equipped on the basis of explicit knowledge enhancement.

[0073] Table 2: Performance comparison on the Goal dataset

[0074] CIDEr METEOR Rouge-L BLEU-1 General Model 3.0 5.9 9.1 10.7 General Model* 3.9 6.4 10.6 14.4 General Model 4.2 6.5 11.0 15.4 Knowledge guidance 3.7 6.4 10.5 14.9 Knowledge Guidance* 4.0 6.6 10.8 15.2 This method 4.1 6.6 11.2 15.3

[0075] Furthermore, the explicit knowledge and implicit knowledge were used as components to perform entity-aware sports video subtitle generation tasks on the basketball dataset NSVA. The effectiveness of dual knowledge enhancement was verified by equipping the explicit game knowledge and implicit visual scene knowledge to the sports video analysis model "Sports Video Analysis on Large-Scale Data" proposed by Richard P. Wildes' team. Among them, there are two modes of analysis models. The first is to use the S3D network as the feature extraction network, and the second is to use the Timesformer (T) network as the feature extraction network. "(full)" means that it comprehensively utilizes video features, basketball features, basket features, player features, and venue features. "list" represents explicit knowledge, and "scene" represents implicit knowledge. "Analysis model*" means that knowledge is equipped. As shown in Table 3, the proposed dual knowledge enhancement improves the performance of existing methods, proving the effectiveness of the proposed dual knowledge enhancement method.

[0076] Table 3: Performance comparison on the NSVA dataset

[0077]

Claims

1. An entity-aware sports video subtitle generation method enhanced by combining explicit knowledge and implicit knowledge, characterized by: Step (1) groups the training video features according to labels and averages each group of features to obtain the central features of the scene, thereby mining the visual scene knowledge hidden in the data; Step (2) gathers all players in a game into a candidate player list so that the model can obtain game-related player knowledge; Step (3) enhancing the feature representation of the video by extracting the spatiotemporal dynamic features of the video; Step (4) using the learnable query vector to adaptively learn scene-related video features; Step (5) uses the attention mechanism to associate entities with video content and learn video features related to entities; In step (6), a scene-to-entity decoder is designed to first decode the scene-related text features and then decode the text features with entity information based on this.

2. The method according to claim 1, characterized in that In step (1), the visual encoder is used to extract the global features of all video clips in the training set. N t represents the number of video clips in the training set, Represents feature dimension; The video features are grouped according to the labels annotated on each video in the training set; the central features of each label scene are obtained by averaging the video features with the same label; in, is a video collection with label r, and the global feature of the jth video in the collection is represented by v j ; K is the number of video features of label r; C r is the central feature of each label; finally, the central feature set C = {C1, C2, C3, …, C9} of 9 labels is obtained, and these 9 central features are defined as implicit scene knowledge.

3. The method according to claim 1, characterized in that In step (2), the game name, team, and all player information corresponding to each video clip are obtained from the basketball knowledge graph; all players in a game are aggregated to form a candidate player list; the global image features of all players in the candidate list are extracted using the visual encoder and the text encoder respectively. and the global name text feature Among them, N e Represents the number of players in the candidate player list; add the image features and name text features according to the corresponding positions to obtain the multimodal player features E m ; HAVE BEEN m =E p +E n W1, (2) Among them, W1 is a learnable matrix that maps video features into text space.

4. The method according to claim 1, characterized in that In step (3), the visual encoder first encodes the 18 frames of images into visual features Then convert it into visual features Along the spatial dimension, the video features are averaged to obtain the spatial features Along the time dimension, the video features are averaged to obtain the time features The spatial features and temporal features are concatenated to obtain the spatiotemporal features V of a given video. st ; Among them, [,] represents the concatenation function.

5. The method according to claim 1, characterized in that In step (4), 18 learnable query vectors are first randomly initialized Then the learnable query vector interacts with the scene knowledge C and the spatiotemporal video features in turn to obtain the video features related to the scene; The interaction process is shown in the following formula: in, and is a randomly initialized learnable matrix, i is the number of attention heads; V i ′ is the interactive feature obtained by the attention mechanism between the query vector and the scene knowledge; δ(·) represents the Softmax normalization function; the concatenated output features of i attention heads [V′1, V′2, …, V′ i ]Through the fully connected layer Integrate to obtain the intermediate feature V′; V i ″ is the feature obtained by the attention mechanism of the intermediate features and spatiotemporal features; the concatenated output features of i attention heads [V″1, V″2, …, V″ i ]Through the fully connected layer Integrate to obtain scene-related features V″ i ; FFN(·) represents the feed-forward layer; a total of 4 VKIM layers are stacked.

6. The method according to claim 1, characterized in that In step (5), the video features related to entities are learned based on the attention mechanism Utilize the scaled dot product attention function to associate entities with scenes and utilize the residual connection operation to enhance feature representation; in, and is a randomly initialized learnable matrix, V scene is the scene-related feature V″ obtained by formula (7).

7. The method according to claim 1, characterized in that In step (6), the designed scene-to-entity decoder first decodes the scene-related text features, and then decodes the text features with entity information based on this; Scene cross attention is used to generate scene-related features in, Represents layer normalization, l represents the current decoding layer, l-1 represents the previous decoding layer; t is the current decoding time, 1:t is the first decoding time to the current decoding time of the autoregressive decoding operation, V scene is the scene-related feature V″ obtained by formula (7); is the previous self-attention layer M self (·) Sentence features output at decoding time t; M S-Att (·) indicates the scene cross attention layer; the first layer embedding representation As shown in formula (11): Among them, the token ω is the result obtained by the word segmenter after performing word segmentation on the input sentence. 0:t-1 represents the token at time t-1, ε(·) represents the token embedding representation extractor, τ pe (·) represents triangular position embedding, which enables the model to distinguish different positions in the sequence and thus better understand the sequence structure; Using entity cross attention M E-Att (·)Entity information V entity Scene-related features Interact so that entity information can be integrated into text features with scene context; use residual connection operations and feedforward networks to enhance feature representation and output entity-related features in, Represents layer normalization; FFN(·) represents the feedforward layer; the final decoder output is obtained by stacking L layers of decoding layers L takes the value of 3; the decoder combines the probability and the word dictionary to decode the corresponding word; the output probability O is obtained through the multi-layer perception layer MLP(·) and the Softmax function δ(·) p ;

Images