Football commentary generation method and device based on visual reasoning and knowledge enhancement

Abstract

The invention provides a football commentary generation method and device based on visual reasoning and knowledge enhancement, and relates to the technical field of multi-modal artificial intelligence, and the method comprises the steps: carrying out the visual reasoning based on competition state information, fine-grained visual information and event positioning guidance information, performing detail enhancement on anonymized and coarse-grained commentary information generated by the end-to-end model according to a visual reasoning result to obtain detail-enhanced commentary information; and based on the internal knowledge base and the external knowledge base, performing knowledge enhancement on the detail-enhanced explanation information to obtain knowledge-enhanced football explanation. According to the football commentary generation method and device based on visual reasoning and knowledge enhancement provided by the invention, through a two-stage processing framework of entity alignment based on visual reasoning and commentary generation based on knowledge enhancement, the workflow of an artificial commentator can be better simulated, and a more natural and information-rich commentary is generated.

Description

Technical Field

[0001] The present application relates to the field of multimodal artificial intelligence technology, and in particular to a method and device for generating football commentary based on visual reasoning and knowledge enhancement. Background Art

[0002] With the development of artificial intelligence technology, automatic commentary generation technology has been widely used in many fields, especially in sports events.

[0003] Traditional automated football commentary generation methods often rely on end-to-end deep learning models, which input a football match video clip of about half a minute in length and output a natural language commentary of the video football clip.

[0004] However, such end-to-end models often lack fine-grained visual information understanding and rich contextual knowledge, resulting in the generated commentary failing to accurately reflect the specific details and background of the real-time game. Summary of the Invention

[0005] The purpose of this application is to provide a method and device for generating football commentary based on visual reasoning and knowledge enhancement. Through a two-stage processing framework of entity alignment based on visual reasoning and commentary generation based on knowledge enhancement, it can better simulate the workflow of human commentators and generate more natural and informative commentary.

[0006] This application provides a football commentary generation method based on visual reasoning and knowledge enhancement, including: Obtain anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; extract game status information and fine-grained visual information based on the target video clip, and use the end-to-end model to obtain event location guidance information; the fine-grained visual information is used to characterize detailed information in the game; perform visual reasoning based on the game status information, the fine-grained visual information, and the event location guidance information, and perform detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; the visual reasoning results include: identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of each event; perform knowledge enhancement on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain knowledge-enhanced football commentary; wherein, the external knowledge base includes: historical statistical data of different football games; the internal knowledge base includes: static background information and dynamically updated events of the current football game.

[0007] Optionally, the game status information includes: temporal team lineup information, a key event timeline, and a historical timeline; the extraction of game status information based on the target video clip includes: generating the temporal team lineup information, the key event timeline, and the historical timeline based on the target video clip and the historical video clip; updating the temporal team lineup information based on the card-playing event and the player replacement event represented by the historical timeline; wherein the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card-playing events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events.

[0008] Optionally, the fine-grained visual information includes: player information, jersey information and alias information of participating teams; the extracting fine-grained visual information based on the target video clip includes: using a visual language model to classify each shot in the target video clip to obtain shot classification information containing each shot category; the shot categories include: close-up shots, medium shots, long shots and off-field shots; based on the player information contained in the temporal team lineup information, the facial image of each player is screened out from the database to obtain a facial image set containing the facial images of all players, and image recognition technology is used to identify facial images that match the facial images in the facial image set from the medium and close-up shots of the target video clip to determine the players included in the target video clip; based on optical character recognition technology, the jerseys appearing in the close-up shots of the target video clip are identified to obtain the jersey information and alias information of the participating teams contained in the target video clip; wherein the medium and close-up shots include: close-up shots, and / or medium shots.

[0009] Optionally, the method of using the end-to-end model to obtain event location guidance information includes: obtaining queries for each cross-attention layer teammate of the end-to-end model, and calculating the average cross-attention between the query and the video frame to obtain the attention weight of each query; aggregating the attention weight of each query to obtain a frame-level importance vector, and obtaining the event location guidance information based on the frame-level importance vector; wherein the frame-level importance vector is used to characterize the importance weight of each video frame in the target video clip when generating anonymized interpretation information.

[0010] Optionally, visual reasoning is performed based on the game status information, the fine-grained visual information, and the event positioning guidance information, and the anonymized, coarse-grained commentary information is enhanced in detail according to the visual reasoning result to obtain detail-enhanced commentary information, including: performing visual reasoning based on the key event timeline, the historical timeline, the shot classification information, the facial image set, the players included in the target video clip, the jersey information, and the participating team alias information, and enhancing the anonymized, coarse-grained commentary information in detail according to the visual reasoning result to obtain detail-enhanced commentary information; wherein, the visual reasoning result is obtained based on a multimodal visual model.

[0011] Optionally, the detail-enhanced commentary information is subjected to knowledge enhancement based on the internal knowledge base and the external knowledge base to obtain the knowledge-enhanced football commentary, including: based on the application methods of relevant knowledge by different commentators in the commentary process, using natural language questions as queries, obtaining historical game data related to the current game from the external knowledge base through a language model, and obtaining real-time updated game status obtained based on context learning from the internal knowledge base; supplementing and correcting the detail-enhanced commentary information based on the historical game data and the real-time updated game status to obtain the knowledge-enhanced football commentary.

[0012] This application also provides a football commentary generation device based on visual reasoning and knowledge enhancement, comprising: An acquisition module is configured to acquire anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; an information extraction module is configured to extract game status information and fine-grained visual information based on the target video clip, and to acquire event location guidance information using the end-to-end model; the fine-grained visual information is used to characterize detailed information in the game; a detail enhancement module is configured to perform visual reasoning based on the game status information, the fine-grained visual information, and the event location guidance information, and to perform detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; the visual reasoning results include identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events; a knowledge enhancement module is configured to perform knowledge enhancement on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain knowledge-enhanced football commentary; wherein the external knowledge base includes historical statistical data from different football games; and the internal knowledge base includes static background information and dynamically updated events of the current football game.

[0013] Optionally, the game status information includes: temporal team lineup information, a key event timeline, and a historical timeline; the information extraction module is specifically used to generate the temporal team lineup information, the key event timeline, and the historical timeline based on the target video clips and the historical video clips; the information extraction module is also specifically used to update the temporal team lineup information based on the card-playing event and the player replacement event represented by the historical timeline; wherein the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card-playing events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events.

[0014] Optionally, the fine-grained visual information includes: player information, jersey information and team alias information; the information extraction module is specifically used to use a visual language model to classify each shot in the target video clip to obtain shot classification information containing each shot category; the shot categories include: close-up shots, medium shots, long shots and off-field shots; the information extraction module is specifically used to filter out the facial image of each player from the database based on the player information contained in the temporal team lineup information, obtain a facial image set containing all players' facial images, and use image recognition technology to identify facial images that match the facial images in the facial image set from the medium and close-up shots of the target video clip to determine the players included in the target video clip; the information extraction module is specifically used to identify the jerseys appearing in the close-up shots of the target video clip based on optical character recognition technology to obtain the jersey information and team alias information of the players included in the target video clip; wherein the medium and close-up shots include: close-up shots and / or medium shots.

[0015] Optionally, the information extraction module is specifically used to obtain the query of each cross-attention layer teammate of the end-to-end model, and calculate the average cross-attention between the query and the video frame to obtain the attention weight of each query; the information extraction module is also specifically used to aggregate the attention weight of each query to obtain a frame-level importance vector, and obtain the event positioning guidance information based on the frame-level importance vector; wherein, the frame-level importance vector is used to characterize the importance weight of each video frame in the target video clip when generating anonymized interpretation information.

[0016] Optionally, the detail enhancement module is specifically used to perform visual reasoning based on the key event timeline, the historical timeline, the shot classification information, the facial image set, the players contained in the target video clip, the jersey information, and the participating team alias information, and enhance the details of the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; wherein the visual reasoning results are obtained based on a multimodal visual model.

[0017] Optionally, the knowledge enhancement module is specifically used to obtain historical game data related to the current game from the external knowledge base through a language model based on the application methods of relevant knowledge by different commentators in the commentary process, using natural language questions as queries, and obtaining real-time updated game status obtained based on context learning from the internal knowledge base; based on the historical game data and the real-time updated game status, the detail-enhanced commentary information is supplemented and corrected to obtain the knowledge-enhanced football commentary.

[0018] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned methods for generating football commentary based on visual reasoning and knowledge enhancement.

[0019] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any of the above-described methods for generating football commentary based on visual reasoning and knowledge enhancement are implemented.

[0020] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-described methods for generating football commentary based on visual reasoning and knowledge enhancement are implemented.

[0021] The present application provides a method and device for generating football commentary based on visual reasoning and knowledge enhancement. First, anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip is obtained. Then, game status information and fine-grained visual information are extracted based on the target video clip, and event location guidance information is obtained using the end-to-end model. The fine-grained visual information is used to represent detailed information in the game. Visual reasoning is performed based on the game status information, the fine-grained visual information, and the event location guidance information, and detail enhancement is performed on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information. The visual reasoning results include: identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events. Finally, knowledge enhancement is performed on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain knowledge-enhanced football commentary. The external knowledge base includes historical statistical data from different football games, and the internal knowledge base includes static background information and dynamically updated events of the current football game. In this way, through the two-stage processing framework of entity alignment based on visual reasoning and explanation generation based on knowledge enhancement, we can better simulate the workflow of human commentators and generate more natural and informative explanations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is one of the flowcharts of the method for generating football commentary based on visual reasoning and knowledge enhancement provided by this application; Figure 2 This is the second flowchart of the football commentary generation method based on visual reasoning and knowledge enhancement provided by this application; Figure 3 This is a schematic diagram of the structure of a football commentary generation device based on visual reasoning and knowledge enhancement provided by this application; Figure 4 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0025] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0026] Professional football commentary possesses several key elements that engage viewers: ① Accurately citing entities: commentators help viewers understand the unfolding events by citing specific details about the players and teams appearing in the sports broadcast footage and appropriately relating them to the ongoing events. ② Understanding the broader context of the game: commentators must not only focus on the current football events but also seamlessly integrate background knowledge into the entire match, providing a comprehensive interpretation of the current state of play. Commentary segments should be continuous, not fragmented, to maintain narrative coherence. ③ Utilizing historical statistics: A well-crafted commentary segment should include not only descriptions but also explanations and commentary. Incorporating historical statistical knowledge is the foundation for these broader perspectives and deeper insights.

[0027] However, previous football commentary models have failed to meet these requirements, primarily due to the following issues: 1. The generated lines use anonymous player and team labels as placeholders and cannot be targeted at specific players in the event; 2. They rely on a series of NLP metrics for evaluation, which often leads to context-related errors, such as an incorrect score announcement after a goal, which is the most crucial information in the entire game. 3. Previous football commentary, which uses live text communications as a data source, is inherently descriptive because the audience does not have access to video. Therefore, compared to live television commentary, these commentaries lack statistical knowledge, which is accompanied by visual information to enhance audience understanding.

[0028] In order to solve the above three key differences between the end-to-end social ceramic commentary model and the human-centered TV commentary, the present application embodiment proposes a GS system, such as Figure 1 As shown, this is a two-stage model that treats football commentary generation as a knowledge-enhanced visual reasoning task. Unlike previous end-to-end models, GS is a two-stage system that transforms football commentary generation into a knowledge-enhanced visual reasoning task. It uses the first stage to implement visual reasoning for entity alignment, and the second stage to use knowledge to correct and re-correct anonymous comments.

[0029] like Figure 1 As shown, in the first stage, GS aligns the anonymous commentary output by the end-to-end commentary model with specific player and team names. Preliminary experiments on human player identification while watching live sports showed that humans rely on various clues to understand details on the football field. In long-view scenes, only 15.5% of players related to the event can be identified solely through player tracking, while other players can be identified through rich context and visual clues. Inspired by this, the embodiments of the present application establish an internal game context construction and a new granularity shot analysis module to reconstruct contextual information in a specific game state and mine visual details that the end-to-end communication generation model cannot capture. GS introduces supervised adjustment with answer-guided reasoning strategies and group-relative strategy optimization on the large language model of the video, improving the entity alignment accuracy on the SN-Caption-test-align benchmark to 71.1%, exceeding the 44.3% of Gemini 2.0 Flash-Lite. It effectively links rough anonymous comments to players involved in specific actions, thereby promoting real comments with precise entity references. In the second phase, GS enriches entity-aligned commentary by combining external and internal football match knowledge, taking it beyond mere descriptive content, thereby providing solutions to Problems 2 and 3 above. Analysis of the practices of live TV commentators reveals their frequent and flexible application of relevant knowledge. Based on this behavior, the present application embodiment develops a football knowledge augmented generation (KAG) system, enabling the model to query external historical statistics and an iteratively updated internal database of match background knowledge, reviewing highlights of the current match through contextual learning. These modules enable GS to provide not only accurate and contextually relevant commentary, but also insightful explanations and comments consistent with professional TV commentary.

[0030] The following describes in detail the football commentary generation method based on visual reasoning and knowledge enhancement provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0031] like Figure 2As shown, an embodiment of the present application provides a method for generating football commentary based on visual reasoning and knowledge enhancement, which may include the following steps 201 to 203: Step 201: Obtain anonymized, coarse-grained commentary information output by the end-to-end model based on the target video clip.

[0032] Illustratively, the method for generating football commentary based on visual reasoning and knowledge enhancement provided in an embodiment of the present application is an anonymized, coarse-grained commentary synthesized based on an existing end-to-end football model (i.e., the above-mentioned end-to-end model). That is, on the basis of the anonymized, coarse-grained commentary generated by the end-to-end model, visual reasoning technology is used to generate an entity-aligned version of football commentary with accurate player and team references. Furthermore, based on the analysis of entity states and football events, combined with information retrieval from internal and external knowledge bases such as dynamic game context information and historical statistical data, a knowledge-enhanced commentary is generated.

[0033] Step 202: extract game status information and fine-grained visual information based on the target video clip, and obtain event location guidance information using the end-to-end model.

[0034] The fine-grained visual information is used to represent detailed information in the game.

[0035] For example, after obtaining the anonymized, coarse-grained commentary generated by the end-to-end model, the first stage of visual reasoning and entity alignment can be carried out. In the first stage, by building a refined game status information database and video clue analysis system, the visual large model is trained to perform anonymous entity reasoning to identify and correspond to the player and team references in the game, and to determine the specific occurrence of the commentary event, so that the commentary has practical meaning.

[0036] For example, the first stage of GS focuses on leveraging various cues to align anonymous entities (including players and teams) in sports game recordings. This alignment is performed through visual reasoning, which includes contextual cues obtained from contextual game state reconstruction, visual cues obtained from fine-grained shot analysis, and guidance from prior knowledge extracted from an end-to-end football base model.

[0037] For example, before performing visual reasoning, it is also necessary to reconstruct the game state information, which includes: temporal team lineup information, key event timeline, and historical timeline.

[0038] Specifically, the step of extracting the game status information based on the target video segment in step 202 may further include the following steps 202a1 and 202a2: Step 202a1: Generate the temporal team lineup information, the key event timeline, and the historical timeline based on the target video clip and the historical video clip.

[0039] Step 202a2: Update the temporal team lineup information based on the card-playing event and the player replacement event represented by the historical timeline.

[0040] Among them, the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events.

[0041] For example, reconstructing the game state based on event timestamps and match progress helps accurately identify the range of players involved and the current game situation. Using a full 90-minute football match (i.e., the aforementioned target video clip) as the subject, key events affecting the game state, such as substitutions, goals, and penalties, are identified. Based on timestamps and event chronology, an internal database of game state information is constructed and dynamically maintained and updated. This allows the on-field lineup and key situations at the time of the segment to be interpreted to narrow the scope of inference.

[0042] For example, the contextual game state is repeatedly updated throughout the game. It mainly includes temporal team lineup information, key event timeline, and historical timeline. The temporal team lineup information records the home team currently active on the court ( ) and the visiting team ( ) The identity of the player, as well as detailed information such as the player's name, jersey number, and position, can be expressed using the following formulas 1 and 2: (Formula 1) (Formula 2) in, Team Indicates the team a 、 h For the two teams playing on the field, Player Indicates the players in the team. For the coach, The position of the player (for example, forward, center, defender, goalkeeper, etc.).

[0043] For example, the Key Events Timeline records the timestamps and details of goals and card events at the current moment of the match. This is because they are the most important match information for tracking match highlights. On the other hand, the Historical Timeline consists of the most recentk It is composed of 100 comments, reflecting recent events, which can be expressed by the following formula 3: , (Formula 3) in, For key events, for t Historical events of the moment, For specific events, such as , play cards (including: red cards and yellow cards), etc. Throughout the game, the ever-changing score and dynamic team lineups are the key factors in determining the outcome of the game. The game score changes with each goal event and is recorded as: ,

[0044] in, P For the players, Score To score, For own goal. The team's lineup will be updated as follows: ,

[0045]

[0046] Among them, the player position attributes when making substitutions Position Changes may occur and can be expressed in the form of P->P' or Q->Q'.

[0047] For example, when a player of a team is replaced, the team's lineup is updated.

[0048] For example, the information obtained by relying solely on long-range shots of the entire football field is often vague and limited. Therefore, in football match broadcasts, the details of the football events are often conveyed to the audience through camera language, such as shot replays and close-up shots of players. This visual information can serve as a key clue to link entities and football events. The embodiment of the present application designs a fine-grained visual information extraction framework. First, video frame analysis is performed based on shot classification (long shot, medium shot, close-up, off-field, etc.), and fine-grained visual information related to game events is identified in medium and close-up shots. Secondly, a photo library of football players is constructed, and facial recognition and jersey number recognition technology are used to accurately identify players appearing in different video clips. Therefore, before performing visual reasoning, it is necessary to extract fine-grained visual information. The fine-grained visual information includes: player information, jersey information, and participating team alias information.

[0049] Specifically, in the above step 202, the step of extracting fine-grained visual information may further include the following steps 202b1 to 202b3: Step 202b1: Classify each shot in the target video clip using a visual language model to obtain shot classification information containing each shot category.

[0050] The lens categories include: close-up lens, medium-shot lens, long-shot lens and off-scene lens.

[0051] Understandably, camera views in football videos are usually categorized into four types: close-up shots , medium shot , long shot And off-field shots In sports broadcast footage of football matches, players appear in close-up and are more likely to be associated with recent events. Therefore, they can provide potential identification information to identify anonymous entities.

[0052] For example, in an embodiment of the present application, fine-grained visual information in a video clip can be captured through the following steps: shot boundary detection and classification, player face recognition, player jersey recognition, and team alias detection.

[0053] For example, for shot boundary detection and classification, the embodiment of the present application may divide a video clip into a series of shots, which can be expressed as follows: ,Afterwards, the visual language model (VLM) is used to classify each shot and obtain the category to which each shot belongs:

[0054] Step 202b2: Based on the player information included in the temporal team lineup information, the facial image of each player is filtered out from the database to obtain a facial image set containing the facial images of all players. Image recognition technology is used to identify facial images that match the facial images in the facial image set from the medium and close-up shots of the target video clip, thereby determining the players included in the target video clip.

[0055] The medium and close-up shots include: close-up shots and / or medium shots.

[0056] For example, for player face recognition, the embodiment of the present application captured facial images of 3213 relevant players from the Soccernet-v2 sports statistics website and created an image database for close-up face registration. First, we need to reconstruct the current lineup according to the above context game state reconstruction steps. and , and then select the players’ facial images from the database as 22 on-field candidates denoted as .

[0057] After that, a professional face recognition model can be used ( ) for a video frame sequence Detect the faces appearing in the dataset and compare them with the candidate face database This process is done by dynamically selecting three key frames , repeated execution to alleviate the problems of image blur and face out-of-frame caused by dynamic lens switching. The final recognition result is marked as .

[0058] For example, the face recognition process can be formally expressed as: , .

[0059] Step 202b3: Identify the jerseys appearing in the close-up shot of the target video clip based on optical character recognition technology to obtain jersey information of the players and alias information of the participating teams contained in the target video clip.

[0060] For example, optical character recognition (OCR) technology based on a visual language model can be used to identify player jerseys. Since jersey names and player numbers are relatively small in detail and prone to blurring and overlapping, this embodiment of the application only uses the visual language model to detect close-up shots. The results of optical character recognition technology can be expressed in the following form:

[0061] in, Used to indicate the player's current action, such as running, dribbling, standing, etc.

[0062] Finally, the fine-grained visual information extracted from the target video clip It can be expressed by the following formula 4: (Formula 4) Exemplarily, for team alias detection, team alias detection refers to associating a team with the color of the jerseys worn in the game. In order to achieve accurate identification, the embodiment of the present application adopts a triple verification mechanism: first, the visual language model (VLM) is called to combine the traditional team color matching knowledge to perform video frame cross-validation; secondly, the large language model (LLM) is used to integrate similar color descriptions (such as "red and blue" and "blue and red stripes" are unified into categories) and filter clear cases. When color ambiguity occurs (for example, when it is necessary to distinguish between "light blue" and "dark blue" teams), the team is locked by the exclusive jersey numbers captured in the video, and these numbers are unique in the current lineup. Finally, the home team color is determined through manual verification With away team colors .

[0063] Exemplarily, for event location guidance information, a 30-second video clip contains many more events than described in the commentary, because the input video is not precisely cut at the event level, but is a fixed-length segment centered on a timestamp. Therefore, it is crucial to search within this fixed-length segment to locate the key football events mentioned in the commentary. Therefore, an embodiment of the present application utilizes a pre-trained end-to-end football commentary model to cleverly obtain the prior knowledge of the connection between the video and the commentary event. Since correct end-to-end football commentary requires an inherent ability to locate key football events from video clips, an embodiment of the present application extracts the cross-attention layer of the end-to-end football commentary model and performs a series of vector transformations to obtain the model's attention coefficient for the input video in seconds as a location clue for key football events.

[0064] Specifically, in the above step 202, the step of obtaining event location guidance information using the end-to-end model may further include the following steps 202c1 and 202c2: Step 202c1: Obtain the query of each cross-attention layer teammate of the end-to-end model, and calculate the average cross-attention between the query and the video frame to obtain the attention weight of each query.

[0065] Step 202c2: Aggregate the attention weights of each query to obtain a frame-level importance vector, and obtain the event location guidance information based on the frame-level importance vector.

[0066] The frame-level importance vector is used to represent the importance weight of each video frame in the target video segment when generating anonymized interpretation information.

[0067] For example, in order to obtain event location guidance information, we first calculate the average cross-attention between the query and the video frame. Indicates in Layer and h In the head, query q With frame n The attention weight between L Taking the average of the layers and all heads, we can get:

[0068] Afterwards, use the query output The L2 criterion is used to calculate the importance of the query, where Q is the total number of queries:

[0069] The final frame-level importance vector is:

[0070] For example, based on the above steps, a vector divided by frame can be obtained W , which represents the relevance of each video frame to the generated commentary. It can be seen as the arousal level of each frame when generating the commentary text, and serves as a basic guide for level events.

[0071] Step 203: Perform visual reasoning based on the game status information, the fine-grained visual information, and the event location guidance information, and perform detail enhancement on the anonymized, coarse-grained commentary information according to the visual reasoning result to obtain detail-enhanced commentary information.

[0072] The visual reasoning results include: identifying and matching player designations and team designations in the game, and matching the occurrence objects of various events.

[0073] For example, the final context clue structure of visual reasoning consists of the following information:

[0074] Specifically, in step 203 above, the step of performing detail enhancement on the anonymized, coarse-grained commentary information according to the visual reasoning result to obtain detail-enhanced commentary information may further include the following step 203a: Step 203a: Perform visual reasoning based on the key event timeline, the historical timeline, the shot classification information, the facial image set, the players included in the target video clip, the jersey information, and the participating team alias information, and perform detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information.

[0075] The visual reasoning result is obtained based on a multimodal visual model.

[0076] For example, the embodiment of the present application first adopts the advanced multimodal visual model Qwen-2.5VL, designs the reasoning steps, and reversely reasons how to use the fine-grained clues collected above (the current lineup on the field, the team affiliation of the jerseys, the faces and jerseys of the players appearing in the close-up shots of the video, and the positioning guidance of key video events) to infer the correct players. This answer-guided visual reasoning method is used to construct training data, and the two fine-tuning strategies of supervised fine-tuning and group relative strategy optimization are used to further optimize the reasoning ability of the multimodal visual model on this task. In the embodiment of the present application, under the prompts of the 30-second video and context, the visual understanding and reasoning ability of the visual language model can be used to infer the team members. Player and team name Team , thereby aligning entities.

[0077] Step 204: perform knowledge enhancement on the detail-enhanced commentary information based on the internal knowledge base and the external knowledge base to obtain knowledge-enhanced football commentary.

[0078] The external knowledge base includes historical statistical data of different football matches; the internal knowledge base includes static background information and dynamic update events of the current football match.

[0079] Exemplarily, the second phase of GS aims to enrich and correct the generated commentary using external historical data and dynamic match data.

[0080] To acquire external knowledge, this embodiment of the application constructs an external knowledge database that integrates historical statistical data, such as game results, player performance, and team results from several past seasons. Using a knowledge-enhanced retrieval generation (KAG) module, using natural language questions as queries, it can retrieve various data types frequently cited by commentators from the external database, generating corresponding historical match data.

[0081] To acquire internal knowledge, the embodiments of the present application utilize fine-grained game data maintained in real time, such as game progress, score changes, etc.; and event information of the game, such as spectator numbers, venues, player personal data, etc., as an internal knowledge base to obtain real-time and dynamic game status information, supporting commentary based on understanding of the game background and previous context.

[0082] Afterwards, an advanced large language model is used to spontaneously generate the precise questions needed to refine the current commentary of the event. Combined with historical match data in an external database (using a knowledge-enhanced retrieval generation module) and real-time updated match status (using contextual learning), a more accurate and in-depth analytical commentary and a more realistic and empathetic commentary are generated.

[0083] Specifically, the above step 204 may further include the following step 204a: Step 204a: Based on the methods by which different commentators apply relevant knowledge during their commentary, natural language questions are used as queries, and historical game data related to the current game is obtained from the external knowledge base through the language model, as well as real-time updated game status obtained based on contextual learning from the internal knowledge base.

[0084] The detail-enhanced commentary information is supplemented and corrected based on the historical game data and the real-time updated game status to obtain the knowledge-enhanced football commentary.

[0085] For example, due to factors such as the commentator's position, speaking habits, personality, and cultural background, live TV commentary presents a variety of styles. Although it is challenging to compare these different commentary styles within a single evaluation framework, human commentators often have a natural preference for incorporating contextual knowledge to improve the accuracy and depth of their insights and to explain and comment on visual scenes. The embodiment of the present application simultaneously analyzes real-time text comments and the English-translated automatic speech recognition (ASR) audio transcription surrounding the timestamp of each real-time text comment. The large language model extracts knowledge through paired explicit query expressions. For example, in the embodiment of the present application, GPT-4o is prompted to identify the previous related knowledge contained in the comments. In addition, the embodiment of the present application also uses a self-questioning mechanism to generate constrained questions based on the extracted knowledge.

[0086] Results show that 15.02% of live TV commentary references various types of knowledge, while only 3.16% of live text commentary contains knowledge, most of which is reported after a goal or at the end of the match. To examine the distribution of knowledge references in human commentary, we employed the same prompts as MatchVision for transcriptions containing knowledge, summarizing the commentary into specific types of standard football events. The primary events that human commentators might reference knowledge about include goals, corner kicks, and card draws. Furthermore, knowledge can be categorized into external and internal knowledge based on the source of information. The former relies on historical statistics from other matches, while the latter refers to static background information and dynamically updated events from the current match. To ensure that the commentary generation model possesses sufficient knowledge similar to that possessed by human commentators, we developed a football KAG system to obtain external statistics and a background match knowledge base to track internal information.

[0087] Modifications to the external football KAG system: The SoccerNet dataset is used to restore data with a well-organized structure, while SoccerRAG is used to interpret structural information into a database format. Building on these excellent pioneering works, we now have a comprehensive database covering six seasons of football matches in the three major leagues. To better align with the search habits of human commentators, this embodiment of the present application improves the football KAG system by incorporating more fine-grained details, increasing the number of query examples, improving SQL query construction, and verifying the execution results. Specifically, the main differences between this application's technical solution and SoccerRAG include: 1. Adding more query examples based on the relevant knowledge used in high-frequency commentary to improve the accuracy of the natural language question description -> SQL query construction. 2. Improving the data structure to increase the dataset's support for fine-grained event-level queries, such as individual goals and cards for players and teams. 3. Adding strict time constraints to the query pattern to protect the database from being affected by future matches when corresponding to the current specified match, and using LLM to carefully check the answer time range.

[0088] Internal Match Context Construction: Unlike the player detection task in scene descriptions, where we provide the jersey numbers of all players in the lineup, here we provide more detailed information about the mentioned players, such as their nationality. To improve the accuracy of ICL and reduce confusion, we separately annotate the key timelines for each event type. In addition, we capture fine-grained information such as the scorer, assist player, and specific goal method (such as penalty, header, or own goal). Other relevant details, including tactics, spectator number, and match venue, are also included to provide comprehensive context.

[0089] Commentary integrated with a large language model: Starting with the first-stage commentary describing current events, we use GPT-4o to generate questions related to external statistical data. The KAG system's responses are then double-checked to eliminate invalid answers. Furthermore, the model is explicitly prompted to reference internal game background knowledge, such as key timelines and detailed player information. This process improves the accuracy and depth of the commentary, bringing it closer to live television commentary.

[0090] The method for generating football commentary based on visual reasoning and knowledge enhancement provided in an embodiment of the present application first obtains anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; then, extracts game status information and fine-grained visual information based on the target video clip, and uses the end-to-end model to obtain event location guidance information; the fine-grained visual information is used to represent detailed information in the game; and visual reasoning is performed based on the game status information, the fine-grained visual information, and the event location guidance information, and the anonymized, coarse-grained commentary information is enhanced in detail based on the visual reasoning results to obtain detail-enhanced commentary information; the visual reasoning results include: identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events; finally, knowledge enhancement is performed on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain a knowledge-enhanced football commentary; wherein the external knowledge base includes: historical statistical data of different football games; and the internal knowledge base includes: static background information and dynamically updated events of the current football game. In this way, through the two-stage processing framework of entity alignment based on visual reasoning and explanation generation based on knowledge enhancement, we can better simulate the workflow of human commentators and generate more natural and informative explanations.

[0091] It should be noted that the method for generating football commentary based on visual reasoning and knowledge enhancement provided in the embodiments of this application can be executed by a football commentary generation device based on visual reasoning and knowledge enhancement, or by a control module within the football commentary generation device based on visual reasoning and knowledge enhancement that is configured to execute the method. In the embodiments of this application, the football commentary generation device based on visual reasoning and knowledge enhancement executing the method is used as an example to illustrate the football commentary generation device based on visual reasoning and knowledge enhancement provided in the embodiments of this application.

[0092] It should be noted that the methods illustrated in the figures of the above-mentioned methods and the methods for generating football commentary based on visual reasoning and knowledge enhancement are each described by way of example in conjunction with one of the figures in the embodiments of this application. In specific implementations, the methods for generating football commentary based on visual reasoning and knowledge enhancement illustrated in the figures of the above-mentioned methods can also be implemented in conjunction with any of the other combinable figures illustrated in the above-mentioned embodiments, and will not be further described here.

[0093] The following describes the football commentary generation device based on visual reasoning and knowledge enhancement provided by this application. The football commentary generation method based on visual reasoning and knowledge enhancement described below and described above can refer to each other.

[0094] Figure 3 This is a structural diagram of a football commentary generation device based on visual reasoning and knowledge enhancement provided in an embodiment of the present application, such as Figure 3 As shown, specifically including: An acquisition module 301 is configured to acquire anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip. An information extraction module 302 is configured to extract game status information and fine-grained visual information based on the target video clip, and to utilize the end-to-end model to acquire event location guidance information. The fine-grained visual information is used to characterize detailed information within the game. A detail enhancement module 303 is configured to perform visual reasoning based on the game status information, the fine-grained visual information, and the event location guidance information, and to perform detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information. The visual reasoning results include identifying and corresponding player and team designations within the game, and matching the occurrence objects of various events. A knowledge enhancement module 304 is configured to perform knowledge enhancement on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain knowledge-enhanced football commentary. The external knowledge base includes historical statistical data from different football matches, and the internal knowledge base includes static background information and dynamically updated events for the current football match.

[0095] Optionally, the game status information includes: temporal team lineup information, key event timeline, and historical timeline; the information extraction module 302 is specifically used to generate the temporal team lineup information, the key event timeline and the historical timeline based on the target video clip and the historical video clip; the information extraction module 302 is also specifically used to update the temporal team lineup information based on the card-playing event and the player replacement event represented by the historical timeline; wherein the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card-playing events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events.

[0096] Optionally, the fine-grained visual information includes: player information, jersey information and team alias information; the information extraction module 302 is specifically used to use a visual language model to classify each shot in the target video clip to obtain shot classification information containing each shot category; the shot categories include: close-up shots, medium shots, long shots and off-field shots; the information extraction module 302 is further specifically used to filter out the facial image of each player from the database based on the player information contained in the temporal team lineup information, obtain a facial image set containing all players' facial images, and use image recognition technology to identify facial images that match the facial images in the facial image set from the medium and close-up shots of the target video clip, and determine the players included in the target video clip; the information extraction module 302 is further specifically used to identify the jerseys appearing in the close-up shots of the target video clip based on optical character recognition technology, and obtain the jersey information and team alias information of the players included in the target video clip; wherein the medium and close-up shots include: close-up shots and / or medium shots.

[0097] Optionally, the information extraction module 302 is specifically used to obtain the query of each cross-attention layer teammate of the end-to-end model, and calculate the average cross-attention between the query and the video frame to obtain the attention weight of each query; the information extraction module 302 is also specifically used to aggregate the attention weight of each query to obtain a frame-level importance vector, and obtain the event positioning guidance information based on the frame-level importance vector; wherein, the frame-level importance vector is used to characterize the importance weight of each video frame in the target video clip when generating anonymized interpretation information.

[0098] Optionally, the detail enhancement module 303 is specifically used to perform visual reasoning based on the key event timeline, the historical timeline, the shot classification information, the facial image set, the players contained in the target video clip, the jersey information, and the participating team alias information, and enhance the details of the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; wherein the visual reasoning results are obtained based on a multimodal visual model.

[0099] Optionally, the knowledge enhancement module 304 is specifically used to obtain historical game data related to the current game from the external knowledge base through a language model based on the application methods of relevant knowledge by different commentators in the commentary process, using natural language questions as queries, and obtain real-time updated game status obtained based on context learning from the internal knowledge base; based on the historical game data and the real-time updated game status, the commentary information with enhanced details is supplemented and corrected to obtain the knowledge-enhanced football commentary.

[0100] The present application provides a football commentary generation device based on visual reasoning and knowledge enhancement. First, anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip is obtained. Then, game status information and fine-grained visual information are extracted based on the target video clip, and event location guidance information is obtained using the end-to-end model. The fine-grained visual information is used to represent detailed information in the game. Visual reasoning is performed based on the game status information, the fine-grained visual information, and the event location guidance information, and detail enhancement is performed on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information. The visual reasoning results include: identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events. Finally, knowledge enhancement is performed on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain a knowledge-enhanced football commentary. The external knowledge base includes historical statistical data from different football games, and the internal knowledge base includes static background information and dynamically updated events of the current football game. In this way, through the two-stage processing framework of entity alignment based on visual reasoning and explanation generation based on knowledge enhancement, we can better simulate the workflow of human commentators and generate more natural and informative explanations.

[0101] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other via the communication bus 440 . The processor 410 may invoke logic instructions in the memory 430 to execute a method for generating football commentary based on visual reasoning and knowledge enhancement, the method comprising: first, obtaining anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; then, extracting game state information and fine-grained visual information based on the target video clip, and using the end-to-end model to obtain event location guidance information; the fine-grained visual information is used to represent detailed information in the game; and visual reasoning is performed based on the game state information, the fine-grained visual information, and the event location guidance information, and detail enhancement is performed on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; the visual reasoning results include identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events; finally, knowledge enhancement is performed on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain a knowledge-enhanced football commentary; wherein the external knowledge base includes historical statistical data from different football games, and the internal knowledge base includes static background information and dynamically updated events of the current football game. In this way, through the two-stage processing framework of entity alignment based on visual reasoning and explanation generation based on knowledge enhancement, we can better simulate the workflow of human commentators and generate more natural and informative explanations.

[0102] In addition, the logical instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0103] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the football commentary generation method based on visual reasoning and knowledge enhancement provided by the above methods, the method comprising: first, obtaining anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; then, extracting game status information and fine-grained visual information based on the target video clip, and using the end-to-end model to obtain event location guidance information; the fine-grained visual information is used to represent detailed information in the game; and Visual reasoning is performed based on the game state information, the fine-grained visual information, and the event location guidance information. The anonymized, coarse-grained commentary information is then enhanced with details based on the visual reasoning results, resulting in detail-enhanced commentary information. The visual reasoning results include identifying and matching player and team designations in the game, as well as matching the occurrence objects of various events. Finally, the detail-enhanced commentary information is knowledge-enhanced based on an internal knowledge base and an external knowledge base, resulting in knowledge-enhanced football commentary. The external knowledge base includes historical statistics from different football matches, while the internal knowledge base includes static background information and dynamically updated events for the current football match. Thus, through a two-stage processing framework of entity alignment based on visual reasoning and commentary generation based on knowledge enhancement, the workflow of a human commentator can be better simulated, generating more natural and informative commentary.

[0104] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned football commentary generation method based on visual reasoning and knowledge enhancement, the method comprising: first, obtaining anonymized, coarse-grained commentary information output by an end-to-end model based on a target video clip; then, extracting game status information and fine-grained visual information based on the target video clip, and using the end-to-end model to obtain event positioning guidance information; the fine-grained visual information is used to represent detailed information in the game; and based on the game status information and the fine-grained visual information, event positioning guidance information is obtained. The system then performs visual reasoning on the anonymized, coarse-grained commentary information and the event location guidance information, and based on the visual reasoning results, performs detail enhancement on the anonymized, coarse-grained commentary information to obtain detail-enhanced commentary information. The visual reasoning results include identifying and matching player and team designations in the game, and matching the occurrence objects of each event. Finally, the detail-enhanced commentary information is subjected to knowledge enhancement based on an internal knowledge base and an external knowledge base to obtain knowledge-enhanced football commentary. The external knowledge base includes historical statistics of different football matches, and the internal knowledge base includes static background information and dynamically updated events of the current football match. In this way, the two-stage processing framework of entity alignment based on visual reasoning and commentary generation based on knowledge enhancement can better simulate the workflow of a human commentator and generate more natural and information-rich commentary.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0106] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A football commentary generation method based on visual reasoning and knowledge enhancement, characterized by: include: Obtain anonymized, coarse-grained explanation information output by the end-to-end model based on the target video clip; Extracting game status information and fine-grained visual information based on the target video clip, and obtaining event location guidance information using the end-to-end model; The fine-grained visual information is used to represent detailed information in the game; performing visual reasoning based on the game state information, the fine-grained visual information, and the event location guidance information, and performing detail enhancement on the anonymized, coarse-grained commentary information according to the visual reasoning result to obtain detail-enhanced commentary information; The visual reasoning results include: identifying and matching player designations and team designations in the game, and matching the occurrence objects of various events; Performing knowledge enhancement on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain a knowledge-enhanced football commentary; The external knowledge base includes historical statistical data of different football matches; the internal knowledge base includes static background information and dynamic update events of the current football match.

2. The method for generating football commentary based on visual reasoning and knowledge enhancement according to claim 1, characterized in that: The game status information includes: temporal team lineup information, key event timeline, and historical timeline; the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events; The extracting of game status information based on the target video segment includes: generating the temporal team lineup information, the key event timeline, and the historical timeline based on the target video clip and the historical video clip; The temporal team lineup information is updated based on the card-playing event and the player replacement event represented by the historical timeline.

3. The method for generating football commentary based on visual reasoning and knowledge enhancement according to claim 2, characterized in that: The fine-grained visual information includes: player information, jersey information, and team nickname information; The extracting fine-grained visual information based on the target video segment includes: Using a visual language model, each shot in the target video clip is classified to obtain shot classification information including each shot category; the shot categories include: close-up shot, medium shot, long shot, and off-scene shot; Based on the player information included in the temporal team lineup information, a facial image of each player is screened from a database to obtain a facial image set including facial images of all players; and using image recognition technology to identify facial images matching the facial images in the facial image set from medium and close-up shots of the target video clip to determine the players included in the target video clip; identifying jerseys appearing in the close-up shot of the target video clip based on optical character recognition technology to obtain jersey information of the players and alias information of the participating teams contained in the target video clip; The medium and close-up shots include: close-up shots and / or medium shots.

4. The method for generating football commentary based on visual reasoning and knowledge enhancement according to claim 3, characterized in that: The obtaining of event location guidance information by using the end-to-end model includes: Obtain the query of each cross-attention layer teammate of the end-to-end model, and calculate the average cross-attention between the query and the video frame to obtain the attention weight of each query; After aggregating the attention weights of each query, a frame-level importance vector is obtained, and the event location guidance information is obtained based on the frame-level importance vector; The frame-level importance vector is used to represent the importance weight of each video frame in the target video segment when generating anonymized interpretation information.

5. The method for generating football commentary based on visual reasoning and knowledge enhancement according to claim 4, characterized in that: The performing visual reasoning based on the game state information, the fine-grained visual information, and the event location guidance information, and performing detail enhancement on the anonymized, coarse-grained commentary information according to the visual reasoning result to obtain detail-enhanced commentary information, includes: performing visual reasoning based on the key event timeline, the historical timeline, the shot classification information, the facial image set, the players included in the target video clip, the jersey information, and the participating team alias information, and performing detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning result to obtain detail-enhanced commentary information; The visual reasoning result is obtained based on a multimodal visual model.

6. The method for generating football commentary based on visual reasoning and knowledge enhancement according to any one of claims 1 to 5, characterized in that: The step of performing knowledge enhancement on the detail-enhanced commentary information based on the internal knowledge base and the external knowledge base to obtain knowledge-enhanced football commentary includes: Based on the methods used by different commentators to apply relevant knowledge during commentary, natural language questions are used as queries, and historical game data related to the current game is obtained from the external knowledge base through a language model, as well as real-time updated game status obtained through contextual learning from the internal knowledge base; The detail-enhanced commentary information is supplemented and corrected based on the historical game data and the real-time updated game status to obtain the knowledge-enhanced football commentary.

7. A football commentary generation device based on visual reasoning and knowledge enhancement, characterized in that: The device comprises: The acquisition module is used to obtain the anonymized, coarse-grained commentary information output by the end-to-end model based on the target video clip; an information extraction module for extracting game status information and fine-grained visual information based on the target video clip, and obtaining event location guidance information using the end-to-end model; the fine-grained visual information is used to represent detailed information in the game; a detail enhancement module configured to perform visual reasoning based on the game state information, the fine-grained visual information, and the event location guidance information, and to perform detail enhancement on the anonymized, coarse-grained commentary information based on the visual reasoning results to obtain detail-enhanced commentary information; the visual reasoning results include identifying and corresponding player designations and team designations in the game, and matching the occurrence objects of various events; A knowledge enhancement module, configured to perform knowledge enhancement on the detail-enhanced commentary information based on an internal knowledge base and an external knowledge base to obtain a knowledge-enhanced football commentary; The external knowledge base includes historical statistical data of different football matches; the internal knowledge base includes static background information and dynamic update events of the current football match.

8. The soccer commentary generation device based on visual reasoning and knowledge enhancement according to claim 7, characterized in that: The game status information includes: temporal team lineup information, key event timeline, and historical timeline; the temporal team lineup information is used to represent the player information of each team currently on the field; the key event timeline is used to represent the timestamps of goal events and card events; the historical timeline is composed of multiple recent commentaries and is used to represent recent events; The information extraction module is specifically configured to generate the temporal team lineup information, the key event timeline, and the historical timeline based on the target video clip and the historical video clip; The information extraction module is further configured to update the temporal team lineup information based on the card-playing event and the player replacement event represented by the historical timeline.

9. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for generating football commentary based on visual reasoning and knowledge enhancement as claimed in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method for generating football commentary based on visual reasoning and knowledge enhancement as claimed in any one of claims 1 to 6 are implemented.

Images