Commentary methods, electronic devices, storage media, and computer program products

By combining multi-camera video streams and structured information, descriptive information is generated, and commentary information is generated using a retrieval system. This solves the problem that viewers cannot fully understand traditional sports commentary and achieves a more accurate and richer commentary effect.

CN123123597APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610976762.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Traditional sports commentary relies on commentators and single-view cameras, which prevents viewers from fully understanding the game and may cause commentators to miss key information. Existing intelligent commentary technology lacks multimodal information fusion, resulting in poor commentary quality.

Method used

The system acquires video streams and structured information from multiple camera positions, generates descriptive information through a model, and combines this with a retrieval system to retrieve and generate narration information. It also utilizes multimodal information to improve the matching degree and accuracy of the narration.

Benefits of technology

Improving the matching degree between commentary information and the state of the game, enhancing the richness and accuracy of commentary information, matching commentary information with user questions, and improving the effect of game commentary.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123597A_ABST
    Figure CN123123597A_ABST
Patent Text Reader

Abstract

This application discloses a method, electronic device, storage medium, and computer program product for commentating on a match. The method includes: acquiring multiple video streams of a first match and structured information of the first match, wherein the multiple video streams are captured from different camera positions among multiple camera angles; obtaining first descriptive information based on the multiple video streams and the structured information using a first model, wherein the first descriptive information describes the footage of the first match; retrieving the first descriptive information and a first question using a first retrieval system to obtain a first retrieval result, wherein the first question represents a question from a first user regarding the first match; and generating first commentary information based on the first retrieval result, wherein the first commentary information is used to commentate on the first match. This application can improve the commentary effect of matches.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for commentating on a competition, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] Traditional sports commentary relies on commentators and a single-view camera, limiting the audience's comprehensive understanding of the game. Furthermore, commentators may miss crucial information due to their limited perspective. To address this, intelligent commentary technology can be employed. Intelligent commentary technology uses artificial intelligence to generate automated, personalized commentary or analysis. Common intelligent commentary techniques include using rule-based systems to generate scripts. These systems can generate scripts based on predefined rules and templates; however, the resulting scripts may not accurately reflect the game's context, leading to a less effective commentary experience. Summary of the Invention

[0003] To address the related technical issues, embodiments of this application provide a method for commentating on a competition, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: Acquire multiple video streams of the first match and the structured information of the first match, wherein the multiple video streams are captured by different camera positions from multiple camera positions; Based on the multiple video streams and the structured information, first description information is obtained through a first model, and the first description information is used to describe the footage of the first match. The first retrieval system retrieves the first description information and the first question to obtain the first retrieval result, wherein the first question is represented as a question from the first user about the first competition. First commentary information is generated based on the first search result, and the first commentary information is used to commentate on the first match.

[0005] In the above scheme, obtaining the first description information based on the multiple video streams and the structured information through the first model includes: The first feature is obtained by extracting features from the multiple video streams using the first model; The structured information is encoded using the first model to obtain a first encoding result; The first historical data is encoded using the first model to obtain a second encoding result. The first historical data includes one or more of the following: historical dialogue records with the first user; historical context of the first model. The first feature, the first encoding result, and the second encoding result are processed by the first model to obtain the first descriptive information.

[0006] In the above scheme, the first feature includes one or more of the following: Events that occurred within the key area of ​​the first match; The movement trajectory information of the participants in the first competition; The first match's overall tactical perspective information; First state information, which describes the state of the participants in the first competition.

[0007] In the above scheme, the first retrieval system is a search enhancement generation (RAG) system, and the step of retrieving the first description information and the first question through the first retrieval system includes: The first retrieval system retrieves information related to the first description information and the first question from the first knowledge base to obtain the first retrieval result. The first knowledge base includes one or more of the following: relevant rules of the first competition; relevant competition data; historical commentary data; and relevant tactical knowledge.

[0008] In the above scheme, generating the first explanatory information based on the first search result includes: The first search result, the first descriptive information, and the first question are processed by the second model to generate the first explanatory information. The first explanatory information is characterized as explanatory information related to the scenario of the first competition and the focus of the first user. The focus is obtained by the second model through analysis of the first question. The focus includes one or more of the following: participants in the first competition who are followed by the first user; areas in the relevant regions of the first competition that are followed by the first user; rules in the relevant rules of the first competition that are followed by the first user; states in the relevant states of the first competition that are followed by the first user; and tactical changes in the tactical changes of the first competition that are followed by the first user. The scenario of the first competition is obtained by the second model through analysis of the first search result and the first descriptive information.

[0009] In the above scheme, obtaining the first description information based on the multiple video streams and the structured information through the first model includes: Based on a first time window, a sequence of video frames within a first time interval is extracted from the multiple video streams, wherein the first time interval is the time interval corresponding to the first time window; The structured information is divided into static information and one or more sets of dynamic information; The static information is used as the background information of the video frame sequence; Based on the time characteristics of each set of dynamic information in the set or multiple sets of dynamic information, target dynamic information matching the video frame sequence is determined from the set or multiple sets of dynamic information. The time characteristics include one or more of the following: the update time of the corresponding dynamic information; the occurrence time of the corresponding event; the effective time interval of the corresponding dynamic information. The video frame sequence, the background information, and the target dynamic information are input into the first model to obtain the first description information.

[0010] In the above scheme, before retrieving the first descriptive information and the first question through the first retrieval system, the method further includes: Receive the first question sent by the first terminal, where the first terminal is the terminal corresponding to the first user; After generating the first explanatory information based on the first search result, the method further includes: The first explanation information is sent to the first terminal, wherein the first explanation information serves as the answer to the first question.

[0011] This application also provides a competition commentary device, including: The first acquisition unit is used to acquire multiple video streams of the first competition and the structured information of the first competition, wherein the multiple video streams are captured by different camera positions among multiple camera positions. The first processing unit is configured to obtain first description information based on the multiple video streams and the structured information through a first model, wherein the first description information is used to describe the footage of the first match. The first retrieval unit is used to retrieve the first description information and the first question through the first retrieval system to obtain the first retrieval result, wherein the first question is characterized as a question from the first user about the first competition; The first generation unit is used to generate first explanatory information based on the first retrieval result, and the first explanatory information is used to explain the first match.

[0012] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor. The processor is used to execute the steps of any of the above-mentioned technical solutions when running the computer program.

[0013] This application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0015] In this embodiment, multiple video streams of a first match and structured information of the first match are acquired, wherein the multiple video streams are captured by different camera positions from multiple camera angles; based on the multiple video streams and the structured information, first descriptive information is obtained through a first model, wherein the first descriptive information is used to describe the footage of the first match; the first descriptive information and a first question are retrieved through a first retrieval system to obtain a first retrieval result, wherein the first question is characterized as a question from a first user about the first match; and first explanatory information is generated based on the first retrieval result, wherein the first explanatory information is used to explain the first match. As can be seen, in this embodiment, the generation of the first description information combines information from two different modalities: multiple video streams and structured information. This makes the generated first description information more closely match the match situation. Since the commentary information in this embodiment is obtained based on the first description information, this embodiment can improve the matching degree between the commentary information and the match status, thereby enhancing the commentary effect. On this basis, this embodiment combines model generation and retrieval. The retrieval system further searches the first description information and the first question generated by the model, and generates commentary information based on the retrieval results. This allows the use of data from the retrieval system to improve the richness and accuracy of the commentary information, and also ensures that the final generated commentary matches the user's question, thereby further improving the commentary effect. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a competition commentary method provided in this application embodiment; Figure 2 A schematic diagram illustrating the process of generating video from a model, provided for an application embodiment of this application; Figure 3 A schematic diagram illustrating the overall steps of an application example provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a competition commentary device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expression, such as one or more, at least one, or at least one item, etc.

[0019] It should also be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0020] It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship such as instruction and being instructed, configuration and being configured.

[0021] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0022] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0023] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0024] The relevant technologies of the embodiments of this application are described below: Traditional sports commentary relies on commentators and a single-view camera, which limits the audience's comprehensive understanding of the game. Furthermore, commentators may miss crucial information due to the limited perspective.

[0025] Besides human commentators, intelligent technology can also be used for match commentary, also known as AI commentary technology. AI commentary is a technology that uses artificial intelligence to generate automated, personalized commentary or analysis. This technology is primarily used in sports events, news reports, and film and television program commentary.

[0026] To achieve intelligent narration, the following methods are provided in related technologies: The first approach is to use rule-based systems for intelligent commentary. Rule-based systems typically generate commentary based on predefined rules and templates. For example, a rule-based system might have a rule that when a football team scores, it generates a commentary similar to "Team A scores!" The advantage of rule-based systems is that they can directly control the output, but the disadvantages are a lack of flexibility and naturalness.

[0027] The second approach involves using deep learning-based systems for intelligent commentary. These systems utilize deep learning models, such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), and Transformer models, to generate commentary. These models can handle more complex language patterns, producing more natural and accurate commentary. In recent years, the emergence of pre-trained language models such as GPT and BERT has significantly improved the performance of such systems. For example, a related paper proposed a deep learning-based system for generating automatic commentary for table tennis matches. This system used RNNs and LSTMs to generate commentary and also employed a method called the "commentary-action mapping model" to map commentary to actions during the match. Another related paper investigated the problem of using deep learning to generate sports commentary. This research used a combined model of a Convolutional Neural Network (CNN) and an LSTM to extract features from match videos and generate commentary based on these features. However, the methods in the aforementioned papers and other related technologies cannot integrate multimodal information, lack understanding of the scene and competition information, and have poor commentary effects.

[0028] The third approach involves using knowledge graph-based systems for intelligent narration. These systems utilize knowledge graphs to provide background information and context, generating more accurate and richer commentary. For example, it might know the historical records of a particular football team and mention this information in the narration. While this approach makes the narration more coherent, the information extracted from the knowledge graph has a low degree of relevance to the context, resulting in poor narration quality.

[0029] The fourth approach is to use multimodal systems for intelligent commentary. Multimodal systems utilize not only textual information but also visual and audio information to generate commentary. For example, it might analyze actions in a video and then generate corresponding commentary. The advantage of multimodal systems is that they can generate more accurate and vivid commentary, but the disadvantage is that they require processing more complex data and models, and single-camera video information is insufficient to ensure accurate commentary. For example, a related paper proposed an automated sports commentary system based on multimodality. This system uses visual and textual information to generate commentary, where visual information is used to detect and recognize actions during the game, and textual information is used to generate commentary. This multimodal system also uses a deep neural network to extract features from visual and textual information. However, these methods suffer from problems such as insufficient feature fusion across multiple modalities, insufficient commentary coherence, lack of user interaction functions, and inability to update commentary knowledge in a timely manner.

[0030] To address at least one of the aforementioned technical problems, this application proposes a method for commentating on a competition. In this method, multiple video streams of a first competition and structured information of the first competition are acquired, the multiple video streams being captured from different camera positions among multiple camera angles; based on the multiple video streams and the structured information, first descriptive information is obtained through a first model, the first descriptive information being used to describe the footage of the first competition; a first retrieval system is used to retrieve the first descriptive information and a first question to obtain a first retrieval result, the first question being characterized as a question from a first user about the first competition; and first commentary information is generated based on the first retrieval result, the first commentary information being used to commentate on the first competition. As can be seen, in this embodiment, the generation of the first description information combines information from two different modalities: multiple video streams and structured information. This makes the generated first description information more closely match the match situation. Since the commentary information in this embodiment is obtained based on the first description information, this embodiment can improve the matching degree between the commentary information and the match status, thereby enhancing the commentary effect. On this basis, this embodiment combines model generation and retrieval. The retrieval system further searches the first description information and the first question generated by the model, and generates commentary information based on the retrieval results. This allows the use of data from the retrieval system to improve the richness and accuracy of the commentary information, and also ensures that the final generated commentary matches the user's question, thereby further improving the commentary effect.

[0031] This application provides a method for commentating on a competition; please refer to [link / reference]. Figure 1 The method includes: Step 101: Obtain multiple video streams of the first match and the structured information of the first match, wherein the multiple video streams are captured by different camera positions from multiple camera positions; It should be noted that the type of the first competition is not limited in the embodiments of this application. The first competition can be any possible competition, such as a sports competition, an e-sports competition, or a dance competition. Taking the first competition as a football match as an example, the first competition can be a European Cup semi-final that is in progress. The participating teams in the first competition can be Spain and Italy. The first competition is in progress at 34 minutes and 14 seconds. The real-time progress of the first competition is that the Spanish team launches a shot attack.

[0032] It should be noted that the embodiments of this application can employ multi-camera shooting for the first match. Multi-camera shooting of the first match refers to simultaneously using multiple cameras to film the first match from different angles or framing perspectives. For example, in the embodiments of this application, multiple video streams can be: Camera 1 (Main Camera): 1920×1080@25fps, focusing on restricted areas Camera 2 (side camera): 1920×1080@25fps, tracking player movement. Camera position 3 (high-altitude camera): 1920×1080@25fps, full-field tactical view Camera position 4 (close-up camera): 1920×1080@25fps, player facial expression capture It is understandable that in the above example, the multiple video streams include four video streams, which are: the video stream shot from camera position 1, the video stream shot from camera position 2, the video stream shot from camera position 3, and the video stream shot from camera position 4.

[0033] For example, each of the multiple video streams is in one-bit MP4 / H.264 encoding format, each video stream has a resolution of 1920×1080@25fps, the number of camera positions is 4-8, and the video latency requirement for the multiple video streams is <200ms. For example, embodiments of this application may also obtain information related to multiple video streams. For example, the information related to multiple video streams may include one or more of the following: timestamp, number of sampled frames, for example, the timestamp may be 34:14.500, the number of sampled frames may be 10 frames per camera position, for example, 1 frame may be sampled at 5 frame intervals, etc.

[0034] For example, the process of this application embodiment is to input the above-mentioned multiple video streams, and obtain the video streams of each camera at the same moment through the multi-camera video frame alignment module. Specifically, multiple video streams or information of multiple video streams can be input. This application embodiment can align the video stream information in time. After alignment, the video frames of multiple video streams are sampled. When sampling the video frames of multiple video streams, sampling can be taken from the current moment backward, with a frame rate of 25, sampling 1 frame at 5 frame intervals, and a sampling quantity of 10 frames / camera. Then, the sampled video frame after multi-camera alignment is returned. For example, the module responsible for aligning the information of multiple video streams can return the aligned sampled video frame to the next module that processes multiple video streams.

[0035] For example, structured information includes one or more of the following: structured match information; knowledge graph; domain knowledge base; wherein, structured match information includes one or more of the following: match overview; match personnel information; match score; match time; match progress; match weather; match location; match status; match events, etc. Match personnel information can be information about players and teams, such as the starting lineup, each player's personal information, team uniform colors, etc. Knowledge graph information can describe the relationships between relevant objects or related knowledge in the first match. For example, knowledge graph information can describe the relationships between different players, and knowledge graph information can also describe the background knowledge of teams in a graph structure, etc.

[0036] It is understood that in the embodiments of this application, multiple video streams and information from other modalities can be fused together using a multimodal model. In order for the multimodal model to better identify the people in the picture, it is necessary to input structured information into the multimodal model. The structured information of the competition (i.e., the structured data of the competition) is obtained through the competition information structuring module and then transmitted to the multimodal model.

[0037] For example, structured match information can be:

[0038] For example, structured match information can be a match overview (match_info), such as the following:

[0039] For example, in addition to obtaining multiple video streams of the first match and the structured information of the first match, first historical data can also be obtained.

[0040] For example, the first historical data may be the historical data of the first competition or the historical data of a competition of the same type as the first competition.

[0041] For example, the first historical data includes one or more of the following: historical conversation records with users (such as the first user); and the historical context of the first model.

[0042] For example, historical dialogue records can be: [33:45] User: "Why wasn't that offside?" [33:46] System: "According to the offside rule, player A was behind an Italian defender when he received the ball, therefore he was not offside." [34:10] User: "How is Player A's performance today?" [34:11] System: "Player A has been active today, completing 3 shots, 2 of which were on target, and covering a distance of 5.2 kilometers." For example, the format of the historical dialogue record is a text sequence, the maximum length of the historical dialogue record is 512 tokens, and the encoding of the historical dialogue record is UTF-8.

[0043] For example, the historical context can be the previously explained text; see [link to previous text]. Figure 2 After the large-scale text model generates the explanatory text, this explanatory text can be input into the first model (i.e., Figure 2 In the multimodal model, the explanatory text input into the first model at this time is equivalent to the previously explanatory text, i.e., the historical explanatory text.

[0044] Step 102: Based on the multiple video streams and the structured information, obtain first description information through the first model. The first description information is used to describe the footage of the first match. For example, the first descriptive information is also called match information description or match description information, etc. For example, the first descriptive information can be a structured match information description, such as {Event: Shot, Team: Team X, Player 1: Player A, Player 2: None, Time: 34:14, Score rewritten: No, Score: 1:0}.

[0045] For example, the first model can be a multimodal model, which can extract features from multiple video streams and structured information and other information of different modalities, and fuse the features of information of different modalities.

[0046] In this embodiment, the input to the first model includes at least two types of data: one is multi-camera video stream, and the other is structured information. To enable the multimodal second model to understand the video content and background at the same moment in the game, the system performs time alignment on the two types of data before inputting them into the first model. See the following practical application for details: In practical applications, obtaining the first description information based on the multiple video streams and the structured information through the first model includes: Based on a first time window, a sequence of video frames within a first time interval is extracted from the multiple video streams, wherein the first time interval is the time interval corresponding to the first time window; The structured information is divided into static information and one or more sets of dynamic information; The static information is used as the background information of the video frame sequence; Based on the time characteristics of each set of dynamic information in the set or multiple sets of dynamic information, target dynamic information matching the video frame sequence is determined from the set or multiple sets of dynamic information. The time characteristics include one or more of the following: the update time of the corresponding dynamic information; the occurrence time of the corresponding event; the effective time interval of the corresponding dynamic information. The video frame sequence, the background information, and the target dynamic information are input into the first model to obtain the first description information.

[0047] It should be noted that the video frame sequence in this embodiment is an exemplary term and can be replaced with any possible term, such as video sequence.

[0048] Understandably, the system first uses the match time or video timestamp as a unified main timeline, and then extracts video sequences from multiple camera positions within the same time interval according to a preset time window (i.e., the first time window), forming a multi-camera video sequence input. For example, the system can take a period before the current moment as the input window, such as 34:12.500-34:14.500, and obtain video frame sequences within this time interval from the main camera position, side camera position, high-altitude camera position, and close-up camera position, respectively. At the same time, the system distinguishes the structured match information data into static and dynamic information. Static information serves as the background input for the entire match, while dynamic information is matched with the time interval of the current video sequence based on its update time, event occurrence time, or effective time interval. After the above alignment, the first model can simultaneously obtain multi-view continuous image information and structured background information of the match within the same time interval. That is, multi-view continuous image information and structured background information of the match within the same time interval can be input into the first model to generate a more accurate structured description of the match footage.

[0049] It is understandable that, since the timestamps of the background information and the target dynamic information are aligned with the timestamps of the video frame sequence, the background information and the target dynamic information are matched in time, and both the background information and the target dynamic information are information that matches the video frame sequence.

[0050] For example, the data input into the first model after alignment can be the following:

[0051] Understandably, the match_info mentioned above can include static information and target dynamic information. For example, static information could be Spain and Italy, while target dynamic information could be information about the players currently on the field.

[0052] For example, the information and structured information of the aligned multiple video streams are input into the first model, and the first model outputs the first description information.

[0053] For example, the first model includes one or more of the following: a video processing branch, a structured information branch; a fusion layer; and an output layer. Understandably, when performing a task, the first model can acquire information from two different input sources (i.e., multiple video streams and structured information) and output initial descriptive information. The first model uses two branches of deep learning: the first branch is a video processing branch, and the second branch is a structured information branch. The video processing branch is used to process video data, while the structured information and historical data processing branch is used to process structured data.

[0054] For example, the video processing branch uses a 3D convolutional neural network (3D-CNN) or a spatio-temporal convolutional neural network (ST-CNN). These networks can capture spatiotemporal information between video frames, such as motion and changes in objects. The output of the video processing branch can be a feature vector that represents the content of the input video.

[0055] For example, the structured information processing branch can use a traditional fully connected neural network (FCN) or a self-attention mechanism (such as a Transformer). Fully connected neural networks or self-attention mechanisms can process structured data, such as starting players, team jersey colors, etc. The output of the structured information processing branch is also a feature vector, which represents the input structured data.

[0056] For example, the above example uses the data processed by the first model to include multiple video streams and structured information. In this embodiment, the data processed by the first model may also include other data, such as first historical data, which is the historical dialogue record with the user (e.g., the first user), the historical context of the first model, etc.

[0057] For example, if the data processed by the first model also includes first historical data, the structured information processing branch can be used to process the first historical data, or an additional branch can be added to process the first historical data.

[0058] For example, the fusion layer of the first model is used to combine the outputs of the video processing branch and the structured information processing branch to form a unified feature representation. This can be achieved through simple concatenation, weighted averaging, or more complex operations (such as feature fusion networks).

[0059] Understandably, when a branch for processing the first historical data is added, the fusion layer is used to combine the outputs of the video processing branch, the structured information processing branch, and the added branch.

[0060] For example, the output layer of the first model uses a Transformer-based text model (such as the Transformer in BERT, GPT, or CLIP) to generate a text description, i.e., the first descriptive information, which is used to describe the first match. The first model can generate the text, i.e., the text of the first descriptive information, by predicting the next word.

[0061] For example, the first model uses a loss function during the training phase of Contrastive Language-Image Pre-training (CLIP), which is the cosine similarity loss between the image and text embeddings. This loss function will encourage the first model to generate a video representation that is semantically close to the target text description.

[0062] For example, the first model can use strategies such as greedy search, bundle search, or sampling to generate a complete output sequence during the generation phase.

[0063] It is understood that the embodiments of this application can realize a multi-source heterogeneous data processing architecture: realize the fusion of three-dimensional features of real-time match scene description (i.e., multiple video streams), historical data (such as historical dialogue data, which can be in the form of text), historical dialogue records (text), and domain knowledge base (such as structured information), breaking through the limitation of a single data source in traditional systems.

[0064] For example, the first description information can be a fused feature vector, the dimension of the first description information is in the range of [1,2048], the data type of the first description information is float32, the update frequency of the first description information is real time, and the update delay shall not exceed 25fps.

[0065] In practical applications, obtaining the first description information based on the multiple video streams and the structured information through the first model includes: The first feature is obtained by extracting features from the multiple video streams using the first model; The structured information is encoded using the first model to obtain a first encoding result; The first historical data is encoded using the first model to obtain a second encoding result. The first historical data includes one or more of the following: historical dialogue records with the first user; historical context of the first model. The first feature, the first encoding result, and the second encoding result are processed by the first model to obtain the first descriptive information.

[0066] It is understood that the embodiments of this application can combine multimodal data such as video streams, structured information and historical data to generate descriptive information, which can integrate multi-dimensional features to improve the narration effect.

[0067] The following describes in detail the data preprocessing and feature extraction process of the first model in this embodiment: For example, video streams can be feature extracted using ST-CNN, with 3D ResNet-50 as the backbone network, as shown in the following formula: F_video = ST-CNN(V_t) = Conv3D(ReLU(BatchNorm(Conv3D(V_t)))) Where V_t: the multi-camera video frame sequence at time t, with dimensions [N,C,T,H,W], where N is the number of cameras, C is the number of channels (RGB=3); T is the length of the time window (10 frames), H and W are the frame height and width respectively, and F_video is the extracted video feature vector with dimensions [N,2048] structured data encoding.

[0068] In practical applications, the first feature includes one or more of the following: Events that occurred within the key area of ​​the first match; The movement trajectory information of the participants in the first competition; The first match's overall tactical perspective information; First state information, which describes the state of the participants in the first competition.

[0069] For example, events occurring within a critical area can be events focused on the critical area, such as goal-line saves or clearances from the penalty area.

[0070] For example, the overall tactical perspective information represents information acquired from higher, wider camera positions or panoramic views to understand the overall situation of the game. It focuses not on the local actions of a particular athlete, but on the spatial distribution, formation changes, and tactical intentions across the entire field.

[0071] For example, the participant status information includes one or more of the following: the participant's facial expression, the participant's actions, and the participant's posture.

[0072] It is understandable that by extracting the above-mentioned multiple types of information, both local details and the overall situation can be obtained simultaneously.

[0073] For example, the structured information of the first competition is processed by a Transformer encoder, as shown in the following formula: F_struct = Transformer_Encoder(Embed(S_t)) Where: S_t: structured information at time t (player information, score, time, etc.), Embed() represents the embedding layer, which is used to convert discrete features into continuous vectors, and F_struct is the structured feature vector with dimensions [1, 768].

[0074] For example, the first historical data is encoded using BERT. The following example uses the first historical data as a sequence such as user historical interaction records or historical dialogue text. For details, please refer to the following example: F_history = BERT_Encoder(H_t) Where H_t is the historical dialogue text sequence, F_history is the historical dialogue feature vector, and F_history is a dimension of [1,768].

[0075] For example, taking the first historical data as historical dialogue records, the first model can extract keywords from the historical dialogue records, and then determine the discussion focus based on the keywords, including the participants, the region, the rules, the status, etc.; then, based on the determined focus, the corresponding explanatory information can be generated and fed back.

[0076] For example, the encoding of the aforementioned multiple video streams, structured information, and first historical data can be performed by the encoding layer of the first model.

[0077] After extracting features from multiple video streams and structured information, feature fusion is required. The multimodal feature fusion mechanism in this application embodiment is described below: First, we introduce the calculation of attention weights. In this embodiment, a multi-head cross-attention mechanism is used to calculate the correlation weights between different modalities: α_ij = softmax(Q_i · K_j^T / √d_k) Where: Q_i: query vector (from modality i), K_j: key vector (from modality j), d_k: key vector dimension, α_ij: attention weight of modality i to modality j.

[0078] For example, modality j can be structured information and first historical data, and modality i can be multiple video streams.

[0079] For example, the importance of each modality can be dynamically adjusted based on the current game state, that is, the weight of each modality can be adjusted. The specific formula can be found in the following formula: w_video = σ(W_v · [F_video; F_struct] + b_v) w_struct = σ(W_s · [F_video; F_struct] + b_s) w_history = σ(W_h · F_history + b_h) Where: σ is the Sigmoid activation function, W_v, W_s, and W_h are learnable weight matrices, b_v, b_s, and b_h are bias vectors, and [;] represents the vector concatenation operation.

[0080] For example, the final fused features (i.e., the first descriptive information) are obtained by weighted summation: F_fused = w_video · F_video + w_struct · F_struct + w_history · F_history Here, F_fused represents the fusion feature.

[0081] The constraint is w_video + w_struct + w_history = 1 Step 103: Search the first description information and the first question through the first retrieval system to obtain the first retrieval result, wherein the first question is represented as a question from the first user about the first competition; For example, the first issue may include one or more of the following: user issues; current issues.

[0082] Step 103 above is an example of searching for the first question. In this embodiment, it is not limited to searching for the first question, nor is it limited to the existence of the first question. For example, when this embodiment is applied to a scenario of automatically generating explanatory text in real time, it is not necessary to search for the first question. Instead, the first description information can be searched directly through the first search system, and explanatory text can be generated based on the search results. For another example, the first question can be replaced with any other possible interactive information, such as a request input by the first user, or instructions conveyed by the first user's actions, expressions, etc.

[0083] For example, the first retrieval system may also be referred to as a retrieval module or retrieval unit. The retrieval module can help the subsequent model to better utilize existing knowledge when generating explanatory information, thereby improving the quality and accuracy of the generated information.

[0084] In practical applications, the first retrieval system is a Retrieval-Augmented Generation (RAG) module. The retrieval of the first descriptive information and the first question through the first retrieval system includes: The first retrieval system retrieves information related to the first description information and the first question from the first knowledge base to obtain the first retrieval result. The first knowledge base includes one or more of the following: relevant rules of the first competition; relevant competition data; historical commentary data; and relevant tactical knowledge.

[0085] Understandably, adding the RAG module can help the model better utilize existing knowledge when generating narration, thereby improving the quality and accuracy of the generated narration.

[0086] Understandably, the goal of the first retrieval system is to retrieve information relevant to the current task from a large knowledge base (i.e., the first knowledge base). In the current task, the first knowledge base could be a database containing a large amount of sports match data, or it could be a pre-trained language model. A vector space-based similarity metric (such as cosine similarity) can be used to retrieve the information most relevant to the description of the current match footage.

[0087] For example, the first retrieval system can index the following data sources in real time: the current match structured description (events, players, scores, etc.), the event knowledge graph (team history, player profiles, rule base), and user historical interaction records (which can be the output of the preference analysis model). The structured information obtained in step 101 may include the current match structured description (events, players, scores, etc.), the event knowledge graph (team history, player profiles, rule base), and the first historical data may include user historical interaction records.

[0088] For example, in this embodiment of the application, the first retrieval system can be RAG. Accordingly, the first retrieval system can use retrieval enhancement methods, such as RAG retrieval of sports knowledge base, to retrieve information related to the current target of interest from external knowledge sources (i.e., the first knowledge base) to supplement the factual basis required for model generation. For example, external knowledge sources (i.e., the first knowledge base) include one or more of the following: rule base, event data, historical commentary segments, tactical knowledge base, etc.

[0089] To improve the accuracy of commentary text and user Q&A content, the system introduces a knowledge graph-enhanced retrieval mechanism before generating commentary information. This mechanism is mainly used to retrieve information related to the current match footage (which can be obtained from multiple video streams), user questions (i.e., the first question), and / or historical commentary context (i.e., the first historical data) from the event knowledge base. This information includes player profiles, team relationships, match rules, historical data, and tactical knowledge. The retrieved information constitutes the first retrieval result, which is used as a supplementary basis for the second model to generate commentary information.

[0090] For example, the knowledge graph-enhanced retrieval mechanism includes entity linking and relational reasoning. Entity linking refers to mapping keywords in the current game scene description or user question to specific entity nodes in the knowledge graph. For instance, the system identifies "player A" and "shot" from "How good was player A's shot?" and links "player A" to the player entity node and "shot" to the game event or technical action node.

[0091] Entities and relations in a knowledge graph can be represented as: Player A – Team – Spain Player A – Position on the field – Forward Player A – Involved in the event – ​​Shot Shot incident – ​​Area where it occurred – Front of the penalty area Offside is a violation of the rules of the game. After the entity linking is completed, the system uses a graph neural network to reason about the entity relationships in the knowledge graph, that is, to perform relational reasoning and obtain extended information related to the current event, the first descriptive information, or the first question. For example, when a user asks "Why wasn't that offside?", that is, the first question is "Why wasn't that offside?", the system not only identifies the rule entity "offside", but also further associates it with the current passing time, the position of the defending player, the position of the attacking player, and the explanation of the offside rule, thereby generating a more accurate answer.

[0092] For example, knowledge graph reasoning can be performed using graph neural networks (GNNs) to determine the entity nodes corresponding to the first question and / or the first descriptive information in the knowledge graph, as well as the related entities and relationships of the first question and / or the first descriptive information in the knowledge graph. The specific formula can be found in the following formula: h_v^{(l+1)} = σ(W^{(l)} · AGG({h_u^{(l)} : u ∈ N(v)})) Where: v represents the entity node that needs to be reasoned about. For example, when the system analyzes "Player A's shot", v can represent the player node "Player A". h_v^(l) represents the representation vector of node v in the l-th layer of the graph neural network. For example, the vector of the "Player A" node can contain information such as its team, position, recent performance, and shot data. N(v) represents the set of neighboring nodes of node v. For example, the neighboring nodes of "Player A" can include "Spain", "forward", "shot event", "penalty area", etc. AGG represents the aggregation function, which is used to summarize the information of neighboring nodes. For example, information from related nodes such as "team", "on-field position", and "current shot event" can be aggregated onto the "Player A" node. W^(l) represents the weight matrix of the l-th layer, which is used to transform the aggregated information so that the model can learn the importance of different relationships. σ represents the activation function, which is used to introduce non-linear expressive power.

[0093] Understandably, through the above reasoning, the system can expand from a single entity to related entities and relationships, thereby obtaining a more complete explanatory basis.

[0094] After completing entity linking and relational reasoning, the system needs to select the most relevant knowledge content from the knowledge base to the current question or match event. This knowledge base can serve as the knowledge source for the first retrieval system. To this end, the system converts the first question, first descriptive information, and / or structured events into query vectors, and converts candidate knowledge from the knowledge base into candidate vectors, then calculates the similarity between the two. The retrieval process is as follows: Current input: "How was the quality of Player A's shot this time?" - Generate query vector q - Calculate similarity with candidate vector k from the knowledge base - Select the knowledge with the highest similarity - Input into the second model to generate an explanation. For example, the knowledge with the highest similarity could be Player A's shot data for this match, shot angle and power information, Player A's historical shot characteristics, etc. The selected knowledge with the highest similarity can be used as the first retrieval result and input into the second model.

[0095] For example, candidate vectors are related to relevant entities and relationships obtained through entity linking and relational reasoning. For instance, if the first question is how good was player A's shot, and the relevant entity obtained through entity linking is that player A's team is Spain, then candidate vectors could be Spain's historical record, etc.

[0096] For example, cosine similarity can be used for knowledge retrieval. That is, the cosine similarity is calculated to determine the similarity between the first question, the first descriptive information, and / or the structured event, etc., and the candidate vector. For example, the following formula can be used for knowledge retrieval: sim(q, k) = (q · k) / (||q|| · ||k||) Where: q represents the query vector, derived from the current first descriptive information, the first dimension Neto, or a combination of both. For example, the user question "How was the quality of Player A's shot this time?" can be converted into the query vector q. k represents the candidate knowledge vector in the knowledge base. For example, knowledge fragments such as "Player A's number of shots in this game," "Player A's historical shot success rate," and "Shot angle evaluation rules" can all be converted into candidate vector k. q·k represents the dot product of the two vectors, used to measure the relevance between the query content and the candidate knowledge. ||q|| and ||k|| represent the lengths of the query vector and the candidate knowledge vector, respectively. sim(q,k) represents the similarity score; the higher the score, the more relevant the candidate knowledge is to the current question or match event.

[0097] For example, when a user asks "Why wasn't that offside?", the system generates a query vector q that is closer to candidate knowledge vectors such as "offside rule," "passing moment," and "defender position," rather than irrelevant knowledge such as "shooting technique" or "player history data." Based on this, the system selects the most relevant rule descriptions and current match event information to assist the second model in generating an accurate answer.

[0098] Adding a Retrieval-Augmented Generation (RAG) module (not strictly necessary) before the second model can help the model better utilize existing knowledge when generating explanatory terms, thereby improving the quality and accuracy of the generated terms. The RAG module consists of a retrieval module and a generation module. Below is the specific process for adding the RAG module: Step 104: Generate first commentary information based on the first search result. The first commentary information is used to commentate on the first match.

[0099] For example, the first commentary information generated could be commentary text, such as: "Player A's shot was of very good quality! From a technical point of view, his shot from the edge of the penalty area had 85% power, and the shooting angle was 15 degrees, which is a relatively tricky angle. From the slow-motion replay, we can see that Player A's shooting motion was very standard, using the inside of his left foot, and the ball's trajectory was low and powerful. Although it was ultimately saved by Donnarumma, it was still a high-quality shooting attempt. It should be noted that Player A's career shooting success rate is 65%, and he is particularly good at poaching shots inside the penalty area. Although he didn't score this time, he demonstrated his excellent shooting technique and finishing ability."

[0100] For example, the first explanatory information can be characterized as explanatory text.

[0101] For example, the format of the first explanatory text is natural language text, the maximum length of the first explanatory text is 200 characters, and the response time of the first explanatory text is <800ms.

[0102] For example, generating the first explanatory information based on the first search result includes: generating the first explanatory information using a second model based on the first search result.

[0103] For example, the second model includes one or more of the following: a large-scale textual model, a question-answering generation model.

[0104] For example, the underlying architecture of the second model is Llama, a large language model based on Transformer. Llama takes a piece of text as input and can handle various NLP tasks, including text generation. In the task of generating the first explanatory information, embodiments of this application use Llama as a conditional generation model.

[0105] For example, the second model is based on the Llama architecture and is adapted to task requirements through two-stage fine-tuning (full fine-tuning and instruction fine-tuning).

[0106] For example, the first retrieval system described above can be used as a retrieval module of the second model, or the retrieval module of the second model can call the first retrieval system. Adding a retrieval module to the second model can help the model better utilize existing knowledge when generating explanatory text, thereby improving the quality and accuracy of the generated text.

[0107] For example, the second model includes a generation module, which is a conditional generation model.

[0108] During the live stream, users can interact at any time. After a user enters a question, the question first goes through the first retrieval system to obtain relevant text information (i.e., the first retrieval result). Then, it combines the video stream, structured information, and first historical data to obtain the competition description (i.e., the first competition description). This description is then input into the second model to obtain the user's answer, which is then returned and displayed. For specific examples, please refer to the following practical applications: In practical applications, before retrieving the first descriptive information and the first question through the first retrieval system, the method further includes: Receive the first question sent by the first terminal, where the first terminal is the terminal corresponding to the first user; After generating the first explanatory information based on the first search result, the method further includes: The first explanation information is sent to the first terminal, wherein the first explanation information serves as the answer to the first question.

[0109] Please see Figure 3 , Figure 3 The question-answering generation model is an example of the second model. The intelligent explanation output by the question-answering generation model is the first explanation information, which can serve as the answer to the user's question.

[0110] It is understood that the above-described solution provided in this application embodiment can be applied to a question-and-answer system, which can automatically generate explanatory information as an answer to user questions to help users understand the first match.

[0111] In this application embodiment, the method of generating the first explanatory information based on the first search result and using the second model can refer to one or more of the following: Method 1: For example, the step of generating the first explanatory information based on the first search result using a second model includes: processing the first search result and additional text information using the second model to generate the first explanatory information.

[0112] For example, taking the second model as a large-scale text model, the design goal of the large-scale text model is to generate a fluent, logically correct commentary containing key information based on a structured description of the match footage and some additional text information. That is, the input of the second model is a structured description of the match footage (i.e., the first description information) and additional text information, and the output is the first commentary information, which can be a single commentary sentence.

[0113] For example, the input to the second model can include two parts: one part is a structured description of the match footage, i.e., the first descriptive information, such as: {Event: Shot, Team: Team X, Player 1: Player A, Player 2: None, Time: 34:14, Score changed: No, Score: 1:0}; the other part is additional text information, such as the previous commentary. This input information will be converted into a text and then fed into the Llama model, i.e., into the second model.

[0114] For example, the input to the second model is the information retrieved by the retrieval module (i.e., the first retrieval result) and the current game screen description (i.e., the first descriptive information), and the output is the generated commentary. In one example, the generation module can also input a first question.

[0115] For example, the generation module can be an Llama model that adapts to task requirements through full fine-tuning and instruction fine-tuning.

[0116] Method 2: Process the first search result, the first description information, and the first question using a second model to generate the first explanation information.

[0117] In practical applications, generating the first explanatory information based on the first search result includes: The first search result, the first descriptive information, and the first question are processed by the second model to generate the first explanatory information. The first explanatory information is characterized as explanatory information related to the scenario of the first competition and the focus of the first user. The focus is obtained by the second model through analysis of the first question. The focus includes one or more of the following: participants in the first competition who are followed by the first user; areas in the relevant regions of the first competition that are followed by the first user; rules in the relevant rules of the first competition that are followed by the first user; states in the relevant states of the first competition that are followed by the first user; and tactical changes in the tactical changes of the first competition that are followed by the first user. The scenario of the first competition is obtained by the second model through analysis of the first search result and the first descriptive information.

[0118] The above practical application takes the analysis of the first problem by the second model as an example. As mentioned above, the embodiments of this application do not limit the existence of the first problem, nor do they limit the analysis of the first problem by the second model. The second model can only input the first search result.

[0119] Understandably, the system can combine real-time interactive content (such as the first question) to identify the explicit or implicit commentary needs expressed by the first user in the current dialogue, and map these needs to the corresponding commentary targets or directions. Examples include user-specified participants, specific areas in the match footage, inquiries about match rules, and changes in the status or tactics of participants the user is interested in. For information that cannot be directly determined from real-time interactive content, the system can further combine multi-camera sensing results, match event data, and enhanced retrieval results for supplementary judgment, thereby determining the commentary content that needs to be prioritized for generation.

[0120] It is understood that in the above practical applications, the embodiments of this application can realize joint modeling of user intent and competition scenario, and can solve the problem of ambiguity in complex questions. In this case, the user intent can be the user's focus target obtained by analyzing the first question through the second model.

[0121] For example, the aforementioned focus can serve as the object of explanation or the direction of explanation, or the object of explanation or the direction of explanation can be derived from the focus.

[0122] Method 3: Process the first search result and the first question using the second model to generate the first explanatory information.

[0123] For example, in this approach, the second model can still determine the first user's focus based on the first question. The scenario is determined based on the first search result; for details, please refer to method 2, which will not be elaborated here.

[0124] For example, the training dataset includes: real-time match question-and-answer pairs, a historical interaction corpus, and a domain knowledge augmentation set. For instance, the real-time match question-and-answer pairs could be sourced from over 500 matches of human commentary, such as the European Championship. The processing of these 500+ match commentary records involves extracting potential question-and-answer relationships from the commentary statements (e.g., "Why was it ruled offside?" → corresponding offside rule explanation). The historical interaction corpus contains 2 million real user question-and-answer records from sports-related smart devices. The annotation dimensions of the historical interaction corpus include question type (rule-based / technical / data-based), sentiment polarity, and follow-up question depth. The domain knowledge augmentation set integrates structured data such as the FIFA official rule base, player career databases, and tactical analysis reports, and constructs a football-related entity relationship graph containing 150,000 entity nodes and 2 million relationship edges.

[0125] For example, the training dataset described above can be used to train the second model, or it can be used for joint training of the first model-retrieval system-second model as a whole.

[0126] For example, in the preliminary stage, the embodiments of this application adopt a course learning strategy, first training basic question answering with rule-based methods (accuracy > 98%), and then progressively adding complex scenarios using a contrastive learning loss function to narrow the vector distance of the same semantic questions (Margin = 0.2).

[0127] For example, the multi-task fine-tuning stage of this application embodiment includes a main task and auxiliary tasks. The main task includes question answering generation (cross-entropy loss), and the auxiliary tasks include intent classification (Focal Loss to solve class imbalance) and knowledge retrieval scoring (Triplet Loss to optimize retrieval relevance). The auxiliary tasks also include assigning weights for joint training, such as assigning the following weights to the main task, intent classification and knowledge retrieval respectively: 0.6, 0.25 and 0.15.

[0128] For example, a user satisfaction reward function can be designed during the reinforcement learning optimization phase: R = 0.7 A+0.2 B+0.1 C Here, A represents the relevance of the answer, which can be evaluated manually; B represents the follow-up question conversion rate, which can be automatically calculated by the system; and C represents the response latency, which can be set with a corresponding threshold. For example, the maximum response latency can be set to 800ms, so the response latency must be <800ms.

[0129] The following describes how embodiments of this application optimize model performance: For example, to optimize computational complexity, knowledge distillation techniques can be used to compress the model (such as the first model, the second model, etc.), as shown in the following formula: L_distill = α · L_CE + (1-α) · L_KD Where: L_CE is the cross-entropy loss, L_KD is the knowledge distillation loss, and α is the balance factor, for example, α can be 0.7.

[0130] For example, for memory optimization, gradient checkpointing can be used to reduce memory usage, specifically: memory_saved = (L-√L) / L × 100% Where memory_saved represents the saved memory, and L represents the number of network layers.

[0131] For example, for a parallel processing architecture, pipelined parallel processing can be used, with a theoretical speedup based on the following formula: Speedup = n / (1 + (n-1) / p) Where Speedup is the theoretical speedup ratio, n is the number of tasks, and p is the number of processors.

[0132] Regarding quality assessment metrics, this application provides several metrics including fusion accuracy, response latency, and user satisfaction. The formula for calculating fusion accuracy can be found in the following formula: Accuracy = (TP + TN) / (TP + TN + FP + FN) Among them, a true positive (TP) characterizes a positive class, and the model (e.g., the first model, the second model) also predicts it as positive; a true negative (TN) characterizes a negative class, and the model also predicts it as negative; a false positive (FP) characterizes a negative class, but the model predicts it as positive; a false negative (FN) characterizes a negative class, but the model predicts it as positive.

[0133] The formula for calculating response latency can be found below: Latency = T_output - T_input Where Latency is the response delay, T_output is the time when the model (e.g., the first model, the second model, etc.) outputs data, and T_input is the time when the model (e.g., the first model, the second model, etc.) inputs data.

[0134] For example, the response latency requirement can be set to be less than 800ms.

[0135] For example, the formula for calculating user satisfaction can refer to the following formula: Satisfaction = 0.4×Relevance + 0.3×Fluency + 0.3×Timeliness Among them, Satisfaction represents user satisfaction, Relevance represents the relevance of the generated first commentary, Fluency represents the fluency of the generated first commentary, and Timeliness represents the timeliness of the generated first commentary.

[0136] It should be noted that the above formulas can be used to optimize the first model, the second model, and other models.

[0137] As can be seen from the above, this application provides a method for intelligent commentary on matches, addressing the problems of lack of flexibility and naturalness in match commentary and poor commentary quality in related technologies. By capturing video information from more angles through multi-camera setups and integrating image information from multimodal video with text information from players and teams, more accurate structured descriptive text is obtained. Furthermore, by leveraging the large-scale text generation capabilities of multimodal models, a more accurate and timely match commentary text is ultimately obtained compared to related technologies. Finally, by utilizing the RAG system, the method effectively supports user interactive question and answer.

[0138] This application's embodiments generate new commentary text based on the query results corresponding to the user's question, real-time video, and historical commentary text. This achieves a novel intelligent commentary method for football matches, not only introducing a new interaction method but also effectively enhancing the viewer's experience. It allows users to receive real-time feedback, encouraging deeper user engagement and increasing user stickiness.

[0139] This application's embodiments creatively construct a multimodal dynamic knowledge-enhanced interactive question-answering system, which has three major differentiating features compared to the unidirectional explanation systems of related technologies: First, a multi-source heterogeneous data processing architecture: achieving three-dimensional feature fusion of real-time match scene description (video stream), historical dialogue records (text), and domain knowledge base (structured information), breaking through the single data source limitation of traditional systems. Second, a dynamic knowledge update mechanism: using the RAG system to index the following data sources in real time: current match structured description (events, players, scores, etc.), event knowledge graph (team history, player profiles, rule base), and user historical interaction records (output of preference analysis model). Third, context-aware question-answering generation: adopting a two-stage attention mechanism. The first stage is timestamp-based semantic alignment to ensure that the answer is synchronized with the match progress, that is, it can align video frames with structured information. The second stage is user intent-match scene joint modeling to solve the ambiguity problem of complex questions.

[0140] This application embodiment can implement a multi-camera spatiotemporal alignment algorithm, i.e., dynamic video stream calibration technology. Specifically, it can be achieved by determining the background information and dynamic information corresponding to the video frames in the structured information. This application embodiment can implement a real-time event data-driven RAG update mechanism. This application embodiment can realize the ability to associate temporal context in user interaction. The system needs to associate the referents of "just now," "this ball," "he," etc. in the user's question with the match timeline, historical commentary text, and structured match events in order to generate an interactive answer that conforms to the current match progress.

[0141] Based on the embodiments described above, this application also provides a competition commentary device, see [link to relevant documentation]. Figure 4 The device includes: The first acquisition unit 401 is used to acquire multiple video streams of the first competition and the structured information of the first competition, wherein the multiple video streams are captured by different camera positions among multiple camera positions. The first processing unit 402 is configured to obtain first description information based on the multiple video streams and the structured information through a first model, wherein the first description information is used to describe the footage of the first match. The first retrieval unit 403 is used to retrieve the first description information and the first question through the first retrieval system to obtain the first retrieval result, wherein the first question is characterized as a question from the first user about the first competition; The first generation unit 404 is used to generate first commentary information based on the first retrieval result, and the first commentary information is used to commentate on the first match.

[0142] In one embodiment, the first processing unit 402 obtains first description information based on the plurality of video streams and the structured information through a first model, including: The first feature is obtained by extracting features from the multiple video streams using the first model; The structured information is encoded using the first model to obtain a first encoding result; The first historical data is encoded using the first model to obtain a second encoding result. The first historical data includes one or more of the following: historical dialogue records with the first user; historical context of the first model. The first feature, the first encoding result, and the second encoding result are processed by the first model to obtain the first descriptive information.

[0143] In one embodiment, the first feature includes one or more of the following: Events that occurred within the key area of ​​the first match; The movement trajectory information of the participants in the first competition; The first match's overall tactical perspective information; First state information, which describes the state of the participants in the first competition.

[0144] In one embodiment, the first retrieval system is a search-enhanced generation (RAG) system, and the first retrieval unit 403 performs retrieval of the first description information and the first question through the first retrieval system, including: The first retrieval system retrieves information related to the first description information and the first question from the first knowledge base to obtain the first retrieval result. The first knowledge base includes one or more of the following: relevant rules of the first competition; relevant competition data; historical commentary data; and relevant tactical knowledge.

[0145] In one embodiment, the first generation unit 404 generates first explanatory information based on the first search result, including: The first search result, the first descriptive information, and the first question are processed by the second model to generate the first explanatory information. The first explanatory information is characterized as explanatory information related to the scenario of the first competition and the focus of the first user. The focus is obtained by the second model through analysis of the first question. The focus includes one or more of the following: participants in the first competition who are followed by the first user; areas in the relevant regions of the first competition that are followed by the first user; rules in the relevant rules of the first competition that are followed by the first user; states in the relevant states of the first competition that are followed by the first user; and tactical changes in the tactical changes of the first competition that are followed by the first user. The scenario of the first competition is obtained by the second model through analysis of the first search result and the first descriptive information.

[0146] In one embodiment, the first processing unit 402 obtains first description information based on the plurality of video streams and the structured information through a first model, including: Based on a first time window, a sequence of video frames within a first time interval is extracted from the multiple video streams, wherein the first time interval is the time interval corresponding to the first time window; The structured information is divided into static information and one or more sets of dynamic information; The static information is used as the background information of the video frame sequence; Based on the time characteristics of each set of dynamic information in the set or multiple sets of dynamic information, target dynamic information matching the video frame sequence is determined from the set or multiple sets of dynamic information. The time characteristics include one or more of the following: the update time of the corresponding dynamic information; the occurrence time of the corresponding event; the effective time interval of the corresponding dynamic information. The video frame sequence, the background information, and the target dynamic information are input into the first model to obtain the first description information.

[0147] In one embodiment, the device further includes a first receiving unit and a first sending unit. Before the first retrieval unit 403 retrieves the first description information and the first question through the first retrieval system, the first receiving unit is used to receive the first question sent by the first terminal, where the first terminal is the terminal corresponding to the first user. After the first generation unit 404 generates the first explanatory information based on the first search result, the first sending unit is used to send the first explanatory information to the first terminal, wherein the first explanatory information serves as the answer to the first question.

[0148] In practical applications, the first acquisition unit 401, the first processing unit 402, the first retrieval unit 403, and the first generation unit 404 can be implemented by the processor in the competition commentary device, and the first receiving unit and the first sending unit can be implemented by the processor in the competition commentary device in combination with the communication interface.

[0149] It should be noted that the above-described examples of the match commentary device are only illustrative of the division of program modules during information processing. In practical applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the match commentary device and the match commentary method embodiments described above belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0150] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, see [link to relevant documentation]. Figure 5 The electronic device includes: The first communication interface 1 is capable of exchanging information with other devices; The first processor 2 is connected to the first communication interface 1 to enable information interaction with other devices. When running a computer program, it executes the methods provided by one or more technical solutions in the above embodiments. The computer program is stored in the first memory 3.

[0151] Specifically, the first processor 2 is used to acquire multiple video streams of the first match and structured information of the first match, wherein the multiple video streams are captured by different camera positions from multiple camera positions; the first processor 2 is used to obtain first descriptive information based on the multiple video streams and the structured information through a first model, wherein the first descriptive information is used to describe the footage of the first match; the first processor 2 is used to search the first descriptive information and a first question through a first retrieval system to obtain a first retrieval result, wherein the first question is characterized as a question from a first user about the first match; the first processor 2 is used to generate first explanatory information based on the first retrieval result, wherein the first explanatory information is used to explain the first match.

[0152] In one embodiment, the first processor 2 obtains first description information based on the plurality of video streams and the structured information through a first model, including: The first feature is obtained by extracting features from the multiple video streams using the first model; The structured information is encoded using the first model to obtain a first encoding result; The first historical data is encoded using the first model to obtain a second encoding result. The first historical data includes one or more of the following: historical dialogue records with the first user; historical context of the first model. The first feature, the first encoding result, and the second encoding result are processed by the first model to obtain the first descriptive information.

[0153] In one embodiment, the first feature includes one or more of the following: Events that occurred within the key area of ​​the first match; The movement trajectory information of the participants in the first competition; The first match's overall tactical perspective information; First state information, which describes the state of the participants in the first competition.

[0154] In one embodiment, the first retrieval system is a search-enhanced generation (RAG) system, and the first processor 2 searches for the first description information and the first question through the first retrieval system, including: The first retrieval system retrieves information related to the first description information and the first question from the first knowledge base to obtain the first retrieval result. The first knowledge base includes one or more of the following: relevant rules of the first competition; relevant competition data; historical commentary data; and relevant tactical knowledge.

[0155] In one embodiment, the first processor 2 generates first explanatory information based on the first search result, including: The first search result, the first descriptive information, and the first question are processed by the second model to generate the first explanatory information. The first explanatory information is characterized as explanatory information related to the scenario of the first competition and the focus of the first user. The focus is obtained by the second model through analysis of the first question. The focus includes one or more of the following: participants in the first competition who are followed by the first user; areas in the relevant regions of the first competition that are followed by the first user; rules in the relevant rules of the first competition that are followed by the first user; states in the relevant states of the first competition that are followed by the first user; and tactical changes in the tactical changes of the first competition that are followed by the first user. The scenario of the first competition is obtained by the second model through analysis of the first search result and the first descriptive information.

[0156] In one embodiment, the first processor 2 obtains first description information based on the plurality of video streams and the structured information through a first model, including: Based on a first time window, a sequence of video frames within a first time interval is extracted from the multiple video streams, wherein the first time interval is the time interval corresponding to the first time window; The structured information is divided into static information and one or more sets of dynamic information; The static information is used as the background information of the video frame sequence; Based on the time characteristics of each set of dynamic information in the set or multiple sets of dynamic information, target dynamic information matching the video frame sequence is determined from the set or multiple sets of dynamic information. The time characteristics include one or more of the following: the update time of the corresponding dynamic information; the occurrence time of the corresponding event; the effective time interval of the corresponding dynamic information. The video frame sequence, the background information, and the target dynamic information are input into the first model to obtain the first description information.

[0157] In one embodiment, before the first processor 2 searches for the first description information and the first question through the first retrieval system, the first communication interface 1 is used to receive the first question sent by the first terminal, where the first terminal is the terminal corresponding to the first user. After the first processor 2 generates the first explanatory information based on the first search result, the first communication interface 1 is used to send the first explanatory information to the first terminal, wherein the first explanatory information serves as the answer to the first question.

[0158] It should be noted that the above-described examples of the match commentary device are only illustrative of the division of program modules during information processing. In practical applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the match commentary device and the match commentary method embodiments described above belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0159] It should be noted that the specific processing procedure of the first communication interface 1 can be understood by referring to the above method.

[0160] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as Bus System 4.

[0161] The first memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0162] The methods disclosed in the embodiments of this application can be applied to the first processor 2, or implemented by the first processor 2. The first processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 2. The first processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 2 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 3. The first processor 2 reads the information in the first memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0163] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

[0164] It is understood that the memory (e.g., the first memory 3) in the embodiments of this application can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0165] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 3 storing a computer program, which can be executed by a first processor 2 of an electronic device to complete the steps described in the aforementioned electronic device-side method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0166] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a first processor 2 of an electronic device to perform the steps described in any of the foregoing methods.

[0167] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for commentating on a match, characterized in that, The methods for commentating on the match include: Acquire multiple video streams of the first match and the structured information of the first match, wherein the multiple video streams are captured by different camera positions from multiple camera positions; Based on the multiple video streams and the structured information, first description information is obtained through a first model, and the first description information is used to describe the footage of the first match. The first retrieval system retrieves the first description information and the first question to obtain the first retrieval result, wherein the first question is represented as a question from the first user about the first competition. First commentary information is generated based on the first search result, and the first commentary information is used to commentate on the first match.

2. The method according to claim 1, characterized in that, The step of obtaining first description information based on the multiple video streams and the structured information through a first model includes: The first feature is obtained by extracting features from the multiple video streams using the first model; The structured information is encoded using the first model to obtain a first encoding result; The first historical data is encoded using the first model to obtain a second encoding result. The first historical data includes one or more of the following: historical dialogue records with the first user; historical context of the first model. The first feature, the first encoding result, and the second encoding result are processed by the first model to obtain the first descriptive information.

3. The method according to claim 2, characterized in that, The first feature includes one or more of the following: Events that occurred within the key area of ​​the first match; The movement trajectory information of the participants in the first competition; The first match's overall tactical perspective information; First state information, which describes the state of the participants in the first competition.

4. The method according to claim 1, characterized in that, The first retrieval system is a search-enhanced generation (RAG) system. The step of retrieving the first descriptive information and the first question using the first retrieval system includes: The first retrieval system retrieves information related to the first description information and the first question from the first knowledge base to obtain the first retrieval result. The first knowledge base includes one or more of the following: relevant rules of the first competition; relevant competition data; historical commentary data; and relevant tactical knowledge.

5. The method according to claim 1, characterized in that, The generation of first explanatory information based on the first search result includes: The first search result, the first descriptive information, and the first question are processed by the second model to generate the first explanatory information. The first explanatory information is characterized as explanatory information related to the scenario of the first competition and the focus of the first user. The focus is obtained by the second model through analysis of the first question. The focus includes one or more of the following: participants in the first competition who are followed by the first user; areas in the relevant regions of the first competition that are followed by the first user; rules in the relevant rules of the first competition that are followed by the first user; states in the relevant states of the first competition that are followed by the first user; and tactical changes in the tactical changes of the first competition that are followed by the first user. The scenario of the first competition is obtained by the second model through analysis of the first search result and the first descriptive information.

6. The method according to claim 1, characterized in that, The step of obtaining first description information based on the multiple video streams and the structured information through a first model includes: Based on a first time window, a sequence of video frames within a first time interval is extracted from the multiple video streams, wherein the first time interval is the time interval corresponding to the first time window; The structured information is divided into static information and one or more sets of dynamic information; The static information is used as the background information of the video frame sequence; Based on the time characteristics of each set of dynamic information in the set or multiple sets of dynamic information, target dynamic information matching the video frame sequence is determined from the set or multiple sets of dynamic information. The time characteristics include one or more of the following: the update time of the corresponding dynamic information; the occurrence time of the corresponding event; the effective time interval of the corresponding dynamic information. The video frame sequence, the background information, and the target dynamic information are input into the first model to obtain the first description information.

7. The method according to claim 1, characterized in that, Before retrieving the first descriptive information and the first question through the first retrieval system, the method further includes: Receive the first question sent by the first terminal, where the first terminal is the terminal corresponding to the first user; After generating the first explanatory information based on the first search result, the method further includes: The first explanation information is sent to the first terminal, wherein the first explanation information serves as the answer to the first question.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.