EXPLANATION VIDEO GENERATION METHOD, DEVICE, SERVER AND COMPUTER PROGRAM

Abstract

A commentary video generating method applied to a commentary server and belonging to the field of artificial intelligence, includes: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object performing a battle action in a battle; generating a commentary data stream based on the battle command frame, the commentary data stream including commentary audio describing at least one battle event, the battle event being triggered when the virtual object performs a battle action; rendering a battle screen based on the battle command frame, generating a battle video stream, the battle video stream including at least one battle video frame; merging the commentary data stream and the battle video stream to generate a commentary video stream, wherein the battle video frames and the commentary audio corresponding to the same battle event in the commentary video stream are time-aligned.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on December 25, 2020, bearing application number 202011560174.5 and entitled "Explanatory video generation method, apparatus, server and storage medium", the entire contents of which are incorporated into this application by reference.

[0002] TECHNICAL FIELD The embodiments of the present application relate to the field of artificial intelligence, and in particular to an instructional video generating method, device, server and storage medium. [Background technology]

[0003] With the rapid development of live streaming technology, live streaming has become a common entertainment and communication method in daily life, and game live streaming has become one of the more popular live streaming methods at present.

[0004] Currently, during live game streaming, the live game streamer needs to provide game commentary based on the game match situation. In the process of generating the game commentary video, processes such as game segment selection, commentary text creation, video editing, audio generation, video synthesis, etc. are manually performed in advance to generate the commentary video, and then the commentary is played back.

[0005] However, the game commentary process in the related art requires human participation in the process of creating commentary videos, which results in a long production flow and high manual operation costs. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide an instructional video generating method, device, server and storage medium, which can reduce the operation cost of generating instructional videos, and the technical solutions are as follows: [Means for solving the problem]

[0007] A commentary video generation method executed by a commentary server, comprising: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs a combat action; A step of performing a battle screen rendering based on the battle command frame and generating a battle video stream, the battle video stream including at least one battle video frame; The method includes a step of merging the commentary data stream and the match video stream to generate a commentary video stream, wherein the match video frames and the commentary audio corresponding to the same match event in the commentary video stream are time-aligned.

[0008] An explanatory video generation device, comprising: an acquisition module for acquiring a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; a first generation module for generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio for explaining at least one combat event, the combat event being used to be triggered when the virtual object performs a combat action; A second generating module is used to perform battle screen rendering based on the battle command frame, generate a battle video stream, and ensure that at least one battle video frame is included in the battle video stream; and a third generation module that merges the commentary data stream with the match video stream to generate a commentary video stream, the commentary video frames and the commentary audio corresponding to the same match event in the commentary video stream being used to time-align.

[0009] 1. A server including a memory having computer readable instructions stored thereon and one or more processors, the computer readable instructions, when executed by the processor, causing the one or more processors to: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs a combat action; performing battle screen rendering based on the battle command frame to generate a battle video stream, the battle video stream including at least one battle video frame; The commentary data stream and the match video stream are merged to generate a commentary video stream, and the commentary audio and the match video frames corresponding to the same match event in the commentary video stream are time-aligned.

[0010] One or more non-volatile readable storage media having computer readable instructions stored thereon, The computer readable instructions, when executed by one or more processors, cause the one or more processors to: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs the combat action; performing battle screen rendering based on the battle command frame to generate a battle video stream, the battle video stream including at least one battle video frame; The commentary data stream and the match video stream are merged to generate a commentary video stream, and the commentary audio and the match video frames corresponding to the same match event in the commentary video stream are time-aligned.

[0011] A computer program product or computer program comprising computer instructions stored on a computer-readable storage medium, wherein a processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, thereby causing the computing device to perform the instructional video generation method provided in the various embodiments above. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an illustrative system architecture diagram shown in one exemplary embodiment of the present application. [Diagram 2] 2 is a flowchart of an instructional video generation method according to an exemplary embodiment of the present application; [Diagram 3] 4 is a flowchart of an instructional video generating method according to another exemplary embodiment of the present application. [Figure 4] FIG. 13 is a setting interface diagram of preset attribute information corresponding to a preset battle event. [Diagram 5] FIG. 2 is a schematic diagram of a match process of a combat video frame and a combat instruction frame shown in an exemplary embodiment of the present application; [Figure 6]4 is a flowchart of a method for identifying a target match event according to an exemplary embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of a match video frame shown in an exemplary embodiment of the present application; [Figure 8] 4 is a flowchart of an instructional video generating method according to another exemplary embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of a process for completely generating a commentary video stream as shown in an exemplary embodiment of the present application. [Figure 10] 1 is a block diagram of a configuration of an explanatory video generating device shown in an exemplary embodiment of the present application. [Figure 11] FIG. 2 is a block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The commentary video generation method presented in the embodiments of this application mainly relates to several aspects of computer vision technology, voice processing technology, and natural language processing technology in the above artificial intelligence software technology.

[0014] Referring to Figure 1, a commentary system architecture diagram shown in an exemplary embodiment of the present application is shown, where the commentary system includes at least one competition terminal 110, a commentary server 120, and a live streaming terminal 130, and the commentary system in the embodiment of the present application is applied to a virtual online commentary scene.

[0015] The battle terminal 110 is a device in which a game-based application program is installed. The game-based application program may be a sports game, a military simulation program, a multiplayer online battle arena (MOBA) game, a big escape shooting game, a simulation game (SLG), etc., and the embodiment of the present application does not limit the type of the game-based application program. The battle terminal 110 may be a smartphone, a tablet, a personal computer, etc. In the embodiment of the present application, in a virtual online commentary game scene, when the battle terminal 110 runs a game-based application program, the user can control the virtual object to perform a battle action in the battle by the battle terminal 110, and accordingly, the battle terminal 110 receives a battle operation command for the user to control the virtual object, and transmits the battle operation command to the commentary server 120, so that the commentary server 120 can perform battle rendering in the commentary server 120 based on the received battle operation command.

[0016] The competition terminal 110 is directly or indirectly connected to the commentary server 120 via a wired or wireless communication method.

[0017] The commentary server 120 is a background server or a business server of a game-based application program, and is used to perform online game commentary and push commentary video streams to other live distribution platforms or live distribution terminals. It may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, and may also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In an embodiment of the present application, the commentary server 120 can be used to receive battle operation commands (battle command frames) sent from multiple battle terminals 110, for example, the commentary server 120 can receive battle operation commands sent from the battle terminals 112 and 111. Based on the analysis of the battle command frames, a commentary data stream is generated, while performing online battle rendering based on the battle command frames, generating a battle video stream in real time, merging the commentary data stream with the battle video stream, generating a commentary data stream, and pushing it to the live distribution terminal 130.

[0018] Preferably, based on the design of the server architecture, the commentary server 120 may include a battle video stream generation server (used to render battle images based on battle command frames and record and generate a battle video stream), a commentary data stream generation server (used to generate a commentary data stream based on battle command frames), and a commentary video stream generation server (used to generate a commentary video stream based on the battle video stream and the commentary data stream).

[0019] The live streaming terminal 130 is directly or indirectly connected to the commentary server 120 via a wired or wireless communication method.

[0020] The live streaming terminal 130 may be a device that runs a live streaming client or a video client, or may be a background server corresponding to the live streaming client or the video client. In an embodiment of the present application, if the live streaming terminal 130 is a device that runs a live streaming client or a video client, it can receive the commentary video stream sent from the commentary server 120, decode the commentary video stream, and play the commentary video on the live streaming client or the video client. Preferably, if the live streaming terminal 130 is a background server corresponding to the live streaming client or the video client, the live streaming terminal 130 can correspondingly receive the commentary video stream sent from the commentary server 120 and push the commentary video stream to the corresponding live streaming client or the video client.

[0021] Referring to FIG. 2, a flowchart of the commentary video generation method shown in an exemplary embodiment of the present application is shown. The embodiment of the present application describes the application of the method to the commentary server shown in FIG. 1 as an example, and the method includes:

[0022] In step 201, a battle command frame is obtained, the battle command frame includes at least one battle operation command, and the battle operation command is used to control a virtual object to perform a battle action in a battle.

[0023] Unlike the related art, in which after a match is completed, commentary text is prepared based on the match video, the commentary text is converted into audio for broadcast, and a commentary video is generated, the application scene of the embodiment of the present application is an online match commentary scene, that is, during the match process, the commentary server automatically generates a corresponding commentary video stream, and pushes the commentary video stream to the live broadcast side for broadcast, thereby improving the timeliness of the generation of the commentary video, and in order to generate the corresponding commentary video in real time during the match process, in a possible embodiment, online match video rendering and online analytical commentary can be realized through analysis of the match command frame.

[0024] Among them, the battle command frame includes at least one battle operation command, which is used to control the virtual object to perform battle actions in a battle, and the battle action refers to controlling the action of the virtual object after the user enters the battle, such as controlling the movement of the virtual object in the virtual environment, controlling the virtual object to use skills, and controlling the virtual object to perform preset game actions.

[0025] Wherein, the terminal can control the virtual object to perform combat actions in a battle according to the battle operation command. For example, when a user launches a game application and touches a control for performing a skill in the game application through the terminal, the terminal can generate a battle operation command according to the user's touch operation, and control the virtual object to perform a skill according to the battle operation command.

[0026] In one embodiment, the battle operation commands are defined in the form of frames, and each battle command frame can include multiple battle operation commands for in-game elements such as player characters and non-player characters (NPCs).

[0027] In step 202, a commentary data stream is generated based on the combat command frame, and the commentary data stream includes commentary audio describing at least one stage of combat events, the combat events being triggered when the virtual object performs a combat action.

[0028] In order to realize online battle commentary and generate commentary videos in real time, an embodiment of the present application provides an online game understanding technology, that is, based on the online battle process of analyzing and understanding the battle command frame, the battle events that need to be explained during the battle process can be obtained.

[0029] Since the battle command frame is a collection of battle operation commands, in a possible embodiment, the commentary server analyzes each battle operation command contained in the battle command frame to accurately calculate the change state of each object attribute value in the virtual environment after receiving each battle command frame, and mines therefrom battle events that require commentary, thereby generating commentary text based on the battle events, and converting the commentary text into commentary audio, thereby realizing a process of generating a commentary data stream through analysis of the battle command frame.

[0030] In one embodiment, the commentary data stream can include commentary text in addition to the commentary audio, so that the commentary text can be added to corresponding commentary video frames in the commentary video stream later when the commentary video stream is synthesized.

[0031] In one illustrative example, if the battle operation command included in the battle command frame is "Sink xx explodes one mixed bomb", the commentary server can calculate information such as the corresponding position, HP (hit points), etc. of each element in the battle in the battle operation command, and based on the calculated position, HP (hit points), etc., when it is determined that there is a large HP (hit points) decrease after a virtual object triggers the mixed bomb in the battle, it can correspondingly determine that the battle event is "Sink xx explodes one mixed bomb, high injury" by analyzing the battle command frame, thereby further generating commentary audio explaining the battle event.

[0032] In step 203, a battle screen rendering is performed based on the battle command frame, and a battle video stream is generated, in which at least one battle video frame is included.

[0033] Based on the principle of generating commentary videos online, when users control virtual objects on different game clients to compete in a game, if it is necessary to generate a commentary video online that is the same as the game process, it is necessary to correspondingly render the game screen in real time, so that there is no need to obtain the game video after the game ends and process the game video to generate the commentary video, and further improve the real-timeness and timeliness of the generation of the commentary video.

[0034] When a user plays a game using a game client installed on a terminal (mobile phone), the game client actually renders attribute changes of each object or element in the game in real time based on the battle operation commands received and the battle operation commands from other users transferred from the server (a background server or business server corresponding to the game client). Based on the above game battle rendering process, in a possible embodiment, a game client may be installed on a commentary server, which is used to receive battle operation commands of the game clients operated by other users, and based on these battle operation commands, it is necessary to render the battle screen in real time and finally generate a commentary video, so that the rendered battle screen needs to be recorded, thereby generating a battle video stream including battle video frames.

[0035] It should be noted that step 202 and step 203 may be performed simultaneously, step 202 may be performed first and then step 203, or step 203 may be performed first and then step 202, and the embodiment of the present application does not limit the order of performing step 202 and step 203.

[0036] In step 204, the commentary data stream and the match video stream are merged to generate a commentary video stream, where match video frames and commentary audio corresponding to the same match event in the commentary video stream are time-aligned.

[0037] In the process of generating the online commentary video provided by this embodiment, the commentary server generates two data streams, one is a commentary data stream and the other is a battle video stream. Due to the differences in the processing processes of the two data, for example, in the process of generating the commentary data stream, an analysis process of the battle command frame needs to be performed, which makes the generation speed slow. In addition, the battle video stream is launched, rendered and recorded after the player loads the game, while the commentary data stream is processed after the battle. Therefore, based on the difference in the processing speed of the two data streams, in the process of synthesizing the commentary video, it is necessary to adapt to the difference in the processing speed of the two data streams and align the two data streams according to a standard to synchronize with the time axis of the commentary video. That is, in the process of generating the commentary video, the commentary server needs to align the battle video frames and commentary audio corresponding to the same battle event in time, that is, it needs to display the battle video frames corresponding to the battle event and start playing the commentary audio corresponding to the battle event at the same time.

[0038] As described above, in the embodiment of the present application, the commentary audio is generated by analyzing the battle command frame online, the commentary audio is rendered to match the battle video, and the commentary video is generated by time-matching the commentary audio with the battle video. By analyzing the battle command frame to generate the commentary video, the commentary video matching the battle can be generated during the battle process, and there is no need to generate the commentary video after the battle, improving the timeliness of the commentary video generation. By generating the commentary video matching the battle during the battle, the need to first record and store the battle process video and then generate the commentary video can be avoided, and the power and storage resources required for recording and storage can be saved. Meanwhile, there is no need to manually create commentary text to generate the commentary video, and an automated commentary video generation process can be realized, which further improves the efficiency of generating the commentary video, improves the matching degree between the commentary video and the battle, and effectively reduces the need for correction due to non-matching, saving the power and calculation resources required for correcting the commentary video.

[0039] Due to the difference in data processing speed between the battle video stream and the commentary data stream, there is a time difference between the commentary data stream and the battle video stream. In the process of synthesizing the commentary video stream, if only the start time of the battle video stream and the commentary data stream are aligned, it is obviously not possible to guarantee that the battle event described by the playing commentary audio is displayed in the displayed battle video frame. Therefore, in a possible embodiment, when time-aligning the battle video stream and the commentary data stream, the commentary server needs to analyze and obtain the correspondence between the battle video frames and the commentary audio, and time-align the battle video frames and commentary audio corresponding to the same battle event.

[0040] Referring to Figure 3, a flowchart of the commentary video generation method shown in another exemplary embodiment of the present application is shown, and the embodiment of the present application describes the application of the method to the commentary server shown in Figure 1 as an example, and the method includes:

[0041] In step 301, a battle command frame is obtained, the battle command frame includes at least one battle operation command, and the battle operation command is used to control a virtual object to perform a battle action in a battle.

[0042] Wherein, the fighting command frame corresponds to a first frame rate, i.e., the fighting command frame is refreshed or obtained according to the first frame rate. In an illustrative example, if the first frame rate is 30 FPS, a fighting command frame is correspondingly obtained every 33 ms, or the time interval between adjacent fighting command frames is 33 ms, and each fighting command frame correspondingly includes a fighting operation command generated within 33 ms.

[0043] In a possible embodiment, the commentary server receives or acquires battle command frames based on a first frame rate, performs battle analysis based on the battle command frames, and acquires attribute information of each object in the battle after performing a battle action corresponding to the battle operation command.

[0044] In step 302, a preset battle event set is obtained, the battle event set includes a plurality of preset battle events, and based on a battle command frame, a virtual object is controlled to perform a corresponding battle action in the battle, and attribute information of each virtual object in the battle after performing the battle action is determined.

[0045] Among them, the attribute information may include position information, HP (hit point) information, speed information, level information, skill information, battle record information, equipment information, score information, etc. of each virtual object in a battle, and the embodiments of the present application do not limit the information types specifically included in the attribute information.

[0046] In a possible embodiment, after receiving a battle command frame, the commentary server controls the virtual objects to perform corresponding battle actions in the battle based on each battle operation command contained in the battle command frame, and after controlling the virtual objects to perform corresponding battle actions in the battle based on the battle command frame, the commentary server accurately calculates attribute information of each object in the virtual environment with each battle operation command, and analyzes and mines battle events for commentary based on the attribute information.

[0047] Preferably, each object in a match may include a virtual object controlled by the user (i.e., a player character), a virtual object controlled by the background (a non-player character NPC), or various virtual buildings within a virtual object, and the embodiments of the present application do not limit the object types included within a match.

[0048] In one illustrative example, if a battle action is "a red / blue BUFF in which a home team hero shoots an away team hero," correspondingly, after the battle action is performed, the attribute information of each object in the match obtained includes "information such as the home team hero HP (hit points), the away team hero HP (hit points), the away team hero position, and the away team hero equipment."

[0049] In one embodiment, the commentary server can preset the attribute information type (attribute information type, i.e., commentary feature dimension) that needs to be analyzed in the online commentary process, so that the required attribute information can be obtained in the online commentary process based on the preset commentary feature dimension.

[0050] In an illustrative example, taking a multiplayer online tactical competitive game as an example, the obtained attribute information can be summarized into four categories, namely, player character (virtual object controlled by a user), NPC, battle, stats, etc. Corresponding attribute information for each category is subdivided, for example, for a battle category, the corresponding attribute information can include battle location, virtual objects included in the battle (type of virtual object and number of virtual objects), battle type, battle purpose, battle time, battle result, etc. For a single virtual object, the corresponding attribute information can include HP (hit points), level, position, equipment, skill, battle record, etc. For an NPC, the corresponding attribute information can include HP (hit points), position, attack skill, etc. For a statistic category, the corresponding attribute information can include score, tower number, win rate, etc.

[0051] In step 303, a plurality of preset battle events are screened for at least one candidate battle event that matches the attribute information.

[0052] In order to realize online battle event mining and understanding, in one embodiment, the commentary server pre-analyzes noteworthy battle events in the commentary scene, presets these battle events in the commentary server to obtain a preset battle event set, and sets corresponding preset attribute information (the preset attribute information is also a preset condition for triggering the preset battle event) for each preset battle event in the preset battle event set, so that during the online commentary process, at least one candidate battle event can be identified based on the preset attribute information and the obtained attribute information.

[0053] Since each preset battle event in a preset battle event set corresponds to preset attribute information, when identifying at least one candidate battle event that matches the attribute information, the commentary server needs to determine whether the attribute information matches the preset attribute information of any preset battle event in the preset battle event set, i.e., it needs to perform an information matching process on the attribute information and the preset attribute information of each preset battle event. As a result, when it is determined that the attribute information matches the preset attribute information of a certain preset battle event in the preset battle event set, the preset battle event corresponding to the matched preset attribute information can be identified as a candidate battle event that matches the attribute information. If the attribute information does not match any of the preset attribute information corresponding to any of the preset battle events, correspondingly, the attribute information does not correspond to a candidate battle event.

[0054] By setting a battle event set in advance, candidate battle events can be quickly screened from the preset battle event set after obtaining attribute information of virtual objects, and compared with generating candidate battle events in real time, the embodiment of the present application can improve the efficiency of identifying candidate battle events. In addition, since the battle events are generated in advance, the power and computing resources required for generating candidate battle events in real time can also be reduced.

[0055] In one embodiment, screening at least one candidate match event that matches the attribute information from the multiple preset match events includes performing an information matching process on the attribute information and preset attribute information of each preset match event in the match event set to obtain target preset attribute information that matches the attribute information, and identifying the candidate match event based on the preset match event corresponding to the target attribute information.

[0056] When it is necessary to identify candidate battle events, the commentary server obtains attribute information of each object in the battle after performing a wartime action, performs information matching processing of each of the attribute information with preset attribute information of each preset battle event in the preset battle event set, obtains target preset attribute information that matches the attribute information, and identifies candidate battle events that match the attribute information based on the preset battle event corresponding to the target preset attribute information.

[0057] In one embodiment, when identifying a candidate match event, attribute information of a virtual object is matched with preset attribute information of a preset match event to obtain a candidate match event, and the screened candidate match events can be matched to a match event within the user's match attention field of view. In this way, not only can the power and computing resources required to generate duplicate commentary videos be reduced by reducing the probability of duplicate commentary on the same match event, but the power and computing resources required to generate inaccurate commentary videos can be reduced by improving the accuracy of the identified final commentary event.

[0058] Accordingly, identifying the candidate battle event based on the preset battle event corresponding to the target attribute information includes identifying the preset battle event corresponding to the target preset attribute information, and determining the preset battle event that satisfies the preset commentary condition in the preset battle event corresponding to the target preset attribute information as the candidate battle event, where the preset commentary condition includes at least one of a battle viewing angle condition and an event overlap condition, and the battle viewing angle condition is that the preset battle event is located within the battle viewing viewing angle, that is, after the attribute information is matched with the preset attribute information corresponding to any preset battle event, the commentary server needs to further determine whether the preset battle event corresponding to the target preset attribute satisfies the preset commentary condition, for example, the commentary server needs to further determine whether the preset battle event corresponding to the target preset attribute is located within the battle viewing angle, and if it is determined that the preset battle event corresponding to the target preset attribute is located within the battle viewing angle, it identifies the preset battle event as the candidate battle event corresponding to the battle command frame, otherwise, if it is determined that the preset battle event is located outside the current battle viewing angle, it deletes the preset battle event from the multiple candidate battle events matched by the attribute information.

[0059] Among them, the event overlap condition is that the number of times that the preset battle event appears within a preset time is less than the number threshold value. That is, when the attribute information matches the preset attribute information corresponding to a certain preset battle event, it is necessary to judge whether the preset battle event is overlapped and explained within the preset time. If there is no overlapping explanation, the preset battle event is determined as a candidate battle event matching the attribute information; otherwise, the preset battle event is deleted from the candidate battle events.

[0060] In one embodiment, a candidate match event may be set to require either a match viewing angle condition or an event overlap condition to be satisfied, or a candidate match event may be set to require both a match viewing angle condition and an event overlap condition to be satisfied simultaneously.

[0061] The preset commentary conditions include at least one of a battle viewing angle condition and an event overlap condition, and by setting as a candidate battle event a preset battle event for which the preset battle event corresponding to the target preset attribute information satisfies the preset commentary condition, not only can the probability of commenting on duplicate battle events be reduced, but the probability of commenting on a battle event that is not located within the battle viewing angle can also be reduced, thereby reducing the power and computing resources required to generate a commentary video that is not located within the battle viewing angle, and reducing the need for corrections due to an inappropriate battle viewing angle, thereby saving the power and computing resources required to correct the commentary video.

[0062] As shown in FIG. 4, this is a setting interface diagram of preset attribute information corresponding to a preset battle event. In the setting interface 401, the preset battle event is "heroes compete against red / blue BUFFs", and the corresponding preset attribute information (trigger condition) may be "home team heroes shoot down red / blue BUFFs of the away team, away team heroes are around the BUFFs, and the home team heroes have good HP (hit points)", etc.

[0063] In step 304, a target match event is screened from the at least one candidate match event.

[0064] Although the attribute information may include multiple candidate match events, only one match event can be commented on at each commentary time, and therefore, in a possible embodiment, when multiple candidate match events are matched to the attribute information, the most suitable match event from the multiple candidate match events needs to be selected as the target match event and subsequent commentary text and commentary audio need to be generated.

[0065] In one embodiment thereof, screening the target match event from the at least one candidate match event may include the steps of:

[0066] First, obtain the event weight corresponding to each candidate match event.

[0067] Wherein, the event weight is the offline event weight or basic event weight corresponding to each candidate match event, that is, the event weight is not directly related to the current match.

[0068] In a possible embodiment, a commentary event scoring model is provided in the commentary server, and the commentary event scoring model is iteratively trained offline by annotating selected commentary events by expert commentary organizers, so that by simply inputting the candidate match events generated in each match command frame into the trained commentary event scoring model, event weights corresponding to each candidate match event can be obtained, and each candidate match event and the corresponding event weights can be stored in the commentary server, so that during the online commentary process, corresponding event weights can be found based on the identified candidate match events.

[0069] In one embodiment, a commentary event scoring model is provided in the commentary server, so that there is no need to store candidate match events and corresponding event weights. During the online commentary process, the commentary server can input each candidate match event into the commentary event scoring model, thereby obtaining an event weight corresponding to each candidate match event.

[0070] In an illustrative example, when three candidate match events are generated based on the match command frame, the event weights corresponding to the obtained three candidate match events are respectively as follows: the event weight corresponding to the candidate match event 1 is 0.6, the event weight corresponding to the candidate match event 2 is 0.7, and the event weight corresponding to the candidate match event 3 is 0.8.

[0071] Second, determine an event score corresponding to each candidate match event based on the importance of each candidate match event within the match.

[0072] Since the event weight obtained in step 1 is an offline event weight and has no direct relation to the current match, if the target match event is selected based only on the offline event weight, the selected target match event may not be the best match event in the match or the match event that the user would like to comment on more; therefore, in a possible embodiment, in addition to the event weight, the commentary server needs to further identify an event score corresponding to each candidate match event by combining the importance of each candidate match event within the match.

[0073] Wherein, the importance corresponding to each candidate battle event is related to at least one of the event occurrence position of the candidate battle event, the virtual object type triggering the candidate battle event, and the number of virtual objects triggering the candidate battle event. That is, if the occurrence position of the battle event is located within the current battle viewing angle, the event score of the battle event is set high accordingly, otherwise the event score is set low. If the number of virtual objects triggering the battle event is large, the event score of the battle event is set high, otherwise the event score of the battle event is set low. If the virtual object triggering the battle event is a main character (or important character) in the battle, the event score of the battle event is set high, otherwise the event score of the battle event is set low, among which the main characters and important characters are preset by the developer.

[0074] In one embodiment, taking a multi-player online competitive game as an example, when determining the event score, the commentary server can obtain an event score corresponding to each candidate battle event comprehensively through two event scoring processes, battle scoring and intra-battle event scoring, of which the battle scoring is related to factors such as the number of battle participants (the more the number of battle participants, the higher the score is set), the battle position (the more important the battle preemption resource, the higher the score is set), and the battle result (the higher the score is set for a battle victory), etc. The intra-battle event scoring is related to the type of hero participating in the battle event (the more important the hero character, the higher the event score is set), the score of the hero participating in the battle event (the higher the score obtained by the hero, the higher the event score is set), etc.

[0075] Preferably, the factors that influence the event score corresponding to a candidate match event may be pre-set by a developer.

[0076] Third, a weighting process is performed on the event scores according to the event weights to obtain event weighted scores corresponding to each candidate match event.

[0077] In a possible embodiment, the event base weight and the online scoring status are combined to obtain an event weighting score corresponding to each candidate match event, and the target match event can be screened from the multiple candidate match events based on the event weighting score. That is, the commentary server can combine the event weight and the event score of the candidate match event to obtain the event weighting score of the candidate match event.

[0078] In one illustrative example, a battle command frame corresponds to three candidate battle events, among which, the event weight corresponding to candidate battle event 1 is 0.6 and the event score is 50, the event weight corresponding to candidate battle event 2 is 0.7 and the event score is 50, and the event weight corresponding to candidate battle event 3 is 0.6 and the event score is 80, and the event weighting scores corresponding to each candidate battle event are as follows: the event weighting score corresponding to candidate battle event 1 is 30, the event weighting score corresponding to candidate battle event 2 is 35, and the event weighting score corresponding to candidate battle event 3 is 42.

[0079] Preferably, when setting an event score, the event score can be based on a sufficient system, or can be based on a percentage system, and the embodiments of the present application are not limited thereto.

[0080] Fourth, the candidate match event with the highest event weighting score is identified as the target match event.

[0081] Since only a single battle event can be commented on at a given commentary time and a higher event weighting score indicates that the battle event is attracting more attention in the offline commentary scene and that the corresponding importance is also higher in the current battle scene, when a target battle event is specified from a plurality of candidate battle events, the candidate battle event with the highest event weighting score is specified as the target battle event.

[0082] In one illustrative example, the event weighting scores corresponding to each candidate match event are as follows: the event weighting score corresponding to candidate match event 1 is 30, the event weighting score corresponding to candidate match event 2 is 35, and the event weighting score corresponding to candidate match event 3 is 42, then the corresponding target match event is candidate match event 3.

[0083] In another possible application scenario, taking a multiplayer online competitive game (including a team battle scene) as an example, when selecting a target battle event from multiple candidate battle events, the commentary server can first select a battle event based on the number of virtual objects included in the team battle. For example, if a battle contains two team battles, and the number of virtual objects corresponding to team battle A is three, and the number of virtual objects corresponding to team battle B is seven, when selecting a battle event, the commentary server can preferentially select the candidate battle event corresponding to team battle B, and further select a target battle event from the multiple candidate battle events corresponding to team battle B, and the selection elements can include virtual object type and virtual object score. For example, team battle B corresponds to three candidate battle events, and the three virtual battle events are respectively performed by virtual object A and virtual object B, among which virtual object A is an important hero character, and correspondingly, the candidate battle event corresponding to virtual object A is selected as the target battle event.

[0084] In a possible application scenario, a target battle event can be specified based on a battle command frame of a single frame. Preferably, when a battle event is specified, when a target battle event cannot be specified based on only a battle command frame of a single frame, a target battle event needs to be specified based on a battle command frame of at least two frames.

[0085] The higher the event weighting score, the higher the attention of the match event in the offline commentary scene and the higher the corresponding importance in the current match scene, so that the candidate match event with the highest event weighting score can be identified as the target match event, and the importance of the match event finally commented can be improved, thus not only improving the user experience but also improving the commentary effect of the match event. In addition, since a focused commentary video is generated for a focused match event, the probability of an unimportant commentary video being generated for an unimportant match event can be reduced, thereby saving the power and computing resources required for generating an unimportant commentary video.

[0086] In step 305, commentary text is generated based on the target match event, and the commentary text is processed to generate a commentary data stream.

[0087] In a possible embodiment, after analyzing and obtaining the corresponding target battle event based on the battle command frame, the commentary server automatically generates commentary text through Natural Language Understanding (NLU) technology, and converts the commentary text into commentary audio through TTS technology, thereby obtaining the commentary data stream and realizing the process of online game understanding.

[0088] Preferably, the commentary audio describes a target match event corresponding to a single target match command frame or multiple target match command frames, and in possible embodiments, the commentary audio is associated with the corresponding target match event or with the frame number of the corresponding match command frame, so that the corresponding commentary audio can be found from the frame number later when synthesizing the commentary audio.

[0089] In step 306, a battle screen rendering is performed based on the battle command frame, and a battle video stream is generated, in which at least one battle video frame is included.

[0090] The embodiment of step 306 can refer to the above embodiment, and the description of this embodiment is omitted here.

[0091] In step 307, a target battle video frame in the battle video stream is identified, the target battle video frame being any one of the battle video frames in the battle video stream, and the battle time corresponding to the target battle video frame is identified as a target battle time, the target battle time being the time from the start of the battle to the target battle video frame.

[0092] Among them, the causes of the difference in data processing speed between the commentary data stream and the battle video stream include the following: The battle video stream is recorded when rendering is started after the user loads the game, while the commentary data stream is generated by analyzing after the players enter the battle, so the recording time of the battle video stream is obviously longer than the battle time, which causes a time difference between the commentary data stream and the battle video stream. Meanwhile, the frame rate of the battle command frame and the recording frame rate of the battle video frame are different, which also causes a time difference between the battle video stream and the commentary data stream. Therefore, it is necessary to analyze the correspondence between the commentary data stream and the battle video stream, so that the battle video frames and commentary audio of the same battle event can be time-aligned to realize the generation of the commentary video stream.

[0093] Regardless of how long the battle video stream is, the commentary is performed with game time (battle time) as the main timeline, so in a possible embodiment, the commentary server sets a time axis in the commentary video stream based on the game time in the game battle, i.e., the commentary server determines the commentary audio corresponding to the battle time by obtaining the target battle time in the battle video frame, which is the time from the start of the battle to the target battle video frame, where the target battle video frame is a video frame in the battle video stream, and therefore the target battle event of the target battle video frame is the time from the start of the battle to the target battle video frame.

[0094] In step 308, the battle command frame generated at the target battle time is identified as a target battle command frame, and the target frame number of the target battle command frame is identified.

[0095] Since the target commentary audio is generated based on the received target match command frame, the target commentary audio describing the target match event can correspond to a frame number corresponding to the target match command frame, and thus, in a possible embodiment, the commentary server can generate a target frame number for the target match command frame based on the match time, thereby identifying the target commentary audio based on the target frame number.

[0096] In one embodiment, the process of determining the target frame number of the target battle command frame may involve determining the target frame number of the target battle command frame based on the target battle time and the first frame rate.

[0097] Since the battle command frame has a preset acquisition or refresh frame rate (i.e., the first frame rate), when determining which battle command frame the target battle time corresponds to, it is necessary to calculate the target frame number of the target battle command frame based on the target battle time and the first frame rate.

[0098] In an illustrative example, when a target battle command frame is generated with a target battle time, and the first frame rate is 30 FPS, i.e., the battle command frames of two adjacent frames are spaced 30 ms apart, and the target battle time is 13 minutes 56 seconds 34 milliseconds, correspondingly, the target frame number of the target battle command frame is the target battle time of the target battle video frame divided by the time interval of the adjacent battle command frames, i.e., the target frame number corresponding to the target battle time of 13 minutes 56 seconds 34 milliseconds is 25334 frames.

[0099] By simply calculating the target battle time and the first frame rate, the target frame number of the target battle command frame can be obtained, which not only improves the efficiency of specifying the target frame number but also saves power and storage resources required for complex calculations.

[0100] In one embodiment, as shown in FIG. 5, a schematic diagram of a matching process between a battle video frame and a battle command frame is shown in an exemplary embodiment of the present application. The identification process of the battle time in the battle video frame is performed by the stream pulling client 510, that is, the stream pulling client 510 pulls the battle video stream from a server generating the battle video stream, and performs battle time identification for each battle video frame included in the battle video stream. The identification process of the battle time includes stream pulling monitoring 511, video decoding 512, time cutting 513, and time identification 514, in which the stream pulling monitoring 511 monitors the generation of the battle video stream and pulls the battle video stream in a timely manner, the video decoding 512 is used to decapsulate the pulled battle video stream to obtain continuous battle video frames, the time cutting 513 is used to cut a partial image including the battle time in the battle video frame to obtain a partial image, and then perform a subsequent time identification process, and the time identification 514 is used to identify the time included in the battle video frame. The stream pulling client 510 can identify the sequence as 1356, that is, the battle time corresponding to the video time of the battle video frame of 36 minutes 21 seconds in the battle video stream is 13 minutes 56 seconds. The time sequence of each battle video frame identified by the stream pulling client 510 forms a time queue 511 and is sent to the commentary service 520, and the commentary service 520 performs an inter-frame matching process. If there is an error in the time identification, that is, the difference between adjacent time sequences is large, the acquired time sequence is processed by time smoothing 516, and then the subsequent game frame matching 517 is performed. In this case, the game frame matching 517 is used to generate a target frame number corresponding to the target game command frame according to the time sequence (target battle time). If the target frame number corresponds to the target battle event, the inter-frame matching 518 is performed, that is, the video time of the battle video frame of 36 minutes 21 seconds in the battle video stream is time-aligned with the commentary audio whose frame number is 25334.

[0101] In step 309, the battle event corresponding to the target frame number is identified as a target battle event, commentary audio for explaining the target battle event is set as target commentary audio, the target commentary audio and the target battle video frame are time-aligned, and a commentary video stream is generated based on the target commentary audio and the target battle video frame having a time-aligned relationship.

[0102] Since none of the battle video frames corresponds to a target battle event, and the target frame number corresponds to a target battle command frame, and the target battle command frame corresponds to a target battle event, the commentary server can find a corresponding target battle event from the commentary data stream based on the target frame number. When the target battle event corresponding to the target frame number is found, the target commentary audio explaining the target battle event and the target battle video frame are time-aligned, i.e., the target battle video frame is displayed and the target commentary audio is played. In one embodiment, the commentary data stream can further include commentary text, and the commentary server can embed the target commentary text corresponding to the target battle video frame at a preset position of the target battle video frame when synthesizing the commentary video stream, and adjust the target commentary audio and the target battle video frame to be at the same time.

[0103] In this embodiment, the attribute information of each object in the battle after the battle action instructed by the battle operation command is analyzed, and the attribute information and the preset attribute information of the preset battle event are matched with the candidate battle event corresponding to the attribute information, so that the objective of automatically analyzing and obtaining the battle event without manual intervention can be realized, and then commentary text and commentary audio are generated based on the battle event, so that the efficiency of generating commentary video can be improved. In addition, the commentary data stream and the battle video stream are adjusted based on the battle time to realize the online merging and generating process of commentary video, so that the video screen and commentary audio of the same battle event are synchronized to improve the commentary effect of the battle event based on the synchronized video screen and commentary audio, and there is no need to artificially cut the battle video, so that the operation cost of online generating commentary video is reduced. In addition, the commentary video and the battle video stream can be adjusted based on the battle time in the battle, so that the need to first record and store the battle process video, or first generate and store the commentary audio before generating the commentary video, can be avoided, so that the power and storage resources required for recording and storage can be saved.

[0104] Because the accuracy of the match time in a match video frame is in seconds, but screen refreshes occur at intervals of ms, in one embodiment, in order to improve the accuracy of identifying the target frame number, the commentary server needs to modify the match time identified from the target match video frame.

[0105] In one embodiment, as shown in Figure 6, a flowchart of a method for identifying a target battle event according to an exemplary embodiment of the present application is illustrated. The embodiment of the present application takes the method as an example for application to the commentary server shown in Figure 1, and the method includes the following:

[0106] In step 601, the image recognition model performs image recognition on the battle time in the target battle video frame, and obtains the image recognition result.

[0107] Since the match time is displayed in the match video frame, in a possible embodiment, the commentary server can obtain the target match time corresponding to the target match video frame by image-identifying the match time in the target match video frame.

[0108] Preferably, the commentary server is provided with an image recognition model, and the target battle video frame is input to the image recognition model to perform image recognition, and the battle time contained in the target battle video frame can be output. Among them, the image recognition model may be a (Deep Neural Networks, DNN) model for processing handwritten digital recognition in the CV field.

[0109] In an exemplary embodiment, as shown in Figure 7, a schematic diagram of a battle video frame shown in an exemplary embodiment of the present application is shown, in which a video time 702 corresponding to the battle video frame is 36 minutes 21 seconds, and a battle time 701 corresponding to the battle video frame is 13 minutes 56 seconds.

[0110] Among these, when performing image recognition on the battle time in the target battle video frame, the target battle video frame can be directly input into the image recognition model to obtain the battle time output from the image recognition model, or time cutting can be performed on the target battle video frame, i.e., a partial image including the battle time is cut and obtained from the target battle video frame, the partial image is input into the image recognition model to obtain the battle time output from the image recognition model.

[0111] In step 602, a match time corresponding to a target match video frame is identified based on the image identification result, and the identified match time is set as a target match time.

[0112] In a possible embodiment, the commentary server can directly identify the time obtained from the image identification result as the target match time corresponding to the target match video frame.

[0113] Preferably, the target battle time included in the battle video frame is in units of seconds, but when calculating the frame number based on the frame rate, it is necessary to be accurate to the millisecond level in order to achieve frame-to-frame consistency. Therefore, in a possible implementation, the commentary server can introduce frequency statistics to accumulate the battle times obtained from the image recognition results by a number of times, thereby obtaining the target battle time in units of milliseconds.

[0114] By image-identifying the target battle time in the target battle video frame using the image identification model, the accuracy of the identified target battle time can be improved, and the matching accuracy of time-matching the target commentary audio and the target battle video frame based on the accurately identified target battle time can be improved. In addition, since the matching accuracy of the time matching is improved, the need for correction due to low matching accuracy can be effectively reduced, and the power and calculation resources required for correcting the commentary video can be saved.

[0115] In one exemplary embodiment, step 602 may include the following steps.

[0116] First, a basic battle time corresponding to a target battle video frame is determined based on the image identification result.

[0117] In a possible embodiment, only the time data obtained from the image identification result is identified as the basic battle time corresponding to the target battle video frame, and the basic battle time is subsequently modified based on the accumulated number and the second frame rate.

[0118] Second, a battle time deviation is determined based on the number of history identifications of the basic battle time and the second frame rate.

[0119] Wherein, the second frame rate is a frame rate corresponding to the fighting video stream, and if the second frame rate is 60 FPS, the time interval between two corresponding adjacent fighting video frames is 17 ms.

[0120] Since the second frame rate can provide a time in milliseconds, in a possible embodiment, the commentary server can calculate the deviation of the actual battle time based on the historical identification number of the basic battle time and the second frame rate, where the historical identification number of the basic battle time is the number of times the basic battle time is identified within a historical identification time period, and the historical identification time period is the time period until the target battle video frame is image-identified.

[0121] In one illustrative example, if the second frame rate is 60 FPS and the basic battle time is 13 minutes and 56 seconds, when the basic battle time is identified for the first time, the corresponding battle time deviation amount is 17 ms, and when the basic battle time is identified for the second time, the corresponding battle time deviation amount is 34 ms.

[0122] Third, the sum of the basic battle time and the battle time deviation is specified as the target battle time.

[0123] Since the battle time deviation is in milliseconds, the target battle time in milliseconds can be obtained by specifying the sum of the battle time deviation and the basic battle time as the target battle time.

[0124] In one illustrative example, if the base match time is 13 minutes 56 seconds and the match time offset is 34 ms, the corresponding target match time may be 13 minutes 56 seconds 34 milliseconds.

[0125] In one exemplary embodiment, the correspondence between the target combat video frames and the target combat command frames is as shown in Tables 1 and 2.

[0126] [Table 1]

[0127] [Table 2]

[0128] As can be seen from the correspondence between Table 1 and Table 2, the target battle time corresponding to the target battle video frame with a video time of 36 minutes 21 seconds is 13 minutes 56 seconds 34 milliseconds, the target frame number of the corresponding target battle command frame is 25334, and the corresponding target battle event is “Cheng xx is shot to death”.

[0129] In this embodiment, by analyzing the historical identification number of the battle time in the battle video frame and combining the frame rate of the battle video stream, the target battle time in milliseconds can be accurately calculated, thereby realizing the matching between the target battle video frame and the target commentary audio in the time dimension, thus not only improving the accuracy of identifying the target battle time, but also improving the accuracy of the matching between frames. In addition, because the accuracy of the matching between frames is improved, the need for correction due to inaccuracy can be effectively reduced, thereby saving the power and computing resources required for correcting the commentary video.

[0130] In a possible application scenario, for games that include multiple virtual objects in a single match, such as multiplayer online competitive games, the match may include multiple virtual objects, and the process of generating the commentary video may include different match viewing viewing angles, where the different match viewing angles may be angles that focus on a certain virtual object. Therefore, when rendering the match screen to generate a match video stream, it is necessary to generate match video streams with different match viewing viewing angles according to the match viewing angles.

[0131] In an illustrative example, as shown in FIG. 8, a flowchart of an explanatory video generation method of another exemplary embodiment of the present application is shown, and the embodiment of the present application describes the application of the method to the explanatory server shown in FIG. 1 as an example, and the method includes:

[0132] In step 801, a battle command frame is obtained, the battle command frame includes at least one battle operation command, and the battle operation command is used to control a virtual object to perform a battle action in a battle.

[0133] In step 802, a commentary data stream is generated based on the combat command frame, the commentary data stream including commentary audio describing at least one stage of combat events, the combat events being triggered when a virtual object performs a combat action.

[0134] The implementation manner of steps 801 and 802 can be referred to in the above embodiment, and the description of this embodiment is omitted here.

[0135] In step 803, a battle screen is rendered based on the battle command frame to obtain a global battle screen.

[0136] Since the battle command frame can include battle operation commands sent from the game client corresponding to different virtual objects (operated by the user), when performing battle screen rendering based on the battle command frame, global rendering is required, and a global battle screen can be obtained after recording.

[0137] In step 804, a target match viewing angle in the match viewing angle is determined, a target match screen is extracted from the global match screen based on the target match viewing angle, and a match video stream corresponding to the target match viewing angle is generated based on the target match screen, in which different match viewing angles correspond to different match video streams.

[0138] In a possible embodiment, the commentary server can obtain match video streams for different match viewing angles so that the user can view the match events occurring at different positions during the commentary process, allowing the user to view the match events occurring at a clearer or more accurate angle.

[0139] The different match viewing viewing angles may be viewing angles centered on different virtual objects, where the virtual objects are virtual objects operated by the users.

[0140] Among these, the method of obtaining a battle video stream corresponding to a different battle viewing angle involves extracting a battle screen with the required battle viewing angle from the global battle screen and recording each different battle screen to generate a battle video stream corresponding to the different battle viewing angles, or distributing the different battle viewing angles to servers with different sound card devices, rendering them in parallel, and recording them to generate a battle video stream corresponding to the different battle viewing angles.

[0141] In step 805, each of the match video streams is merged with the commentary data stream to generate commentary video streams corresponding to each of the different match viewing angles.

[0142] Based on correspondingly generating the battle video streams of different battle viewing angles, in the process of generating the commentary video stream, it is necessary to merge the different battle video streams and the commentary data stream, thereby generating the commentary video streams corresponding to the different battle viewing angles.

[0143] Preferably, for a scene that generates commentary video streams corresponding to different match viewing angles, the commentary server can directly push the commentary video streams of different match viewing angles to the live streaming platform or client, so that the live streaming platform or client can switch and play the match viewing angles as needed, or can push only the target commentary data stream corresponding to the required match viewing angle to the live streaming platform or client based on the needs of different live streaming platforms and clients.

[0144] In the embodiment of the present application, different commentary video streams can be generated according to different match viewing angles, so that different commentary video streams can be pushed accurately according to the needs of different platforms, so that the accuracy of the pushed commentary video streams can be improved; or, when playing commentary video streams, switching between different match viewing angles can be realized, so that the diversity of commentary video streams can be improved. Because commentary video streams can be pushed accurately to different platforms, the need for correction due to inaccurate pushing can be reduced, so that the power and computing resources required for correcting the pushed commentary video streams can be saved.

[0145] As shown in FIG. 9, a schematic diagram of the process of completely generating a commentary video stream shown in an exemplary embodiment of the present application is shown. The commentary server receives game commands 901 (battle operation commands), one is to generate a commentary data stream by obtaining game information and TTS voice synthesis, and one is to generate a battle video stream according to the game commands. Among them, the process of generating a commentary data stream includes: game core conversion (Game Core) 902 (i.e., analyzing the battle command frame), commentary feature 903 (i.e., obtaining attribute information of each object in the battle), event generation 904 (i.e., identifying at least one candidate battle event matched according to the attribute information), event selection 905 (i.e., selecting a target battle event from multiple candidate battle events), and TTS voice synthesis 906 (i.e., generating commentary text according to the target battle event, and performing TTS processing to obtain commentary audio). The process of generating a battle video stream includes battle rendering 907 (i.e., performing battle rendering based on a battle command to generate a battle screen), rendering live broadcast (Outside Broadcast, OB) scheduling 908 (i.e., rendering and obtaining battle screens corresponding to different battle viewing angles), video recording 909 (recording the battle screen to generate a battle video stream), and video push 910 (pushing the battle video stream to a server for synthesizing a commentary video stream). Once the battle video stream and commentary data stream are obtained, a commentary video 911 can be generated by multi-matching the battle video stream and the commentary data stream.

[0146] Referring to FIG. 10, a block diagram of a commentary video generating device shown in an exemplary embodiment of the present application is shown. The commentary video generating device can be realized as a part or the whole of a server. The commentary video generating device includes: an acquisition module 1001 for acquiring a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; a first generation module 1002 for generating a commentary data stream based on the battle command frame, the commentary data stream including commentary audio explaining at least one battle event, the battle event being triggered when the virtual object performs a battle action; a second generation module 1003 for rendering a battle screen based on the battle command frame, generating a battle video stream, and the battle video stream including at least one battle video frame; and a third generation module 1004 for merging the commentary data stream and the battle video stream, generating a commentary video stream, and the commentary audio being used to time-align the battle video frame and the commentary audio corresponding to the same battle event in the commentary video stream.

[0147] Preferably, the third generation module 1004 includes a first identification unit used to identify a target battle video frame in the battle video stream, the target battle video frame being any one of the frames in the battle video stream, a battle time corresponding to the target battle video frame being the target battle time, the target battle time being the time from the start of the battle to the target battle video frame, a second identification unit used to identify a battle command frame generated at the target battle time as a target battle command frame and to identify a target frame number of the target battle command frame, and a time alignment unit used to identify a battle event corresponding to the target frame number as a target battle event, to set commentary audio for explaining the target battle event as target commentary audio, to time-align the target commentary audio and the target battle video frame, and to generate a commentary video stream based on the target commentary audio and the target battle video frame having a time-aligned relationship.

[0148] Preferably, the match command frame corresponds to a first frame rate, and the second identification unit is further used for identifying the target frame number of the target match command frame based on the target match time and the first frame rate.

[0149] Preferably, the first identification unit is further used to image-identify the battle time in the target battle video frame using an image recognition model, obtain an image identification result, identify the battle time corresponding to the target battle video frame based on the image identification result, and set the identified battle time as a target battle time.

[0150] Preferably, the frame rate of the battle video stream is a second frame rate, and the first identification unit further identifies a basic battle time corresponding to the target battle video frame based on the image identification result, and identifies a battle time offset based on a historical identification count of the basic battle time and the second frame rate, the historical identification count of the basic battle time being the number of times the basic battle time is identified within a historical identification time stage, and is used to identify the sum of the basic battle time and the battle time offset as the target battle time.

[0151] Preferably, the first generation module 1002 further includes: a third identification unit used to obtain a preset battle event set, the battle event set including a plurality of preset battle events; a third identification unit used to control a virtual object to perform corresponding battle actions in a battle based on a battle command frame, and identify attribute information of each object in the battle after performing the battle action; a fourth identification unit used to screen at least one candidate battle event matching the attribute information from the plurality of preset battle events; a screening unit used to screen a target battle event from at least one of the candidate battle events; and a first generation unit used to generate commentary text based on the target battle event, process the commentary text, and generate the commentary video stream.

[0152] Preferably, the fourth identification unit further performs an information matching process on the attribute information and the preset attribute information of each of the preset battle events in the battle event set, obtains target preset attribute information matching the attribute information, and is used to identify candidate battle events based on the preset battle events corresponding to the target preset attribute information.

[0153] Preferably, the fourth identification unit further identifies a preset battle event corresponding to the target preset attribute information, and sets a preset battle event that satisfies a preset commentary condition in the preset battle event corresponding to the target preset attribute information as a candidate battle event, and the preset commentary condition includes at least one of a battle viewing angle condition and an event overlap condition, wherein the battle viewing angle condition is that the preset battle event is located within a battle viewing angle, and the event overlap condition is that the number of times the preset battle event appears within a preset time is less than a number threshold.

[0154] Preferably, the screening unit is further used for obtaining an event weight corresponding to each of the candidate match events; identifying an event score corresponding to each of the candidate match events based on the importance of each of the candidate match events in a match, the importance being related to at least one of an event occurrence position of the candidate match event, a virtual object type that triggers the candidate match event, and a number of virtual objects that trigger the candidate match event; performing a weighting process on the event score according to the event weight to obtain an event weighting score corresponding to each of the candidate match events; and identifying the candidate match event having the highest event weighting score as the target match event.

[0155] Preferably, the second generation module 1003 includes a second generation unit used to perform battle screen rendering based on the battle command frame, obtain a global battle screen, and identify a target battle viewing angle in the battle viewing angle, and a third generation unit used to extract a target battle screen from the global battle screen based on the target battle viewing angle, and generate a battle video stream corresponding to the target battle viewing angle based on the target battle screen, in which different battle viewing angles correspond to different battle video streams.

[0156] The third generation module 1004 includes a fourth generation unit that is used to merge each of the match video streams with the commentary data stream to generate the commentary video streams corresponding to each of the different match viewing angles.

[0157] As described above, in the embodiment of the present application, the commentary audio is generated and the battle video is rendered by analyzing the battle command frame online, and the commentary audio and the battle video are time-matched to generate the commentary video. By analyzing the battle command frame and generating the commentary video, the commentary video matching the battle can be generated during the battle process, and there is no need to generate the commentary video after the battle, improving the timeliness of the commentary video generation; on the other hand, there is no need to manually create the commentary text and generate the commentary audio, and an automated commentary video generation process can be realized, further improving the efficiency of the commentary video generation.

[0158] For the purpose of illustration, the explanatory video generating device provided in the above embodiment is described by only taking the above-mentioned functional module division as an example, in actual application, the above-mentioned function allocation can be completed by different functional modules as required, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-mentioned functions. In addition, the explanatory video generating device and the explanatory video generating method embodiments provided in the above embodiment belong to the same concept, and their specific realization process is shown in detail in the method embodiment, and the description is omitted here.

[0159] 11, a configuration block diagram of a server provided in an embodiment of the present application is shown, which is used to implement the commentary video generating method performed by the commentary server in the above embodiment.

[0160] Specifically, the server 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The server 1100 further includes a basic input / output system (Input / Output system, I / O system) 1106 that assists in transmitting information between each device in the server, and a mass storage device 1107 for storing an operating system 1113, an application program 1114, and other program modules 1115.

[0161] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109, such as a mouse, keyboard, etc., for a user to input information. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110, which is connected to the system bus 1105. The basic input / output system 1106 may further include an input / output controller 1110 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, a printer, or other type of output device.

[0162] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) that is connected to the system bus 1105. The mass storage device 1107 and associated computer-readable storage media provide non-volatile storage for the server 1100. That is, the mass storage device 1107 may include a computer-readable storage medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0163] Without loss of generality, the computer readable storage medium may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable storage instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Flash memory, or other solid-state memory, CD-ROM, Digital Versatile Disc (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device. Of course, those skilled in the art will recognize that the computer storage media is not limited to the above. The above-mentioned system memory 1104 and mass storage device 1107 may be collectively referred to as memory.

[0164] The memory stores one or more programs, the one or more programs are arranged to be executed by one or more central processing units 1101, the one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 1101 executes the one or more programs to implement the explanatory video generation method provided by each of the above method embodiments.

[0165] According to various embodiments of the present application, the server 1100 may be implemented by a remote server connected to a network, such as the Internet, via a network, i.e., the server 1100 may be connected to a network 1112 via a network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 may be connected to another type of network or remote server system (not shown).

[0166] The memory further includes one or more programs, the one or more programs being stored in the memory, and the one or more programs including performing steps executed by the commentary server in the method provided by the embodiments of the present application.

[0167] In an embodiment of the present application, a computer-readable storage medium is further provided, storing at least one instruction, at least one program, code set or instruction set in the storage medium, and the at least one instruction, the at least one program, code set or instruction set is loaded into a processor and executed to realize the commentary video generation method described in the above aspect.

[0168] According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored on a computer readable storage medium, the computer instructions being read by a processor of a computing device from the computer readable storage medium and executed by the processor to cause the computing device to perform the instructional video generation method provided in various preferred embodiments of the above aspect.

[0169] Other embodiments of the present application will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present application, which variations, uses or adaptations follow the general principles of the present application and include common general knowledge or commonly used techniques in the art that are not disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the present application being indicated by the following claims.

[0170] As will be appreciated, the present invention is not limited to the precise configuration described above and illustrated in the drawings, and various modifications and variations are possible without departing from the spirit and scope of the present invention. The scope of the present application is limited only by the appended claims.

Claims

1. A commentary video generation method executed by a commentary server, comprising: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs the combat action; A step of performing a battle screen rendering based on the battle command frame and generating a battle video stream, the battle video stream including at least one battle video frame; merging the commentary data stream and the match video stream to generate a commentary video stream, wherein the match video frames and the commentary audio corresponding to the same match event in the commentary video stream are time-aligned; The step of merging the commentary data stream and the match video stream to generate a commentary video stream includes: identifying a target match video frame in the match video stream, the target match video frame being any one match video frame in the match video stream; A step of specifying a match time corresponding to the target match video frame as a target match time, the target match time being a time from a start of the match to the target match video frame; specifying a battle command frame generated at the target battle time as a target battle command frame, and specifying a target frame number of the target battle command frame; specifying a battle event corresponding to the target frame number as a target battle event, and setting commentary audio for explaining the target battle event as target commentary audio; time-aligning the target commentary audio and the target match video frames; generating a commentary video stream based on the time-aligned target commentary audio and target fight video frames.

2. the combat command frame corresponds to a first frame rate; The step of identifying a target frame number of the target battle command frame includes:

2. The method of claim 1, further comprising determining the target frame number of the target fight command frame based on the target fight duration and the first frame rate.

3. The step of specifying a match time corresponding to the target match video frame as a target match time includes: A step of image-classifying the battle time in the target battle video frame using an image recognition model to obtain an image recognition result; The method according to claim 1, further comprising a step of: identifying a match time corresponding to the target match video frame based on the image identification result; and setting the identified match time as a target match time.

4. the frame rate of the competition video stream is a second frame rate; The step of identifying a match time corresponding to the target match video frame based on the image identification result and setting the match time as a target match time includes: determining a basic match time corresponding to the target match video frame based on the image identification result; a step of determining a battle time deviation amount based on the number of history identifications of the basic battle time and the second frame rate, the number of history identifications of the basic battle time being the number of times the basic battle time is identified within a history identification time stage; The method according to claim 3, further comprising a step of specifying a sum of the basic battle time and the battle time deviation as the target battle time.

5. A commentary video generation method executed by a commentary server, comprising: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs the combat action; A step of performing a battle screen rendering based on the battle command frame and generating a battle video stream, the battle video stream including at least one battle video frame; merging the commentary data stream and the match video stream to generate a commentary video stream, wherein the match video frames and the commentary audio corresponding to the same match event in the commentary video stream are time-aligned; The step of generating a commentary data stream based on the combat command frame includes: obtaining a preset battle event set, the battle event set including a plurality of preset battle events; controlling, based on the battle command frame, the execution of a corresponding battle action in a battle of virtual objects; specifying attribute information of each virtual object in the battle after performing the wartime action; screening at least one candidate battle event that matches the attribute information from the plurality of preset battle events; screening a target match event from at least one of the candidate match events; generating descriptive text based on the target match events; and performing a text-to-speech process on the descriptive text to generate a descriptive data stream.

6. The step of screening at least one candidate battle event matching the attribute information from the plurality of preset battle events includes: performing information matching processing on the attribute information and preset attribute information of each of the preset battle events in the battle event set, and acquiring target preset attribute information matching the attribute information; 6. The method of claim 5, further comprising: identifying candidate match events based on preset match events corresponding to the target preset attribute information.

7. The step of identifying a candidate competition event based on a preset competition event corresponding to the target preset attribute information includes: A method according to claim 6, characterized in that a step of identifying a preset battle event corresponding to the target preset attribute information, and setting a preset battle event in the preset battle event corresponding to the target preset attribute information that satisfies a preset commentary condition as a candidate battle event, the preset commentary condition including at least one of a battle viewing angle condition and an event overlap condition, the battle viewing angle condition being that the preset battle event is located within a battle viewing angle, and the event overlap condition being that the number of times the preset battle event appears within a preset time is less than a number threshold.

8. The step of screening a target match event from at least one of the candidate match events includes: obtaining an event weight corresponding to each of the candidate match events; determining an event score corresponding to each of the candidate match events based on the importance of each of the candidate match events in a match, the importance being related to at least one of an event occurrence location of the candidate match event, a virtual object type that triggers the candidate match event, and a number of virtual objects that trigger the candidate match event; weighting the event scores by the event weights to obtain event weighted scores corresponding to each of the candidate match events; and identifying the candidate match event having the highest event weighting score as the target match event.

9. A commentary video generation method executed by a commentary server, comprising: obtaining a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio describing at least one combat event, the combat event being triggered when the virtual object performs the combat action; A step of performing a battle screen rendering based on the battle command frame and generating a battle video stream, the battle video stream including at least one battle video frame; merging the commentary data stream and the match video stream to generate a commentary video stream, wherein the match video frames and the commentary audio corresponding to the same match event in the commentary video stream are time-aligned; The step of rendering a battle screen based on the battle command frame and generating a battle video stream includes: performing battle screen rendering based on the battle command frame to obtain a global battle screen; A step of specifying a target match viewing angle in the match viewing angle; A step of extracting a target match screen from the global match screen based on a target match viewing viewing angle, and generating a match video stream corresponding to the target match viewing viewing angle based on the target match screen, the corresponding match video stream being different depending on the match viewing viewing angle; The method according to claim 1, wherein the step of merging the commentary data stream with the match video stream to generate a commentary video stream includes a step of merging each of the match video streams with the commentary data stream to generate the commentary video stream corresponding to each of the match viewing angles.

10. an acquisition module for acquiring a battle command frame, the battle command frame including at least one battle operation command, the battle operation command being used to control a virtual object to perform a battle action in a battle; a first generation module for generating a commentary data stream based on the combat command frame, the commentary data stream including commentary audio for explaining at least one combat event, the combat event being used to be triggered when the virtual object performs a combat action; A second generating module is used to perform battle screen rendering based on the battle command frame, generate a battle video stream, and ensure that at least one battle video frame is included in the battle video stream; a third generation module that merges the commentary data stream and the match video stream to generate a commentary video stream, the commentary video stream being used to time-align the match video frames and the commentary audio corresponding to the same match event in the commentary video stream; The third generation module: A first determination unit is used to determine a target battle video frame in the battle video stream, the target battle video frame being any one of the battle video frames in the battle video stream, and to determine a battle time corresponding to the target battle video frame as a target battle time, the target battle time being a time from a battle start to the target battle video frame; a second identification unit for identifying the battle command frame generated at the target battle time as a target battle command frame and for identifying a target frame number of the target battle command frame; a time alignment unit used to identify a battle event corresponding to the target frame number as a target battle event, time-align a commentary audio for explaining the target battle event as a target commentary audio, and time-align the target commentary audio and the target battle video frame, and generate a commentary video stream based on the target commentary audio and the target battle video frame having a time-aligned relationship.

11. A server comprising a memory in which computer-readable instructions are stored and one or more processors, the computer-readable instructions, when executed by the processors, causing the one or more processors to perform the steps of the method according to any one of claims 1 to 9.

12. A computer program comprising: A computer program which, when executed by one or more processors, causes said one or more processors to carry out the steps of the method according to any one of claims 1 to 9.

Images