Game live broadcast data processing method and system

Abstract

The present application discloses a method for processing game live broadcast data, which includes: the host end converts the game data into a data packet of a preset type and encapsulates it with specific video data and audio data; the encapsulated data is used as a live stream and distributed to the audience end through a CDN; the audience end obtains the live stream, parses the game data and plays it back. The present application also discloses a game live broadcast data processing system, device, computer equipment and computer-readable storage medium. As a result, through special processing of the game data, the existing live broadcast mode and CDN network can be used for data distribution without modification during the game live broadcast, which is suitable for large-scale audiences to watch game live broadcasts and reduces server costs and development costs.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular to a game live broadcast data processing method, system, device, computer equipment and computer-readable storage medium. Background Art

[0002] The basic architecture of the current online game system is that the online game server runs or verifies the game logic and maintains the game status and data; the online game client establishes a connection with the server, sends game control signals to the server, and the server executes the game control instructions and returns the execution results to the client for display.

[0003] This architecture is designed for participating in games and is suitable for this purpose. However, for large audiences watching a game, event, virtual concert, or other live broadcasts hosted by a network-like gaming system, its shortcomings become apparent. For example, bidirectional communication between the client and server increases server computing costs and significantly increases the complexity of the server cluster. Network gaming protocols are not universal, making it difficult to efficiently utilize services deployed at the edge nodes of a CDN (Content Delivery Network). If replaying recordings is needed later, the storage and playback methods for the recordings require additional development. Furthermore, if video is used for content distribution, it is impossible to allow viewers to move around the scene while watching 3D content, which is common in gaming scenarios. In short, developing a live broadcast solution that can support large audiences for network-like gaming systems presents numerous challenges, requires extensive development, and is extremely expensive in terms of server costs. Summary of the Invention

[0004] The main purpose of this application is to propose a game live broadcast data processing method, system, device, computer equipment and computer-readable storage medium, aiming to solve the above technical problems.

[0005] To achieve the above objectives, an embodiment of the present application provides a method for processing game live broadcast data, the method comprising:

[0006] The host converts the game data into a data packet of a preset type and encapsulates it with specific video and audio data;

[0007] The encapsulated data is used as a live stream and distributed to the audience through CDN;

[0008] The audience obtains the live stream, parses the game data and plays it back.

[0009] Optionally, converting the game data into a data packet of a preset type and encapsulating the data packet with the specific video data and audio data includes:

[0010] Convert each frame of the game playback data into an SEI data packet;

[0011] Generate black screen video data and audio data of equal length according to the time length of each frame of playback data;

[0012] Adding the SEI data packet to the black screen video data to obtain mixed video data;

[0013] After the mixed video data and the audio data are encapsulated, they are pushed to the content distribution network via the RTMP protocol.

[0014] Optionally, converting the replay data of each frame of the game into an SEI data packet further includes:

[0015] Generate full game data every preset time period and convert it into SEI data packets.

[0016] Optionally, generating black screen video data of equal length includes:

[0017] Generate low-resolution, low-bitrate H.264 video data representing black screen data of equal length.

[0018] Optionally, generating audio data of equal time length includes:

[0019] Generate low sampling rate, low bit rate AAC audio data representing silence data of equal length; or

[0020] Encode game sounds or commentary sounds of equal length to generate AAC data packets.

[0021] Optionally, before distributing the encapsulated data as a live stream to the viewer via a CDN, the method further includes:

[0022] The encapsulated data is sliced ​​on the live broadcast server and converted into a live broadcast stream in the form of HLS.

[0023] Optionally, obtaining the live stream and parsing the game data for playback includes:

[0024] Obtain the live stream from the CDN;

[0025] Parsing the mixed video data and the audio data from the live stream;

[0026] disassembling the data packet from the mixed video data to restore the game data;

[0027] The game is replayed at the audience end according to the game data.

[0028] In addition, to achieve the above-mentioned purpose, the embodiment of the present application further provides a game live broadcast data processing system, the system comprising:

[0029] The host side is used to convert game data into a preset type of data packet, encapsulate it with specific video and audio data, and push it to the CDN;

[0030] The CDN is used to distribute the data packaged by the anchor end as a live stream to the audience end;

[0031] The audience terminal is used to obtain the live stream from the CDN, parse the game data and play it back.

[0032] To achieve the above-mentioned purpose, the present application also provides a method for processing game live broadcast data, which is applied to a broadcaster end. The method includes:

[0033] Converting each frame of the game playback data into a data packet of a preset type;

[0034] Generate black screen video data and audio data of equal length according to the time length of each frame of playback data;

[0035] Adding the data packet to the black screen video data to obtain mixed video data;

[0036] After the mixed video data and the audio data are encapsulated, they are pushed to the CDN via the RTMP protocol.

[0037] Optionally, converting the replay data of each frame of the game into a data packet of a preset type includes:

[0038] Whenever the game generates a frame of playback data, the frame of playback data is converted into an SEI data packet in H.264 or H.265 format.

[0039] Optionally, converting the replay data of each frame of the game into a data packet of a preset type further comprises:

[0040] Generate full game data every preset time period and convert it into SEI data packets.

[0041] Optionally, generating black screen video data of equal length includes:

[0042] Generate low-resolution, low-bitrate H.264 video data representing black screen data of equal length.

[0043] Optionally, generating audio data of equal time length includes:

[0044] Generate low sampling rate, low bit rate AAC audio data representing silence data of equal length; or

[0045] Encode game sounds or commentary sounds of equal length to generate AAC data packets.

[0046] Optionally, the step of adding the data packet to the black screen video data to obtain mixed video data further includes:

[0047] When the data packet contains the full data, the mixed video data generated after being added to the black screen video data is a key frame.

[0048] To achieve the above-mentioned purpose, the embodiment of the present application further provides a game live broadcast data processing device, which is applied to the anchor terminal, and the device includes:

[0049] A conversion module, configured to convert each frame of the game playback data into a data packet of a preset type;

[0050] A generating module, configured to generate black screen video data and audio data of equal length according to the time length of each frame of playback data;

[0051] A mixing module, configured to add the data packet to the black screen video data to obtain mixed video data;

[0052] The encapsulation module is used to encapsulate the mixed video data and the audio data and then push the data to the CDN via the RTMP protocol.

[0053] To achieve the above objectives, the present invention further provides a method for processing game live broadcast data, which is applied to a viewer terminal. The method includes:

[0054] Obtain a live stream from the CDN, wherein the live stream is obtained by the host converting the game data into a data packet of a preset type, encapsulating it with the black screen video data and audio data, and then pushing the stream to the CDN;

[0055] Parsing mixed video data and the audio data from the live stream, wherein the mixed video data includes the black screen video data and the additional data packet;

[0056] disassembling the data packet from the mixed video data to restore the game data;

[0057] The game is replayed based on the game data.

[0058] Optionally, the method further includes:

[0059] When the audio data is AAC audio data representing silence data, discarding the audio data;

[0060] When the audio data is an AAC data packet generated by encoding the game sound or commentary sound, the audio data is parsed to obtain the game sound or commentary sound, and played together with the game data.

[0061] To achieve the above-mentioned purpose, the embodiment of the present application further provides a game live broadcast data processing device, which is applied to the audience end, and the device includes:

[0062] An acquisition module is used to obtain a live stream from a CDN, wherein the live stream is a stream in which the host converts the game data into a data packet of a preset type, encapsulates it with the black screen video data and audio data, and then pushes the stream to the CDN;

[0063] A parsing module, configured to parse the live stream to obtain mixed video data and the audio data, wherein the mixed video data includes the black screen video data and the additional data packet;

[0064] a disassembly module, configured to disassemble and assemble the data packet from the mixed video data to restore the game data;

[0065] A playing module is used to replay the game according to the game data.

[0066] To achieve the above-mentioned purpose, an embodiment of the present application also provides a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the game live broadcast data processing method as described above is implemented.

[0067] To achieve the above-mentioned purpose, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the game live broadcast data processing method as described above is implemented.

[0068] The game live broadcast data processing method, system, device, computer equipment and computer-readable storage medium proposed in the embodiments of the present application are based on the game live broadcast content distribution mechanism constructed based on the existing live broadcast CDN network. By converting the game data into SEI data packets and then encapsulating them together with specific video data (black screen video data) and audio data (silent data), the CDN can process the game data as an ordinary live broadcast stream and distribute the game content using the existing live broadcast CDN network architecture without modification. This is not only suitable for large-scale audiences to watch game live broadcasts, but also can greatly reduce server costs and development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of the architecture of the current game data processing solution;

[0070] Figure 2 This is a schematic diagram of the architecture of the current first game live broadcast data distribution solution;

[0071] Figure 3 This is a schematic diagram of the architecture of the second current game live broadcast data distribution solution;

[0072] Figure 4 This is an architectural diagram of a game live broadcast data processing system proposed in the first embodiment of the present application;

[0073] Figure 5 This is a flowchart of a method for processing game live broadcast data proposed in the second embodiment of the present application;

[0074] Figure 6 for Figure 5 Detailed flow chart of step S200;

[0075] Figure 7 for Figure 5 Detailed flow chart of step S204;

[0076] Figure 8 This is a flowchart of another form of a specific embodiment of the game live broadcast data processing method based on the second embodiment;

[0077] Figure 9 This is a flowchart of a method for processing game live broadcast data proposed in the third embodiment of the present application;

[0078] Figure 10 This is a flowchart of a method for processing game live broadcast data proposed in the fourth embodiment of the present application;

[0079] Figure 11 This is a module diagram of a game live broadcast data processing device proposed in the fifth embodiment of the present application;

[0080] Figure 12 This is a module diagram of a game live broadcast data processing device proposed in the sixth embodiment of the present application;

[0081] Figure 13 This is a schematic diagram of the hardware architecture of a computer device proposed in the seventh embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0083] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0084] Example 1

[0085] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the current game data processing solution.

[0086] Most current games utilize an "event loop" mechanism to advance the game, processing input data, updating the game world, and generating output on a frame-by-frame basis. User input is processed, the game's internal state is updated, and the resulting graphics are rendered and presented to the user. For online games, updates to the game's internal state occur on the server side (the game server). The game server supports the operation of online games, accepting control data from numerous game clients, executing the game logic, and transmitting the game status back to the client, which then displays it to the user. The game server can also synchronize the game status with all other players (their clients), ensuring that everyone is playing the same game.

[0087] See also Figure 2 , Figure 2 This is a schematic diagram of the architecture of the current first game live broadcast data distribution solution.

[0088] exist Figure 1 On this basis, if you want other users (audiences) to watch the game, that is, to broadcast the game live, the audience (client) must also connect to the server. From this structure, it can be seen that when the number of viewers continues to increase, the server burden will continue to increase, and eventually the server will not be able to support so many viewers.

[0089] See also Figure 3 , Figure 3 This is a schematic diagram of the architecture of the current second game live broadcast data distribution solution.

[0090] against Figure 2 The current solution to the problem is to increase the number of game servers, synchronize the internal state of the game to all these added servers, and then let the audience (the client) connect to these servers.

[0091] Although the problem can be solved by adding game servers, a complete set of game server software must be run on each server. These software usually have high requirements for hardware resources. Moreover, because they are all dedicated software, it is difficult to share a system with other services, and the deployment cost is very high.

[0092] The most common live streaming services in China are based on RTMP (Real Time Messaging Protocol), a streaming media transmission protocol that transmits audio and video data in tags. The most commonly used audio and video codec is a combination of H.264 and AAC (Advanced Audio Coding). H.264, also known as MPEG-4 AVC (MPEG-4 Part 10, Advanced Video Coding), is a block-oriented, motion-compensated video coding standard.

[0093] This live streaming service can accept data transmitted by the content production side (anchor side) via the RTMP protocol, and provide FLV (Flash Video, a streaming media format) format data to the content consumption side (viewer side) via HTTP (Hyper Text Transfer Protocol, Hypertext Transfer Protocol) and HTTPS (Hyper Text Transfer Protocol over Secure Socket Layer, Hypertext Transfer Security Protocol). However, game data is not encapsulated in the FLV format, nor is it data in the form of H.264 or AAC, so it is not possible to directly use this service to transmit game data. Even if the content production side forcibly encapsulates the game data into the FLV tag and sends it to the server, the server will not further distribute this data.

[0094] Based on the above reasons, this application proposes a new mechanism that allows game data to be packaged and transmitted like a live broadcast, fully reusing the existing live broadcast CDN infrastructure. In other words, the game data is pushed via the RTMP protocol, distributed through the CDN network, and the viewer pulls the FLV data via the HTTP protocol, then unpacks the FLV data and provides it to the game software for playback; or alternatively, the game data is pushed via the RTMP protocol, sliced ​​and converted to a live stream in the form of HLS (HTTP Live Streaming, an HTTP-based adaptive bitrate streaming protocol) on the server side.

[0095] See also Figure 4 , Figure 4 This is an architecture diagram of a game live broadcast data processing system proposed in the first embodiment of this application. In this embodiment, the game live broadcast data processing system mainly includes, but is not limited to, the anchor terminal 2, the audience terminal 4, and the CDN 6.

[0096] The host terminal 2 is the client where the host of the live broadcast room is located, and the audience terminal 4 is the client where the users (audience) of the live broadcast room are located. The client can be a terminal device such as a PC (Personal Computer), a mobile phone, a tablet computer, a portable computer, a wearable device, etc.

[0097] The anchor terminal 2 is used to convert the game data into a data packet of a preset type and encapsulate it with specific video data and audio data, so that the encapsulated data can be distributed as a live stream through the existing CDN 6.

[0098] Specifically, when a game generates a frame of playback data, it is converted into an SEI (Supplemental Enhancement Information) packet of the H.264 or H.265 format, with the User Data Unregistered (UTD) type. This packet can contain custom data of any type and length.

[0099] The game generates "full data" every certain period of time (e.g., 3 seconds). In traditional game spectator scenarios, full data refers to all the data transmitted to a new viewer when they connect to the game server, as opposed to the partial data of other connected and currently watching viewers.

[0100] Furthermore, based on the duration of the content contained in this frame of game data, low-resolution, low-bitrate H.264 video data representing black screen data of equal duration is generated. Additionally, low-sampling-rate, low-bitrate AAC audio data representing silent data of equal duration is generated. An SEI data packet representing the game data is added to the video data. If the SEI data packet contains the full data, the resulting mixed video data after being added to the video data is a key frame.

[0101] It is worth noting that if the game itself contains more complex sounds, such as background music, singing, etc., or if the game needs to be narrated, the AAC audio data representing the silent data will be changed to the AAC data packet after the above sound encoding.

[0102] Finally, the mixed video data and the AAC audio data (or AAC data packet) are encapsulated and streamed to the CDN 6 via the RTMP protocol.

[0103] Through the above processing, the live broadcast server can treat this data as a standard live broadcast stream for processing and forwarding, and no modification is required to the existing live broadcast facilities.

[0104] The CDN 6 is used to distribute the data packaged by the anchor terminal 2 as a live stream to the viewer terminal 4.

[0105] In this embodiment, the encapsulated data can be regarded as a standard live broadcast stream, which is processed and forwarded by reusing the existing live broadcast mode (through the existing CDN network) and distributed to each viewer terminal 4. The specific distribution process is similar to the existing network live broadcast and will not be repeated here.

[0106] Optionally, in other embodiments, the encapsulated data may be sliced ​​and converted into a live stream in the form of HLS on the live server side, and then distributed to each viewer terminal 4 via the CDN 6. HLS is a live content distribution method that independently transmits each small segment (e.g., 1 second) of live video using the standard HTTP protocol. The viewer terminal 4 plays these videos continuously in sequence to achieve the purpose of watching the live broadcast.

[0107] Optionally, when the CDN 6 distributes the encapsulated data, it can also save the live video as in ordinary live broadcasting. That is, this embodiment is fully compatible with existing video storage solutions.

[0108] The audience terminal 4 is used to obtain the live stream from the CDN 6, and obtain the game data after parsing for playback.

[0109] The audience terminal 4 obtains the live stream through HTTP-FLV, HLS and other normal live viewing methods, obtains the H.264 video data and AAC audio data (or AAC data packet) after parsing, discards the black screen video data and silent audio data, restores the SEI data packet to the original game data, and hands it over to the game software for playback.

[0110] It is worth noting that if the viewer terminal 4 includes a player with P2P functionality, the P2P functionality can be directly reused to save bandwidth. The P2P functionality is based on HLS live streaming slicing. After each user (viewer terminal) downloads different parts of the slice, data is exchanged between users. This means that a single user does not need to download the entire data from the server, but can obtain the entire data by exchanging data between users, thereby saving server bandwidth.

[0111] For example, there are now two users, each downloading the first half and the second half of a live broadcast slice file. After the two users exchange their data, both users have the complete data. At this time, the bandwidth consumed by the server becomes half of the expected bandwidth, that is, the savings rate is 50%. P2P itself does not make any modifications to the transmitted data, so under the premise that the HLS format can achieve the purpose of the embodiment of this application, the P2P function can also be directly applied to this application.

[0112] In addition, although this embodiment is described using electronic games as an example, electronic games can also be replaced by other systems built on game engines, such as interactive movies and variety shows based on virtual characters, and their data can also be packaged and distributed in a similar manner to the game data, reusing the existing online live broadcast CDN infrastructure.

[0113] Example 2

[0114] like Figure 5 The figure shows a flowchart of a method for processing live game data according to the first embodiment of the present application. It is understood that the flowchart in this method embodiment is not intended to limit the order in which the steps are executed. Some steps in the flowchart may be added or deleted as needed. The method will be described below using the live game data processing system as the execution subject.

[0115] The method comprises the following steps:

[0116] S200: The host converts the game data into a data packet of a preset type and encapsulates it with specific video data and audio data.

[0117] For more details, see Figure 6, is a detailed flowchart of step S200. It is understood that this flowchart is not intended to limit the order in which the steps are performed. As needed, some steps in the flowchart may be added or deleted. In this embodiment, step S200 specifically includes:

[0118] S2000: Convert each frame of the game playback data into an SEI data packet.

[0119] When the game generates a frame of playback data, the frame is converted into an H.264 or H.265 SEI packet of type User data unregistered. This packet can contain custom data of any type and length.

[0120] The full amount of data is generated once every preset time period (eg, 3 seconds) and is also converted into a SEI data packet.

[0121] S2002: Generate black screen video data and audio data of equal length according to the time length of each frame of playback data.

[0122] In this embodiment, based on the duration of the content contained in each frame of game data, low-resolution, low-bitrate H.264 video data representing black screen data of equal duration is generated. Additionally, low-sampling-rate, low-bitrate AAC audio data representing silent data of equal duration is generated.

[0123] It is worth noting that if the game itself contains more complex sounds, such as background music, singing, commentary, etc., the AAC audio data representing silent data will be changed to an AAC data packet encoded by the sound output from the game.

[0124] S2004: Add the SEI data packet to the black screen video data to obtain mixed video data.

[0125] The SEI data packet representing the game data is added to the black screen video data. If the SEI data packet contains the full data, the mixed video data generated after adding the data to the video data is a key frame.

[0126] S2006: After encapsulating the mixed video data and the audio data, push the data to the CDN via the RTMP protocol.

[0127] The mixed video data and the AAC audio data (or AAC data packet) are encapsulated as an RTMP Tag, and pushed to the CDN (specifically, transmitted to the live broadcast server in the CDN) via the RTMP protocol, so that the encapsulated data can be distributed as a live broadcast stream through the CDN.

[0128] Back to Figure 5 , S202, the encapsulated data is distributed as a live stream to the audience through CDN.

[0129] In this embodiment, the encapsulated data can be regarded as a standard live stream, which is processed and forwarded by reusing the existing live broadcast mode (through the existing CDN network) and distributed to each viewer. The specific distribution process is similar to that of existing network live broadcasts and will not be repeated here.

[0130] Optionally, in other embodiments, the encapsulated data can also be sliced ​​and converted into a live stream in the form of HLS on the live server side, and then distributed to each viewer side through the CDN.

[0131] Optionally, when the CDN distributes the encapsulated data, it can also save the live video as in ordinary live broadcasts. That is, this embodiment is fully compatible with existing video storage solutions.

[0132] S204: The audience obtains the live stream, parses the game data and plays it back.

[0133] For more details, see Figure 7 , is a detailed flowchart of step S204. It is understood that this flowchart is not intended to limit the order in which the steps are performed. As needed, some steps in the flowchart may be added or deleted. In this embodiment, step S204 specifically includes:

[0134] S2040: Obtain the live stream from the CDN.

[0135] The audience side obtains the live stream by pulling the stream from the CDN in a normal way of watching live broadcasts, such as HTTP-FLV and HLS.

[0136] S2042: Parse the mixed video data and the audio data from the live stream.

[0137] By parsing the live stream in the FLV or HLS format, the mixed video data (black screen video data and additional SEI data packets) and the audio data (AAC audio data or AAC data packets) can be obtained.

[0138] If the audio data is AAC audio data representing silent data, it can be discarded directly; if the audio data is an encoded AAC data packet representing game background music, singing, commentary, etc., it does not need to be discarded and needs to be played together with the game data on the audience end.

[0139] S2044: Disassemble and assemble the SEI data packet from the mixed video data to restore the game data.

[0140] The mixed video data includes both H.264 video data representing black screen data and the SEI data packet in the additional information. The viewer discards the black screen video data and restores the original game data by disassembling and assembling the SEI data packet.

[0141] S2046: Replay the game at the audience end according to the game data.

[0142] By handing over the game data to the game software at the audience end for playback, the audience end can watch the game scene.

[0143] This embodiment focuses on encapsulating H.264 video (here, a black screen video plus game control data stored in the additional data section) and AAC audio, pushing them via the RTMP protocol, and multiplexing them over a CDN network for data distribution. Both FLV and HLS are transmission formats and protocols capable of encapsulating both H.264 and AAC, so either method can achieve the desired effect.

[0144] It is worth noting that if the viewer end includes a player with P2P function, the P2P function can be directly reused to save bandwidth.

[0145] In addition, although this embodiment is described using electronic games as an example, electronic games can also be replaced by other systems built on game engines, such as interactive movies and variety shows based on virtual characters, and their data can also be packaged and distributed in a similar manner to the game data, reusing the existing online live broadcast CDN infrastructure.

[0146] like Figure 8 FIG. 1 is a flowchart of another embodiment of the method for processing game live data based on the second embodiment. Figure 8 In the method, the game live broadcast data processing method includes:

[0147] (1) The SEI encapsulation module on the host side encapsulates each frame of the electronic game data into an SEI data packet, forming an H.264 stream representing the game data. The game generates full data every 3 seconds.

[0148] (2) The blank screen H.264 data generator generates blank screen H.264 data (black screen video data) of equal length according to the time length of the content contained in each frame of game data.

[0149] (3) The H.264 mixer mixes the H.264 stream of the game data and the H.264 data of the blank screen to obtain a mixed H.264 stream (mixed video data).

[0150] The H.264 stream can be replaced with other encoding formats that support the insertion of custom data packets, such as H.265.

[0151] (4) The AAC encoder encodes the game commentary sound to generate AAC data packets of the same length, forming an AAC stream. The commentary sound can also be other sounds in the game, such as background music.

[0152] (5) The RTMP encapsulator encapsulates the mixed H.264 stream and the AAC stream into an RTMP tag and pushes the stream to the live cloud CDN via the RTMP protocol.

[0153] (6) Reuse the live cloud CDN to distribute the encapsulated live stream.

[0154] Optionally, the live broadcast server can also slice and convert the live broadcast stream into HLS format and then distribute it.

[0155] Optionally, if video recording needs to be saved, this process is fully compatible with the existing video recording saving process.

[0156] (7) The live streaming module on the audience side pulls the stream from the CDN via HTTP-FLV.

[0157] When the live stream is converted to HLS format for distribution, viewers can also access the live stream via HTTP-HLS. Additionally, P2P functionality is optional. If the viewer supports P2P, P2P can be enabled to exchange live slices between users.

[0158] (8) The stream parsing module parses the live stream in FLV format, obtains the above-mentioned H.264 and AAC data, and discards the black screen video data.

[0159] (9) The AAC decoder decodes the AAC data packet and obtains the game commentary sound for playback.

[0160] (10) The SEI disassembly and assembly module disassembles and assembles the SEI data packet, restores it to the original game data, and gives it to the electronic game software for playback, so that the audience in the live broadcast room (audience-end users) can watch the game scene.

[0161] The live game data processing method proposed in this embodiment is based on the live game content distribution mechanism built on the existing live CDN network. By converting game data into SEI packets and encapsulating them with specific video data (black screen video data) and audio data (silent data), the CDN can process the game data as a normal live stream. The viewer can then parse the live stream to extract the original game data for playback. This embodiment can utilize the existing live CDN network architecture to distribute game content without modification. This is not only suitable for large-scale audiences watching live games, but also can significantly reduce server and development costs.

[0162] Example 3

[0163] like Figure 9 The figure below is a flowchart of a method for processing live game data according to the third embodiment of this application. It should be understood that the flowchart in this method embodiment does not limit the order in which the steps are executed. Some steps in the flowchart may be added or deleted as needed. The following describes the method using the host terminal as the execution subject.

[0164] The method comprises the following steps:

[0165] S300: Convert each frame of the game playback data into a data packet of a preset type.

[0166] In this embodiment, after the game generates a frame of playback data, the frame is converted into an H.264 or H.265 SEI data packet of type User data unregistered. This packet can contain custom data of any type and length.

[0167] The full amount of data is generated once every preset time period (eg, 3 seconds) and is also converted into a SEI data packet.

[0168] S302: Generate black screen video data and audio data of equal length according to the time length of each frame of playback data.

[0169] In this embodiment, based on the duration of the content contained in each frame of game data, low-resolution, low-bitrate H.264 video data representing black screen data of equal duration is generated. Additionally, low-sampling-rate, low-bitrate AAC audio data representing silent data of equal duration is generated.

[0170] It is worth noting that if the game itself contains more complex sounds, such as background music, singing, commentary, etc., the AAC audio data representing silent data will be changed to an AAC data packet encoded by the sound output from the game.

[0171] S304: Add the data packet to the black screen video data to obtain mixed video data.

[0172] The SEI data packet representing the game data is added to the black screen video data. If the SEI data packet contains the full data, the mixed video data generated after adding the data to the video data is a key frame.

[0173] S306: After encapsulating the mixed video data and the audio data, push the data to the CDN via the RTMP protocol.

[0174] The mixed video data and the AAC audio data (or AAC data packet) are encapsulated as an RTMP Tag, and pushed to the CDN (specifically, transmitted to the live broadcast server in the CDN) via the RTMP protocol, so that the encapsulated data can be distributed as a live broadcast stream through the CDN.

[0175] The game live broadcast data processing method proposed in this embodiment can be based on the game live broadcast content distribution mechanism built on the existing live broadcast CDN network. By converting the game data into an SEI data packet and then encapsulating it together with specific video data (black screen video data) and audio data (silent data), the CDN can process the game data as an ordinary live broadcast stream. The existing live broadcast CDN network architecture can be used without modification to distribute game content. This is not only suitable for large-scale audiences to watch game live broadcasts, but also can greatly reduce server costs and development costs.

[0176] Example 4

[0177] like Figure 10 The figure shows a flowchart of a method for processing live game data according to the fourth embodiment of the present application. It should be understood that the flowchart in this method embodiment is not intended to limit the order in which the steps are executed. Some steps in the flowchart may be added or deleted as needed. The method will be described below using the viewer terminal as the execution subject.

[0178] The method comprises the following steps:

[0179] S400: Obtain the live stream from the CDN.

[0180] The audience side obtains the live stream by pulling the stream from the CDN in a normal way of watching live broadcasts, such as HTTP-FLV and HLS.

[0181] S402: Parse the mixed video data and the audio data from the live stream.

[0182] By parsing the live stream in the FLV or HLS format, the mixed video data (black screen video data and additional SEI data packets) and the audio data (AAC audio data or AAC data packets) can be obtained.

[0183] If the audio data is AAC audio data representing silent data, it can be discarded directly; if the audio data is an encoded AAC data packet representing game background music, singing, commentary, etc., it does not need to be discarded and needs to be played together with the game data on the audience end.

[0184] S404: disassemble and assemble the data packet from the mixed video data to restore the game data.

[0185] The mixed video data includes both H.264 video data representing black screen data and the SEI data packet in the additional information. The viewer discards the black screen video data and restores the original game data by disassembling and assembling the SEI data packet.

[0186] S406: replay the game according to the game data.

[0187] By handing over the game data to the game software at the audience end for playback, the audience end can watch the game scene.

[0188] The live game data processing method proposed in this embodiment can leverage the existing live game content distribution mechanism built on CDN networks. The live stream is specially processed by the host and then encapsulated. The viewer then pulls and parses the live stream from the CDN, discarding any black screen or silent data to restore the original game data for playback. This embodiment can leverage the existing CDN network architecture for game content distribution without modification, making it suitable for large audiences viewing live games while significantly reducing server and development costs.

[0189] Example 5

[0190] like Figure 11 As shown, a module diagram of a game live broadcast data processing device 50 is proposed in the fifth embodiment of the present application. In this embodiment, the game live broadcast data processing device 50 is applied to the anchor end. The game live broadcast data processing device 50 can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can perform specific functions. The following description will specifically introduce the functions of each program module of this embodiment.

[0191] In this embodiment, the game live broadcast data processing device 50 includes:

[0192] The conversion module 500 is used to convert each frame of the game playback data into a data packet of a preset type.

[0193] In this embodiment, after the game generates a frame of playback data, the frame is converted into an H.264 or H.265 SEI data packet of type User data unregistered. This packet can contain custom data of any type and length.

[0194] The full amount of data is generated once every preset time period (eg, 3 seconds) and is also converted into a SEI data packet.

[0195] The generating module 502 is configured to generate black screen video data and audio data of equal length according to the time length of each frame of playback data.

[0196] In this embodiment, based on the duration of the content contained in each frame of game data, low-resolution, low-bitrate H.264 video data representing black screen data of equal duration is generated. Additionally, low-sampling-rate, low-bitrate AAC audio data representing silent data of equal duration is generated.

[0197] It is worth noting that if the game itself contains more complex sounds, such as background music, singing, commentary, etc., the AAC audio data representing silent data will be changed to an AAC data packet encoded by the sound output from the game.

[0198] The mixing module 504 is configured to add the data packet to the black screen video data to obtain mixed video data.

[0199] The SEI data packet representing the game data is added to the black screen video data. If the SEI data packet contains the full data, the mixed video data generated after adding the data to the video data is a key frame.

[0200] The encapsulation module 506 is configured to encapsulate the mixed video data and the audio data and then stream the encapsulated data to the CDN via the RTMP protocol.

[0201] The mixed video data and the AAC audio data (or AAC data packet) are encapsulated as an RTMP Tag, and pushed to the CDN (specifically, transmitted to the live broadcast server in the CDN) via the RTMP protocol, so that the encapsulated data can be distributed as a live broadcast stream through the CDN.

[0202] The game live broadcast data processing device proposed in this embodiment can be based on the game live broadcast content distribution mechanism built on the existing live broadcast CDN network. By converting the game data into SEI data packets and encapsulating them together with specific video data (black screen video data) and audio data (silent data), the CDN can process the game data as an ordinary live broadcast stream and distribute the game content using the existing live broadcast CDN network architecture without modification. This is not only suitable for large-scale audiences to watch game live broadcasts, but also can greatly reduce server costs and development costs.

[0203] Example 6

[0204] like Figure 12 As shown, a module diagram of a game live broadcast data processing device 60 is proposed in the sixth embodiment of the present application. In this embodiment, the game live broadcast data processing device 60 is applied to the audience end. The game live broadcast data processing device 60 can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can perform specific functions. The following description will specifically introduce the functions of each program module of this embodiment.

[0205] In this embodiment, the game live broadcast data processing device 60 includes:

[0206] The acquisition module 600 is configured to acquire the live stream from the CDN.

[0207] In this embodiment, the acquisition module 600 can pull the stream from the CDN to obtain the live stream through a normal way of watching live broadcasts, such as HTTP-FLV and HLS.

[0208] The parsing module 602 is configured to parse the live stream to obtain the mixed video data and the audio data.

[0209] By parsing the live stream in the FLV or HLS format, the mixed video data (black screen video data and additional SEI data packets) and the audio data (AAC audio data or AAC data packets) can be obtained.

[0210] If the audio data is AAC audio data representing silent data, it can be discarded directly; if the audio data is an encoded AAC data packet representing game background music, singing, commentary, etc., it does not need to be discarded and needs to be played together with the game data on the audience end.

[0211] The restoration module 604 is configured to disassemble and assemble the SEI data packet from the mixed video data to restore the game data.

[0212] The mixed video data includes both H.264 video data representing black screen data and the SEI data packet in the additional information. The viewer discards the black screen video data and restores the original game data by disassembling and assembling the SEI data packet.

[0213] The playing module 606 is configured to replay the game according to the game data.

[0214] By handing over the game data to the game software at the audience end for playback, the audience end can watch the game scene.

[0215] The live game data processing device proposed in this embodiment can leverage the existing live game content distribution mechanism built on CDN networks. The live stream is specially processed by the host and then encapsulated. The viewer then pulls and parses the live stream from the CDN, discarding any black screen or silent data, and restoring the original game data for playback. This embodiment can utilize the existing CDN network architecture for game content distribution without modification, making it suitable for large audiences viewing live games while significantly reducing server and development costs.

[0216] Example 7

[0217] like Figure 13 FIG2 is a schematic diagram of the hardware architecture of a computer device 20 according to the seventh embodiment of the present application. In this embodiment, the computer device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23 that can be interconnected via a system bus. It should be noted that Figure 13 Only a computer device 20 having components 21-23 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. In this embodiment, the computer device 20 may be the host terminal or the viewer terminal, etc.

[0218] The memory 21 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 can be an internal storage unit of the computer device 20, such as the hard disk or memory of the computer device 20. In other embodiments, the memory 21 can also be an external storage device of the computer device 20, such as a plug-in hard disk equipped on the computer device 20, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 21 can also include both the internal storage unit of the computer device 20 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed on the computer device 20, such as the program code of the game live data processing device in the embodiment. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or are to be output.

[0219] In some embodiments, the processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 22 is generally used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is used to execute program code stored in the memory 21 or process data.

[0220] The network interface 23 may include a wireless network interface or a wired network interface. The network interface 23 is generally used to establish a communication connection between the computer device 20 and other electronic devices. For example, the network interface 23 is used to connect the computer device 20 to an external server or terminal via a network, and to establish a data transmission channel and a communication connection between the computer device 20 and the external server or terminal. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0221] In this embodiment, the game live broadcast data processing device stored in the memory 21 can also be divided into one or more program modules and executed by one or more processors (processor 22 in this embodiment) to complete this application.

[0222] Example 8

[0223] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the game live broadcast data processing method as described above.

[0224] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0225] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0226] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0227] The above are only preferred embodiments of the embodiments of the present application, and do not limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the embodiments of the present application.

Claims

1. A method for processing game live broadcast data, characterized in that: The method comprises: The host converts the game data into a data packet of a preset type and encapsulates it with specific video and audio data; The encapsulated data is used as a live stream and distributed to the audience through the content delivery network CDN; The audience obtains the live stream, parses the game data and plays it back; The step of converting the game data into a data packet of a preset type and encapsulating the data packet with the specific video data and audio data includes: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Generate black screen video data and audio data of equal length according to the time length of each frame of playback data; Adding the SEI data packet to the black screen video data to obtain mixed video data; After the mixed video data and the audio data are encapsulated, they are pushed to the content distribution network via the RTMP protocol.

2. The game live broadcast data processing method according to claim 1, characterized in that: The converting of each frame of the game playback data into an SEI data packet further comprises: Generate full game data every preset time period and convert it into SEI data packets.

3. The game live broadcast data processing method according to claim 1 or 2, characterized in that: The generating of black screen video data of equal length includes: Generate low-resolution, low-bitrate H.264 video data representing black screen data of equal length.

4. The game live broadcast data processing method according to claim 2, characterized in that: Generating audio data of equal time length includes: Generate low sampling rate, low bit rate AAC audio data representing silence data of equal length; or Encode game sounds or commentary sounds of equal length to generate AAC data packets.

5. The game live broadcast data processing method according to any one of claims 1 to 2, characterized in that: The method further includes, before distributing the encapsulated data as a live stream to the viewer via the CDN: The encapsulated data is sliced ​​on the live broadcast server and converted into a live broadcast stream in the form of HLS.

6. The method for processing game live broadcast data according to any one of claims 1 to 2, characterized in that: The acquiring of the live stream and parsing of the game data for playback includes: Obtain the live stream from the CDN; Parsing the mixed video data and the audio data from the live stream; disassembling the data packet from the mixed video data to restore the game data; The game is replayed at the audience end according to the game data.

7. A game live broadcast data processing system, characterized in that: The system comprises: The host side is used to convert game data into a preset type of data packet, encapsulate it with specific video data and audio data, and push it to the content delivery network CDN; The CDN is used to distribute the data packaged by the anchor end as a live stream to the audience end; The viewer terminal is used to obtain the live stream from the CDN, parse the game data and play it back; The host terminal is also used for: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Generate black screen video data and audio data of equal length according to the time length of each frame of playback data; Adding the SEI data packet to the black screen video data to obtain mixed video data; After the mixed video data and the audio data are encapsulated, they are pushed to the content distribution network via the RTMP protocol.

8. A method for processing game live broadcast data, applied to the host end, characterized in that: The method comprises: Converting each frame of the game replay data into a data packet of a preset type; generating black screen video data and audio data of the same length according to the time length of each frame of the replay data; Adding the data packet to the black screen video data to obtain mixed video data; After encapsulating the mixed video data and the audio data, the mixed video data and the audio data are pushed to the content distribution network CDN via the RTMP protocol; The process of converting the game data into a data packet of a preset type and encapsulating the data packet with specific video data and audio data includes: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Correspondingly, the data packet is added to the black screen video data to obtain mixed video data, including: The SEI data packet is added to the black screen video data to obtain mixed video data.

9. The game live broadcast data processing method according to claim 8, characterized in that: The step of converting the replay data of each frame of the game into a data packet of a preset type includes: Whenever the game generates a frame of playback data, the frame of playback data is converted into a supplemental enhancement information SEI data packet in H.264 or H.265 format.

10. The game live broadcast data processing method according to claim 9, characterized in that: The step of converting the replay data of each frame of the game into a data packet of a preset type further comprises: Generate full game data every preset time period and convert it into SEI data packets.

11. The method for processing game live broadcast data according to any one of claims 8 to 10, characterized in that: The generating of black screen video data of equal length includes: Generate low-resolution, low-bitrate H.264 video data representing black screen data of equal length.

12. The method for processing game live broadcast data according to any one of claims 8 to 10, characterized in that: Generating audio data of equal time length includes: Generate low sampling rate, low bit rate AAC audio data representing silence data of equal length; or Encode game sounds or commentary sounds of equal length to generate AAC data packets.

13. The method for processing game live broadcast data according to claim 10, wherein: The step of adding the data packet to the black screen video data to obtain mixed video data further comprises: When the data packet contains the full data, the mixed video data generated after being added to the black screen video data is a key frame.

14. A game live broadcast data processing device, applied to the anchor terminal, characterized in that: The device comprises: A conversion module, configured to convert each frame of the game playback data into a data packet of a preset type; A generating module, configured to generate black screen video data and audio data of equal length according to the time length of each frame of playback data; A mixing module, configured to add the data packet to the black screen video data to obtain mixed video data; An encapsulation module is used to encapsulate the mixed video data and the audio data and push the data to a content delivery network CDN via the RTMP protocol; Wherein, the conversion module is further used for: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Wherein, the mixing module is further used for: The SEI data packet is added to the black screen video data to obtain mixed video data.

15. A method for processing game live broadcast data, applied to the audience end, characterized in that: The method comprises: Obtain a live stream from a content delivery network (CDN), wherein the live stream is generated by the host converting the game data into a data packet of a preset type, encapsulating it with black screen video data and audio data, and then pushing the stream to the CDN; Parsing mixed video data and the audio data from the live stream, wherein the mixed video data includes the black screen video data and the additional data packet; disassembling the data packet from the mixed video data to restore the game data; replaying the game based on the game data; The step of converting the game data into a data packet of a preset type, encapsulating the data packet with the black screen video data and the audio data, and then streaming the data to the CDN includes: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Generate black screen video data and audio data of equal length according to the time length of each frame of playback data; Adding the SEI data packet to the black screen video data to obtain mixed video data; After the mixed video data and the audio data are encapsulated, they are pushed to the CDN via the RTMP protocol.

16. The method for processing game live broadcast data according to claim 15, wherein: The method further comprises: When the audio data is AAC audio data representing silent data, the audio data is discarded; when the audio data is an AAC data packet generated by encoding game sounds or commentary sounds, the audio data is parsed to obtain the game sounds or commentary sounds, and played together with the game data.

17. A game live broadcast data processing device, applied to the audience end, characterized in that: The device comprises: An acquisition module is used to obtain a live stream from a content delivery network (CDN), wherein the live stream is a stream in which the host converts the game data into a data packet of a preset type, encapsulates it with black screen video data and audio data, and then pushes the stream to the CDN; A parsing module, configured to parse the live stream to obtain mixed video data and the audio data, wherein the mixed video data includes the black screen video data and the additional data packet; a disassembly module, configured to disassemble and assemble the data packet from the mixed video data to restore the game data; A playback module, configured to replay the game based on the game data; The step of converting the game data into a data packet of a preset type, encapsulating the data packet with the black screen video data and the audio data, and then streaming the data to the CDN includes: Convert each frame of the game playback data into a supplementary enhancement information SEI data packet; Generate black screen video data and audio data of equal length according to the time length of each frame of playback data; Adding the SEI data packet to the black screen video data to obtain mixed video data; After the mixed video data and the audio data are encapsulated, they are pushed to the CDN via the RTMP protocol.

18. A computer device, characterized in that: The computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the game live broadcast data processing method as described in any one of claims 8 to 13 or 15 to 16 is implemented.

19. A computer-readable storage medium, characterized in that A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the game live broadcast data processing method as described in any one of claims 1 to 6 or 8 to 13 or 15 to 16 is implemented.

20. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, the game live broadcast data processing method as described in any one of claims 1 to 6 or 8 to 13 or 15 to 16 is implemented.

Images