Live broadcast processing method and device, terminal and storage medium

Abstract

The invention discloses a live broadcast processing method and device, a terminal and a storage medium, and the method comprises the steps: obtaining the game audio and video data of a target game at the current moment through a game engine of the target game played by an anchor user in a game live broadcast process, and enabling the target game to run on a target system platform; the game engine is controlled to obtain environment audio and video data on the target system platform through the cross-platform interface, and the environment audio and video data is obtained by calling an audio and video collection assembly configured by the target system platform to conduct data collection on the environment where the anchor user is located at the current moment; and generating live broadcast data by adopting the game audio and video data and the environment audio and video data, and pushing the live broadcast data to a server, so that the server issues the live broadcast data to an audience terminal for playing. The convenience of game live broadcast can be improved, and the live broadcast effect of game live broadcast can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, specifically to the field of communication technology, and in particular to a live broadcast processing method, device, terminal and storage medium. Background Art

[0002] With the development of Internet technology and live broadcast technology, online game live broadcast has been proposed. Online game live broadcast can be referred to as game live broadcast, which is a live broadcast method of broadcasting the game process in real time over the Internet; the live game process mentioned here can be understood as: the game screen displayed on the terminal screen of the real-time live game player (i.e. the anchor user).

[0003] At present, game players need to use third-party software (such as live broadcast software) to broadcast game live. The third-party software mainly records the game screen displayed on the game player's terminal screen, and then sends the recorded screen to the audience terminal for display to realize game live broadcast. In this way, game players must use third-party software to realize game live broadcast, which is less convenient; and in this way, the terminal not only needs to run the game application during the game live broadcast, but also consumes a lot of processing resources to run the third-party software, which affects the terminal's operating performance, causing the game to freeze during operation, and further affects the live broadcast effect of the game live broadcast. Summary of the invention

[0004] The embodiments of the present application provide a live broadcast processing method, device, terminal and storage medium, which can not only effectively improve the convenience of game live broadcast, but also save processing resources and reduce the phenomenon of game jams during operation, thereby improving the live broadcast effect of game live broadcast.

[0005] On the one hand, an embodiment of the present application provides a live broadcast processing method, the method comprising:

[0006] During the live broadcast of the game, the game audio and video data of the target game at the current moment is obtained through the game engine of the target game played by the anchor user; the target game is run on a target system platform, and the target system platform is configured with an audio and video acquisition component;

[0007] Control the game engine to obtain environmental audio and video data on the target system platform through a cross-platform interface, wherein the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment;

[0008] The game audio and video data and the environmental audio and video data are used to generate live broadcast data, and the live broadcast data is pushed to a server, so that the server sends the live broadcast data to the audience terminal for playback.

[0009] On the other hand, an embodiment of the present application provides a live broadcast processing device, the device comprising:

[0010] An acquisition unit is used to acquire game audio and video data of a target game at a current moment through a game engine of the target game played by the anchor user during the live broadcast of the game; the target game is run on a target system platform, and the target system platform is configured with an audio and video acquisition component;

[0011] The acquisition unit is further used to control the game engine to acquire the environmental audio and video data on the target system platform through a cross-platform interface, wherein the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment;

[0012] The processing unit is used to generate live broadcast data using the game audio and video data and the environmental audio and video data, and push the live broadcast data to the server, so that the server sends the live broadcast data to the audience terminal for playback.

[0013] On the other hand, an embodiment of the present application provides a terminal, the terminal including an input interface and an output interface, the terminal further including:

[0014] processor and computer storage media;

[0015] Among them, the processor is suitable for implementing one or more instructions, the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned live broadcast processing method.

[0016] On the other hand, an embodiment of the present application provides a computer storage medium, which stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the above-mentioned live broadcast processing method.

[0017] On the other hand, an embodiment of the present application provides a computer program product, which includes one or more instructions; when one or more instructions in the computer program product are executed by a processor, the above-mentioned live broadcast processing method is implemented.

[0018] In the process of live broadcasting of the game, the embodiment of the present application can obtain the game audio and video data of the target game at the current moment and the ambient audio and video data on the host user side through the game engine of the target game played by the host user, thereby generating live broadcast data using the game audio and video data and the ambient audio and video data, and sending the live broadcast data to the audience terminal through the server for playback, so that the audience user can not only watch the game screen on the host user side, but also listen to the game audio data on the host user side and watch the ambient audio and video data on the host user side at the same time, enriching the live broadcast content of the game live broadcast and effectively improving the live broadcast effect of the game live broadcast. In addition, by using the game engine to realize the generation of live broadcast data, the host user does not need to use third-party software to broadcast the game live, which not only improves the convenience of game live broadcasting, but also enables the terminal to save processing resources during the game live broadcast without consuming additional processing resources to run third-party software, thereby avoiding affecting the terminal's operating performance due to the third-party software occupying more operating memory, thereby reducing the phenomenon of freezing during the operation of the game, and further improving the live broadcast effect of the game live broadcast. Furthermore, by developing a cross-platform interface, no matter which system platform the target game runs on, the game engine can collect data from the corresponding system platform through the cross-platform interface. This will make game live streaming not limited to a single system platform and can effectively improve the applicability of game live streaming. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1a It is a schematic diagram of a cross-platform audio and video streaming solution provided in an embodiment of the present application;

[0021] Figure 1b This is a schematic diagram of a transmission of live broadcast data provided by an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a live broadcast processing method provided by an embodiment of the present application;

[0023] Figure 3 is a flowchart of a live broadcast processing method provided by another embodiment of the present application;

[0024] Figure 4a It is a schematic diagram of adding a watermark on a game screen provided by an embodiment of the present application;

[0025] Figure 4b It is a schematic diagram of adding stickers to a game screen provided by an embodiment of the present application;

[0026] Figure 4c It is a schematic diagram of a process of performing video post-processing on a game screen and an environment screen provided by an embodiment of the present application;

[0027] Figure 4d It is a schematic diagram of superimposing an environment screen on a game screen provided by an embodiment of the present application;

[0028] Figure 4e is another schematic diagram of superimposing an environment screen on a game screen provided in an embodiment of the present application;

[0029] Figure 4f is a schematic diagram of transferring a mixed image from a GPU to a CPU provided by an embodiment of the present application;

[0030] Figure 4g is another schematic diagram of transmitting a mixed image from a GPU to a CPU provided by an embodiment of the present application;

[0031] Figure 4h is a schematic diagram of a mixing process provided by an embodiment of the present application;

[0032] Figure 4i It is a schematic diagram of a video encoding and audio encoding process provided by an embodiment of the present application;

[0033] Figure 4j This is a schematic diagram of a process for streaming live data provided by an embodiment of the present application;

[0034] Figure 5 is a flowchart of another live broadcast processing method provided by another embodiment of the present application;

[0035] Figure 6 It is a structural schematic diagram of a live broadcast processing device provided in an embodiment of the present application;

[0036] Figure 7 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] In the embodiments of the present application, the games mentioned later all refer to: electronic games running on the system platform in the terminal. The terminals mentioned here can be: smart phones, computers (such as tablets, laptops, desktop computers, etc.), smart wearable devices (such as smart watches, smart glasses), smart voice interaction devices, smart home appliances (such as smart TVs), vehicle terminals or aircraft, etc. The system platform in the terminal can be understood as an operating system (OS), which may include but is not limited to: Android system (Android system), Windows system (Windows system), IOS system (a mobile operating system), Mac system (a graphical operating system based on the Unix kernel), Unix operating system (an operating system that supports multi-tasking), Hongmeng system, etc.

[0039] It should be noted that the embodiments of the present application do not limit the specific types of games mentioned later. For example, from the perspective of the implementation conditions of electronic games, the games mentioned later may be stand-alone games or online games. Among them, stand-alone games refer to games that are completed locally on the terminal and do not require a network; correspondingly, online games refer to games that require the terminal and the game server to rely on the network to complete together and can support multiple users to participate at the same time. For another example, from the perspective of the running carrier of electronic games, the games mentioned later may be terminal games, mobile games, or online games, and so on. Among them, terminal games refer to games that run in game applications installed on the system platform of personal computers (such as desktop computers, laptops, etc.); mobile games refer to games that run in game applications installed on the system platform of mobile devices (such as mobile phones, tablet computers, smart watches, etc.); online games refer to games that run in web pages provided on the operating platform in the terminal.

[0040] Any game can be configured with a game engine, which refers to an engine that can be used to develop games and draw and display each game screen during the running of the game. Exemplary, the game engine mentioned in the embodiment of the present application can be Unreal Engine (UE), Unity (a game engine that can support real-time 3D (three-dimensional) interactive content creation and operation), or Cocos (a game engine with open source, free, lightweight, high performance and other characteristics), etc. Among them, Unreal Engine is a 3A (AAA, the highest level) next-generation game engine, and there are multiple versions of it, such as UE4 (Unreal Engine 4, the 4th version of Unreal Engine), UE5 (Unreal Engine 5, the 5th version of Unreal Engine), etc.

[0041] In order to improve the convenience and effect of game live broadcast, the present application embodiment proposes a cross-platform audio and video streaming solution. Figure 1a As shown, the general principle of the scheme is as follows: for a game running on any system platform, during the live broadcast of the game by the anchor user, the anchor terminal (i.e., the terminal used by the anchor user) can use the game engine of the game to collect the game screen and camera data (i.e., the environment screen (or camera screen) corresponding to the environment where the anchor user is located) and other video data on the corresponding system platform at the current moment, and collect the game audio data and microphone data (i.e., the audio data corresponding to the environment where the anchor user is located) and other audio data on the corresponding system platform at the current moment; secondly, perform video post-processing on the collected video data, and perform video encoding on the video post-processed video data (i.e., formatting the video data), and perform audio post-processing on the collected audio data, and perform audio encoding on the audio post-processed audio data (i.e., formatting the audio data); then, the video encoding results and the audio encoding results can be encapsulated to obtain live broadcast data, so that the live broadcast data can be pushed to the server (a server that can provide live broadcast services), and the server performs data forwarding to forward the live broadcast data to each audience terminal (i.e., the terminal used by the audience user) for playback (such as Figure 1b As shown), each audience user can watch the game process of the anchor user in real time through the audience terminal.

[0042] Among them, the video post-processing mentioned above refers to: the processing of reprocessing (adjusting) the video data; for example, it may include but is not limited to: adding watermarks, adding stickers, adding filters and other operations. The audio post-processing mentioned above refers to: the processing of reprocessing (adjusting) the audio data; for example, it may include but is not limited to: voice changing (i.e. changing the timbre of the sound), pitch changing (i.e. changing the pitch of the sound), mixing and resampling and other operations. It should be noted that the video post-processing and audio post-processing mentioned above can be used as optional operations. In other words, after collecting video data such as game screen and camera data, it is also possible to directly encode the video data without performing video post-processing; similarly, after collecting audio data such as game audio data and microphone data, it is also possible to directly encode the audio data without performing audio post-processing.

[0043] Based on the above description, it should be noted that: the above is an example of the anchor terminal as the execution subject to explain the general principle of the solution; in actual application, the solution can also be executed by the game application in the anchor terminal, or by the game engine in the anchor terminal, without limitation. Figure 1bIt is just an example to list several audience clients, and it is not limited to this; and the server mentioned above can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc. Moreover, the embodiment of the present application does not limit the communication method between the server and any terminal (such as the anchor terminal and the audience terminal); it can achieve communication through any communication method, such as mobile communication based on the fourth generation mobile communication technology (4G), the fifth generation mobile communication technology (5G), Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), or computer network communication based on TCP / IP Protocol Suite (TCP / IP) and User Datagram Protocol (UDP).

[0044] Practice has shown that the cross-platform audio and video streaming solution proposed in the embodiment of the present application can have at least the following beneficial effects: ① It can realize the collection and encoding of audio and video data for the system platform located in the PC (computer) end or the mobile end, and has high compatibility and applicability. ② This solution can not only stream game screens and game audio data, but also stream microphone data and camera screens (i.e., environmental screens), which can enrich the live content of game live broadcasts and improve the live effect of game live broadcasts. ③ It supports the processing of audio data and video data (i.e., post-processing), so that some effects that are not convenient for front-end display can be completed through post-processing, enriching the display effect of audio data and video data, so that audience users can watch richer live content and improve the live effect of game live broadcasts. ④ By using the game engine to collect data and generate live broadcast data, the host user does not need to use third-party software to broadcast the game live. This not only improves the convenience of game live broadcast, but also allows the terminal to run third-party software without consuming additional processing resources during the game live broadcast, thereby saving processing resources and avoiding the impact of third-party software occupying more running memory on the terminal's running performance, thereby reducing the phenomenon of game freezes during operation and improving the live broadcast effect of the game.

[0045] Based on the cross-platform audio and video streaming solution mentioned above, the embodiment of the present application proposes a live broadcast processing method. The execution subject of the live broadcast processing method can be the anchor terminal, or the game application in the anchor terminal, or the game engine in the anchor terminal, etc. For the sake of ease of explanation, the embodiment of the present application takes the anchor terminal executing the live broadcast processing method as an example. Please refer to Figure 2 As shown, the live broadcast processing method may include the following steps S201-S204:

[0046] S201, during the live broadcast of the game, obtaining the game audio and video data of the target game at the current moment through the game engine of the target game played by the anchor user.

[0047] The target game can be any game, such as a PC game, a mobile game, or an online game. The target game runs on a target system platform, which refers to the operating platform in the anchor terminal; and the target system platform can be configured with an audio and video acquisition component, which refers to a component for collecting audio and video data, which can specifically include a camera and a microphone. It should be noted that the audio and video acquisition component configured by the target operating system can be an audio and video acquisition component carried by the anchor terminal, or an audio and video acquisition component external to the anchor terminal, and there is no limitation on this.

[0048] The game audio and video data of the target game at the current moment may include: game screen and game audio data; the game audio data mentioned here can be understood as: background music that is triggered to play due to specific behaviors or specific operations of the anchor user during the target game, such as the theme song of the game, music during battles, sound effects of clicking a button, etc. Among them, the current moment refers to: the latest moment (i.e. the current moment) displayed by the system clock of the anchor terminal during the target game; it should be understood that the moment displayed by the system clock of the anchor terminal can be automatically updated as time passes. For example, if the latest moment displayed by the system clock of the anchor terminal is 10:40, the current moment is 10:40; after 10 minutes, the system clock of the anchor terminal will display 10:50, and the current moment at this time is 10:50.

[0049] Since the game engine of the target game can draw and render the game screen of the target game in real time during the running of the target game, and the game engine of the target game can have its own screen acquisition interface (i.e., an interface for acquiring the game screen) and its own audio acquisition interface (i.e., an interface for acquiring the game audio data of the target game); therefore, when the anchor terminal executes step S201, it can directly acquire the game screen of the target game at the current moment from the game engine through the screen acquisition interface of the game engine, and acquire the game audio data of the target game at the current moment through the audio acquisition interface of the game engine, thereby obtaining the audio and video data of the target game at the current moment.

[0050] S202, controlling the game engine to obtain ambient audio and video data on the target system platform through a cross-platform interface.

[0051] Among them, the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment. As can be seen from the above, the audio and video acquisition component may include a camera and a microphone; then the environmental audio and video data collected by the audio and video acquisition component may include: the environmental picture corresponding to the environment where the anchor user is located collected by the camera, and the environmental audio data corresponding to the environment where the anchor user is located collected by the microphone. Further, the environmental picture may include at least one of the following: the anchor's face image obtained by collecting the anchor user's face, the image obtained by shooting other objects in the environment where the anchor user is located except the anchor user, and so on. The environmental audio data may include at least one of the following: the sound made by the anchor user during the game (such as the sound of the game commentary, the sound of communicating with teammates), the background music played by the anchor user during the game, and so on.

[0052] The cross-platform interface mentioned in the embodiment of the present application can also be called a full-platform interface, which can be used to transmit data with the full platform (i.e., all system platforms). In a specific implementation, the cross-platform interface can be an interface that supports both video data (i.e., image data) and audio data transmission; in this case, a plurality of sub-interfaces can be encapsulated in the cross-platform interface, one sub-interface is adapted to one system platform, different sub-interfaces are adapted to different system platforms, any sub-interface is generated according to the platform characteristics of the corresponding system platform, and any sub-interface is used to transmit video data and audio data on the corresponding system platform.

[0053] Then, correspondingly, when the anchor terminal executes step S202, it can directly control the game engine to call the cross-platform interface, so that the cross-platform interface selects a sub-interface that is compatible with the target system platform from multiple sub-interfaces inside, and calls the selected sub-interface to obtain the environmental image and environmental audio data from the target system platform. It can be seen that the embodiment of the present application can make the cross-platform interface compatible with each system platform by encapsulating multiple sub-interfaces into one interface (i.e., a cross-platform interface), so that no matter which system platform the environmental audio and video data is obtained on, it can be achieved by calling the cross-platform interface, and the anchor terminal does not need to pay attention to which specific sub-interface should be called. Such a unified calling method can achieve no perception of the upper layer of the application and improve the user experience.

[0054] In another specific implementation, the cross-platform interface may include: a first cross-platform interface and a second cross-platform interface; wherein the first cross-platform interface is an interface for transmitting video data, and the second cross-platform interface is an interface for transmitting audio data. In this case, when the anchor terminal executes step S202, it may control the game engine to obtain the environment picture on the target system platform through the first cross-platform interface; and control the game engine to obtain the environment audio data on the target system platform through the second cross-platform interface.

[0055] Furthermore, a plurality of first sub-interfaces are encapsulated in the cross-platform first interface, one first sub-interface is adapted to one system platform, different first sub-interfaces are adapted to different system platforms, any first sub-interface is generated according to the platform characteristics of the corresponding system platform, and any first sub-interface is used to transmit video data on the corresponding system platform. Then, correspondingly, when the anchor terminal controls the game engine to obtain the environment screen on the target system platform through the cross-platform first interface, it can control the game engine to call the cross-platform first interface, so that the cross-platform first interface selects the first sub-interface adapted to the target system platform from the multiple first sub-interfaces inside, and calls the selected first sub-interface to obtain the environment screen from the target system platform.

[0056] Similarly, a cross-platform second interface encapsulates multiple second sub-interfaces, one second sub-interface is adapted to one system platform, different second sub-interfaces are adapted to different system platforms, any second sub-interface is generated according to the platform characteristics of the corresponding system platform, and any second sub-interface is used to transmit audio data on the corresponding system platform. Then, when the anchor terminal controls the game engine to obtain the ambient audio data on the target system platform through the cross-platform second interface, it can control the game engine to call the cross-platform second interface, so that the cross-platform second interface selects the second sub-interface adapted to the target system platform from the multiple second sub-interfaces inside, and calls the selected second sub-interface to obtain the ambient audio data from the target system platform.

[0057] It can be seen that the embodiment of the present application encapsulates multiple first sub-interfaces into one interface (i.e., a cross-platform first interface), and encapsulates multiple second sub-interfaces into one interface (i.e., a cross-platform second interface), so that the anchor terminal can use a unified calling method to obtain environmental images and environmental audio data for each system platform, thereby achieving unawareness of the upper layer of the application and improving the user experience.

[0058] S203, using the game audio and video data and the environmental audio and video data to generate live broadcast data.

[0059] As can be seen from the above, the game audio and video data includes: game screen and game audio data; the environmental audio and video data includes: environmental screen and environmental audio data corresponding to the environment where the anchor user is located. Then correspondingly, when the anchor terminal executes step S203, it can respectively perform video encoding on the game screen in the game audio and video data, and perform audio encoding on the game audio data in the game audio and video data, to obtain the encoding result of the game audio and video data, and the encoding result of the game audio and video data includes: the encoded game screen and the encoded game audio data; similarly, the environmental screen in the environmental audio and video data can be video encoded, and the environmental audio data in the environmental audio and video data can be audio encoded, to obtain the encoding result of the environmental audio and video data, and the encoding result of the environmental audio and video data includes: the encoded environmental screen and the encoded environmental audio data. Then, the encoding results of the game audio and video data and the encoding results of the environmental audio and video data can be encapsulated according to the preset encapsulation format to obtain live broadcast data.

[0060] Alternatively, considering that the game screen and the environment screen are two independent screens, in order to avoid the problem that each viewer terminal uses different display modes to display the two screens, so that the display forms of the two screens in different viewer terminals are inconsistent, the two screens can be mixed into one screen before video encoding, so as to achieve the unified display form of the two screens in each viewer terminal. Similarly, considering that the game audio data and the environment audio data are two independent audio data, in order to avoid the problem that each viewer terminal uses different playback modes to play the two audio data, so that the playback effects of the two audio data in different viewer terminals are inconsistent, the two audio data can be mixed into one audio data before audio encoding, so as to achieve the unified playback effects of the two audio data in each viewer terminal. Based on this, when the anchor terminal executes step S203, it can perform screen mixing processing on the game screen and the environment screen to obtain a mixed screen; and perform video encoding on the mixed screen to obtain a video data stream. And, the game audio data and the environment audio data are mixed to obtain a mixing result; and the mixing result is audio encoded to obtain an audio data stream. Then, the video data stream and the audio data stream are encapsulated according to a preset encapsulation format to obtain live broadcast data.

[0061] Among them, the preset encapsulation format mentioned above refers to: a pre-set encapsulation format according to actual needs or experience values; and the encapsulation format refers to a format used to package the results of video encoding and audio encoding, and merge them into a file in a specific way. Exemplarily, the preset encapsulation format mentioned above can be an FLV (Flash Video) encapsulation format, which is a streaming media format, which has the characteristics of simple encapsulation, and the encapsulated audio and video files obtained by the FLV encapsulation format have the characteristics of small size, and the suffix of the encapsulated audio and video files obtained by the FLV encapsulation format is .flv. Based on this, it can be seen that the embodiment of the present application adopts the FLV encapsulation format to perform the encapsulation operation, which can make the encapsulation process simpler and improve the encapsulation efficiency on the one hand, and on the other hand, the volume of the live broadcast data (i.e., audio and video files) obtained by the encapsulation operation can be smaller, thereby reducing the network resources required for transmitting the live broadcast data and shortening the data transmission time, improving the data transmission efficiency, and thus improving the real-time performance of the live broadcast. Optionally, the preset encapsulation format may also be an MP4 encapsulation format, where MP4 is an extended media format defined by the MPEG-4 (a digital multimedia compression format) video standard and the AAC (Advance Audio Coding) audio standard.

[0062] S204, pushing the live broadcast data to the server, so that the server sends the live broadcast data to the viewer terminal for playback.

[0063] Among them, streaming refers to the process of encapsulating the collected audio and video data and transmitting it to the server. In other words, the essence of the anchor terminal pushing the live data to the server can be understood as: using the streaming media transmission protocol to send the live data to the server. The streaming media transmission protocol mentioned here is a protocol for transmitting multimedia through the web (network). Correspondingly, after receiving the live data, the server can send the live data to each currently connected audience terminal by data forwarding, so that each corresponding audience terminal plays the live data. Among them, each currently connected audience terminal refers to: the audience terminal that has established a communication connection with the server at the current moment.

[0064] It should be noted that the embodiment of the present application does not limit the streaming media transmission protocol adopted by the anchor terminal, and it can be set according to actual needs. For example, the streaming media transmission protocol can be RTMP (Real Time Message Protocol), which is an application layer protocol that can be used to solve the problems of multiplexing and packetizing of multimedia data transmission streams; by using the RTMP protocol to push live data to the server, the live data push process can have the characteristics of high reliability and strong compatibility. For another example, the streaming media transmission protocol can be RTP (Real-time Transport Protocol), or the streaming media transmission protocol can also be RTSP (Real Time Streaming Protocol), and so on.

[0065] In the process of live broadcasting of the game, the embodiment of the present application can obtain the game audio and video data of the target game at the current moment and the ambient audio and video data on the host user side through the game engine of the target game played by the host user, thereby generating live broadcast data using the game audio and video data and the ambient audio and video data, and sending the live broadcast data to the audience terminal through the server for playback, so that the audience user can not only watch the game screen on the host user side, but also listen to the game audio data on the host user side and watch the ambient audio and video data on the host user side at the same time, enriching the live broadcast content of the game live broadcast and effectively improving the live broadcast effect of the game live broadcast. In addition, by using the game engine to collect data and generate live broadcast data, the host user does not need to use third-party software to broadcast the game live, which not only improves the convenience of game live broadcasting, but also enables the terminal to save processing resources during the game live broadcasting without consuming additional processing resources to run third-party software, thereby avoiding affecting the terminal's operating performance due to the third-party software occupying more operating memory, thereby reducing the phenomenon of freezing during the game operation, and thus improving the live broadcast effect of the game live broadcast. Furthermore, by developing a cross-platform interface, no matter which system platform the target game runs on, the game engine can collect data from the corresponding system platform through the cross-platform interface. This will make game live streaming not limited to a single system platform and can effectively improve the applicability of game live streaming.

[0066] Based on the above Figure 2 The embodiment of the live broadcast processing method shown in the figure further proposes a more specific live broadcast processing method; in the embodiment of the application, the live broadcast processing method executed by the anchor terminal is still used as an example for description. Figure 3 As shown, the live broadcast processing method may include the following steps S301-S308:

[0067] S301, during the live broadcast of the game, the game audio and video data of the target game at the current moment is obtained through the game engine of the target game played by the anchor user, and the game audio and video data may include: game screen and game audio data.

[0068] S302, controlling the game engine to obtain environmental audio and video data on the target system platform through a cross-platform interface, where the environmental audio and video data may include: environmental images and environmental audio data corresponding to the environment where the host user is located.

[0069] S303, performing image mixing processing on the game image and the environment image to obtain a mixed image.

[0070] In a specific implementation, the anchor terminal can directly perform image mixing processing on the game screen and the environment screen to obtain a mixed image. Alternatively, the anchor terminal can perform video post-processing on the game screen and video post-processing on the environment screen; then, the video post-processed game screen and the video post-processed environment screen are mixed to obtain a mixed image. Specifically, the specific implementation method of each step can be referred to the following description:

[0071] (1) Video post-processing for game images may include at least one of the following:

[0072] (1) The process of adding a watermark to the game screen. A watermark refers to a semi-transparent icon that can be added to a picture (screen) to prevent others from stealing the picture. In this case, the specific implementation method of performing video post-processing on the game screen may include: the anchor terminal may obtain the watermark preset by the anchor user, and add the preset watermark to the game screen. Alternatively, the anchor terminal may also use the logic of automatically adding watermarks to perform video post-processing on the game screen. Specifically, the anchor terminal obtains the user identifier of the anchor user (such as a user nickname, game account, etc.), uses the obtained user identifier to generate a watermark, and adds the generated watermark to the game screen. It should be noted that the embodiments of the present application do not limit the specific method of adding a watermark to the game screen. For example, a watermark may be added to the center of the game screen, or a few positions may be randomly selected on the game screen and watermarks may be added to the randomly selected positions, such as Figure 4a By adding a watermark or other personal identification on the game screen, the game screen can be effectively prevented from being stolen by others, thus improving the security of the game screen.

[0073] (2) Adding stickers to the game screen, where stickers refer to patterns that can be added to a picture (screen). In this case, the specific implementation of post-processing the game screen may include: the anchor terminal may obtain stickers preset by the anchor user, and add the preset stickers to the game screen. Alternatively, the anchor terminal may also use the logic of automatically adding stickers to perform post-processing on the game screen. Specifically, the anchor terminal may obtain the historical game behavior performed by the anchor user at the previous moment of the current moment, and determine whether the current moment is a game highlight moment based on the historical game behavior; if the current moment is determined to be a game highlight moment based on the historical game behavior, the stickers associated with the game highlight moment are obtained, and the obtained stickers are added to the game screen. By automatically identifying game highlight moments and automatically adding stickers at game highlight moments, the fun of game live broadcasts can be enhanced.

[0074] Among them, the highlight moment of the game refers to the moment when a set of game tasks distributed in a gradient is completed during the game. It is understandable that for different games, the corresponding game tasks distributed in a gradient may be different. For example, for multiplayer online tactical competitive games (a game in which at least two teams of game players fight against each other), the game tasks distributed in a gradient may include but are not limited to: N kill tasks, end tasks, and so on. Among them, the N kill task refers to the task of continuously eliminating N game objects, and N can be three, four, five, etc.; the end task refers to the task of eliminating the opponent's player when the opponent's player is in a continuous killing state. For elimination games (a game that matches and eliminates a certain number of identical game elements (such as fruits, animal avatars, etc.), the game tasks distributed in a gradient may include but are not limited to: tasks of continuously eliminating a large number of game elements, tasks of eliminating a large number of game elements at one time, tasks of refreshing historical records of the total score of the game, and so on.

[0075] Take the target game as a multiplayer online tactical competitive game as an example, and assume that the game task distributed according to the gradient is a pentakill task. If the historical game behavior performed by the anchor user at the moment before the current moment is: after having eliminated four game objects in a row, he continues to eliminate the fifth game object, then the current moment can be determined as the game highlight moment based on the historical game behavior. At this time, the stickers associated with the game highlight moment can be obtained, and the obtained stickers can be added to the game screen, such as Figure 4b shown.

[0076] (3) Adding a filter to the game screen. A filter refers to an image processing tool that can be used to change the display effect of an image (screen). In this case, the specific implementation of post-processing the game screen may include: the anchor terminal may obtain a first filter preset by the anchor user for the game screen, and add the preset first filter to the game screen. Alternatively, the anchor terminal may also use the logic of automatically adding filters to post-process the game screen. Specifically, the anchor terminal may determine the display effect of the game screen, and the display effect includes at least one of the following: the brightness of the screen, the contrast of the screen, and the hue of the screen; then, determine whether the display effect meets the effect adjustment condition, and the effect adjustment condition includes at least one of the following: the brightness of the screen is less than the brightness threshold, the contrast of the screen is less than the contrast threshold, and the hue of the screen is a preset hue. If the display effect meets the effect adjustment condition, the target filter may be obtained, and the target filter may be added to the game screen to adjust the display effect of the game screen. By automatically identifying the display effect of the game screen and determining whether to add a filter based on the display effect, it is possible to avoid the final game screen having too low brightness or contrast, or abrupt color tones, thereby ensuring the display effect of the final game screen.

[0077] Among them, the embodiment of the present application does not limit the specific implementation method of obtaining the target filter. For example, a filter pre-set based on the experience value can be obtained as the target filter. For another example, a filter can be selected as the target filter according to the display effect of the game screen according to the filter selection strategy, and the filter selection strategy is used to indicate that the display effect of the selected filter must be better than the display effect of the game screen, and the difference between the display effect of the selected filter and the display effect of the game screen must be less than the difference threshold; for example, taking the display effect including the brightness of the screen as an example, the filter selection strategy can be used to indicate that the brightness corresponding to the selected filter must be greater than the brightness corresponding to the game screen, and the difference between the two brightnesses is less than the difference threshold. By obtaining the target filter through this filter selection strategy, it can be ensured that the game screen is distorted due to the large change in the display effect while improving the display effect of the game screen based on the target filter. In addition, adding a target filter to the game screen can be understood as: using the target filter to adjust the display effect of the game screen so that the adjusted display effect matches the display effect corresponding to the target filter.

[0078] It should be noted that the above is only an exemplary list of several video post-processing methods for game screens, and it is not an exhaustive list. Moreover, in actual applications, when the video post-processing for game screens includes multiple processes such as adding watermarks, adding stickers, and adding filters, the embodiments of the present application do not limit the execution order of each process. For example, you can first add a watermark to the game screen, then add stickers to the game screen, and finally add a filter to the game screen; for another example, you can first add a filter to the game screen, then add stickers to the game screen, and finally add a watermark to the game screen, and so on.

[0079] (ii) Video post-processing for the environment image may include at least one of the following:

[0080] (1) Beautification processing of the host's face image in the environment screen. Beautification processing refers to beautification processing of the face image, which may include but is not limited to: skin smoothing processing (i.e., processing to improve the fineness of the face image by removing flaws in the face image), spot removal processing (i.e., removing spots in the face image), etc. In this case, the specific implementation of video post-processing of the environment screen may include: spot recognition in the host's face image, and if at least one spot is recognized, then removing the recognized spot in the host's face image. In addition, flaws (such as wrinkles) in the host's face image may also be recognized, and the recognized flaws may be removed in the host's face image.

[0081] (2) Adding a filter to the environment screen. In this case, the specific implementation method of performing video post-processing on the environment screen may include: the anchor terminal may obtain the second filter preset by the anchor user for the environment screen, and add the preset second filter to the environment screen. Alternatively, the anchor terminal may also use the logic of automatically adding filters to perform video post-processing on the environment screen. Specifically, obtain the anchor user's filter usage history record, and the filter usage history record includes: each filter used by the anchor user in the history and the usage time of each filter; then, according to the usage time of each filter in the filter usage history record, a filter can be selected from the filters used by the anchor user in the history, and the selected filter can be added to the environment screen, and the usage time of the selected filter is later than the usage time of other filters.

[0082] (3) Adding stickers to the environment screen. In this case, the specific implementation of post-processing the environment screen may include: the anchor terminal may obtain the target sticker pre-set by the anchor user for the anchor face image, and then add the target sticker to the anchor face image in the environment screen. Alternatively, the anchor terminal may randomly select some stickers from the sticker library and add the selected stickers to the environment screen.

[0083] It should be noted that the above only lists several types of video post-processing for environmental screens by way of example, and is not exhaustive; for example, the environmental screen may also be mosaicked, which is an image processing method that can degrade the color scale details in an image area and create a disrupted color block effect, making the image area appear as if it is composed of small grids. Moreover, in actual applications, when the video post-processing for environmental screens includes multiple processes such as beauty processing, adding filters, and adding stickers, the embodiments of the present application do not limit the execution order of each process. For example, the host's facial image in the environmental screen may be beautified first, then a filter may be added to the environmental screen, and finally stickers may be added to the game screen. Figure 4c As shown; for another example, you can first add stickers to the environment screen, then add filters to the environment screen, and finally beautify the host's face image in the environment screen, and so on.

[0084] (III) The above-mentioned screen mixing process can be understood as screen overlay process, which can be a process of overlaying the environment screen on the game screen, or a process of overlaying the game screen on the environment screen, without limitation. For the sake of convenience, the following description will take the case of overlaying the environment screen on the game screen as an example.

[0085] In one embodiment, when the anchor terminal performs screen mixing processing on the game screen after video post-processing and the environment screen after video post-processing to obtain the mixed screen, a default area can be determined from the game screen after video post-processing. The default area is a preset area for superimposing and displaying the environment screen, for example, it can be an area located in the lower left corner or lower right corner of the game screen. Secondly, the anchor terminal can scale the environment screen after video post-processing according to the area size of the default area, so that the screen size of the scaled environment screen matches the area size of the default area; then, the scaled environment screen can be superimposed on the game screen after video post-processing to obtain a mixed screen. Among them, the scaling process can include a reduction process or an enlargement process; when the area size of the default area is smaller than the screen size of the environment screen after video post-processing, the scaling process includes a reduction process; when the area size of the default area is larger than the screen size of the environment screen after video post-processing, the scaling process includes an enlargement process.

[0086] In another embodiment, in order to prevent the environment screen from blocking the key content in the game screen, the anchor terminal performs image mixing processing on the game screen after video post-processing and the environment screen after video post-processing. When obtaining the mixed image, the image content in the default area of ​​the game screen after video post-processing can be identified, and it can be determined whether the identified image content is the key content. The key content can be set according to actual needs, such as the content of the game character used by the anchor user, or the content of the game props used by the anchor user. If the identified image content is the key content, the transparency of the environment screen after video post-processing can be increased, such as increasing the transparency of the environment screen after video post-processing to a preset transparency; and adjusting the image size of the environment screen after video post-processing according to the area size of the default area to obtain an adjusted environment screen, the image size of the adjusted environment screen matches the area size of the default area, and the transparency of the adjusted environment screen is the preset transparency. Then, the adjusted environment screen is superimposed on the default area in the game screen after video post-processing to obtain a mixed image. Taking the default area as the area located in the lower right corner of the game screen as an example, through this embodiment, the schematic diagram of superimposing the environment screen on the game screen can be seen in Figure 4d shown.

[0087] In another embodiment, when the anchor terminal performs image mixing processing on the game screen after video post-processing and the environment screen after video post-processing to obtain a mixed image, it can perform image recognition processing on the game screen after video post-processing to identify the blank area in the game screen after video post-processing, and the blank area may refer to the area that does not contain any image content. Then, the environment screen after video post-processing can be scaled according to the area size of the blank area to obtain a scaled environment screen; and the scaled environment screen can be superimposed on the blank area in the game screen after video post-processing to obtain a mixed image. Exemplarily, through this embodiment, the schematic diagram of superimposing the environment screen on the game screen can be seen in Figure 4e shown.

[0088] It should be noted that the implementation method of the host terminal directly mixing the game screen and the environment screen is similar to the above-mentioned implementation method of mixing the game screen after video post-processing and the environment screen after video post-processing, which will not be repeated here. In addition, the entire screen mixing process can be performed in the GPU (Graphics Processing Unit) of the host terminal, so that the final mixed screen is a kind of GPU texture data. By superimposing the environment screen containing the host's face image on the game screen, the function of displaying the game screen while displaying the host user's face image can be realized.

[0089] S304, performing video encoding on the mixed picture to obtain a video data stream.

[0090] In a specific implementation, since the mixed picture is a GPU texture data, and the video encoding process is performed in the CPU (Central Processing Unit) of the anchor terminal, for the mixed picture, the mixed picture can be first transferred from the GPU to the CPU, and then the mixed picture is video encoded in the CPU to obtain a video data stream. Furthermore, when the mixed picture is video encoded in the CPU, a data format conversion is required to convert the mixed picture into the YUV data required for CPU encoding, and then the YUV data is video encoded using the video encoding protocol to obtain video stream data. The embodiment of the present application does not limit the video encoding protocol, for example, it can be the H264 protocol or the H265 protocol, etc. The YUV mentioned here is a color encoding method that uses brightness and chroma to specify the color of a pixel; Y in YUV represents brightness (Luminance, Luma), and U and V represent chroma (Chrominance, Chroma), and chroma defines two aspects of color: hue and saturation.

[0091] The mixed image can be transferred from the GPU to the CPU by calling a system API (application programming interface) to read the mixed image stored in the GPU into the RAM (Random Access Memory) in the CPU; for example, see Figure 4f As shown, the readpixel interface (a system API) can be called on the opengl (pipeline) to read the mixed image from the GPU to the CPU.

[0092] Furthermore, considering that transferring the mixed picture by calling the system API will block the thread used when calling the system API, the thread will not be unblocked until the complete mixed picture is read into the CPU, which is extremely time-consuming; and the entire process is completed by the CPU, so a part of the CPU cycle will be occupied. Based on this, the embodiment of the present application also proposes to use PBO (Pixel Buffer Object) to optimize the process of transferring the mixed picture from the GPU to the CPU. PBO can use DMA (Direct Memory Access) technology to quickly transfer data between the GPU and the CPU without occupying the CPU's operating cycle, and supports asynchronous DMA transmission, so PBO can be used to quickly upload and download data. In this case, the method of transferring the mixed picture from the GPU to the CPU can be referred to. Figure 4g As shown in the figure: the mixed picture is loaded into PBO through GPU, and then the mixed picture data is read out from PBO asynchronously through CPU. It can be seen that since the first step in this process is GPU operation, it will not occupy the CPU's operating cycle, which can improve the CPU's operating performance.

[0093] S305, mixing the game audio data and the environment audio data to obtain a mixing result.

[0094] In a specific implementation, the anchor terminal can directly perform mixing processing on the game audio data and the environmental audio data to obtain a mixing result; wherein, mixing processing refers to the process of integrating multiple audio data into one audio track (i.e., audio track). Alternatively, considering that the audio parameters of the game audio data and the audio parameters of the environmental audio data may be different, in order to ensure the subsequent audio playback effect, the game audio data and the environmental audio data can be resampled first to make the audio parameters of the two audio data consistent, and then perform mixing processing. In this case, the specific implementation process of step S305 can be as follows:

[0095] First, audio resampling parameters can be obtained. The audio resampling parameters refer to audio parameters used for resampling, which may include at least one of the following: sampling frequency, sampling bit number, and audio channel number; any audio data may include multiple time points and amplitude values ​​at each time point, and the sampling bit number mentioned here is used to indicate that the amplitude value is represented by a numerical value of M digits, where M is a positive integer. It should be understood that the audio parameters of the resampled game audio data are the same as the audio resampling parameters, and the audio parameters of the resampled environmental audio data are also the same as the audio resampling parameters.

[0096] The specific implementation method of the anchor terminal obtaining the audio resampling parameter may be: obtaining a preset audio parameter as the audio resampling parameter. Alternatively, an audio parameter is selected from the audio parameters of the game audio data and the audio parameters of the environmental audio data as the audio resampling parameter. Alternatively, the anchor terminal may send a parameter acquisition request to the server to request the server to dynamically determine the audio resampling parameter according to the audio playback performance of each currently connected audience terminal, and return the determined audio resampling parameter; then, the anchor terminal may receive the audio resampling parameter returned by the server. Among them, the audio playback performance of any audience terminal can be used to indicate the audio parameters supported by the corresponding audience terminal, such as sampling frequency, sampling bit number and number of audio channels; then, when the server determines the audio resampling parameter according to the audio playback performance of each currently connected audience terminal, it can group each audience terminal according to the audio playback performance of each audience terminal, and the audience terminals in the same group have the same audio playback performance, count the number of audience terminals in each group, and use the audio parameters indicated by the audio playback performance of the audience terminals in the group corresponding to the largest number as the audio resampling parameter. In this way, the audio resampling parameters finally determined can be adapted to most of the viewer terminals, thereby ensuring that the audio data can be played normally in most of the viewer terminals.

[0097] Then, the audio resampling parameters can be used to resample the game audio data and the environment audio data respectively, and the resampled game audio data and the resampled environment audio data can be mixed to obtain a mixing result. Specifically, the resampled game audio data and the resampled environment audio data can be directly mixed to obtain a mixing result; or, the resampled game audio data can be audio-post-processed, and the resampled environment audio data can be audio-post-processed, so as to mix the audio-post-processed game audio data and the audio-post-processed environment audio data to obtain a mixing result, such as Figure 4h shown.

[0098] The audio post-processing for the resampled game audio data may include at least one of the following: volume adjustment (such as lowering the treble, increasing the bass, etc.), and equalization processing; the equalization processing mentioned here refers to adjusting the volume of each track in the audio data so that the volume of each track reaches an appropriate ratio. The audio post-processing for the resampled ambient audio data may include at least one of the following: noise reduction processing and voice and tone changing processing; the voice and tone changing processing mentioned here includes at least one of the following: changing the timbre of the sound and changing the pitch of the sound.

[0099] S306, performing audio encoding on the mixing result to obtain an audio data stream.

[0100] In a specific implementation, the audio coding standard can be used to perform audio coding on the mixing result to obtain an audio data stream; the embodiment of the present application does not limit the audio coding standard, for example, it can be AAC (Advance Audio Coding), PCM (Pulse Code Modulation), etc. Taking the AAC audio coding standard as an example, and the H264 video coding protocol as an example, the schematic diagram of the host terminal performing video coding and audio coding can be seen in Figure 4i It should be noted that the video data stream obtained by performing video encoding based on the H264 protocol can also be called an H264 code stream, and the audio data stream obtained by performing audio encoding based on AAC can also be called an AAC code stream.

[0101] It should be noted that the embodiment of the present application does not limit the execution order between steps S303-S304 and steps S305-S306. For example, steps S303-S304 may be executed first, and then steps S305-S306; or, steps S305-S306 may be executed first, and then steps S303-S304; or, steps S303-S304 and steps S305-S306 may be executed simultaneously.

[0102] S307, encapsulating the video data stream and the audio data stream according to a preset encapsulation format to obtain live broadcast data.

[0103] S308, pushing the live broadcast data to the server, so that the server sends the live broadcast data to the viewer terminal for playback.

[0104] In steps S307-S308, the preset encapsulation format may be the FLV encapsulation format, that is, the anchor terminal may encapsulate the video data stream (such as H264 code stream) and the audio data stream (such as AAC code stream) according to the FLV encapsulation format to obtain live broadcast data (i.e., a file in the FLV encapsulation format), and then push the live broadcast data to the server through the RMTP protocol. Figure 4j shown.

[0105] In the process of live broadcasting of the game, the embodiment of the present application can obtain the game audio and video data of the target game at the current moment and the ambient audio and video data on the host user side through the game engine of the target game played by the host user, thereby generating live broadcast data using the game audio and video data and the ambient audio and video data, and sending the live broadcast data to the audience terminal through the server for playback, so that the audience user can not only watch the game screen on the host user side, but also listen to the game audio data on the host user side and watch the ambient audio and video data on the host user side at the same time, enriching the live broadcast content of the game live broadcast and effectively improving the live broadcast effect of the game live broadcast. In addition, by using the game engine to collect data and generate live broadcast data, the host user does not need to use third-party software to broadcast the game live, which not only improves the convenience of game live broadcasting, but also enables the terminal to save processing resources during the game live broadcasting without consuming additional processing resources to run third-party software, thereby avoiding affecting the terminal's operating performance due to the third-party software occupying more operating memory, thereby reducing the phenomenon of freezing during the game operation, and thus improving the live broadcast effect of the game live broadcast. Furthermore, by developing a cross-platform interface, no matter which system platform the target game runs on, the game engine can collect data from the corresponding system platform through the cross-platform interface. This will make game live streaming not limited to a single system platform and can effectively improve the applicability of game live streaming.

[0106] Furthermore, based on the above Figure 3 The live broadcast processing method embodiment shown in the figure, the present application embodiment proposes another live broadcast processing method; see Figure 5 As shown, the process of the live broadcast processing method is roughly as follows:

[0107] 1. Adapt the audio and video data collection of each system platform. During the live broadcast of the game, the game engine of the target game played by the anchor user is used to obtain the game screen and game audio data of the target game at the current moment, as well as the environmental screen and environmental audio data of the anchor user at the current moment.

[0108] 2. Perform video post-processing on the game screen and the environment screen respectively, such as adding watermarks, stickers, filters, mosaics, etc.; resample the game audio data and the environment audio data respectively, and perform audio post-processing on the resampled game audio data and the resampled environment audio data respectively, such as voice changing, pitch changing, noise reduction, etc.

[0109] 3. Perform image mixing processing on the game screen after video post-processing and the environment screen after video post-processing to obtain a mixed image; mix the game audio data after audio post-processing and the environment audio data after audio post-processing to obtain a mixing result.

[0110] 4. Convert the mixed image into a data format, and encode the result of the data format conversion (i.e., YUV data) according to the H264 protocol to obtain an H264 bitstream (i.e., video data stream); at the same time, encode the mixing result according to the AAC audio coding standard to obtain an ACC bitstream (i.e., audio data stream).

[0111] 5. Encapsulate the encoded H264 code stream and AAC code stream according to the FLV encapsulation format to obtain live broadcast data.

[0112] 6. Push the encapsulated live data through the RTMP protocol to send the live data to the server.

[0113] Through the above process, the game screen currently rendered by the game engine can be pushed to the server quickly and easily, with the characteristics of flexible configuration, stable performance and strong compatibility. In addition, the effect of cross-platform push can be achieved, solving the problem of data conversion between different platforms. At the same time, the collected audio and video data is post-processed, and some effects that are not convenient for front-end display can be completed through post-processing, such as adding watermarks, anti-counterfeiting marks, etc., and the audio can be processed with voice-changing special effects; it can be seen that the embodiment of the present application has made cross-platform adaptation and performance improvement for each channel of video and audio content, and also provides a post-processing interface, which can post-process the audio and video content according to needs, and facilitate expansion and secondary development. In addition, the embodiment of the present application can also perform special optimization on the intermediate links of the entire link, which can greatly improve the overall performance while ensuring functional stability.

[0114] It can be understood that in the specific implementation of the above-mentioned live broadcast processing method embodiment, relevant data such as user information (such as the environmental image on the anchor user side, the user identification of the anchor user, etc.) is involved. When the above method embodiment is applied to a specific product or technology, these relevant data are collected with the user's permission or consent, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0115] Based on the description of the above live broadcast processing method embodiment, the present application embodiment also discloses a live broadcast processing device; see Figure 6 , the live broadcast processing device can run the following units:

[0116] The acquisition unit 601 is used to acquire the game audio and video data of the target game at the current moment through the game engine of the target game played by the anchor user during the game live broadcast; the target game is run on the target system platform, and the target system platform is configured with an audio and video acquisition component;

[0117] The acquisition unit 601 is further used to control the game engine to acquire the environmental audio and video data on the target system platform through the cross-platform interface, wherein the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment;

[0118] The processing unit 602 is used to generate live broadcast data using the game audio and video data and the environmental audio and video data, and push the live broadcast data to the server, so that the server sends the live broadcast data to the audience terminal for playback.

[0119] In one implementation, the game audio and video data includes: game screen and game audio data; the environmental audio and video data includes: environmental screen and environmental audio data corresponding to the environment where the anchor user is located; accordingly, when the processing unit 602 is used to generate live broadcast data using the game audio and video data and the environmental audio and video data, it can be specifically used to:

[0120] Performing image mixing processing on the game image and the environment image to obtain a mixed image; and performing video encoding on the mixed image to obtain a video data stream;

[0121] Performing mixing processing on the game audio data and the environment audio data to obtain a mixing result; and performing audio encoding on the mixing result to obtain an audio data stream;

[0122] The video data stream and the audio data stream are encapsulated according to a preset encapsulation format to obtain live broadcast data.

[0123] In another implementation manner, when the processing unit 602 is used to perform screen mixing processing on the game screen and the environment screen to obtain a mixed screen, it can be specifically used to:

[0124] Performing video post-processing on the game screen and performing video post-processing on the environment screen;

[0125] The game screen after video post-processing and the environment screen after video post-processing are mixed to obtain a mixed screen.

[0126] In another implementation manner, when the processing unit 602 is used to perform video post-processing on the game screen, it can be specifically used to:

[0127] Obtaining the user ID of the anchor user;

[0128] The acquired user identifier is used to generate a watermark, and the generated watermark is added to the game screen.

[0129] In another implementation manner, when the processing unit 602 is used to perform video post-processing on the game screen, it can be specifically used to:

[0130] Obtaining historical game behaviors performed by the anchor user at a moment before the current moment;

[0131] If the current moment is determined to be a game highlight moment according to the historical game behavior, a sticker associated with the game highlight moment is obtained, and the obtained sticker is added to the game screen;

[0132] The game highlight moment refers to the moment when a group of game tasks distributed in a gradient are completed during the game.

[0133] In another implementation manner, when the processing unit 602 is used to perform video post-processing on the game screen, it can be specifically used to:

[0134] Determining a display effect of the game screen, wherein the display effect includes at least one of the following: screen brightness, screen contrast, and screen hue;

[0135] If the display effect satisfies the effect adjustment condition, a target filter is obtained, and the target filter is added to the game screen to adjust the display effect of the game screen;

[0136] The effect adjustment condition includes at least one of the following: the brightness of the picture is less than a brightness threshold, the contrast of the picture is less than a contrast threshold, and the hue of the picture is a preset hue.

[0137] In another implementation, the environment screen includes: a host face image obtained by collecting the face of the host user;

[0138] The video post-processing for the environmental screen includes at least one of the following: beautification processing for the host's facial image in the environmental screen, adding filters to the environmental screen, and adding stickers to the host's facial image.

[0139] In another embodiment, when the processing unit 602 is used to perform image mixing processing on the game image after video post-processing and the environment image after video post-processing to obtain a mixed image, it can be specifically used to:

[0140] Performing screen recognition processing on the game screen after video post-processing to identify a blank area in the game screen after video post-processing;

[0141] Scaling the environment picture after the video post-processing according to the area size of the blank area to obtain a scaled environment picture;

[0142] The scaled environment picture is superimposed on a blank area in the game picture after the video post-processing to obtain a mixed picture.

[0143] In another embodiment, when the processing unit 602 is used to perform image mixing processing on the game image after video post-processing and the environment image after video post-processing to obtain a mixed image, it can be specifically used to:

[0144] Identify the screen content in a default area of ​​the game screen after video post-processing, where the default area is a preset area for overlaying and displaying the environment screen;

[0145] If the identified picture content is key content, the transparency of the environment picture after the video post-processing is increased, and the picture size of the environment picture after the video post-processing is adjusted according to the area size of the default area to obtain an adjusted environment picture;

[0146] The adjusted environment picture is superimposed on the default area in the game picture after the video post-processing to obtain a mixed picture.

[0147] In another implementation manner, when the processing unit 602 is used to mix the game audio data and the environment audio data to obtain a mixing result, it can be specifically used to:

[0148] Acquire audio resampling parameters, where the audio resampling parameters include at least one of the following: sampling frequency, sampling bit number, and audio channel number;

[0149] Resampling the game audio data and the environment audio data respectively by using the audio resampling parameters;

[0150] The resampled game audio data and the resampled environment audio data are mixed to obtain a mixing result.

[0151] In another implementation, when the processing unit 602 is used to perform mixing processing on the resampled game audio data and the resampled environment audio data to obtain a mixing result, it can be specifically used to:

[0152] Performing audio post-processing on the resampled game audio data, and performing audio post-processing on the resampled environment audio data;

[0153] The game audio data after the audio post-processing and the environmental audio data after the audio post-processing are mixed to obtain a mixing result.

[0154] In another implementation manner, when the processing unit 602 is used to obtain the audio resampling parameters, it can be specifically used to:

[0155] Sending a parameter acquisition request to the server to request the server to dynamically determine the audio resampling parameters according to the audio playback performance of each currently connected viewer terminal, and return the determined audio resampling parameters;

[0156] Receive the audio resampling parameters returned by the server.

[0157] In another implementation, the cross-platform interface includes: a cross-platform first interface and a cross-platform second interface; the environmental audio and video data includes: environmental images and environmental audio data corresponding to the environment where the anchor user is located;

[0158] Accordingly, when the acquisition unit 601 is used to control the game engine to acquire the ambient audio and video data on the target system platform through the cross-platform interface, it can be specifically used to:

[0159] Controlling the game engine to obtain the environment screen on the target system platform through the cross-platform first interface;

[0160] The game engine is controlled to obtain the environmental audio data on the target system platform through the cross-platform second interface.

[0161] In another implementation manner, the cross-platform first interface encapsulates a plurality of first sub-interfaces, one first sub-interface is adapted to one system platform, and different first sub-interfaces are adapted to different system platforms;

[0162] Accordingly, when the acquisition unit 601 is used to control the game engine to acquire the environment screen on the target system platform through the cross-platform first interface, it can be specifically used to:

[0163] Control the game engine to call the cross-platform first interface, so that the cross-platform first interface selects a first sub-interface adapted to the target system platform from multiple first sub-interfaces inside, and calls the selected first sub-interface to obtain the environment screen from the target system platform.

[0164] In another implementation manner, the cross-platform second interface encapsulates a plurality of second sub-interfaces, one second sub-interface is adapted to one system platform, and different second sub-interfaces are adapted to different system platforms;

[0165] Accordingly, when the acquisition unit 601 is used to control the game engine to acquire the ambient audio data on the target system platform through the cross-platform second interface, it can be specifically used to:

[0166] Control the game engine to call the cross-platform second interface, so that the cross-platform second interface selects a second sub-interface adapted to the target system platform from multiple second sub-interfaces inside, and calls the selected second sub-interface to obtain the environmental audio data from the target system platform.

[0167] According to another embodiment of the present application, Figure 6 The various units in the live broadcast processing device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, other units can also be included based on the live broadcast processing device. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0168] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 or Figure 3 A computer program (including one or more instructions) for each step involved in the corresponding method shown in , to construct Figure 6 The live broadcast processing device shown in the embodiment of the present application is used to implement the live broadcast processing method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computing device through the computer-readable storage medium, and run therein.

[0169] In the process of live broadcasting of the game, the embodiment of the present application can obtain the game audio and video data of the target game at the current moment and the ambient audio and video data on the host user side through the game engine of the target game played by the host user, thereby generating live broadcast data using the game audio and video data and the ambient audio and video data, and sending the live broadcast data to the audience terminal through the server for playback, so that the audience user can not only watch the game screen on the host user side, but also listen to the game audio data on the host user side and watch the ambient audio and video data on the host user side at the same time, enriching the live broadcast content of the game live broadcast and effectively improving the live broadcast effect of the game live broadcast. In addition, by using the game engine to collect data and generate live broadcast data, the host user does not need to use third-party software to broadcast the game live, which not only improves the convenience of game live broadcasting, but also enables the terminal to save processing resources during the game live broadcasting without consuming additional processing resources to run third-party software, thereby avoiding affecting the terminal's operating performance due to the third-party software occupying more operating memory, thereby reducing the phenomenon of freezing during the game operation, and thus improving the live broadcast effect of the game live broadcast. Furthermore, by developing a cross-platform interface, no matter which system platform the target game runs on, the game engine can collect data from the corresponding system platform through the cross-platform interface. This will make game live streaming not limited to a single system platform and can effectively improve the applicability of game live streaming.

[0170] Based on the description of the above method embodiment and device embodiment, the present application embodiment also provides a terminal. Figure 7 , the terminal at least includes a processor 701, an input interface 702, an output interface 703 and a computer storage medium 704. The processor 701, the input interface 702, the output interface 703 and the computer storage medium 704 in the terminal can be connected via a bus or other means. The computer storage medium 704 can be stored in the memory of the terminal, the computer storage medium 704 is used to store a computer program, the computer program includes one or more instructions, and the processor 701 is used to execute one or more instructions in the computer program stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to realize the corresponding method flow or corresponding function.

[0171] In one embodiment, the processor 701 described in the embodiment of the present application can be used to perform a series of live broadcast processing, specifically including: in the process of game live broadcast, through the game engine of the target game played by the anchor user, obtaining the game audio and video data of the target game at the current moment; the target game is run on the target system platform, and the target system platform is configured with an audio and video acquisition component; controlling the game engine to obtain the environmental audio and video data on the target system platform through a cross-platform interface, and the environmental audio and video data is obtained by calling the audio and video acquisition component configured on the target system platform at the current moment to collect data on the environment where the anchor user is located; using the game audio and video data and the environmental audio and video data to generate live broadcast data, and streaming the live broadcast data to the server, so that the server sends the live broadcast data to the audience terminal for playback, and so on.

[0172] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in the terminal for storing computer programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer storage medium provides a storage space, which stores the operating system of the terminal. In addition, a computer program is also stored in the storage space, which includes one or more instructions suitable for being loaded and executed by the processor 701, and these instructions can be one or more program codes. It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor. In one embodiment, the processor can load and execute one or more instructions stored in the computer storage medium to implement the above-mentioned related Figure 2 or Figure 3 The corresponding steps in the method embodiment shown.

[0173] In the process of live broadcasting of the game, the embodiment of the present application can obtain the game audio and video data of the target game at the current moment and the ambient audio and video data on the host user side through the game engine of the target game played by the host user, thereby generating live broadcast data using the game audio and video data and the ambient audio and video data, and sending the live broadcast data to the audience terminal through the server for playback, so that the audience user can not only watch the game screen on the host user side, but also listen to the game audio data on the host user side and watch the ambient audio and video data on the host user side at the same time, enriching the live broadcast content of the game live broadcast and effectively improving the live broadcast effect of the game live broadcast. In addition, by using the game engine to collect data and generate live broadcast data, the host user does not need to use third-party software to broadcast the game live, which not only improves the convenience of game live broadcasting, but also enables the terminal to save processing resources during the game live broadcasting without consuming additional processing resources to run third-party software, thereby avoiding affecting the terminal's operating performance due to the third-party software occupying more operating memory, thereby reducing the phenomenon of freezing during the game operation, and thus improving the live broadcast effect of the game live broadcast. Furthermore, by developing a cross-platform interface, no matter which system platform the target game runs on, the game engine can collect data from the corresponding system platform through the cross-platform interface. This will make game live streaming not limited to a single system platform and can effectively improve the applicability of game live streaming.

[0174] It should be noted that, according to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes one or more instructions, and the one or more instructions are stored in a computer storage medium. A processor of a computer device (such as a terminal) reads one or more instructions from the computer storage medium, and the processor executes the one or more instructions, so that the computer device executes the above Figure 2 or Figure 3 It should be understood that the above disclosure is only the preferred embodiment of the present application, and certainly cannot be used to limit the scope of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A live broadcast processing method, characterized in that: The method comprises: During the live broadcast of the game, the game audio and video data of the target game at the current moment is obtained through the game engine of the target game played by the anchor user; the target game is run on a target system platform, and the target system platform is configured with an audio and video acquisition component; Control the game engine to obtain environmental audio and video data on the target system platform through a cross-platform interface, wherein the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment; The game audio and video data and the environmental audio and video data are used to generate live broadcast data, and the live broadcast data is pushed to a server, so that the server sends the live broadcast data to the audience terminal for playback.

2. The method according to claim 1, characterized in that The game audio and video data includes: game screen and game audio data; the environmental audio and video data includes: environmental screen and environmental audio data corresponding to the environment where the anchor user is located; The generating of live broadcast data by using the game audio and video data and the environmental audio and video data includes: Performing image mixing processing on the game image and the environment image to obtain a mixed image; and performing video encoding on the mixed image to obtain a video data stream; Performing mixing processing on the game audio data and the environment audio data to obtain a mixing result; and performing audio encoding on the mixing result to obtain an audio data stream; The video data stream and the audio data stream are encapsulated according to a preset encapsulation format to obtain live broadcast data.

3. The method according to claim 2, characterized in that The performing image mixing processing on the game image and the environment image to obtain a mixed image includes: Performing video post-processing on the game screen and performing video post-processing on the environment screen; The game screen after video post-processing and the environment screen after video post-processing are mixed to obtain a mixed screen.

4. The method according to claim 3, characterized in that The performing video post-processing on the game screen includes: Obtaining the user ID of the anchor user; The acquired user identifier is used to generate a watermark, and the generated watermark is added to the game screen.

5. The method according to claim 3, characterized in that The performing video post-processing on the game screen includes: Obtaining historical game behaviors performed by the anchor user at a moment before the current moment; If the current moment is determined to be a game highlight moment according to the historical game behavior, a sticker associated with the game highlight moment is obtained, and the obtained sticker is added to the game screen; The game highlight moment refers to the moment when a group of game tasks distributed in a gradient are completed during the game.

6. The method according to claim 3, characterized in that The performing video post-processing on the game screen includes: Determining a display effect of the game screen, wherein the display effect includes at least one of the following: screen brightness, screen contrast, and screen hue; If the display effect satisfies the effect adjustment condition, a target filter is obtained, and the target filter is added to the game screen to adjust the display effect of the game screen; The effect adjustment condition includes at least one of the following: the brightness of the picture is less than a brightness threshold, the contrast of the picture is less than a contrast threshold, and the hue of the picture is a preset hue.

7. The method according to claim 3, characterized in that The environment screen includes: a host face image obtained by collecting the face of the host user; The video post-processing for the environmental picture includes at least one of the following: beautification processing for the host's facial image in the environmental picture, adding filters to the environmental picture, and adding stickers to the host's facial image.

8. The method according to any one of claims 3 to 7, characterized in that: The mixing process of the game screen after the video post-processing and the environment screen after the video post-processing to obtain the mixed screen includes: Performing screen recognition processing on the game screen after video post-processing to identify a blank area in the game screen after video post-processing; Scaling the environment picture after the video post-processing according to the area size of the blank area to obtain a scaled environment picture; The scaled environment picture is superimposed on a blank area in the game picture after the video post-processing to obtain a mixed picture.

9. The method according to any one of claims 3 to 7, characterized in that: The mixing process of the game screen after the video post-processing and the environment screen after the video post-processing to obtain the mixed screen includes: Identify the screen content in a default area of ​​the game screen after video post-processing, where the default area is a preset area for overlaying and displaying the environment screen; If the identified screen content is key content, the transparency of the environment screen after the video post-processing is increased, and the screen size of the environment screen after the video post-processing is adjusted according to the area size of the default area to obtain an adjusted environment screen; The adjusted environment picture is superimposed on the default area in the game picture after the video post-processing to obtain a mixed picture.

10. The method according to claim 2, characterized in that The mixing process of the game audio data and the environment audio data to obtain a mixing result includes: Acquire audio resampling parameters, where the audio resampling parameters include at least one of the following: sampling frequency, sampling bit number, and audio channel number; Resampling the game audio data and the environment audio data respectively by using the audio resampling parameters; The resampled game audio data and the resampled environment audio data are mixed to obtain a mixing result.

11. The method according to claim 10, characterized in that The mixing process is performed on the resampled game audio data and the resampled environment audio data to obtain a mixing result, including: Performing audio post-processing on the resampled game audio data, and performing audio post-processing on the resampled environment audio data; The game audio data after the audio post-processing and the environmental audio data after the audio post-processing are mixed to obtain a mixing result.

12. The method according to claim 10, characterized in that The step of obtaining audio resampling parameters includes: Sending a parameter acquisition request to the server to request the server to dynamically determine the audio resampling parameters according to the audio playback performance of each currently connected viewer terminal, and return the determined audio resampling parameters; Receive the audio resampling parameters returned by the server.

13. The method according to claim 1, characterized in that The cross-platform interface includes: a cross-platform first interface and a cross-platform second interface; the environmental audio and video data includes: environmental images and environmental audio data corresponding to the environment where the anchor user is located; The controlling the game engine to obtain the ambient audio and video data on the target system platform through a cross-platform interface includes: Controlling the game engine to obtain the environment screen on the target system platform through the cross-platform first interface; The game engine is controlled to obtain the environmental audio data on the target system platform through the cross-platform second interface.

14. The method according to claim 13, characterized in that The cross-platform first interface encapsulates a plurality of first sub-interfaces, one first sub-interface is adapted to one system platform, and different first sub-interfaces are adapted to different system platforms; The controlling the game engine to obtain the environment screen on the target system platform through the cross-platform first interface includes: Control the game engine to call the cross-platform first interface, so that the cross-platform first interface selects a first sub-interface adapted to the target system platform from multiple first sub-interfaces inside, and calls the selected first sub-interface to obtain the environment screen from the target system platform.

15. The method according to claim 13, characterized in that The cross-platform second interface encapsulates a plurality of second sub-interfaces, one second sub-interface is adapted to one system platform, and different second sub-interfaces are adapted to different system platforms; The controlling the game engine to obtain the environmental audio data on the target system platform through the cross-platform second interface includes: Control the game engine to call the cross-platform second interface, so that the cross-platform second interface selects a second sub-interface adapted to the target system platform from multiple second sub-interfaces inside, and calls the selected second sub-interface to obtain the environmental audio data from the target system platform.

16. A live broadcast processing device, characterized in that: include: An acquisition unit is used to acquire game audio and video data of a target game at a current moment through a game engine of the target game played by the anchor user during the live broadcast of the game; the target game is run on a target system platform, and the target system platform is configured with an audio and video acquisition component; The acquisition unit is further used to control the game engine to acquire the environmental audio and video data on the target system platform through a cross-platform interface, wherein the environmental audio and video data is obtained by calling the audio and video acquisition component configured by the target system platform to collect data on the environment where the anchor user is located at the current moment; The processing unit is used to generate live broadcast data using the game audio and video data and the environmental audio and video data, and push the live broadcast data to the server, so that the server sends the live broadcast data to the audience terminal for playback.

17. A terminal, comprising an input interface and an output interface, characterized in that: Also includes: processor and computer storage media; The processor is suitable for implementing one or more instructions, the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the live broadcast processing method as described in any one of claims 1-15.

18. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the live broadcast processing method as described in any one of claims 1-15.

19. A computer program product, characterized in that The computer program product includes one or more instructions; when one or more instructions in the computer program are executed by the processor, the live broadcast processing method as described in any one of claims 1-15 is implemented.

Images