Method for processing live broadcast data and related device

Abstract

The invention provides a method and device for processing live broadcast data, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps executed by a client: acquiring supplementary enhancement information of a target live broadcast from a server; determining a target playing progress of the target live broadcast by analyzing the supplementary enhancement information; and based on the target playing progress, synchronizing the current playing progress of the target live broadcast in the client with the target playing progress. According to the method, the video stream and the supplementary enhancement information of the target live broadcast from the server are acquired, so that the client can analyze the supplementary enhancement information and determine the target playing progress of the target live broadcast. And based on the target playing progress indicated by the supplemental enhancement information, the client synchronizes the playing progress of the client with the target playing progress, so that the content seen by the audience is ensured to be consistent with the content of the stream pushed by the anchor, and the live broadcast watching experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium and computer program product for processing live broadcast data. Background Art

[0002] In some live interactive scenarios, for example, when live streaming, the connection template has limitations on the number of connections it supports, resulting in obvious functional differences between single-player and multiplayer connections. Based on the solutions of the prior art, similar functions mostly use WebSocket connections and API polling requests to ensure real-time data updates. However, based on the existing live broadcast solutions, in some interactive scenarios, such as when the anchor is PKing, the video screen seen by users watching the live broadcast may have data delays compared to the actual PK screen of the anchor. In order to overcome the above defects, some solutions based on the prior art draw relevant gameplay on the anchor side and push the stream to users for viewing through the broadcast tool. However, this method has poor interactivity with users and is highly dependent on the clarity of the video stream. Summary of the invention

[0003] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for processing live broadcast data.

[0004] In one aspect of the present application, a method for processing live broadcast data at a client is provided, wherein the method comprises:

[0005] Obtain supplementary enhancement information of the target live broadcast from the server;

[0006] Determining a target playback progress of the target live broadcast by parsing the supplementary enhancement information;

[0007] Based on the target playback progress, the current playback progress of the target live broadcast in the client is synchronized with the target playback progress.

[0008] In one aspect of the present application, a method for processing live broadcast data at a server is provided, wherein the method comprises:

[0009] Encode the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information;

[0010] The supplementary enhancement information is sent to a client watching the target live broadcast.

[0011] In one aspect of the present application, a device for processing live broadcast data at a client is provided, wherein the device comprises:

[0012] A device for obtaining supplementary enhancement information of a target live broadcast from a server;

[0013] A device for determining a target playback progress of a target live broadcast by parsing the supplementary enhancement information;

[0014] A device for synchronizing the current playback progress of a target live broadcast in a client with the target playback progress based on the target playback progress indicated by the supplementary enhancement information.

[0015] In one aspect of the present application, a device for processing live broadcast data at a server is provided, wherein the device comprises:

[0016] Device for encoding based on the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information;

[0017] A device for sending the supplemental enhancement information to a client watching the target live broadcast.

[0018] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of an embodiment of the present application.

[0019] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0020] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0021] In the solution provided by the embodiment of the present application, by obtaining the target live video stream and supplementary enhancement information from the server, the client can parse the supplementary enhancement information and determine the target playback progress of the target live. Based on the target playback progress indicated by the supplementary enhancement information, the client synchronizes its own playback progress with the target playback progress to ensure that the content seen by the audience is consistent with the content pushed by the anchor, thereby improving the live viewing experience; the supplementary enhancement information contains the layout instructions of the target scene, and the client displays the interactive page of the target scene according to these layout instructions, allowing the anchor to freely set the position and size of the grid displaying the interactive object, which is synchronized to the user end in real time through the supplementary enhancement information, thereby improving the user's viewing experience and interactivity; the live audience can view the real-time data contained in the supplementary enhancement information through interactive operations such as hovering. For example, in e-sports live broadcasts, the audience can view the real-time data of the players and the economic curve of the game in real time by hovering over the hero avatar. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0024] Figure 1 A flow chart of a method for processing live broadcast data provided by an embodiment of the present application is shown;

[0025] Figure 2 A schematic diagram showing an exemplary live interactive page according to an embodiment of the present application is shown;

[0026] Figure 3 A schematic diagram showing an exemplary creation of a live interactive page including a display area of ​​multiple grids according to an embodiment of the present application is shown;

[0027] Figure 4 A schematic diagram of an exemplary process of adjusting the layout of a display area based on an interactive operation according to an embodiment of the present application is shown;

[0028] Figure 5 A schematic diagram of the structure of a client and a server for processing live data provided by an embodiment of the present application is shown;

[0029] Figure 6 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0030] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0033] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0034] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0035] Figure 1 A flow chart of a method for processing live broadcast data provided by an embodiment of the present application is shown. The method at least includes steps S101, S102 and S103 executed by the client, and steps S201 and S202 executed by the server.

[0036] In an actual live broadcast scenario, the live broadcast system of the embodiment of the present application includes a live broadcast end, a client end, and a server end. The relationship between the live broadcast end, the client end, and the server end is as follows:

[0037] Live broadcast end (i.e. the terminal where the anchor is located): The anchor conducts live broadcast activities through the live broadcast end and pushes the stream to the server end. The live broadcast end can generate supplementary enhancement information, including playback progress information and layout adjustment instructions, etc., which will be sent to the client through the server end so that the audience can synchronously see the anchor's live broadcast content and changes in the interactive interface.

[0038] Server: As the intermediary between the live broadcast end and the client, the server receives the live video stream from the live broadcast end and generates supplementary enhancement information. Then, the server sends the live video stream and the processed supplementary enhancement information to the client to ensure that the client can correctly synchronize the live broadcast progress and display the interactive interface.

[0039] Client (i.e. the terminal where the audience watching the live broadcast is located): The audience watches the live broadcast through the client. The client receives the video stream and supplementary enhancement information from the server, parses the supplementary enhancement information to determine the playback progress of the target live broadcast, and adjusts the display of the interactive page according to the layout instructions in the supplementary enhancement information.

[0040] The server includes but is not limited to network hosts, single network servers, multiple network server sets or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0041] Among them, the terminal devices corresponding to the live broadcast end and the client include but are not limited to computers, mobile phones, tablets, smart watches, bracelets and other terminal devices.

[0042] Among them, the supplemental enhancement information (SEI) of the embodiment of the present application is a type of information defined in a video coding standard (such as H.264 / AVC or H.265 / HEVC), which can carry non-video data in a video stream. The supplemental enhancement information is part of a video coding standard such as H.264 / H.265, which allows additional information to be embedded in a video bitstream. This information may include timestamps, copyright information, user-defined data, etc. In live synchronization, supplemental enhancement information can be embedded in a specific frame of a video stream, which includes the playback timestamp of the current frame.

[0043] Reference Figure 1 In step S201, encoding is performed based on the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information.

[0044] The playback progress information includes various information that can be used to indicate the playback progress and synchronize the live video content, such as the timestamp of the video frame, the playback status, the current playback time or the total playback time.

[0045] Among them, the target live broadcast is the live broadcast conducted in the live broadcast terminal.

[0046] The supplementary enhancement information includes a timestamp and / or a sequence number of a current video frame.

[0047] According to one embodiment, when a triggering target scene is detected, the supplementary enhancement information is sent to a client watching the target live broadcast in step S201.

[0048] The target scene includes various scenes that need to display multiple live interactive objects on the playback interface, such as a multi-player live broadcast scene or initiating a PK game.

[0049] In step S202, the supplemental enhancement information is sent to a client watching the target live broadcast.

[0050] Optionally, the supplementary enhancement information is inserted into a target live broadcast stream to obtain the target live broadcast stream, and the target live broadcast stream is sent to the client.

[0051] Optionally, the supplementary enhancement information including the timestamp and sequence number of the current video frame is broadcast to the client via a Socket, wherein the Socket includes but is not limited to a WebSocket, a TCP Socket, and the like.

[0052] The server may, upon receiving the supplementary enhancement information acquisition request sent by the client, execute steps S201 and S202 to generate corresponding supplementary enhancement information and return it to the client. Alternatively, upon detecting a triggering target scene, the server may execute steps S201 and S202 to generate corresponding supplementary enhancement information and return it to the client.

[0053] In step S101, supplementary enhancement information of the target live broadcast is obtained from the server.

[0054] Optionally, the client actively sends a supplementary enhancement information acquisition request to the server, thereby receiving the supplementary enhancement information returned based on the request.

[0055] In step S102, the target playback progress of the target live broadcast is determined by parsing the supplementary enhancement information.

[0056] The target playback progress indicates the playback time point that the client should reach at the current moment.

[0057] Specifically, the corresponding timestamp information is parsed from the payload of the supplementary enhancement information according to the predefined format. Then, the target playback progress is calculated according to the parsed timestamp information, the time point when the client starts to play the target live broadcast, and the current time point of the client.

[0058] For example, the time point when the client starts playing the target live broadcast is recorded as T1, the timestamp parsed from the payload of the supplementary enhancement information is T2, and the current playback time of the client is t. The target playback progress C is calculated by the following formula C=T2-T1-t, which represents the playback time point that the client should reach at the current moment.

[0059] In step S103, the current playback progress of the target live broadcast in the client is synchronized with the target playback progress.

[0060] Specifically, step S103 includes step S1031 and step S1032.

[0061] In step S1031, based on the current playback progress of the client and the target playback progress, it is determined whether the current playback progress of the client needs to be adjusted.

[0062] Optionally, if the current playback progress of the client deviates from the target playback progress, it is determined that the current playback progress of the client needs to be adjusted. Alternatively, the time difference between the current playback progress of the client and the target playback progress is calculated, and if the time difference is greater than a predetermined threshold, it is determined that the current playback progress of the client needs to be adjusted.

[0063] In step S1032, if it is determined that the current playback progress of the client needs to be adjusted, the current playback progress of the client is adjusted to be consistent with the target playback progress.

[0064] The method can adjust the current playback progress of the client to be consistent with the target playback progress in a variety of ways. For example, the playback speed or time axis can be adjusted according to the time difference between the current playback progress of the client and the target playback progress.

[0065] According to one embodiment, the supplemental enhancement information includes layout instructions of the target scene, and the method includes step S203 executed by the server and step S104 executed by the client.

[0066] In step S203, a corresponding layout instruction is generated according to the number of live broadcast objects interacting in the live broadcast interaction page, and the layout instruction is added to the supplementary enhancement information and sent to the client.

[0067] The live interactive page is used to display multiple interactive objects, including hosts, viewers, etc. connected in the target scene.

[0068] The layout instructions are used to indicate the display positions and sizes of multiple interactive objects on the live interactive page.

[0069] The server may generate corresponding layout instructions based on the number of interactive objects and the default layout settings. For example, layout templates corresponding to different numbers of interactive objects may be pre-stored to generate corresponding layout instructions.

[0070] Optionally, the method indicates in the layout instruction that the relevant data of the corresponding interactive objects are displayed in the display area of ​​each interactive object, for example, the personal relevant information of the anchor is displayed in the anchor PK scene. For another example, in the scene of game live broadcast, the audience can view the real-time data of the player being broadcast and the economic curve chart of the game in real time by hovering over the player's avatar.

[0071] In step S104, the corresponding live interactive page is displayed according to the layout instructions in the supplementary enhancement information, where the layout instructions are used to indicate the display positions and sizes of multiple interactive objects on the live interactive page.

[0072] According to one embodiment, the embodiment of the present application displays the live content corresponding to each interactive object in the live interactive page in the form of a grid.

[0073] For example, refer to Figure 2 The grid layout of the exemplary voice chat room scene shown includes the room owner and 8 microphone users.

[0074] Optionally, the supplementary enhancement information according to this embodiment includes instruction information for dynamically adjusting each grid to adapt to different playback scenarios or user needs. For example, when multiple anchors are online at the same time, the supplementary enhancement information can instruct the client's player to use different grid layouts to optimize the viewing experience.

[0075] Optionally, the supplemental enhancement information may also include encoding parameters, such as changes in resolution or frame rate, which may affect the size and display effect of the grid.

[0076] Specifically, the step 104 of this embodiment includes steps S1041 to S1043.

[0077] In step S1041, based on the layout instruction in the supplementary enhancement information, the view parameters of each grid to be displayed are obtained. The view parameters include various parameters, such as the shape and size of the grid, the resolution of the video content, etc.

[0078] In step S1042, based on the display area size in the client, the coordinates of each grid are calculated respectively.

[0079] In step S1043, based on the calculated coordinates of each grid, a display area corresponding to each grid is created in the live interactive page.

[0080] The following is an example to illustrate the process of creating a live interactive page with a display area containing multiple grids in an embodiment of the present application.

[0081] Reference Figure 3 In this example, the process starts with "business createBizView", which means that the business logic triggers the operation of creating a view. Then get the width of the display area and calculate the coordinates. Create a transparent business view (bizView) based on the size of the display area. Draw a transparent cellView on each video grid and add it to bizView.

[0082] in, Figure 3 The flowchart on the left is the main business process, describing the basic steps to create a business view. The flowchart on the right is the branch process under specific conditions, that is, the special case when the business end (end B) is already in the confluence.

[0083] Referring to the flowchart on the left, the process of creating and managing business views includes:

[0084] Call the createBizView method to start creating a business view;

[0085] Get the width of the display area;

[0086] Perform coordinate calculation to determine the position of the view on the interface;

[0087] Create a transparent bizView according to the size of the display area;

[0088] Draw a transparent cellView on each video grid and add it to bizView;

[0089] Next, for the business end (B end), add bizView to the parent view of renderView, align the frame, ensure that bizView is correctly displayed on the interface, and finally return bizView to the business end; for the client (C end), bizView is overlaid on the screen by the business end to complete the display of the view.

[0090] Reference Figure 3 The flowchart on the right includes:

[0091] Get the width of the display area;

[0092] Check whether it is in the confluence state, that is, whether it has been merged with other video streams;

[0093] If yes, directly use the width of hostNode (video preview area); if not, ask the service end for the width through the protocol;

[0094] If getting the width fails, use the screen width (if it is a portrait screen) or height (if it is a landscape screen) directly.

[0095] In this example, in the step of "getting the display area width" or "calculating coordinates", the supplementary enhancement information is received and parsed, and the layout of the bizView is adjusted according to the supplementary enhancement information. For example, if the supplementary enhancement information indicates a new layout change, the system can adjust the position and size of the cellView in the bizView accordingly.

[0096] According to one embodiment, the supplementary enhancement information also includes layout adjustment instructions for the live interactive page, and the method also includes steps S204 and S205 executed by the server, and step S105 executed by the client.

[0097] In step S204, a corresponding layout adjustment instruction is generated based on the interactive operation of the live broadcast end or the client in the live broadcast interactive page.

[0098] Among them, the interactive operations include various interactions that can be performed with the live broadcast object, such as displaying the anchor ranking, personal information and classification information, user microphone operations, etc.

[0099] The interactive operation includes an interactive operation that can affect the view display in the playback interface, for example, an operation performed by the anchor in a multi-person chat room to zoom in / out his own display area, or to zoom in / out the display area of ​​other live broadcast objects.

[0100] Optionally, in step S204, a corresponding layout adjustment instruction is generated according to the setting operation of the live broadcast object on the live broadcast end on the display layout.

[0101] For example, refer to Figure 2 In the voice chat room interface shown, the host can freely set the size, position, etc. of each grid.

[0102] In step S205, the supplementary enhancement information including the layout adjustment instruction is sent to the client.

[0103] In step S105, based on the layout adjustment instruction in the supplementary enhancement information from the server, the corresponding display area in the live interactive page is adjusted.

[0104] Specifically, the client obtains the layout adjustment instruction by parsing the supplemental enhancement information, and adjusts the corresponding display area in the interactive page according to the layout adjustment instruction.

[0105] The following is an example to illustrate the process of adjusting the layout of the display area based on the interactive operation in an embodiment of the present application.

[0106] Reference Figure 4 The flowchart shown in the figure describes the interactive operations between the anchor and the audience in the RTC (real-time communication) system, including etc. The flowchart is divided into several parts, each of which involves different components and operations. The components involved in the flowchart include RTC, xinteract, xinteractbiz, xinteractbiz_job, Databus and MySQL. These components work together by sending messages and updating data.

[0107] The following is a detailed description of the flow chart:

[0108] Operations of anchor A and anchor B: anchor A and anchor B can perform the operations of "enlarging themselves" or "enlarging others".

[0109] The process of amplifying oneself: Anchor A performs the operation of amplifying himself; obtains the connection session information and performs parameter verification; calls the SwitchTemplate function to amplify himself; updates the data of anchor A in the user records; obtains the new connection session information; sends a Data Channel message to update the template; updates the connection layout of amplified anchor A; updates the connection layout in SEI; sends a Data Channel message to send a toast notification to the room.

[0110] The process of amplifying others' operations: anchor A performs the operation of amplifying anchor B; obtains the connection session information and performs parameter verification; calls SwitchTemplate to amplify anchor B; updates anchor A's data in user records; calculates the streaming bitrate of each room in the session; updates anchor B's data in user stream control; sends Data Channel messages to update stream control data; updates the streaming bitrate; obtains new connection session information; sends Data Channel messages to update templates; updates the connection layout and amplifies anchor B; updates the connection layout in SEI; sends Data Channel messages to send toast notifications to the room.

[0111] The process of a user who is magnified by others leaving: When the magnified user leaves, the user leaving callback is triggered; the data in the user records is updated; the UniversalUser message is sent; the UniversalUser message is consumed; the connection session information is obtained; if the current users in the session are less than 3, SwitchTemplate is called to restore all users to the original grid layout; if the users in the session are greater than or equal to 3, SwitchTemplate is called to restore the user who magnified X to the original grid layout; if there are more than 3 people, SessionPubControlChange is called to calculate the streaming bitrate of each room in the session; if the streaming bitrate of anchor A / anchor B needs to be updated, the user stream control data will be updated, and the Data Channel message will be sent to update the stream control data, and then the streaming bitrate of anchor A / anchor B will be updated.

[0112] According to one embodiment, the method further includes steps S206 and S207 performed by the server, and step S106 performed by the client.

[0113] In step S206, based on the screen interactive elements set by the anchor at the live broadcast end, element display information related to the screen interactive elements is generated.

[0114] The screen interactive elements include various virtual elements that can be displayed on the screen and interact with the live audience. Optionally, the screen interactive elements are real-time interactive stickers, which can be static pictures, dynamic GIFs or animation effects. For example, the anchor can add a "Follow" sticker in the live broadcast, and the user clicks on the sticker to follow the anchor, which increases the interactivity and fun of the live broadcast.

[0115] The element display information includes but is not limited to the type, initial position, size or animation effect of the element.

[0116] In step S207, the supplementary enhancement information including the element display information is sent to the client device.

[0117] In step S106, based on the element display information in the supplementary enhancement information from the server, the screen interaction element is displayed at a corresponding position in the live interactive page.

[0118] Specifically, the client obtains the element display information by parsing the supplementary enhancement information, and displays the screen interactive element at a corresponding position in the live interactive page based on the original material type, initial position, size or animation effect contained in the element display information.

[0119] According to the method of the embodiment of the present application, by obtaining the target live video stream and supplementary enhancement information from the server, the client can parse the supplementary enhancement information and determine the target playback progress of the target live. Based on the target playback progress indicated by the supplementary enhancement information, the client synchronizes its own playback progress with the target playback progress to ensure that the content seen by the audience is consistent with the content pushed by the anchor, thereby improving the live viewing experience; the supplementary enhancement information contains the layout instructions of the target scene, and the client displays the interactive page of the target scene according to these layout instructions, allowing the anchor to freely set the position and size of the grid displaying the interactive object, which is synchronized to the user end in real time through the supplementary enhancement information, thereby improving the user's viewing experience and interactivity; the live audience can view the real-time data contained in the supplementary enhancement information through interactive operations such as hovering. For example, in e-sports live broadcasts, the audience can view the real-time data of the players and the economic curve of the game in real time by hovering over the hero avatar.

[0120] Figure 5 A schematic diagram of the structure of a client and a server for processing live data provided in an embodiment of the present application is shown.

[0121] The client includes: a device for obtaining supplementary enhancement information of a target live broadcast from a server (hereinafter referred to as "information receiving device 101"), a device for determining a target playback progress of the target live broadcast by parsing the supplementary enhancement information (hereinafter referred to as "information parsing device 102"), and a device for synchronizing the playback progress of the target live broadcast in the client with the target playback progress based on the target playback progress (hereinafter referred to as "playback synchronization device 103").

[0122] The server includes: a device for encoding the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information (hereinafter referred to as "information generating device 201"), and a device for sending the supplementary enhancement information to the client watching the target live broadcast (hereinafter referred to as "information sending device 202").

[0123] Reference Figure 5 The information generating device 201 encodes the playback progress information of the target live broadcast and generates corresponding supplementary enhancement information.

[0124] The playback progress information includes various information that can be used to indicate the playback progress and synchronize the live video content, such as the timestamp of the video frame, the playback status, the current playback time or the total playback time.

[0125] Among them, the target live broadcast is the live broadcast conducted in the live broadcast terminal.

[0126] The supplementary enhancement information includes a timestamp and / or a sequence number of a current video frame.

[0127] According to one embodiment, when a triggering target scene is detected, the information generating device 201 sends the supplementary enhancement information to a client watching the target live broadcast.

[0128] The target scene includes various scenes that need to display multiple live interactive objects on the playback interface, such as a multi-player live broadcast scene or initiating a PK game.

[0129] The information sending device 202 sends the supplementary enhancement information to the client watching the target live broadcast.

[0130] Optionally, the information sending device 202 inserts the supplementary enhancement information into the live broadcast stream of the target live broadcast, obtains the target live broadcast stream, and sends the target live broadcast stream to the client.

[0131] Optionally, the information sending device 202 broadcasts the supplementary enhancement information including the timestamp and sequence number of the current video frame to the client via Socket, wherein Socket includes but is not limited to WebSocket, TCP Socket, etc.

[0132] The server may, upon receiving the supplementary enhancement information acquisition request sent by the client, execute the operations of the information generating device 201 and the information sending device 202 to generate corresponding supplementary enhancement information and return it to the client. Alternatively, upon detecting a triggering target scene, the server executes the operations of the information generating device 201 and the information sending device 202 to generate corresponding supplementary enhancement information and return it to the client.

[0133] The information receiving device 101 obtains the supplementary enhancement information of the target live broadcast from the server.

[0134] Optionally, the client actively sends a supplementary enhancement information acquisition request to the server, thereby receiving the supplementary enhancement information returned based on the request.

[0135] The information parsing device 102 determines the target playback progress of the target live broadcast by parsing the supplementary enhancement information.

[0136] The target playback progress indicates the playback time point that the client should reach at the current moment.

[0137] Specifically, the information parsing device 102 parses the corresponding timestamp information from the payload of the supplementary enhancement information according to the predefined format, and then calculates the target playback progress according to the parsed timestamp information, the time point when the client starts to play the target live broadcast, and the current time point of the client.

[0138] For example, the time point when the client starts playing the target live broadcast is recorded as T1, the timestamp parsed from the payload of the supplementary enhancement information is T2, and the current playback time of the client is t. The target playback progress C is calculated by the following formula C=T2-T1-t, which represents the playback time point that the client should reach at the current moment.

[0139] The playback synchronization device 103 synchronizes the current playback progress of the target live broadcast in the client with the target playback progress based on the target playback progress.

[0140] Specifically, the playback synchronization device 103 determines whether the current playback progress of the client needs to be adjusted based on the current playback progress of the client and the target playback progress.

[0141] Optionally, if the current playback progress of the client deviates from the target playback progress, it is determined that the current playback progress of the client needs to be adjusted. Alternatively, the time difference between the current playback progress of the client and the target playback progress is calculated, and if the time difference is greater than a predetermined threshold, it is determined that the current playback progress of the client needs to be adjusted.

[0142] Next, if it is determined that the current playing progress of the client needs to be adjusted, the playing synchronization device 103 adjusts the current playing progress of the client to make it consistent with the target playing progress.

[0143] The playback synchronization device 103 can adjust the current playback progress of the client to make it consistent with the target playback progress in a variety of ways. For example, the playback speed or time axis can be adjusted according to the time difference between the current playback progress of the client and the target playback progress.

[0144] According to one embodiment, the supplementary enhancement information includes layout instructions of the target scene, the server also includes a layout instruction generating device, and the client also includes an interactive page display device.

[0145] The layout instruction generating device generates corresponding layout instructions according to the number of live broadcast objects interacting in the live broadcast interactive page, and then adds the layout instructions to the supplementary enhancement information and sends it to the client.

[0146] The live interactive page is used to display multiple interactive objects, including hosts, viewers, etc. connected in the target scene.

[0147] The layout instructions are used to indicate the display positions and sizes of multiple interactive objects on the live interactive page.

[0148] The layout instruction generating device may generate corresponding layout instructions based on the number of interactive objects and the default layout settings. For example, layout templates corresponding to different numbers of interactive objects may be pre-stored to generate corresponding layout instructions.

[0149] Optionally, the layout instruction generating device may instruct to display the relevant data of the corresponding interactive objects in the display area of ​​each interactive object, for example, displaying the personal relevant information of the anchor in the anchor PK scene. For another example, in the scene of game live broadcast, the audience can view the real-time data of the live broadcast player and the economic curve chart of the game in real time by hovering over the hero avatar.

[0150] The interactive page display device displays the corresponding live interactive page according to the layout instructions in the supplementary enhancement information, and the layout instructions are used to indicate the display position and size of multiple interactive objects on the live interactive page.

[0151] According to one embodiment, the embodiment of the present application displays the live content corresponding to each interactive object in the live interactive page in the form of a grid.

[0152] For example, refer to Figure 2 The grid layout of the exemplary voice chat room scene shown includes the room owner and 8 microphone users.

[0153] Optionally, the supplementary enhancement information according to this embodiment includes instruction information for dynamically adjusting each grid to adapt to different playback scenarios or user needs. For example, when multiple anchors are online at the same time, the supplementary enhancement information can instruct the client's player to use different grid layouts to optimize the viewing experience.

[0154] Optionally, the supplemental enhancement information may also include encoding parameters, such as changes in resolution or frame rate, which may affect the size and display effect of the grid.

[0155] Specifically, the interactive page display device of this embodiment performs the following operations:

[0156] Based on the layout instructions in the supplementary enhancement information, the view parameters of each grid to be displayed are obtained. The view parameters include various parameters, such as the shape and size of the grid, the resolution of the video content, etc.

[0157] Next, based on the display area size in the client, the coordinates of each grid are calculated respectively.

[0158] Next, based on the calculated coordinates of each grid, a display area corresponding to each grid is created in the live interactive page.

[0159] According to one embodiment, the supplementary enhancement information also includes layout adjustment instructions for the live interactive page, the server also includes a layout adjustment instruction generating device and a layout adjustment instruction sending device, and the client also includes a display adjustment device.

[0160] The layout adjustment instruction generating device generates corresponding layout adjustment instructions based on the interactive operation in the live broadcast end or the client.

[0161] Among them, the interactive operations include various interactions that can be performed with the live broadcast object, such as displaying the anchor ranking, personal information and classification information, user microphone operations, etc.

[0162] The interactive operation includes an interactive operation that can affect the view display in the playback interface, for example, an operation performed by the anchor in a multi-person chat room to zoom in / out his own display area, or to zoom in / out the display area of ​​other live broadcast objects.

[0163] Optionally, the layout adjustment instruction generating device generates a corresponding layout adjustment instruction according to the setting operation of the live broadcast object on the live broadcast end on the display layout.

[0164] For example, refer to Figure 2 In the voice chat room interface shown, the host can freely set the size, position, etc. of each grid.

[0165] The layout adjustment instruction sending device sends the supplementary enhancement information including the layout adjustment instruction to the client device.

[0166] The display adjustment device adjusts the corresponding display area in the live interactive page based on the layout adjustment instruction in the supplementary enhancement information from the server.

[0167] Specifically, the client obtains the layout adjustment instruction by parsing the supplemental enhancement information, and adjusts the corresponding display area in the interactive page according to the layout adjustment instruction.

[0168] According to an embodiment, the supplementary enhancement information further includes element display information, the server further includes an element information generating device and an element information sending device, and the client further includes an element display device.

[0169] The element information generating device generates element display information related to the screen interactive elements based on the screen interactive elements set by the anchor at the live broadcast end.

[0170] The screen interactive elements include various virtual elements that can be displayed on the screen and interact with the live audience. Optionally, the screen interactive elements are real-time interactive stickers, which can be static pictures, dynamic GIFs or animation effects. For example, the anchor can add a "Follow" sticker in the live broadcast, and the user clicks on the sticker to follow the anchor, which increases the interactivity and fun of the live broadcast.

[0171] The element display information includes but is not limited to the type, initial position, size or animation effect of the element.

[0172] The element information sending device sends the supplementary enhancement information including the element display information to the client device.

[0173] The element display device displays the screen interactive element at a corresponding position in the live interactive page based on the element display information in the supplementary enhancement information from the server.

[0174] Specifically, the client obtains the element display information by parsing the supplementary enhancement information, and displays the screen interactive element at a corresponding position in the live interactive page based on the original material type, initial position, size or animation effect contained in the element display information.

[0175] According to the device of the embodiment of the present application, by obtaining the video stream and supplementary enhancement information of the target live broadcast from the server, the client can parse the supplementary enhancement information and determine the target playback progress of the target live broadcast. Based on the target playback progress indicated by the supplementary enhancement information, the client synchronizes its own playback progress with the target playback progress to ensure that the content seen by the audience is consistent with the content pushed by the anchor, thereby improving the live viewing experience; the supplementary enhancement information contains the layout instructions of the target scene, and the client displays the interactive page of the target scene according to these layout instructions, allowing the anchor to freely set the position and size of the grid displaying the interactive object, which is synchronized to the user end in real time through the supplementary enhancement information, thereby improving the user's viewing experience and interactivity; the live audience can view the real-time data contained in the supplementary enhancement information through interactive operations such as hovering. For example, in e-sports live broadcasts, the audience can view the real-time data of the players and the economic curve of the game in real time by hovering over the hero avatar.

[0176] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0177] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0178] Figure 5The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0179] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0180] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0181] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0182] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0183] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0184] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0185] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0186] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0191] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0193] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for processing live broadcast data at a client, wherein: The method comprises: Obtain supplementary enhancement information of the target live broadcast from the server; Determine the target playback progress of the target live broadcast by parsing the supplementary enhancement information; Based on the target playback progress, the current playback progress of the target live broadcast in the client is synchronized with the target playback progress.

2. The method according to claim 1, wherein: The step of determining the target playback progress of the target live broadcast by parsing the supplementary enhancement information includes: Parsing corresponding timestamp information from the payload of the supplementary enhancement information according to a predefined format; The target playback progress is calculated based on the parsed timestamp information, the time point when the client starts playing the target live broadcast, and the current time point of the client.

3. The method according to claim 1 or 2, wherein: The supplementary enhancement information also includes a layout instruction of a live interactive page, and the method further includes: According to the layout instructions in the supplementary enhancement information, the corresponding live interactive page is displayed, and the layout instructions are used to indicate the display positions and sizes of multiple interactive objects on the live interactive page.

4. The method according to claim 3, wherein: The method displays the live content corresponding to each interactive object in the live interactive page in the form of a grid, and the displaying of the corresponding live interactive page according to the layout instruction in the supplementary enhancement information includes: Based on the layout instructions in the supplementary enhancement information, obtain the view parameters of each grid to be displayed; Based on the display area size in the client, calculate the coordinates of each grid respectively; Based on the calculated coordinates of each grid, a display area corresponding to each grid is created in the live interactive page.

5. The method according to claim 3 or 4, wherein: The supplementary enhancement information also includes a layout adjustment instruction of the live interactive page, and the method further includes: Based on the layout adjustment instructions in the supplementary enhancement information from the server, adjust the corresponding display area in the live interactive page.

6. The method according to claim 1 or 2, wherein: The supplementary enhancement information also includes element display information, and the method further includes: Based on the element display information in the supplementary enhancement information from the server, the screen interactive element is displayed at a corresponding position in the live interactive page.

7. A method for processing live broadcast data at a server, wherein: The method comprises: Encode the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information; The supplementary enhancement information is sent to a client watching the target live broadcast.

8. The method according to claim 7, wherein: The supplementary enhancement information also includes a layout instruction of a live interactive page, and the method further includes: According to the number of live broadcast objects interacting in the live broadcast interactive page, a corresponding layout instruction is generated, and the layout instruction is added to the supplementary enhancement information and sent to the client.

9. The method according to claim 7 or 8, wherein: The supplementary enhancement information also includes a layout adjustment instruction of the live interactive page, and the method further includes: Generate corresponding layout adjustment instructions based on the interactive operations of the live broadcast end or the client in the live broadcast interactive page; The supplementary enhancement information including the layout adjustment instruction is sent to the client.

10. The method according to claim 7 or 8, wherein: The supplementary enhancement information also includes element display information, and the method further includes: Based on the screen interactive elements set by the anchor on the live broadcast end, generating element display information related to the screen interactive elements; The supplemental enhancement information including the element display information is sent to the client device.

11. A device for processing live broadcast data at a client, wherein: The device comprises: A device for obtaining supplementary enhancement information of a target live broadcast from a server; A device for determining a target playback progress of a target live broadcast by parsing the supplementary enhancement information; A device for synchronizing the current playback progress of the target live broadcast in the client with the target playback progress based on the target playback progress.

12. A device for processing live broadcast data at a server, wherein: The device comprises: Device for encoding based on the playback progress information of the target live broadcast to generate corresponding supplementary enhancement information; A device for sending the supplemental enhancement information to a client watching the target live broadcast.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 6, or the method of any one of claims 7 to 10.

14. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 6, or to implement the method according to any one of claims 7 to 10.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 10.

Images