Live broadcast processing method and related equipment

Abstract

The embodiment of the invention provides a live broadcast processing method and related equipment, and the method comprises the steps: obtaining a video code stream of a live broadcast video, the video code stream comprises the video content of the live broadcast video and the auxiliary enhancement information of the live broadcast video; the auxiliary enhancement information is used for recording live scene content of the live video; performing analysis processing on the video code stream to obtain video content and auxiliary enhancement information; playing the analyzed video content; and in the playing process of the video content, presenting the live broadcast scene content based on the auxiliary enhancement information. According to the embodiment of the invention, the live broadcast content presented in the live broadcast scene can be enriched, and the content presentation flexibility in the live broadcast process is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and specifically to a live broadcast processing method and related equipment, and in particular to a live broadcast processing method, a live broadcast processing device, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of Internet technology, it is becoming more and more popular to watch live videos through various live broadcast platforms on the Internet. The so-called live broadcast refers to the process of using the Internet to synchronously display the real-time events happening on the side of a host or anchor object to the viewers. However, in the current live broadcast scene, only the live broadcast screen of the anchor object can be presented. For example, when the anchor object is broadcasting a game, only the game operation screen of the anchor object during the live broadcast of the game can be presented, which makes the presented live content too single and the live content presentation is not flexible enough. Summary of the invention

[0003] The embodiments of the present application provide a live broadcast processing method and related equipment, which can enrich the live broadcast content presented in the live broadcast scenario and improve the flexibility of content presentation during the live broadcast process.

[0004] On the one hand, an embodiment of the present application provides a live broadcast processing method, the method comprising:

[0005] Obtaining a video code stream of a live video, where the video code stream includes the video content of the live video and auxiliary enhancement information of the live video; the auxiliary enhancement information is used to record the live scene content of the live video;

[0006] Analyze and process the video code stream to obtain video content and auxiliary enhancement information;

[0007] Play the video content obtained by parsing; and,

[0008] During the playback of video content, the live scene content is presented based on the auxiliary enhancement information.

[0009] On the one hand, an embodiment of the present application provides a live broadcast processing method, the method comprising:

[0010] Collect video content of live video and collect live scene content of live video;

[0011] Generate auxiliary enhancement information for live video based on live scene content;

[0012] The video content and the auxiliary enhancement information are encoded to obtain a video code stream of the live video.

[0013] On the one hand, an embodiment of the present application provides a live broadcast processing device, the device comprising:

[0014] An acquisition unit is used to acquire a video code stream of a live video, wherein the video code stream includes video content of the live video and auxiliary enhancement information of the live video; the auxiliary enhancement information is used to record live scene content of the live video;

[0015] A processing unit, used to parse and process the video code stream to obtain video content and auxiliary enhancement information;

[0016] The processing unit is also used to play the video content obtained by analysis; and, during the playback of the video content, present the live scene content based on the auxiliary enhancement information.

[0017] On the one hand, an embodiment of the present application provides a live broadcast processing device, the device comprising:

[0018] A processing unit, used to collect video content of a live video and collect live scene content of the live video;

[0019] The processing unit is further used to generate auxiliary enhancement information of the live video based on the live scene content;

[0020] The encoding unit is used to encode the video content and the auxiliary enhancement information to obtain a video code stream of the live video.

[0021] In one aspect, an embodiment of the present application provides a computer device, the computer device comprising:

[0022] a processor suitable for executing a computer program;

[0023] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the live broadcast processing method as described above is implemented.

[0024] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is loaded by a processor and executed as the above-mentioned live broadcast processing method.

[0025] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the above-mentioned live broadcast processing method is implemented.

[0026] In an embodiment of the present application, a video code stream of a live video is obtained, and the video code stream includes the video content of the live video and the auxiliary enhancement information of the live video, and the auxiliary enhancement information is used to record the live scene content of the live video; it can be seen that by adding the auxiliary enhancement information to the video code stream, the video code stream can include the live scene content of the live video, and thus the video code stream can provide richer information in addition to the video content of the live video. The video code stream is parsed and processed to obtain the video content and the auxiliary enhancement information; the video content obtained by the analysis is played; and during the playback of the video content, the live scene content is presented based on the auxiliary enhancement information. It can be seen that during the live broadcast process, the embodiment of the present application can not only present the video content of the live video, but also present the live scene content related to the live video according to the auxiliary enhancement information, enriching the live content presented by the live scene and improving the flexibility of content presentation during the live broadcast process. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A schematic diagram of the architecture of a live broadcast processing system provided for an exemplary embodiment of the present application;

[0029] Figure 2 A live broadcast processing flow chart provided for an exemplary embodiment of the present application;

[0030] Figure 3 A schematic diagram of a live scene content collection selection interface provided by an exemplary embodiment of the present application;

[0031] Figure 4 A flowchart of a live broadcast processing method provided for an exemplary embodiment of the present application;

[0032] Figure 5a A schematic diagram of presenting live scene content provided by an exemplary embodiment of the present application;

[0033] Figure 5b A schematic diagram of presenting live scene content provided by an exemplary embodiment of the present application;

[0034] Figure 5c A schematic diagram of a prompt information output method provided for an exemplary embodiment of the present application;

[0035] Figure 5d A schematic diagram of displaying different live broadcast scene contents provided for an exemplary embodiment of the present application;

[0036] Figure 5e A schematic diagram of selecting a display position of live scene content provided by an exemplary embodiment of the present application;

[0037] Figure 6 A flowchart of a live broadcast processing method provided for an exemplary embodiment of the present application;

[0038] Figure 7 A schematic diagram of a process for collecting live scene content of a live video provided by an exemplary embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of a live broadcast processing device provided for an exemplary embodiment of the present application;

[0040] Fig. 9 A structural schematic diagram of a live broadcast processing device provided for another exemplary embodiment of the present application;

[0041] Fig.10 A schematic diagram of the structure of a computer device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The following is an introduction to the technical terms involved in the embodiments of the present application:

[0044] 1. Live Broadcast

[0045] Live broadcast is a technical means that uses the Internet to break the limitation of space and synchronously present the real-time events on the side of one or more anchor objects to the viewing objects. At present, the so-called live broadcast usually refers to online live broadcast or video live broadcast. It absorbs and continues the advantages of the Internet, uses video to broadcast live online, publishes product display, game teaching, beauty teaching and other content on the Internet, and synchronizes the content to the viewing objects by using the intuitive, fast and geographically unlimited characteristics of the Internet. A live broadcast usually involves one or more anchor objects. Among them, the anchor object refers to the content provider of the live broadcast, and the live broadcast device on the side of the anchor object refers to the device used by the anchor object. Live broadcast usually refers to the video content of the live broadcast video collected by the anchor object through the live broadcast device in the live broadcast scene, that is, the video content of the live broadcast video usually refers to the real-time event content on the side of the anchor object collected by the live broadcast device. The viewing object refers to the content viewer of the live broadcast. The video content collected from the anchor object side will be transmitted to the viewing object via the Internet, and the viewing object can watch these contents in real time.

[0046] 2. Live Video

[0047] Live video refers to the video obtained by collecting video content through live broadcast equipment in a live broadcast scene. Live video can include a frame sequence consisting of multiple video frames, and each video frame can contain part or all of the video content of the live video.

[0048] When encoding live video, video frames can be divided into I frames, P frames and B frames according to whether the video frames need to refer to other video frames to decode the video content of the video frames.

[0049] I frame (Intra-coded picture), also known as key frame, is an independent frame that carries all information and can be decoded independently without referring to other images. It can be simply understood as a static picture.

[0050] A P frame (Predictive-coded Picture) indicates the difference between this video frame and the previous I frame or P frame. The previous I frame or P frame needs to be referenced during the video encoding process.

[0051] B frame (Bidirectionally predicted picture), also known as bidirectional difference frame, means that B frame records the difference between the current video frame and the previous and next video frames. In other words, when encoding the video, the B frame not only refers to the previous video frame, but also to the next video frame.

[0052] In one implementation, a video frame can be encoded into one or more macroblocks, each of which can include one or more data units (Network Abstraction Layer Unit, NALU). A data unit is also called a network abstraction layer data unit, which refers to a data unit in video encoding that satisfies network transmission. The video content of a live video can be stored in a data unit.

[0053] 3. Supplemental Enhancement Information (Sei)

[0054] Auxiliary enhancement information can be used to carry some auxiliary information that is not related to the video content of the live video, such as content description information of the live video and additional data of the live video. In an embodiment of the present application, the auxiliary enhancement information can also be used to record the live scene content of the live video. The live scene content may include at least one of the following: ① The operation performed by the anchor object of the live video in the live scene. Schematically, the live scene includes a game live scene, and the operation performed by the anchor object of the live video in the live scene may be a game operation, such as operating a game character in a live device to launch skills, move, etc. ② The environment in which the anchor object is located in the live scene, which may include but is not limited to: the distance between the anchor object and the live device used by the anchor object, the light intensity of the environment in which the anchor object is located, the temperature of the environment in which the anchor object is located, the humidity of the environment in which the anchor object is located, etc. ③ The object state of the anchor object in the live scene, the object state may include emotional state (such as happy, excited, etc.), physical state (such as body temperature, heart rate, etc.), attention state (such as attention is greater than a threshold, it is considered that the anchor object is focused in the live scene), etc. ④ The configuration of the live scene. Schematically, taking the game live broadcast scene as an example, the configuration of the live broadcast scene may include but is not limited to: key configuration, sensitivity configuration, sound effect configuration, etc. The so-called key configuration refers to the keys configured for game operations in the game live broadcast scene. For example, the host object controls the game character to move, jump, attack, etc. in the game. These operations require the host object to press the corresponding keys in the game to complete. Then, the corresponding keys can be configured for these operations. For example, the shift key on the keyboard can be configured to control the movement of the game character, and the ctrl key can be configured to control the jump of the game character. Sensitivity configuration may include but is not limited to: the sensitivity of skill release, the sensitivity of key operation, etc. Sound effect configuration refers to the sound effects configured for the game live broadcast scene. It should be understood that depending on different game scenes, there may be certain differences in key configuration, sensitivity configuration, and sound effect configuration.

[0055] In an embodiment of the present application, the auxiliary enhancement information will be encapsulated in the video code stream of the live video and transmitted to the decoding end together, so the auxiliary enhancement information and the video content can be presented synchronously.

[0056] It should be noted that in this application, the relevant data involved in the live broadcast processing process, for example: the operations performed by the anchor object of the live video in the live broadcast scene, the environment in which the anchor object is located, the object status of the anchor object in the live broadcast scene, the configuration of the live broadcast scene, etc. When the above embodiments of this application are applied to specific products or technologies, the permission or consent of the object must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and comply with the principles of legality, legitimacy and necessity, and do not involve the acquisition of data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data involved in the embodiments of this application are obtained after the object has been authorized separately. In addition, when obtaining the separate authorization of the object, the purpose of the relevant data involved is indicated to the object.

[0057] The embodiment of the present application provides a live broadcast processing solution, the general principle of which is as follows:

[0058] 1. Processing flow at the encoding end:

[0059] ① Collect the video content of the live video and the live scene content of the live video. ② Generate auxiliary enhancement information of the live video based on the live scene content. ③ Encode the video content and the auxiliary enhancement information to obtain the video code stream of the live video, and send the video code stream of the live video to the decoding end.

[0060] Through the above scheme, auxiliary enhancement information is added to the video bitstream, so that the video bitstream can include the live scene content of the live video, thereby enabling the video bitstream to provide richer information in addition to the video content of the live video.

[0061] 2. Processing flow at the decoding end:

[0062] ① Obtain the video bitstream of the live video sent by the encoder, which includes the video content of the live video and the auxiliary enhancement information of the live video. ② Parse the video bitstream to obtain the video content and the auxiliary enhancement information. ③ Play the video content obtained by the analysis; and, during the playback of the video content, present the live scene content based on the auxiliary enhancement information.

[0063] Through the above scheme, during the live broadcast process, not only the video content of the live video can be presented, but also the live scene content related to the live video can be presented according to the auxiliary enhancement information, thereby enriching the live content presented by the live scene and improving the flexibility of content presentation during the live broadcast process.

[0064] Next, the live broadcast processing system provided by the embodiment of the present application is described. Figure 1 , is a schematic diagram of the architecture of a live broadcast processing system provided by an exemplary embodiment of the present application. The live broadcast processing system 10 may include a live broadcast device 101 and a rendering device 102. The live broadcast device 101 is located at the encoding end, and the live broadcast device can be a device used by the anchor object; the rendering device is located at the decoding end, and the rendering device can be a device used by the viewing object. The live broadcast device 101 can be a terminal or a server, and the rendering device 102 can be a terminal or a server. A communication connection can be established between the live broadcast device 101 and the rendering device 102. Among them, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0065] For ease of understanding, let's combine Figure 2 , describing the entire interaction process between the live broadcast device 101 and the presentation device 102 in the embodiment of the present application. Figure 2 , is a live broadcast processing flow chart provided by an exemplary embodiment of the present application. The live broadcast processing flow is as follows:

[0066] (1) The live broadcast device 101 mainly involves the following steps s11-s17:

[0067] s11. Capturing the video content of the live video. When the host is performing a live video broadcast, the live broadcast device 101 can capture the video content of the live video through a first acquisition device. The first acquisition device can be a hardware component provided in the live broadcast device 101. For example, the first acquisition device can be a common camera, a stereo camera, a light field camera, etc. provided in a terminal device. The first acquisition device can also refer to a hardware device connected to the live broadcast device 101, such as a camera connected to a server.

[0068] s12. Encode the video content. The live video includes a frame sequence consisting of multiple video frames, each video frame is divided into one or more macroblocks, each macroblock includes one or more data units, and the video content of the live video is stored in the data unit.

[0069] s13. Determine the live scene content to be collected.

[0070] In one implementation, for different live videos, the live scene contents of the live videos to be collected are the same or different. For example, if the live video is a game live video, the live scene contents of the live video to be collected may include the operations performed by the host object of the live video in the live scene. For another example, if the live video is a dress-up live video, the live scene contents of the live video to be collected may include the environment and object status of the host object of the live video in the live scene. In addition, for the same type of live video, such as a game type live video, the live scene contents of the live video to be collected may be different or the same depending on the game scene.

[0071] In another implementation, a live scene content collection selection interface may be provided, such as Figure 3 As shown, it is a schematic diagram of a live scene content collection selection interface provided by an exemplary embodiment of the present application. The live scene content collection selection interface 31 includes live scene content 1 and live scene content 2. The host object can select the live scene content to be collected from the live content collection selection interface according to the collection requirements, and collect the selected live scene content through the collection device. If the host object selects live scene content 2, the live broadcast device can collect the live scene content 2 of the live video through the collection device.

[0072] s14. Collecting the live scene content of the live video. The live broadcast device 101 can collect the live scene content of the live broadcast video through the second collection device. The second collection device can be a hardware component set in the live broadcast device 101. For example, the second collection device can be various sensors set in the terminal device. The first collection device can also refer to a hardware device connected to the live broadcast device 101, such as various sensors connected to the server. The sensors here can include but are not limited to: temperature sensors, distance sensors, humidity sensors, heart rate sensors, etc.

[0073] s15. Use auxiliary enhancement information frame to encapsulate live scene content. The live broadcast device 101 can generate auxiliary enhancement information based on the collected live broadcast scene content, and store the auxiliary enhancement information in the auxiliary enhancement information frame. Schematically, the auxiliary enhancement information includes operation information, so the collected live broadcast scene content includes the operations performed by the anchor object of the live broadcast video in the live broadcast scene, and the operation point coordinates, operation speed and operation direction can be determined according to the operations performed by the anchor object of the live broadcast video in the live broadcast scene, and the operation information is generated according to the operation point coordinates, operation speed and operation direction. The operation point coordinates here can refer to the contact point coordinates between the anchor object and the screen of the live broadcast device.

[0074] s16. Determine whether it is necessary to insert an auxiliary enhancement information frame before the target reference video frame. When it is necessary to transmit additional auxiliary enhancement information, that is, if the live broadcast device collects live scene content and generates auxiliary enhancement information, it can be determined that additional auxiliary enhancement information needs to be transmitted. At this time, the auxiliary enhancement information frame can be inserted before the target reference video frame so that the live scene content can be presented together with the video content. When it is not necessary to transmit additional auxiliary enhancement information, that is, if the live broadcast device does not collect live scene content, it can be determined that it is not necessary to transmit additional auxiliary enhancement information. At this time, it can be determined that it is not necessary to insert an auxiliary enhancement information frame between the target reference video frames. The frame sequence includes one or more reference video frames, and the target reference video frame is a reference video frame determined from one or more reference video frames. The reference video frame may include an I frame or a P frame. The reason for inserting the auxiliary enhancement information frame before the I frame or the P frame here is that when the I frame and the P frame are encoded, since the I frame does not depend on other video frames, and the P frame only depends on its previous video frame, inserting the auxiliary enhancement information frame before the I frame or the P frame will not affect the encoding of the video frame.

[0075] s17. Encode the frame sequence with the auxiliary enhancement information frame added to obtain a video code stream, and send the video code to the presentation device 102. As an implementation method, the video code stream can be sent to a content delivery network (CDN), and the presentation device 102 can subsequently pull the video code stream of the live video from the CDN.

[0076] (2) The presenting device 102 mainly involves the following steps s21-s24:

[0077] s21. Obtain the video code stream of the live video sent by the live broadcast device 101. In one implementation, the presentation device 102 may pull the video code stream of the live video from the CDN.

[0078] s22. Decode the video code stream: The presentation device 102 may decode the video code stream to obtain a data unit storing video content and a target data unit storing auxiliary enhancement information.

[0079] s23. Parse to obtain auxiliary enhancement information and video content. As an implementation method, in order to display the live scene content and the video content synchronously, it is necessary to parse and obtain the corresponding auxiliary enhancement information while parsing and obtaining the video content of each video frame. Therefore, the presentation device 102 can obtain the video content from the data unit storing the video content and obtain the auxiliary enhancement information from the target data unit.

[0080] s24, UI (User Interface) display: The presentation device 102 can play the video content obtained by parsing; and, during the playback of the video content, present the live scene content based on the auxiliary enhancement information. Specifically, draw the video content, and draw the live scene content based on the auxiliary enhancement information, then render the drawn video content and the drawn live scene content at the same time, play the rendered video content, and present the rendered live scene content at the same time. Through the auxiliary enhancement information, the live scene content of the live video can be better presented, thereby enriching the live content presented by the live scene.

[0081] It should be understood that the live broadcasting scenarios involved in the embodiments of the present application may include but are not limited to: game live broadcasting scenarios, dressing live broadcasting scenarios, sports live broadcasting scenarios, music live broadcasting scenarios, etc. Correspondingly, the video content of the live broadcasting video may include game teaching, dressing teaching, sports competitions, concerts, etc. The following two examples illustrate the live broadcasting processing flow involved in the embodiments of the present application:

[0082] Example 1: Game live streaming scenario

[0083] When the host object is conducting a live broadcast of game teaching, the live broadcast device 101 can collect the game teaching content (i.e., video content) of the live broadcast of game teaching by the host object. In addition, the live broadcast device 101 will also collect the game operations performed by the host object during the live broadcast of game teaching, as well as the environment in which the host object is located during the live broadcast of game teaching, such as the game operation including operating a certain game character to move on the live broadcast device. Then the live broadcast device 101 can determine the coordinates of the game operation point on the live broadcast device, the operation direction and operation speed when operating the game character to move based on the game operation performed by the host object during the live broadcast of game teaching, and generate operation information based on the coordinates of the game operation point, the operation direction and the operation speed, and generate environmental information according to the environment in which the host object is located during the live broadcast of game teaching. Auxiliary enhancement information is generated according to the operation information and environmental information. Subsequently, the game teaching content and auxiliary enhancement information of the live broadcast of game teaching by the host object are encoded and processed to obtain a video code stream of the live broadcast video of the game, and the video code stream is sent to the presentation device 102.

[0084] When the viewing object (i.e., the viewer) wants to watch the live broadcast of the game teaching, the viewing object can use the presentation device 102 to obtain the video code stream of the live broadcast of the game, and parse the video code stream to obtain the game teaching content and auxiliary enhancement information. The presentation device 102 can play the game teaching content in the playback interface, and draw and display the operation points during the game operation based on the operation point coordinates in the first display interface (i.e., the interface covered on the playback interface and having a certain degree of transparency), as well as display the operation speed, operation direction, and the environment in which the host object is located. The viewing object can watch the game teaching content, and at the same time understand the operations performed by the host object during the live broadcast of the game teaching and the environment in which the host object is located during the live broadcast of the game teaching from the first display interface, so that the viewing object can more comprehensively understand the detailed operations of the host object and the environment in which the host object is located, which can help the viewing object better learn the game skills of the host object.

[0085] Example 2: Sports live broadcast scenario

[0086] When the host is broadcasting live sports, the live broadcast device 101 can collect the sports game (i.e., video content) of the live broadcast of the host. In addition, the live broadcast device 101 also collects the environment in which the host is broadcasting live sports, such as the light intensity of the host is XX when the host is broadcasting live sports. The live broadcast device 101 can generate auxiliary enhancement information based on the environment in which the host is broadcasting live sports, encode the video content and the auxiliary enhancement information, obtain the video code stream of the live sports video, and send the video code stream to the presentation device 102.

[0087] When the viewing object (i.e., the viewer) wants to watch a live sports broadcast, the viewing object can use the presentation device 102 to obtain the video code stream of the live sports broadcast video from the live broadcast device, and parse the video code stream to obtain the sports game and auxiliary enhancement information. The presentation device 102 can play the sports game in the playback interface, and display the environment in which the host object is located when the live sports broadcast is performed in the second display interface (i.e., an interface that covers the playback interface and has a certain degree of transparency). The viewing object can watch the sports game while understanding the environment in which the host object is located when the live sports broadcast is performed, which can help to more comprehensively understand the environment in which the host object is located and feel the sports atmosphere.

[0088] In the embodiment of the present application, the live broadcast device 101 collects the video content of the live broadcast video, collects the live broadcast scene content of the live broadcast video, and generates the auxiliary enhancement information of the live broadcast video based on the live broadcast scene content. The video content and the auxiliary enhancement information are encoded and processed to obtain the video code stream of the live broadcast video, and the video code stream of the live broadcast video is sent to the presentation device 102; it can be seen that by adding the auxiliary enhancement information to the video code stream, the live broadcast scene content of the live broadcast video can be included in the video code stream, so that the video code stream can provide more abundant information in addition to the video content of the live broadcast video. The presentation device 102 obtains the video code stream of the live broadcast video, which includes the video content of the live broadcast video and the auxiliary enhancement information of the live broadcast video; the video code stream is parsed and processed to obtain the video content and the auxiliary enhancement information; the video content obtained by the analysis is played; and, during the playback of the video content, the live broadcast scene content is presented based on the auxiliary enhancement information. It can be seen that in the live broadcast process, the embodiment of the present application can not only present the video content of the live broadcast video, but also present the live broadcast scene content related to the live broadcast video according to the auxiliary enhancement information, enriching the live broadcast content presented by the live broadcast scene and improving the flexibility of content presentation during the live broadcast process.

[0089] Next, the live broadcast processing method provided by the embodiment of the present application is explained from the decoding end.

[0090] See also Figure 4 , is a flowchart of a live broadcast processing method provided in an embodiment of the present application. The live broadcast processing method can be executed by a presentation device, and the live broadcast processing method can include the following steps S401-S403:

[0091] S401. Obtain a video code stream of a live video, where the video code stream includes video content of the live video and auxiliary enhancement information of the live video; the auxiliary enhancement information is used to record live scene content of the live video.

[0092] The live scene content may include at least one of the following: ① The operation performed by the host object of the live video in the live scene. ② The environment in which the host object is located in the live scene. ③ The object state of the host object in the live scene. ④ The configuration of the live scene. Accordingly, the auxiliary enhancement information may include at least one of the following: ① Operation information. The operation information is used to describe the operation performed by the host object in the live scene, and the operation information may include at least one of the following: operation point coordinates, operation direction, operation speed, etc.; ② Environmental information. Environmental information is used to describe the environment in which the host object is located in the live scene. Environmental information may include at least one of the following: the distance between the host object and the live device used by the host object, the light intensity of the environment in which the host object is located, the temperature of the environment in which the host object is located, and the humidity of the environment in which the host object is located. ③ State information. The state information is used to describe the object state of the host object in the live scene. The state information includes at least one of the following: emotional state information (such as happy, sad, excited, etc.), physical state information (such as body temperature, heart rate, etc.), and attention state information. ④ Configuration information. The configuration information is used to describe the configuration of the live broadcast scene, and the configuration information includes at least one of the following: key configuration information, sensitivity configuration information, and sound effect configuration information.

[0093] S402: Analyze the video code stream to obtain video content and auxiliary enhancement information.

[0094] In one implementation, the video bitstream includes an auxiliary enhancement information frame, the auxiliary enhancement information frame includes one or more macroblocks, each macroblock includes one or more data units, and the auxiliary enhancement information is encapsulated in the data unit. The format of the data unit in the video bitstream is shown in Table 1: the data unit includes a start code field, a header data field, and a data unit body field, the start code field is used to indicate the encoding start position, the header data field is used to indicate the type of data encapsulated in the data unit, when the header data field of the data unit is a first value (such as 0x06 in Table 1), it indicates that the data unit is encapsulated with auxiliary enhancement information. When the header data field of the data unit is a third value (such as 0x25 / 65 in Table 1), it indicates that the data unit is encapsulated with I frame video content, and when the header data field of the data unit is a fourth value (such as 0x21 / 61 in Table 1), it indicates that the data unit is encapsulated with P frame video content.

[0095] Table 1

[0096]

[0097] Among them, the data unit body field can be filled in according to the data format corresponding to the data to be encoded. In one implementation, when the auxiliary enhancement information is encapsulated in the data unit, the data unit body field is filled in according to the data format of the auxiliary enhancement information frame to be encoded. The data format of the auxiliary enhancement information frame is shown in Table 2. The data unit body field includes a start code field (start code), a data unit type field (NALU type), a payload type field (Payloadtype), a video service identification field (UUID), an auxiliary information field, an auxiliary information length field, and an end alignment code field (0x80). The field value of the start code field is 0x00000001, and the length of the start code field is 4 bytes.

[0098] Table 2

[0099]

[0100] The meanings of the fields included in the data unit body field are as follows:

[0101] Data unit type field: indicates the type of data encapsulated in the data unit. When the data unit type field is the first value (such as 0x06 in Table 2), it indicates that the data unit encapsulates auxiliary enhancement information. The length of the data unit type field is 1 byte.

[0102] Payload type field: used to indicate whether the encoding format of the auxiliary enhancement information complies with the preset encoding format. When the Payload type field is the second value (such as 0x05), it indicates that the encoding format of the auxiliary enhancement information complies with the preset encoding format, and the preset encoding format includes the H264 standard format. The length of the Payload type field is 1 byte.

[0103] Video service identification field: used to indicate the service identification code of live video in live broadcast scenario. For live video in different live broadcast scenarios, the service identification code of live video is different and can be determined by the service party. The length of the video service identification field is 16 bytes.

[0104] Auxiliary information field: used to store auxiliary enhancement information. The auxiliary enhancement information stored in the auxiliary information field may be stored in a format specified by the service provider. The length of the auxiliary information field may be N bytes.

[0105] Auxiliary information length field: used to indicate the length of the auxiliary enhancement information. The length of the auxiliary information can be calculated and determined based on the auxiliary enhancement information; the length of the auxiliary information length field can be 2 bytes.

[0106] End alignment code field; used to indicate the end position of the encoding. In the data format of the auxiliary enhanced information frame, the field value of the end alignment code field can be 0x80, that is, the end is aligned with 0x80 after the encoding is completed, and the length of the end alignment code field is 1 byte.

[0107] As can be seen from the above, the auxiliary enhancement information is encapsulated in the data unit, and parsing and processing the video code stream to obtain the auxiliary enhancement information can specifically include: reading the header data field of the data unit; when the target data unit with the header data field as the first value is read, the auxiliary enhancement information is read from the auxiliary information field of the target data unit.

[0108] It should be understood that the video code stream includes multiple video frames, each video frame includes one or more macroblocks, each macroblock may include one or more data units, and the video content in the video frame is stored in the corresponding data unit. At this time, the video content of the live video can be read from the corresponding data unit with reference to the above-mentioned process of reading the auxiliary enhancement information, which will not be repeated here.

[0109] S403: Play the video content obtained by the analysis; and, during the playback of the video content, present the live scene content based on the auxiliary enhancement information.

[0110] Among them, in the embodiments of the present application, different methods can be used to present the live scene content. In addition, according to the different live scene contents of the live video recorded by the auxiliary enhancement information, during the playback of the video content, the method of presenting the live scene content based on the auxiliary enhancement information can be flexibly set to improve the flexibility of presenting the live scene content. Next, the specific implementation method of presenting the live scene content based on the auxiliary enhancement information during the playback of the video content is described:

[0111] (1) The auxiliary enhancement information includes operation information, which includes operation point coordinates, operation speed and operation direction.

[0112] During the playback of video content, presenting live scene content based on auxiliary enhancement information can specifically include: during the playback of video content, restoring the operations performed by the host object of the live video in the live scene based on the operation information, which can ensure that the operations of the host object are restored to a certain extent. In one implementation, a first display interface is output, and the first display interface is covered on the playback interface of the video content, and the transparency of the first display interface is higher than the preset transparency threshold; in the first display interface, the operation point is drawn and displayed based on the coordinates of the operation point, and the operation point is controlled to move in the first display interface according to the operation speed and operation direction to restore the operations performed by the host object in the live scene; wherein, the preset transparency threshold is set according to demand, such as the preset transparency threshold can be 80%, 70%, etc.

[0113] Indicatively, Figure 5a FIG. 1 is a schematic diagram of presenting live scene content provided by an exemplary embodiment of the present application. Figure 5a In the example, a first display interface 501 is output, and the first display interface 501 is covered on the playback interface 502 of the video content. The operation point (such as Figure 5a Then, according to the operation speed V and the operation direction (such as Figure 5a The control unit 3 controls the operation point 3 to move in the first display interface 501 (eg, move to the left).

[0114] In another implementation, while the operation point is drawn and displayed in the first display interface, the operation speed and operation direction can be displayed in the first display interface. Figure 5b FIG. 1 is a schematic diagram of presenting live scene content provided by another exemplary embodiment of the present application. Figure 5b In the first interface, the operation point is drawn and displayed based on the coordinates of the operation point (such as Figure 5b The operation point 1, the operation point 2 and the operation point 3 are displayed, and the operation direction 51 and the operation speed 52 are displayed in the first display interface.

[0115] In addition, in an embodiment of the present application, the presentation device can store the operation information in the entire live broadcast scene, and based on the operation information, generate the operation record data of the anchor object. When it is detected that the live video playback ends, the operation record data of the anchor object can be sent to the corresponding application to directly reproduce the operation of the anchor object in the corresponding application. Schematically, the live broadcast scene includes a game teaching live broadcast scene of a certain game. The presentation device can store the operation information in the entire game live broadcast scene, and based on the operation information, generate the game operation record data of the anchor object. When it is detected that the game teaching live broadcast video ends, the game operation record data of the anchor object can be sent to the game application corresponding to the game, so as to directly reproduce the game operation of the anchor object in the game application.

[0116] (2) Auxiliary enhancement information includes configuration information.

[0117] During the playback of the video content, presenting the live scene content based on the auxiliary enhancement information may specifically include: during the playback of the video content, restoring the configuration of the live scene based on the configuration information. For example, the configuration information includes key configuration information (such as configuring the shift key on the keyboard to control the movement of the game character), and the configuration of the live scene can be restored based on the configuration information, that is, configuring the shift key on the keyboard to control the movement of the game character. For another example, the configuration information includes picture attribute configuration information (such as configuring the picture saturation to XX), and the picture attribute configuration of the live scene can be restored based on the configuration information, that is, configuring the picture saturation to XX.

[0118] Optionally, during the playback of the video content, the configuration of the live broadcast scene can be output in the display interface based on the configuration information, and in the process of outputting the configuration of the live broadcast scene, a restore option can be output. The audience object can trigger (such as clicking or double-clicking) the restore option, and in response to the trigger operation on the restore option, the configuration of the live broadcast scene is restored based on the configuration information.

[0119] (3) Auxiliary enhancement information includes environmental information or status information.

[0120] During the playback of video content, presenting the live scene content based on the auxiliary enhancement information may specifically include: during the playback of the video content, outputting prompt information based on the auxiliary enhancement information, wherein if the auxiliary enhancement information includes environmental information, the prompt information is used to prompt the environment in which the host object of the live video is located in the live scene; if the auxiliary enhancement information includes status information, the prompt information is used to prompt the object status of the host object in the live scene.

[0121] The output method of the prompt information includes any of the following: ① Outputting the prompt information in the second display interface, the second display interface is covered on the playback interface of the video content, and the transparency of the second display interface is higher than the preset transparency threshold. As an implementation method, the prompt information can be output in the first area of ​​the second display interface. The first area can be any area in the second display interface, and the embodiment of the present application does not limit this. Schematically, as Figure 5c FIG. 1 is a schematic diagram of a prompt information output method provided by an exemplary embodiment of the present application. Figure 5c In the first area 53 of the second display interface 503 , prompt information “light intensity XX” and “humidity XX” is output.

[0122] It should be understood that the first display interface and the second display interface may be different or the same display interface.

[0123] ② Outputting prompt information in the playback interface of the video content. In one implementation, the prompt information can be output in the second area of ​​the playback interface. The second area can be any area in the playback interface, such as the upper area, the lower area, the left area, etc. of the playback interface, which is not limited in the embodiments of the present application.

[0124] It should be understood that the method of outputting prompt information in the first area or the second area can be flexibly set, such as scrolling the prompt information in the first area or the second area, or fixedly outputting the prompt information in the first area or the second area, and the embodiments of the present application are not limited to this.

[0125] ③ Output prompt information in text format.

[0126] ④ Output the prompt information in multimedia mode. Multimedia modes may include but are not limited to: animation mode, audio and video mode, etc. For example, the light intensity of the current environment of the anchor object is XX and the humidity is XX in the form of voice.

[0127] (4) Auxiliary enhancement information includes environmental information.

[0128] During the playback of video content, presenting live scene content based on auxiliary enhancement information may specifically include: simulating the playback screen of the video content based on environmental information to form a simulated screen, the attributes of which are consistent with the attributes of the live screen presented by the live device used by the anchor object of the live video in the live scene, and outputting the simulated screen. Among them, the attributes of the simulated screen are consistent with the attributes of the live screen presented by the live device used by the anchor object of the live video in the live scene, which means: simulating the attributes of the screen in the same environment, where the attributes may include: saturation, color, etc. For example, simulating the color and saturation of the playback screen based on environmental information to obtain a simulated screen with the same color and saturation as the playback screen.

[0129] (5) The auxiliary enhancement information includes at least two of operation information, configuration information, environment information and status information.

[0130] As an implementation method, during the playback of the video content, presenting the live scene content based on the auxiliary enhancement information may specifically include: during the playback of the video content, outputting a third display interface, and presenting different live scene contents in the third display interface or in the playback interface of the video content based on the auxiliary enhancement information. Different live scene contents may be presented in different or the same areas of the third display interface, and similarly, different live scene contents may also be presented in different or the same areas of the playback interface. Schematically, as Figure 5d FIG. 1 is a schematic diagram of presenting live scene content provided by an exemplary embodiment of the present application. Figure 5dIn the embodiment, the auxiliary enhancement information includes operation information (such as operation speed and operation direction) and environmental information (such as "light intensity XX, humidity XX"). Based on the auxiliary enhancement information, the operation speed and operation direction can be presented in area 54 of the third display interface 504, and the light intensity XX and humidity XX of the environment in which the host object is located in the live broadcast scene can be presented in area 55.

[0131] As another implementation method, during the playback of video content, presenting live scene content based on auxiliary enhancement information may specifically include: classifying the auxiliary enhancement information, and during the playback of the video content, displaying different live scene contents in layers according to the classification results.

[0132] The so-called layered display may include the following situations: ① Different live scene contents are displayed in different display interfaces, and different display interfaces are all covered on the playback interface of the video content. The transparency of each display interface is higher than the preset transparency threshold, and the transparency of different display interfaces is different. Schematically, the auxiliary enhancement information includes operation information, environmental information and configuration information. During the playback of the video content, display interface 1, display interface 2 and display interface 3 are output, wherein display interface 1, display interface 2 and display interface 3 are all covered on the playback interface of the video content, the transparency of each display interface is higher than the preset transparency threshold, and the transparency of display interface 1, display interface 2 and display interface 3 increases in sequence. At this time, the operation point can be drawn and displayed in display interface 1 based on the coordinates of the operation point, and the operation point can be controlled to move in the first display interface according to the operation speed and operation direction to restore and replay the operations performed by the anchor object in the live scene. The environment in which the anchor object is located in the live scene is displayed in display interface 2 based on the environmental information, and the configuration of the live scene is output in display interface 3 based on the configuration information. ② At least two live scene contents among different live scene contents are displayed in one display interface, and other live scene contents among different live scene contents are displayed in another display interface. Schematically, the auxiliary enhancement information includes operation information, environmental information and status information. Based on the operation information and environmental information, the operation point, operation speed and operation direction can be displayed in display interface 1, and the environment in which the host object is located in the live scene can be displayed in display interface 1. Based on the status information, the object status of the host object in the live scene is displayed in display interface 2. ③ At least one live scene content among different live scene contents is displayed in one display interface, and other live scene contents are displayed in the playback interface of the video content. Schematically, the auxiliary enhancement information includes status information and environmental information. Based on the status information, the object status of the host object in the live scene can be displayed in display interface 1, and based on the environmental information, the environment in which the host object is located in the live scene can be displayed in the playback interface.

[0133] It should be understood that the embodiments of the present application do not impose any restrictions on the hierarchical display of different live scene contents. The audience object can choose how to display different live scene contents in layers according to needs. In the embodiments of the present application, a content display selection interface can be provided. The content display selection interface includes the locations where various types of live scene contents are displayed. The audience object can select the location where the live scene content is to be displayed in the content display selection interface. Schematically, as shown in FIG. Figure 5e FIG. 1 is a schematic diagram of selecting a display position of live scene content provided by an exemplary embodiment of the present application. Figure 5e In the embodiment of the present application, the audience object can choose to display the live scene content 1 in the playback interface and choose to display the live scene content 2 in the display interface, then the presentation device can display the live scene content 1 in the playback interface 506 and display the live scene content 2 in the display interface 507. In addition, in the embodiment of the present application, the audience object can also select the live scene content to be displayed in the content display selection interface. If only the operation information is displayed in the content display selection interface, then the operation point can be drawn and displayed based on the operation point coordinates in the first display interface based on the operation information, and the operation point can be controlled to move in the display interface according to the operation speed and operation direction to restore and replay the operation performed by the anchor object in the live scene.

[0134] In an embodiment of the present application, a video code stream of a live video is obtained, the video code stream includes the video content of the live video and auxiliary enhancement information of the live video, the auxiliary enhancement information is used to record the live scene content of the live video; the video code stream is parsed and processed to obtain the video content and the auxiliary enhancement information; the video content obtained by the analysis is played; and during the playback of the video content, the live scene content is presented based on the auxiliary enhancement information. Through the above scheme, during the live broadcast process, not only the video content of the live video can be presented, but also the live scene content related to the live video can be presented according to the auxiliary enhancement information, which enriches the live content presented by the live scene and improves the flexibility of content presentation during the live broadcast process.

[0135] Next, the live broadcast processing method involved in the encoding end is explained.

[0136] See also Figure 6 , is a flowchart of a live broadcast processing method provided by another exemplary embodiment of the present application. The live broadcast processing method can be executed by a live broadcast device, and the live broadcast processing method can include the following steps S601-S603:

[0137] S601: Collect video content of a live video and collect live scene content of the live video.

[0138] The live broadcast scene content includes at least one of the following: operations performed by the host object of the live broadcast video in the live broadcast scene, the environment in which the host object is located in the live broadcast scene, and the object state of the host object in the live broadcast scene.

[0139] In one implementation, the live scene content to be collected can be determined based on the type of live video. Illustratively, the types of live videos can include, but are not limited to: game live type, music live type, outfit live type, and so on. For example, the live video is a game live video, and the types of game live videos include game live types. According to the game live type, the live scene content to be collected includes: the operations performed by the host object of the live video in the live scene and the environment in which the host object is located in the live scene. For another example, the live video is a concert live video, and the types of concert live videos include music live types. According to the music live type, the live scene content to be collected includes: the environment in which the host object is located in the live scene.

[0140] In another implementation, the live scene content to be collected is determined according to the collection requirements of the host object of the host video. Figure 3 In the live scene content collection interface shown, the host object can select the live scene content to be collected in the live scene content collection interface according to the collection requirements. When the selection operation is detected, the selected live scene content is determined as the live scene content to be collected. Indicatively, the host object wants to collect the operations performed by the host object of the live video in the live scene and present them to the viewing object. Then, the host object selects the operations performed by the host object of the live video in the live scene in the live scene in the live scene collection interface. In response to the selection operation, the operations performed by the host object of the live video in the live scene are determined as the live scene content to be collected.

[0141] S602: Generate auxiliary enhancement information for the live video based on the live scene content.

[0142] S603: Encode the video content and the auxiliary enhancement information to obtain a video code stream of the live video.

[0143] In one implementation, the live video includes a frame sequence consisting of multiple video frames. Encoding the video content and the auxiliary enhancement information to obtain a video code stream of the live video may include: adding the auxiliary enhancement information to the auxiliary enhancement information frame of the live video, determining the adding position of the auxiliary enhancement information frame in the frame sequence, and adding the auxiliary enhancement information frame to the frame sequence according to the adding position; encoding the frame sequence to which the auxiliary enhancement information frame is added to form a video code stream.

[0144] Among them, the frame sequence includes one or more reference video frames, which include I frames or P frames; each reference video frame corresponds to a corresponding acquisition time. In this implementation mode, the specific process of determining the addition position of the auxiliary enhancement information frame in the frame sequence may include the following steps: s31, obtaining the acquisition time of the auxiliary enhancement information, and obtaining the acquisition time of each reference video frame in the frame sequence. Specifically, the time of acquiring the live scene content can be determined as the acquisition time of the auxiliary enhancement information. s32, in the frame sequence, determine the target reference video frame whose time difference between the acquisition time and the acquisition time of the auxiliary enhancement information is less than the time difference threshold. The time difference threshold can be set according to requirements. Schematically, the acquisition time of the auxiliary enhancement information is T1, the acquisition time of the reference video frame 1 in the frame sequence is T2, and the acquisition time of the reference video frame 2 in the frame sequence is T3. If the time difference between T1 and T2 is less than the time difference threshold, the reference video frame 1 is determined to be the target reference video frame. s33, obtain the target position of the target reference video frame in the frame sequence. s34, determine the preceding adjacent position of the target position as the addition position of the auxiliary enhancement information frame. The preceding adjacent position here refers to a position that is located before the target position and is adjacent to the target position.

[0145] As an implementation method, the live broadcast device provides a buffer pool, which is used to cache video frames whose time difference between the acquisition time and the system time is less than a threshold, where N is a positive integer greater than or equal to 1. In the implementation method, the acquisition time of the reference video frame cached by the buffer pool is obtained, and the target reference video frame whose time difference between the acquisition time and the acquisition time of the auxiliary enhancement information is less than the time difference threshold is determined from the buffer pool. By searching the target reference video frame from the buffer pool, the complexity of searching the target reference video frame can be reduced, thereby quickly determining the target reference video frame, and then quickly determining the adding position of the auxiliary enhancement information frame.

[0146] Among them, adding auxiliary enhancement information to the auxiliary enhancement information frame of the live video can specifically include: ① dividing the auxiliary enhancement information frame into one or more macroblocks, each macroblock includes one or more data units; the data unit includes a header data field and a data unit body field; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field. ② Storing the auxiliary enhancement information in the auxiliary information field of the target data unit. ③ Calculating the length of the auxiliary enhancement information, and setting the field value of the auxiliary information length field in the target data unit based on the length of the auxiliary enhancement information. ④ Setting the value of the header data field of the target data unit storing the auxiliary enhancement information to a first value; and, setting the value of the payload type field in the target data unit to a second value.

[0147] In the embodiment of the present application, the video content of the live video is collected, and the live scene content of the live video is collected; the auxiliary enhancement information of the live video is generated based on the live scene content; the video content and the auxiliary enhancement information are encoded and processed to obtain a video code stream of the live video. It can be seen that by adding auxiliary enhancement information to the video code stream, the live scene content of the live video can be included in the video code stream, and then the video code stream can provide richer information in addition to the video content of the live video, which can enrich the live content presented by the live scene at the decoding end and improve the flexibility of content presentation during the live broadcast process.

[0148] In some feasible embodiments, different live scene contents correspond to different collection methods. The following is an explanation of the live scene contents of the live video collection:

[0149] (1) When the auxiliary enhancement information includes operation information, the operation information includes the coordinates of the operation point, the operation speed and the operation direction. Preset operation events in the live broadcast device are pre-set, and the preset operation events may include one or more of the following: a pressing event on the live broadcast device, a moving event in the live broadcast device, a release event generated after a pressing event, a moving out event, etc. The live broadcast device can expose the preset operation events to application developers. For different live broadcast devices, the detection of preset operation events is also different. The live broadcast scene content of the live broadcast video may include: in the live broadcast scene, detecting the operation events in the live broadcast device used by the anchor object. If a preset operation event is detected, the application programming interface of the live broadcast device is called to obtain the operation point coordinates corresponding to the preset operation event. Figure 7 As shown, a schematic diagram of a process for collecting live scene content of live video provided by an exemplary embodiment of the present application. Taking a certain operating system as an example, the following process can be used to collect live scene content of live video: s71. In the live scene, the touch screen detector (setOnTouchListener) is used to detect the operation event in the live device used by the anchor object. s72. If a preset operation event is detected, the preset operation event (List of MotionEvent) is obtained, and the operation point coordinates corresponding to the preset operation event are determined. s73. The API (Application Programming Interface) of the live device is called to obtain the operation point coordinates corresponding to the preset operation event.

[0150] Optionally, s74, introduce a Vsvnc (Vertical Synchronization) mechanism: when the screen of the live broadcast device is refreshed, the Vsvnc mechanism is used to trigger the API of the live broadcast device to obtain the operation point coordinates corresponding to the preset operation event, that is, the Vsvnc mechanism is used to trigger the re-acquisition of the preset operation event and determine the operation point coordinates corresponding to the preset operation event. The Vsvnc mechanism can be used to trigger the re-acquisition of the preset operation event and determine the operation point coordinates corresponding to the preset operation event when the screen of the live broadcast device is refreshed every N milliseconds.

[0151] In addition, when a preset operation event is detected, the acceleration sensor and the magnetic sensor can be called to detect the operation speed and operation direction corresponding to the preset operation event. Specifically, the value of the magnetic sensor is affected by the environmental magnetic field, and the magnetic sensor can be called in advance to detect the magnetic vector V of the environment where the anchor object is located. M , the direction of the magnetic vector is horizontal north-south direction, and the magnetic vector is fixed in the same environment; and the acceleration sensor is called to detect the gravity vector V when placed horizontally G , the gravity vector V G The direction of the magnetic force vector V is vertically downward. M and the gravity vector V G Perform difference multiplication to obtain the difference vector V H , based on the gravity vector V G , gravity vector V G and the difference vector V H Construct three-dimensional coordinates (V x , V y , V z ). When a preset operation event is detected, acceleration will be generated. At this time, the acceleration sensor is called to read the acceleration vector and calculate the acceleration vector and the three-dimensional coordinates (V x , V y , V z ) is determined as the angle between each coordinate (i.e. V x , V y , V z ) is an angle change value (Rx, Ry, Rz), and the operation speed and operation direction corresponding to the preset operation event can be determined according to the angle change value.

[0152] (2) When the auxiliary enhancement information includes environmental information, the environmental information includes at least one of the following: the distance between the anchor object and the live broadcast device used by the anchor object, the light intensity of the environment in which the anchor object is located, the temperature of the environment in which the anchor object is located, and the humidity of the environment in which the anchor object is located.

[0153] When the environmental information includes the distance between the object and the live broadcast device used by the anchor object, the distance sensor is called to obtain the distance between the anchor object and the live broadcast device used by the anchor object. When the environmental information includes the light intensity of the environment where the anchor object is located, the light sensor can be called to collect the light intensity of the environment where the anchor object is located. When the environmental information includes the temperature of the environment where the anchor object is located, the temperature sensor can be called to collect the temperature of the environment where the anchor object is located. When the environmental information includes the humidity of the environment where the anchor object is located, the humidity sensor can be called to collect the humidity of the environment where the anchor object is located.

[0154] (3) When the auxiliary enhancement information includes state information, the state information includes at least one of the following: emotional state information, physical state information, and attention state information.

[0155] When the status information includes an emotional state, the facial information of the host object can be obtained, and the facial information can be identified to obtain the emotional state of the host object. When the status information includes physical status information, such as the physical status information includes heart rate, the heart rate of the host object can be collected through a heart rate sensor. When the status information includes attention status information, the attention status information can be determined by the duration of the host object continuously watching the live broadcast device. The longer the duration, the more focused the host object's attention. Collecting the live scene content of the live video can include: obtaining the duration of the host object continuously watching the screen of the live broadcast device, and determining the attention status information based on the obtained duration.

[0156] In summary, different collection methods can be flexibly adopted for different live broadcast scene contents, thereby effectively ensuring that richer information is provided in addition to the video content of the live video during the live broadcast process.

[0157] Next, the live broadcast processing device provided in the embodiment of the present application is described.

[0158] See also Figure 8 , Figure 8 is a structural diagram of a live broadcast processing device provided in an embodiment of the present application. The live broadcast processing device may be a computer program (including program code) in a computer device. For example, the live broadcast processing device may be an application software in a computer device. The live broadcast processing device may be used to execute Figure 4 Some or all of the steps in the method embodiment shown. Figure 8 , the live broadcast processing device includes the following units:

[0159] The acquisition unit 801 is used to acquire a video code stream of a live video, where the video code stream includes the video content of the live video and auxiliary enhancement information of the live video; the auxiliary enhancement information is used to record the live scene content of the live video;

[0160] The processing unit 802 is used to parse the video code stream to obtain video content and auxiliary enhancement information;

[0161] The processing unit 802 is further configured to play the video content obtained by the analysis; and, during the playback of the video content, present the live scene content based on the auxiliary enhancement information.

[0162] The live broadcast scene content includes at least one of the following: operations performed by the host object of the live broadcast video in the live broadcast scene, the environment in which the host object is located in the live broadcast scene, and the object state of the host object in the live broadcast scene;

[0163] The auxiliary enhancement information includes at least one of the following: operation information, configuration information, environment information and status information; the operation information is used to describe the operation performed by the anchor object in the live broadcast scene, the configuration information is used to describe the configuration of the live broadcast scene; the environment information is used to describe the environment where the anchor object is located in the live broadcast scene; the status information is used to describe the object status of the anchor object in the live broadcast scene;

[0164] The operation information includes at least one of the following: operation point coordinates, operation speed, and operation direction; the configuration information includes at least one of the following: key configuration information, sensitivity configuration information, and sound effect configuration information; the environmental information includes at least one of the following: the distance between the anchor object and the live broadcast device used by the anchor object, the light intensity of the environment in which the anchor object is located, the temperature of the environment in which the anchor object is located, and the humidity of the environment in which the anchor object is located;

[0165] The state information includes at least one of the following: emotional state information, physical state information, and attention state information.

[0166] The auxiliary enhancement information includes operation information; the processing unit 802 is specifically configured to:

[0167] During the playback of the video content, the operations performed by the host object who plays back the live video in the live scene are restored based on the operation information.

[0168] The operation information includes the coordinates of the operation point, the operation speed and the operation direction; the processing unit 802 is specifically used to:

[0169] Outputting a first display interface, where the first display interface covers the playback interface of the video content, and the transparency of the first display interface is higher than a preset transparency threshold;

[0170] Drawing and displaying the operating point based on the operating point coordinates in the first display interface; and,

[0171] The operation point is controlled to move in the first display interface according to the operation speed and the operation direction to restore and replay the operations performed by the anchor object in the live broadcast scene.

[0172] The auxiliary enhancement information includes configuration information; the processing unit 802 is specifically configured to:

[0173] During the playback of the video content, the configuration of the live scene is restored based on the configuration information.

[0174] The processing unit 802 is specifically configured to:

[0175] During the playback of the video content, outputting prompt information based on the auxiliary enhancement information;

[0176] Among them, if the auxiliary enhancement information includes environmental information, the prompt information is used to prompt the environment in which the host object of the live video is located in the live scene; if the auxiliary enhancement information includes status information, the prompt information is used to prompt the object status of the host object in the live scene.

[0177] The output mode of the prompt information includes any of the following:

[0178] Outputting prompt information in a second display interface, where the second display interface covers the playback interface of the video content, and the transparency of the second display interface is higher than a preset transparency threshold;

[0179] Outputting prompt information in the playback interface of the video content;

[0180] Output prompt information in text form;

[0181] Output prompt information in multimedia format.

[0182] The auxiliary enhancement information includes environmental information; the processing unit 802 is specifically configured to:

[0183] The playback screen of the video content is simulated based on the environmental information to form a simulated screen, and the attributes of the simulated screen are consistent with the attributes of the live screen presented by the live broadcast device used by the anchor object of the live broadcast video in the live broadcast scene;

[0184] Output simulation image.

[0185] The video bitstream includes an auxiliary enhancement information frame, the auxiliary enhancement information frame includes one or more macroblocks, each macroblock includes one or more data units; the auxiliary enhancement information is encapsulated into the data unit;

[0186] The data unit includes a header data field and a data unit body field; when the header data field of the data unit is a first value, it indicates that auxiliary enhancement information is encapsulated in the data unit; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field; when the payload type field is a second value, it indicates that the encoding format of the auxiliary enhancement information complies with a preset encoding format; the auxiliary information field is used to store the auxiliary enhancement information; and the auxiliary information length field is used to indicate the length of the auxiliary enhancement information;

[0187] The processing unit 802 is specifically used to: read the header data field of the data unit;

[0188] When a target data unit whose header data field is a first value is read, the auxiliary enhancement information is read from the auxiliary information field of the target data unit.

[0189] In an embodiment of the present application, a video code stream of a live video is obtained, and the video code stream includes the video content of the live video and the auxiliary enhancement information of the live video, and the auxiliary enhancement information is used to record the live scene content of the live video; it can be seen that by adding the auxiliary enhancement information to the video code stream, the video code stream can include the live scene content of the live video, and thus the video code stream can provide richer information in addition to the video content of the live video. The video code stream is parsed and processed to obtain the video content and the auxiliary enhancement information; the video content obtained by the analysis is played; and during the playback of the video content, the live scene content is presented based on the auxiliary enhancement information. It can be seen that during the live broadcast process, the embodiment of the present application can not only present the video content of the live video, but also present the live scene content related to the live video according to the auxiliary enhancement information, enriching the live content presented by the live scene and improving the flexibility of content presentation during the live broadcast process.

[0190] See also Fig. 9 , Fig. 9 is a structural diagram of a live broadcast processing device provided in an embodiment of the present application. The live broadcast processing device may be a computer program (including program code) in a computer device. For example, the live broadcast processing device may be an application software in a computer device. The live broadcast processing device may be used to execute Figure 6 Some or all of the steps in the method embodiment shown. Fig. 9 , the live broadcast processing device includes the following units:

[0191] The processing unit 901 is used to collect video content of the live video and collect live scene content of the live video;

[0192] The processing unit 901 is further configured to generate auxiliary enhancement information of the live video based on the live scene content;

[0193] The encoding unit 902 is used to encode the video content and the auxiliary enhancement information to obtain a video code stream of the live video.

[0194] The processing unit 901 is further configured to:

[0195] Determine the live scene content to be collected based on the type of live video; or,

[0196] Determine the live scene content to be collected based on the collection requirements of the host of the live video.

[0197] The auxiliary enhancement information includes operation information, and the operation information includes operation point coordinates, operation speed, and operation direction; the processing unit 901 is specifically used to:

[0198] In the live broadcast scenario, detect the operation events in the live broadcast device used by the anchor object;

[0199] If a preset operation event is detected, the application programming interface of the live broadcast device is called to obtain the operation point coordinates corresponding to the preset operation event; and

[0200] Calling the acceleration sensor and the magnetic sensor to detect the operation speed and operation direction corresponding to the preset operation event;

[0201] The preset operation event includes one or more of the following: a press event, a move event, a release event generated after a press event, and a move-out event.

[0202] The live video includes a frame sequence consisting of multiple video frames; the encoding unit 902 is specifically used to:

[0203] Adding auxiliary enhancement information to the auxiliary enhancement information frame of the live video;

[0204] Determine an adding position of the auxiliary enhancement information frame in the frame sequence, and add the auxiliary enhancement information frame to the frame sequence according to the adding position;

[0205] The frame sequence with the auxiliary enhancement information frame added is encoded to form a video code stream.

[0206] The frame sequence includes one or more reference video frames, and the reference video frames include independent frames or forward prediction frames; the encoding unit 902 is specifically used to:

[0207] Acquiring the acquisition time of the auxiliary enhancement information and acquiring the acquisition time of each reference video frame in the frame sequence;

[0208] Determine, in the frame sequence, a target reference video frame whose time difference between the acquisition time and the acquisition time of the auxiliary enhancement information is less than a time difference threshold;

[0209] Obtaining a target position of a target reference video frame in a frame sequence;

[0210] A preceding adjacent position of the target position is determined as a position for adding the auxiliary enhancement information frame.

[0211] The encoding unit 902 is specifically used for:

[0212] The auxiliary enhancement information frame is divided into one or more macroblocks, each macroblock includes one or more data units; the data unit includes a header data field and a data unit body field; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field;

[0213] storing the auxiliary enhancement information in an auxiliary information field of the target data unit;

[0214] Calculating the length of the auxiliary enhancement information, and setting a field value of an auxiliary information length field in a target data unit based on the length of the auxiliary enhancement information;

[0215] Setting the value of the header data field of the target data unit storing the auxiliary enhancement information to a first value; and

[0216] The value of the payload type field in the target data unit is set to the second value.

[0217] In the embodiment of the present application, the video content of the live video is collected, and the live scene content of the live video is collected; the auxiliary enhancement information of the live video is generated based on the live scene content; the video content and the auxiliary enhancement information are encoded and processed to obtain a video code stream of the live video. It can be seen that by adding auxiliary enhancement information to the video code stream, the live scene content of the live video can be included in the video code stream, and then the video code stream can provide richer information in addition to the video content of the live video, which can enrich the live content presented by the live scene at the decoding end and improve the flexibility of content presentation during the live broadcast process.

[0218] Next, the computer device provided in the embodiment of the present application is described.

[0219] Furthermore, the present application embodiment also provides a schematic diagram of the structure of a computer device. The schematic diagram of the structure of the computer device can be found in Fig.10 The computer device may include: a processor 1001, an input device 1002, an output device 1003 and a memory 1004. The processor 1001, the input device 1002, the output device 1003 and the memory 1004 are connected via a bus. The memory 1004 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the memory 1004.

[0220] In one embodiment, the computer device may be the above-mentioned presentation device. In this embodiment, the processor 1001 executes the following operations by running the program instructions in the memory 1004:

[0221] Obtaining a video code stream of a live video, where the video code stream includes the video content of the live video and auxiliary enhancement information of the live video; the auxiliary enhancement information is used to record the live scene content of the live video;

[0222] Analyze and process the video code stream to obtain video content and auxiliary enhancement information;

[0223] Play the video content obtained by parsing; and,

[0224] During the playback of video content, the live scene content is presented based on the auxiliary enhancement information.

[0225] The live broadcast scene content includes at least one of the following: operations performed by the host object of the live broadcast video in the live broadcast scene, the environment in which the host object is located in the live broadcast scene, and the object state of the host object in the live broadcast scene;

[0226] The auxiliary enhancement information includes at least one of the following: operation information, configuration information, environment information and status information; the operation information is used to describe the operation performed by the anchor object in the live broadcast scene; the configuration information is used to describe the configuration of the live broadcast scene; the environment information is used to describe the environment where the anchor object is located in the live broadcast scene; the status information is used to describe the object status of the anchor object in the live broadcast scene;

[0227] Among them, the operation information includes at least one of the following: operation point coordinates, operation speed, and operation direction; the configuration information includes at least one of the following: key configuration information, sensitivity configuration information, and sound effect configuration information; the environmental information includes at least one of the following: the distance between the anchor object and the live broadcast device used by the anchor object, the light intensity of the environment in which the anchor object is located, the temperature of the environment in which the anchor object is located, and the humidity of the environment in which the anchor object is located; the state information includes at least one of the following: emotional state information, physical state information, and attention state information.

[0228] The auxiliary enhancement information includes operation information. When the processor 1001 presents the live scene content based on the auxiliary enhancement information during the playback of the video content, the processor 1001 may specifically perform the following operations:

[0229] During the playback of the video content, the operations performed by the host object who plays back the live video in the live scene are restored based on the operation information.

[0230] The operation information includes the coordinates of the operation point, the operation speed, and the operation direction. When the processor 1001 restores and replays the operation performed by the anchor object in the live broadcast scene based on the operation information during the playback of the video content, the processor 1001 may specifically perform the following operations:

[0231] Outputting a first display interface, where the first display interface covers the playback interface of the video content, and the transparency of the first display interface is higher than a preset transparency threshold;

[0232] Drawing and displaying the operating point based on the operating point coordinates in the first display interface; and,

[0233] The operation point is controlled to move in the first display interface according to the operation speed and the operation direction to restore and replay the operations performed by the anchor object in the live broadcast scene.

[0234] The auxiliary enhancement information includes configuration information; when the processor 1001 presents the live scene content based on the auxiliary enhancement information during the playback of the video content, the processor 1001 may specifically perform the following operations:

[0235] During the playback of the video content, the configuration of the live scene is restored based on the configuration information.

[0236] In the process of playing the video content, when the processor 1001 presents the live scene content based on the auxiliary enhancement information, it can specifically perform the following operations:

[0237] During the playback of the video content, outputting prompt information based on the auxiliary enhancement information;

[0238] Among them, if the auxiliary enhancement information includes environmental information, the prompt information is used to prompt the environment in which the host object of the live video is located in the live scene; if the auxiliary enhancement information includes status information, the prompt information is used to prompt the object status of the host object in the live scene.

[0239] The output mode of the prompt information includes any of the following:

[0240] Outputting prompt information in a second display interface, where the second display interface covers the playback interface of the video content, and the transparency of the second display interface is higher than a preset transparency threshold;

[0241] Outputting prompt information in the playback interface of the video content;

[0242] Output prompt information in text form;

[0243] Output prompt information in multimedia format.

[0244] The auxiliary enhancement information includes environmental information. When the processor 1001 presents the live scene content based on the auxiliary enhancement information during the playback of the video content, the processor 1001 may specifically perform the following operations:

[0245] The playback screen of the video content is simulated based on the environmental information to form a simulated screen, and the attributes of the simulated screen are consistent with the attributes of the live screen presented by the live broadcast device used by the anchor object of the live broadcast video in the live broadcast scene;

[0246] Output simulation image.

[0247] The video bitstream includes an auxiliary enhancement information frame, the auxiliary enhancement information frame includes one or more macroblocks, each macroblock includes one or more data units; the auxiliary enhancement information is encapsulated into the data unit;

[0248] The data unit includes a header data field and a data unit body field; when the header data field of the data unit is a first value, it indicates that auxiliary enhancement information is encapsulated in the data unit; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field; when the payload type field is a second value, it indicates that the encoding format of the auxiliary enhancement information complies with a preset encoding format; the auxiliary information field is used to store the auxiliary enhancement information; and the auxiliary information length field is used to indicate the length of the auxiliary enhancement information;

[0249] When the processor 1001 parses the video code stream and obtains the auxiliary enhancement information, it can specifically perform the following operations:

[0250] Read the header data field of the data unit;

[0251] When a target data unit whose header data field is a first value is read, the auxiliary enhancement information is read from the auxiliary information field of the target data unit.

[0252] In an embodiment of the present application, a video code stream of a live video is obtained, and the video code stream includes the video content of the live video and the auxiliary enhancement information of the live video, and the auxiliary enhancement information is used to record the live scene content of the live video; it can be seen that by adding the auxiliary enhancement information to the video code stream, the video code stream can include the live scene content of the live video, and thus the video code stream can provide richer information in addition to the video content of the live video. The video code stream is parsed and processed to obtain the video content and the auxiliary enhancement information; the video content obtained by the analysis is played; and during the playback of the video content, the live scene content is presented based on the auxiliary enhancement information. It can be seen that during the live broadcast process, the embodiment of the present application can not only present the video content of the live video, but also present the live scene content related to the live video according to the auxiliary enhancement information, enriching the live content presented by the live scene and improving the flexibility of content presentation during the live broadcast process.

[0253] In another embodiment, the computer device may be the above-mentioned live broadcast device. In this embodiment, the processor 1001 performs the following operations by running the program instructions in the memory 1004:

[0254] Collect video content of live video and collect live scene content of live video;

[0255] Generate auxiliary enhancement information for live video based on live scene content;

[0256] The video content and the auxiliary enhancement information are encoded to obtain a video code stream of the live video.

[0257] The processor 1001 may further perform the following operations:

[0258] Determine the live scene content to be collected based on the type of live video; or,

[0259] Determine the live scene content to be collected based on the collection requirements of the host of the live video.

[0260] The auxiliary enhancement information includes operation information, and the operation information includes operation point coordinates, operation speed, and operation direction. When the processor 1001 collects the live scene content of the live video, it can specifically perform the following operations:

[0261] In the live broadcast scenario, detect the operation events in the live broadcast device used by the anchor object;

[0262] If a preset operation event is detected, the application programming interface of the live broadcast device is called to obtain the operation point coordinates corresponding to the preset operation event; and

[0263] Calling the acceleration sensor and the magnetic sensor to detect the operation speed and operation direction corresponding to the preset operation event;

[0264] The preset operation event includes one or more of the following: a press event, a move event, a release event generated after a press event, and a move-out event.

[0265] The live video includes a frame sequence composed of multiple video frames; when the processor 1001 encodes the video content and the auxiliary enhancement information to obtain the video code stream of the live video, the processor 1001 may specifically perform the following operations:

[0266] Adding auxiliary enhancement information to the auxiliary enhancement information frame of the live video;

[0267] Determine an adding position of the auxiliary enhancement information frame in the frame sequence, and add the auxiliary enhancement information frame to the frame sequence according to the adding position;

[0268] The frame sequence with the auxiliary enhancement information frame added is encoded to form a video code stream.

[0269] The frame sequence includes one or more reference video frames, and the reference video frames include independent frames or forward prediction frames. When the processor 1001 determines the adding position of the auxiliary enhancement information frame in the frame sequence, it may specifically perform the following operations:

[0270] Acquiring the acquisition time of the auxiliary enhancement information and acquiring the acquisition time of each reference video frame in the frame sequence;

[0271] Determine, in the frame sequence, a target reference video frame whose time difference between the acquisition time and the acquisition time of the auxiliary enhancement information is less than a time difference threshold;

[0272] Obtaining a target position of a target reference video frame in a frame sequence;

[0273] A preceding adjacent position of the target position is determined as a position for adding the auxiliary enhancement information frame.

[0274] When the processor 1001 adds the auxiliary enhancement information to the auxiliary enhancement information frame of the live video, it may specifically perform the following operations:

[0275] The auxiliary enhancement information frame is divided into one or more macroblocks, each macroblock includes one or more data units; the data unit includes a header data field and a data unit body field; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field;

[0276] storing the auxiliary enhancement information in an auxiliary information field of the target data unit;

[0277] Calculating the length of the auxiliary enhancement information, and setting a field value of an auxiliary information length field in a target data unit based on the length of the auxiliary enhancement information;

[0278] Setting the value of the header data field of the target data unit storing the auxiliary enhancement information to a first value; and

[0279] The value of the payload type field in the target data unit is set to the second value.

[0280] In the embodiment of the present application, the video content of the live video is collected, and the live scene content of the live video is collected; the auxiliary enhancement information of the live video is generated based on the live scene content; the video content and the auxiliary enhancement information are encoded and processed to obtain a video code stream of the live video. It can be seen that by adding auxiliary enhancement information to the video code stream, the live scene content of the live video can be included in the video code stream, and then the video code stream can provide richer information in addition to the video content of the live video, which can enrich the live content presented by the live scene at the decoding end and improve the flexibility of content presentation during the live broadcast process.

[0281] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above Figure 4 and Figure 6 The method in the corresponding embodiment, therefore, will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0282] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device can perform the above Figure 4 and Figure 6 The method in the corresponding embodiment will therefore not be described in detail here.

[0283] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0284] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A live broadcast processing method, characterized in that: include: Acquire a video code stream of a live video, wherein the video code stream includes video content of the live video and auxiliary enhancement information of the live video; The auxiliary enhancement information is used to record the live scene content of the live video; Parsing the video code stream to obtain the video content and the auxiliary enhancement information; Play the video content obtained by parsing; as well as, During playback of the video content, the live scene content is presented based on the auxiliary enhancement information.

2. The method according to claim 1, characterized in that The live broadcast scene content includes at least one of the following: an operation performed by the host object of the live broadcast video in the live broadcast scene, an environment where the host object is located in the live broadcast scene, and an object state of the host object in the live broadcast scene; The auxiliary enhancement information includes at least one of the following: operation information, configuration information, environment information and status information; the operation information is used to describe the operation performed by the anchor object in the live broadcast scene; the configuration information is used to describe the configuration of the live broadcast scene; the environment information is used to describe the environment where the anchor object is located in the live broadcast scene; the status information is used to describe the object status of the anchor object in the live broadcast scene; Among them, the operation information includes at least one of the following: operation point coordinates, operation speed, and operation direction; the configuration information includes at least one of the following: key configuration information, sensitivity configuration information, and sound effect configuration information; the environmental information includes at least one of the following: the distance between the anchor object and the live broadcast device used by the anchor object, the light intensity of the environment in which the anchor object is located, the temperature of the environment in which the anchor object is located, and the humidity of the environment in which the anchor object is located; the state information includes at least one of the following: emotional state information, physical state information, and attention state information.

3. The method according to claim 1 or 2, characterized in that The auxiliary enhancement information includes operation information; and during the playback of the video content, presenting the live scene content based on the auxiliary enhancement information includes: During the playback of the video content, the operation performed by the host object of the live video in the live scene is restored and replayed based on the operation information.

4. The method according to claim 3, characterized in that The operation information includes the coordinates of the operation point, the operation speed, and the operation direction; during the playback of the video content, restoring and replaying the operation performed by the anchor object in the live broadcast scene based on the operation information includes: Outputting a first display interface, where the first display interface covers the playback interface of the video content, and the transparency of the first display interface is higher than a preset transparency threshold; Drawing and displaying the operation point based on the operation point coordinates in the first display interface; and, The operation point is controlled to move in the first display interface according to the operation speed and the operation direction, so as to restore and replay the operation performed by the anchor object in the live broadcast scene.

5. The method according to claim 1 or 2, characterized in that: The auxiliary enhancement information includes configuration information; and during the playback of the video content, presenting the live scene content based on the auxiliary enhancement information includes: During playback of the video content, the configuration of the live broadcast scene is restored based on the configuration information.

6. The method according to claim 1 or 2, characterized in that: The presenting of the live scene content based on the auxiliary enhancement information during the playback of the video content includes: During playback of the video content, outputting prompt information based on the auxiliary enhancement information; Among them, if the auxiliary enhancement information includes environmental information, the prompt information is used to prompt the environment in which the host object of the live video is located in the live scene; if the auxiliary enhancement information includes status information, the prompt information is used to prompt the object status of the host object in the live scene.

7. The method according to claim 6, characterized in that The output method of the prompt information includes any of the following: Outputting the prompt information in a second display interface, where the second display interface covers the playback interface of the video content, and the transparency of the second display interface is higher than a preset transparency threshold; Outputting the prompt information in the playback interface of the video content; Outputting the prompt information in text form; The prompt information is output in a multimedia manner.

8. The method according to claim 1 or 2, characterized in that: The auxiliary enhancement information includes environmental information; and during the playback of the video content, presenting the live scene content based on the auxiliary enhancement information includes: Performing simulation processing on the playback screen of the video content based on the environmental information to form a simulated screen, wherein the attributes of the simulated screen are consistent with the attributes of the live screen presented by the live broadcast device used by the anchor object of the live broadcast video in the live broadcast scenario; The simulation picture is output.

9. The method according to claim 1, characterized in that The video code stream includes an auxiliary enhancement information frame, the auxiliary enhancement information frame includes one or more macroblocks, each of the macroblocks includes one or more data units; the auxiliary enhancement information is encapsulated into the data unit; The data unit includes a header data field and a data unit body field; When the header data field of the data unit is a first value, it indicates that the data unit is encapsulated with auxiliary enhancement information; the body field of the data unit includes a payload type field, an auxiliary information field, and an auxiliary information length field; when the payload type field is a second value, it indicates that the encoding format of the auxiliary enhancement information complies with a preset encoding format; the auxiliary information field is used to store the auxiliary enhancement information; The auxiliary information length field is used to indicate the length of the auxiliary enhancement information; Parsing the video code stream to obtain the auxiliary enhancement information includes: Reading a header data field of the data unit; When a target data unit whose header data field is a first value is read, the auxiliary enhancement information is read from the auxiliary information field of the target data unit.

10. A live broadcast processing method, characterized in that: include: Collecting video content of a live video and collecting live scene content of the live video; Generating auxiliary enhancement information of the live video based on the live scene content; The video content and the auxiliary enhancement information are encoded to obtain a video code stream of the live video.

11. The method according to claim 10, characterized in that The method further comprises: Determine the live scene content to be collected according to the type of the live video; or, According to the collection requirements of the host object of the live video, the live scene content to be collected is determined.

12. The method according to claim 10, characterized in that The auxiliary enhancement information includes operation information, and the operation information includes operation point coordinates, operation speed and operation direction; The collecting of the live scene content of the live video includes: In a live broadcast scenario, detecting an operation event in a live broadcast device used by the anchor object; If a preset operation event is detected, calling the application programming interface of the live broadcast device to obtain the operation point coordinates corresponding to the preset operation event; and Calling an acceleration sensor and a magnetic sensor to detect an operation speed and an operation direction corresponding to the preset operation event; The preset operation event includes one or more of the following: a pressing event, a moving event, a releasing event generated after the pressing event, and a moving-out event.

13. The method according to claim 10, characterized in that The live video includes a frame sequence composed of a plurality of video frames; the encoding process of the video content and the auxiliary enhancement information to obtain a video code stream of the live video includes: Adding the auxiliary enhancement information to the auxiliary enhancement information frame of the live video; Determining an adding position of the auxiliary enhancement information frame in the frame sequence, and adding the auxiliary enhancement information frame to the frame sequence according to the adding position; The frame sequence with the auxiliary enhancement information frame added is encoded to form a video code stream.

14. The method according to claim 13, characterized in that The frame sequence includes one or more reference video frames, and the reference video frames include independent frames or forward prediction frames; The determining the adding position of the auxiliary enhancement information frame in the frame sequence includes: Acquiring the acquisition time of the auxiliary enhancement information and acquiring the acquisition time of each reference video frame in the frame sequence; Determine, in the frame sequence, a target reference video frame whose time difference between the acquisition time and the acquisition time of the auxiliary enhancement information is less than a time difference threshold; Acquire a target position of the target reference video frame in the frame sequence; A preceding adjacent position of the target position is determined as a position for adding the auxiliary enhancement information frame.

15. The method according to claim 13, characterized in that The adding the auxiliary enhancement information to the auxiliary enhancement information frame of the live video includes: Divide the auxiliary enhancement information frame into one or more macroblocks, each of the macroblocks includes one or more data units; the data unit includes a header data field and a data unit body field; the data unit body field includes a payload type field, an auxiliary information field, and an auxiliary information length field; storing the auxiliary enhancement information in an auxiliary information field of a target data unit; Calculating the length of the auxiliary enhancement information, and setting a field value of an auxiliary information length field in the target data unit based on the length of the auxiliary enhancement information; Setting the value of the header data field of the target data unit storing the auxiliary enhancement information to a first value; and The value of the payload type field in the target data unit is set to a second value.

16. A live broadcast processing device, characterized in that: include: An acquisition unit, configured to acquire a video code stream of a live video, wherein the video code stream includes video content of the live video and auxiliary enhancement information of the live video; The auxiliary enhancement information is used to record the live scene content of the live video; A processing unit, configured to parse the video code stream to obtain the video content and the auxiliary enhancement information; The processing unit is further used to play the video content obtained by parsing; And, during the playback of the video content, the live scene content is presented based on the auxiliary enhancement information.

17. A live broadcast processing device, characterized in that: include: A processing unit, used to collect video content of a live video and collect live scene content of the live video; The processing unit is further configured to generate auxiliary enhancement information of the live video based on the live scene content; The encoding unit is used to encode the video content and the auxiliary enhancement information to obtain a video code stream of the live video.

18. A computer device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the live broadcast processing method according to any one of claims 1 to 9 is executed, or the live broadcast processing method according to any one of claims 10 to 15 is executed.

19. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program. When the computer program is executed by the processor, the live broadcast processing method as described in any one of claims 1 to 9 is executed, or the live broadcast processing method as described in any one of claims 10 to 15 is executed.

20. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the live broadcast processing method as described in any one of claims 1-9, or implements the live broadcast processing method as described in any one of claims 10-15.

Images