Method and apparatus for processing live video stream, method and apparatus for playing, electronic device, medium and program product

Abstract

Embodiments of the present application provide a live video stream processing method and device, a playing method and electronic equipment, and relate to the fields of cloud technology, block chain, multimedia technology and artificial intelligence. The method comprises: obtaining a live video stream; if a time corresponding to a first video frame of the live video stream matches a triggering time of preset recommendation information, determining target recommendation information corresponding to attribute information of the first video frame from at least one item of preset recommendation information; inserting the target recommendation information converted into a preset format into a SEI (Supplemental Enhancement Information) supplementary enhancement information unit of the first video frame to form a target SEI frame, and obtaining a processed live video stream. The present application realizes simultaneous playing of the recommendation information and the live video stream, achieves zero delay between the playing content of the recommendation information and the live video stream, fully combines the attribute information of the live video stream to obtain matching target recommendation information, and makes the subsequent process of displaying the recommendation while playing the video stream more integrated and smooth.

Description

Technical Field

[0001] This application relates to the fields of cloud technology, blockchain, multimedia technology, and artificial intelligence technology. Specifically, this application relates to a method and apparatus for processing live video streams, a method and apparatus for playing live video streams, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of internet technology, live streaming has become a common way to disseminate information. In related technologies, recommendation pushes are typically implemented in two ways during live streaming: independently pushing the live stream and recommendation data, or retrieving recommendation data from the server at specific times during video playback on the playback device.

[0003] However, both of the above methods of recommendation push involve fetching recommendation data after the playback terminal receives the video stream, which causes a time delay between the video stream and the recommendation data, and the recommendation data cannot correspond to the playback content of the video stream. Summary of the Invention

[0004] This application provides a method, apparatus, method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing live video streams, which enables simultaneous playback of recommendation information and live video streams, and fully integrates the attribute information of the live video stream for pushing and displaying recommendation information.

[0005] According to one aspect of the embodiments of this application, a method for processing live video streams is provided, the method comprising:

[0006] Obtain the live video stream;

[0007] If the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, the target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information.

[0008] The target recommendation information, converted to a preset format, is inserted into the SEI (supplemental enhancement information) unit of the first video frame to form a target SEI frame, thereby obtaining the processed live video stream.

[0009] According to another aspect of the embodiments of this application, a method for playing a live video stream is provided, the method comprising:

[0010] Get the live video stream to be played;

[0011] The target SEI frame in the live video stream to be played is parsed to obtain the first video frame and target recommendation information;

[0012] If it is determined that the target recommendation information matches the attribute information of the first video frame, the target recommendation information and the first video frame are played together.

[0013] According to another aspect of the embodiments of this application, a processing apparatus for live video streams is provided. The apparatus includes an acquisition module, a target recommendation information determination module, and a target recommendation information insertion module, wherein...

[0014] The acquisition module is used to acquire live video streams;

[0015] If the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, the target recommendation information determination module is used to determine the target recommendation information corresponding to the attribute information of the first video frame from at least one preset recommendation information.

[0016] The target recommendation information insertion module is used to insert target recommendation information converted to a preset format into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, so as to obtain the processed live video stream.

[0017] Optionally, the attribute information of the first video frame includes the type information of the playback information corresponding to the live video stream and the event information of the playback scene corresponding to the first video frame;

[0018] When the target recommendation information determination module determines the target recommendation information corresponding to the attribute information of the first video frame from at least one preset recommendation information, it specifically performs any one of the following operations:

[0019] Determine the target recommendation information corresponding to the type information of the live video stream from at least one preset recommendation information;

[0020] Target recommendation information corresponding to the event information of the first video frame is determined from at least one preset recommendation information.

[0021] Optionally, the time corresponding to the first video frame matches the trigger time of the preset recommendation information, including at least one of the following:

[0022] The time corresponding to the first video frame is the same as the preset trigger time for the recommendation information;

[0023] The event information corresponding to the first video frame is the event information of any preset scene.

[0024] Optionally, the matching of the time corresponding to the first video frame with the preset trigger time of the recommendation information further includes:

[0025] The trigger time of the preset recommendation information is corrected according to the preset time offset, and the corrected time is determined.

[0026] If the corrected time is the same as the time corresponding to the first video frame, then the time corresponding to the first video frame is determined to match the trigger time of the preset recommendation information.

[0027] Optionally, when the target recommendation information insertion module inserts the target recommendation information, converted to a preset format, into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, it specifically performs the following:

[0028] The target recommendation information, converted to a preset format, and the corresponding identifier of the recommendation information are inserted into the SEI unit of the first video frame to form a target SEI frame, so that the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information.

[0029] According to another aspect of the embodiments of this application, a live video stream playback device is provided, the device including an acquisition module, a parsing module, and a playback module, wherein,

[0030] The acquisition module is used to acquire the live video stream to be played.

[0031] The parsing module is used to parse the target SEI frame in the live video stream to be played, and obtain the first video frame and target recommendation information;

[0032] If it is determined that the target recommendation information matches the attribute information of the first video frame, the playback module is used to play the target recommendation information and the first video frame together.

[0033] Optionally, when the parsing module parses the target SEI frame in the live video stream to be played, it is specifically used for:

[0034] If a video frame containing an identifier with recommendation information is detected in the live video stream, the video frame with the identifier with recommendation information is identified as the target SEI frame, and the target SEI frame is parsed.

[0035] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of any of the above methods.

[0036] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above methods.

[0037] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0038] The beneficial effects of the technical solutions provided in this application are:

[0039] In the live video stream processing method provided in this application embodiment, by acquiring the live video stream, when the time corresponding to the first video frame of the live video stream matches the trigger time of preset recommendation information, target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information. The target recommendation information is converted into a preset format and inserted into the SEI unit of the first video frame to form a target SEI frame, thereby obtaining the processed live video stream.

[0040] Since the system architecture of a video stream can be divided into a video coding layer (VCL) and a network abstraction layer (NAL), each video frame in the video stream can include both a VCL layer and a NAL layer. Within the same video frame, the time information (i.e., timestamps) corresponding to the VCL layer and the NAL layer are identical. The NAL layer includes supplementary enhancement information units, through which additional information can be added to the video stream. Therefore, using the live video stream processing method provided in this application embodiment, when the time corresponding to the first video frame of the live video stream matches the trigger time of preset recommendation information, target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information, and this target recommendation information is converted into a preset format. The target recommendation information converted to the preset format is inserted into the SEI unit of the first video frame to form a target SEI frame. This enables synchronous playback of the first video frame and the target recommendation information during live streaming, achieving zero latency between the target recommendation information and the playback content of the live video stream.

[0041] Furthermore, since the target recommendation information corresponds to the attribute information of the first video frame, the live video stream processing method provided in this application embodiment can also fully combine the attribute information of the video frame to obtain matching target recommendation information. Because the target recommendation information corresponds to the attribute information of the first video frame, the subsequent process of displaying recommendation information while playing the video stream is more integrated and smooth, providing a prerequisite guarantee for the efficient push and display of recommendation information. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0043] Figure 1 An architecture diagram of the live video stream processing system provided in an embodiment of this application is shown;

[0044] Figure 2 A flowchart of a live video stream processing system provided in an embodiment of this application is shown;

[0045] Figure 3 A flowchart illustrating a method for processing a live video stream according to an embodiment of this application is shown;

[0046] Figure 4 A schematic diagram illustrating the setting method of at least one piece of recommendation information in an embodiment of this application is shown;

[0047] Figure 5 This illustration shows a schematic diagram of an SEI unit corresponding to a live video stream in H.264 format in an embodiment of this application;

[0048] Figure 6 A framework diagram of a method for processing live video streams provided in an embodiment of this application is shown;

[0049] Figure 7 A flowchart illustrating the method for processing live video streams in an embodiment of this application is shown;

[0050] Figure 8 A flowchart illustrating a live video stream playback method provided in an embodiment of this application is shown;

[0051] Figure 9 This illustration shows a schematic diagram of the display area for target recommendation information in an embodiment of this application;

[0052] Figure 10 A flowchart illustrating a method for playing a live video stream according to an embodiment of this application is shown;

[0053] Figure 11 A schematic diagram of a live video stream processing apparatus provided in an embodiment of this application is shown;

[0054] Figure 12 This illustration shows a schematic diagram of a live video stream playback device provided in an embodiment of this application;

[0055] Figure 13 A schematic diagram of the structure of an electronic device to which this application embodiment applies is shown. Detailed Implementation

[0056] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0057] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] First, let's introduce and explain several terms used in this application:

[0060] Live video streaming: Utilizing the internet and streaming media technologies for live broadcasting, video, by integrating rich elements such as images, text, and sound, offers a vivid and engaging experience, gradually becoming the mainstream expression method for live broadcasting in the internet technology field. Live video streaming creates a strong sense of presence through authentic and vivid communication, attracting attention and achieving a memorable and lasting impact, enabling truthful, intuitive, and comprehensive promotion.

[0061] SEI (Supplemental Enhancement Information) falls under the category of bitstream and provides a method for adding extra information to the video bitstream. It is a feature of video compression standards such as H.264 and H.265. The basic characteristics of SEI are as follows: it is not a mandatory option in the decoding process; it may be helpful in the decoding process (error tolerance and correction); and it is integrated into the video bitstream. Inserting SEI information into the video stream allows this inserted information to travel along with other video content through the transmission link to the consumer, achieving a real-time synchronized playback effect with the video stream.

[0062] WebView: An embedded browser within a native application. Within a WebView, JavaScript code (JSON, JavaScriptObject Notation) can communicate with the native application code and can also call native APIs (Application Programming Interfaces) to integrate system-level functions. As a built-in browser within the app, the WebView can display not only linked content but also recommended information.

[0063] CDN (Content Delivery Network): An intelligent virtual network built on top of existing networks. Relying on edge servers deployed in various locations, it uses a central platform's load balancing, content distribution, and scheduling modules to allow users to access the content they need from the nearest location, reducing network congestion and improving user access response speed and hit rate. The key technologies of CDN mainly include content storage and distribution technologies.

[0064] SDK (Software Development Kit): A collection of development tools used to create application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

[0065] NAL: In the H.264 / AVC (Advanced Video Coding) video coding standard, the entire system framework is divided into two layers: the video coding layer and the network abstraction layer. The former is responsible for effectively representing the content of the video data, while the latter is responsible for formatting the data and providing header information to ensure that the data is suitable for transmission over various channels and storage media. One frame of data represents one NAL unit.

[0066] In related technologies, recommendation data is retrieved only after the live video stream is received, resulting in a time delay between the live video stream and the recommendation data, and the recommendation data cannot be directly correlated with the content of the live video stream. To address this, this application provides a method, apparatus, method, apparatus, electronic device, computer-readable storage medium, and computer program product for processing live video streams. This enables simultaneous playback of recommendation information and live video streams, fully integrating the attribute information of the live video stream to obtain matching target recommendation information. Because the target recommendation information corresponds to the attribute information of the first video frame, the subsequent display of recommendation information while the video stream is playing is more seamless and smooth, providing a prerequisite guarantee for the efficient push and display of recommendation information.

[0067] The live video stream processing method provided in this application embodiment can be implemented based on cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the Internet industry, in the future, every item may have its own identification mark, which will need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately, and various industry data will all require strong system support, which can only be achieved through cloud computing.

[0068] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources through the network in an on-demand and easily scalable manner. In a broader sense, cloud computing refers to the delivery and usage model of services, meaning obtaining necessary services through the network in an on-demand and easily scalable manner. These services can be IT and software, internet-related, or other services. Driven by the development of the internet, real-time data streams, the diversification of connected devices, and the demands of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Unlike previous parallel distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.

[0069] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0070] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0071] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0072] The live video stream processing method provided in this application embodiment can also be implemented based on blockchain technology. Specifically, the data used in processing the live video stream, such as the preset trigger time of recommendation information, at least one preset recommendation information, event information of preset scenarios, and identifiers of recommendation information, can be stored on the blockchain.

[0073] The live video stream processing method provided in this application embodiment can be implemented based on Artificial Intelligence (AI) technology. For example, the attribute information of the first video frame can be determined by calling a trained neural network model. AI is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain optimal results. With the research and progress of artificial intelligence technology, it has been widely studied and applied in many fields. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0074] The trained neural network model mentioned above can be a neural network model based on Computer Vision (CV) technology. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0075] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0076] To facilitate understanding of the application value of the live video stream processing method and the live video stream playback method provided in the embodiments of this application, the following description is based on a specific application scenario. Figure 1 An architecture diagram of the live video stream processing system provided in an embodiment of this application is shown. Figure 2 A flowchart of a live video stream processing system provided in an embodiment of this application is shown.

[0077] The live video stream processing system provided in this application embodiment can be applied to any scenario requiring live streaming, including but not limited to game live streaming and shopping live streaming. To better illustrate the practicality of the solution in this application embodiment, the solution will be described below using a game live streaming scenario as an example.

[0078] like Figure 1As shown, the live video stream processing system can include a console, a director's console, an origin server, a live streaming platform, a CDN, and a playback client. The origin server can be implemented through a server, specifically any local origin server or any cloud origin server; this application does not impose any restrictions on this. The playback client can be any terminal device, an application on the terminal device, or a mini-program within the application. The playback client can also be categorized based on the operating mechanism of the terminal device, such as web playback clients and device playback clients. Device playback clients include, but are not limited to, Android playback clients and iOS (an operating mechanism) playback clients. Specifically, in conjunction with... Figure 2 This describes the processing procedure of each specific device in the live video stream processing system:

[0079] Step S10: Configure at least one preset recommendation message and the mapping relationship between the preset recommendation message and the live video stream through the console (also known as the recommendation information console), and send the configured preset recommendation message to the origin server. Store the mapping relationship between the preset recommendation message and the live video stream in the relationship module of the SDK.

[0080] Step S20: Collect live data in the game thread through the broadcast control console, convert the collected live data into a live video stream, and push the live video stream to the source server.

[0081] Step S30: Receive the live video stream sent by the broadcast control station from the source station, and receive the recommendation information sent by the control console. Call the relationship module in the SDK to obtain the correspondence between at least one preset recommendation information and the live video stream. Based on the correspondence between the at least one preset recommendation information and the live video stream, determine the target recommendation information corresponding to the live video stream from the at least one preset recommendation information, and insert the target recommendation information into the SEI unit of the first video frame of the live video stream to form a target SEI frame, thus obtaining the processed live video stream with the inserted target SEI frame.

[0082] Step S40: Pull the processed live video stream with the inserted target SEI frame from the origin server through the live streaming platform, and push the processed live video stream with the inserted target SEI frame to the CDN provider.

[0083] Step S50: The CDN provider stores and distributes the processed live video stream after inserting the target SEI frame.

[0084] Step S60: The playback terminal plays the live video stream after receiving the processed live video stream with the inserted target SEI frame sent by the CDN provider. Video frames in the live video stream that are not target SEI frames can be played directly.

[0085] If it is determined that the live video stream includes the target SEI frame, the parsing module in the SDK is called to parse the processed live video stream after the insertion of the target SEI frame, so as to obtain the first video frame and target recommendation information corresponding to the target SEI frame.

[0086] After obtaining the first video frame and target recommendation information corresponding to the target SEI frame on the playback end, the rendering module in the SDK can be called again. As the first video frame is played, the target recommendation information is rendered onto the specified area of ​​the playback page corresponding to the first video frame, so that the target recommendation information can be played synchronously with the first video frame.

[0087] Optionally, the specific execution operations of each device in the above system can be stored in the SDK, so that each device can perform its corresponding operation by calling the SDK. For example, after configuring at least one preset recommendation information and the correspondence between at least one preset recommendation information and the live video stream through the console, the correspondence can be stored in the SDK, so that the origin server can obtain at least one preset recommendation information and the correspondence between at least one preset recommendation information and the live video stream by calling the SDK.

[0088] Figure 3 A flowchart illustrating a method for processing a live video stream according to an embodiment of this application is shown. The execution entity of this live video stream processing method can be a live video stream processing device. Optionally, the live video stream processing device can include, but is not limited to, a terminal device or a server; optionally, the server can be a cloud server. The terminal device can be a video playback device, video recommendation device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, or wearable device, etc. This method can be implemented as an application or as a plugin or functional module of an existing application with video processing capabilities. For example, it can be a new functional module of a video application that has live video streaming functionality. Of course, the live streaming application can also have other functions, such as shopping, music, social networking, and gaming functions. The live video stream processing method can be implemented by a processor calling computer-readable instructions stored in memory.

[0089] Optionally, this method can be executed by a user terminal, such as, but not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, wearable electronic devices, AR / VR devices, etc. For example, this method can be applied to any cloud origin station to receive a live video stream sent by the broadcast control station and at least one piece of recommendation information sent by the control station, and insert the target recommendation information into the live video stream to obtain a processed live video stream with the inserted recommendation information.

[0090] like Figure 3 As shown, the method includes steps S110 to S130.

[0091] Step S110: Obtain the live video stream.

[0092] Live video streams can be obtained by converting the format of video captured by any image acquisition device. This image acquisition device can be any image acquisition device on the broadcast control panel. When any video is captured by the image acquisition device, it can be converted into formats such as FLV (flash video, streaming media format), AVI (audio video interleave), ASF (advanced stream format), H.264, and H.265 to form a live video stream. This live video stream is then pushed to the execution body of the live video stream processing method, or the execution body of the live video stream processing method pulls the live video stream from the corresponding image acquisition device. The video format conversion can be performed by the image acquisition device itself, or by the broadcast control panel corresponding to the image acquisition device. The live video stream format can be FLV, AVI, ASF, etc.

[0093] The live video stream can be for any live streaming scenario, and this application embodiment does not impose any restrictions on it. For example, the live video stream can be for games, or for shopping, music, etc.

[0094] Step S120: If the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, determine the target recommendation information corresponding to the attribute information of the first video frame from at least one preset recommendation information.

[0095] The time corresponding to the first video frame can be the time of the first video frame relative to the live video stream, that is, the display / playback timestamp corresponding to the first video frame. This application embodiment does not limit the method for determining the time corresponding to the first video frame. Specifically, the time corresponding to the first video frame can be determined by the acquisition time corresponding to the first video frame during the video acquisition process. Alternatively, when the execution body of the live video stream processing method receives the live video stream, the reception time of the first video frame can be determined based on the system clock of the execution body, and the reception time of the first video frame can be determined as the time corresponding to the first video frame.

[0096] The preset trigger time for recommendation information, i.e., the time when the target recommendation information is inserted into the live video stream, can be determined according to the actual situation, and this application does not impose any restrictions on it. It is understood that multiple preset trigger times for recommendation information can be set, and different target recommendation information can be set for different trigger times.

[0097] Matching the time corresponding to the first video frame with the preset trigger time of the recommendation information can include, but is not limited to, the time corresponding to the first video frame being the same as the preset trigger time of the recommendation information.

[0098] The attribute information of the first video frame may include, but is not limited to, the time corresponding to the first video frame, the specific content of the playback information corresponding to the first video frame, and the storage space occupied by the first video frame. This application does not impose any restrictions on this.

[0099] The specific form of the recommendation information includes, but is not limited to, images, links, animations, and audio, and this application does not impose any restrictions on this. In the embodiments of this application, the specific content corresponding to the target recommendation information is preferably content corresponding to the attribute information of the first video frame. Of course, the specific content corresponding to the target recommendation information can also be any content unrelated to the attribute information of the first video frame.

[0100] Figure 4 A schematic diagram illustrating the setting method of at least one piece of recommendation information in an embodiment of this application is shown. For example... Figure 4 As shown, the console allows control over when recommendations are triggered and their attributes can be configured. The recommendation triggering interface can include attributes for each recommendation, such as the recommendation number (…). Figure 4 The titles of the recommended information ("Recommended Information 1", "Recommended Information 2", "Recommended Information 3") and the recommended information titles ( Figure 4 The "Title 1", "Title 2", and "Title 3" in the title, and the images in the recommended information ( Figure 4 The "Recommended Information 1 Image", "Recommended Information 2 Image", and "Recommended Information 3 Image" in the text, and the push trigger control for recommended information (( Figure 4The "Push 1", "Push 2", and "Push 3" options allow users to push recommended information to the playback device by clicking the corresponding push trigger control on the recommendation information trigger interface. The recommendation information configuration interface can include currently configured recommendation information (such as...). Figure 4 The example shown (recommendation information number 16) includes the title, number, link, and image of the recommendation information. It also includes an "Upload" control for the image, and "Add," "Save," and "Modify" controls for the currently configured recommendation information. Clicking the "Upload" control allows you to upload an image related to the currently configured recommendation information. Clicking the "Save" control saves the configured content of the currently configured recommendation information. Clicking the "Modify" control allows you to modify the configured content of the currently configured recommendation information. Clicking the "Add" control switches to the configuration interface for other recommendation information.

[0101] A pre-defined correspondence between the attribute information of the first video frame and the target recommendation information can be established and stored in the execution body of the live video stream processing method. When the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, the target recommendation information corresponding to the attribute information of the first video frame can be determined from at least one preset recommendation information based on the correspondence.

[0102] Step S130: The target recommendation information converted to a preset format is inserted into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, so as to obtain the processed live video stream.

[0103] As described above, since the specific forms of target recommendation information include images, links, animations, audio, etc., and the format of the live video stream can be FLV, AVI, ASF, H.264, H.265, etc., the two formats are different. The target recommendation information needs to be converted into a preset format that is the same as the format of the live video stream in order to insert the target recommendation information converted into the preset format into the first video frame of the live video stream.

[0104] Alternatively, both the target recommendation information and the live video stream can be converted to the same target format. For example, both the target recommendation information and the first video frame can be converted to JSON format. The target recommendation information converted to the preset format is then inserted into the SEI unit of the first video frame to form a target SEI frame, resulting in a processed live video stream. This allows the processed live video stream to be parsed according to the target format to obtain the target recommendation information and the first video frame, enabling simultaneous playback of the target recommendation information and the first video frame.

[0105] In the live video stream processing method provided in this application embodiment, by acquiring the live video stream, when the time corresponding to the first video frame of the live video stream matches the trigger time of preset recommendation information, target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information. The target recommendation information is converted into a preset format and inserted into the SEI unit of the first video frame to form a target SEI frame, thereby obtaining the processed live video stream.

[0106] Since the system architecture of a video stream can be divided into a video coding layer and a network abstraction layer, the time information corresponding to the VCL layer and the NAL layer is the same for each video frame in the video stream. The NAL layer includes an SEI unit, through which additional information can be added to the video stream. Therefore, using the live video stream processing method provided in this application embodiment, when the time corresponding to the first video frame of the live video stream matches the trigger time of preset recommendation information, target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information, and this target recommendation information is converted into a preset format. The target recommendation information converted to the preset format is inserted into the SEI unit of the first video frame to form a target SEI frame. This enables synchronous playback of the first video frame and the target recommendation information during live streaming, achieving zero latency between the target recommendation information and the playback content of the live video stream.

[0107] Furthermore, since the target recommendation information corresponds to the attribute information of the first video frame, the live video stream processing method provided in this application embodiment can also fully combine the attribute information of the video frame to obtain matching target recommendation information. Because the target recommendation information corresponds to the attribute information of the first video frame, the subsequent process of displaying recommendation information while playing the video stream is more integrated and smooth, providing a prerequisite guarantee for the efficient push and display of recommendation information.

[0108] Optionally, the attribute information of the first video frame includes the type information of the playback information corresponding to the live video stream and the event information of the playback scene corresponding to the first video frame;

[0109] Target recommendation information corresponding to the attribute information of the first video frame is determined from at least one preset recommendation information, including any one of the following:

[0110] Determine the target recommendation information corresponding to the type information of the live video stream from at least one preset recommendation information;

[0111] Target recommendation information corresponding to the event information of the first video frame is determined from at least one preset recommendation information.

[0112] The type of playback information corresponding to the live video stream can be determined based on the specific content of the playback information, i.e., the content displayed on the live stream interface during the live stream. This application does not impose any limitations on this. For example, when the live video stream is used for shopping recommendations, and the recommended product is a down jacket, then the type of playback information corresponding to the live video stream can be "down jacket".

[0113] When a live video stream is used for game streaming, the type information of the playback information can be determined based on the game type. For example, the type information ("game_type") of the playback information for a game live stream can include action games, fighting games, strategy games, role-playing games, sports games, real-time combat games, etc. Taking a live video stream used for streaming a real-time combat game as an example, the type of the playback information for this live video stream is real-time combat game.

[0114] Furthermore, the playback information type can be specifically determined for each type of game. For example, for real-time battle games, it can include "Game 1", "Game 2", "Game 3", etc. Taking a live video stream used for "Game 1" as an example, the playback information type corresponding to this live video stream is "Game 1".

[0115] In this implementation, different type tags can be set for the type information of different live video streams. A type tag is used to uniquely identify the type information of the live video stream corresponding to that type tag. This application embodiment does not limit the specific form of the type tag and can configure it according to actual application needs. Optionally, taking a live video stream as an example, the type tag corresponding to the live video stream can be the name, numeric identifier, letter identifier, etc., of the type information of the live video stream. For example, in the above example, real-time battle games can include "Game 1", "Game 2", "Game 3", etc., the type tag corresponding to "Game 1" can be set to "game_type=1", the type tag corresponding to "Game 2" to "game_type=2", and the type tag corresponding to "Game 3" to "game_type=3".

[0116] Optionally, different labels (“ads_type”) can be set for each of the at least one preset recommendation information. Taking a recommendation information as an example, the label corresponding to the recommendation information can be the title, numerical identifier, letter identifier, etc., where the numerical identifier can be the image number when the recommendation information is an image. “ads_type = 0” can be set to indicate the end of the previous event notification, and “ads_type = n-1” can be set to indicate a specific event. Here, n represents the number of specific recommendation information items. The value of n can be determined according to the actual number of recommendation information items set; this application does not impose any restrictions on this.

[0117] Based on a pre-defined first correspondence between the type information of the live video stream and the recommended information, after determining the type information of the live video stream, the target recommended information corresponding to the type information of the live video stream can be determined from at least one pre-defined set of recommended information according to the pre-defined first correspondence. For example, taking the example of real-time battle games including "Game 1", "Game 2", "Game 3", etc., as mentioned above, "ads_type = 0" can specifically represent the end of the current game, or specifically represent the recommended information when entering the game state from the pre-game introduction (relative to the state of "in-game" being "idle time"). "ads_type = 1" can specifically represent the recommended information during idle time, such as the recommended information when a game character is "temporarily dead". "ads_type = 2" can specifically represent the recommended information corresponding to "Game 1". "ads_type = 3" can specifically represent the recommended information corresponding to "Game 2". "ads_type = 4" can specifically represent the recommended information corresponding to "Game 3".

[0118] The event information (i.e., "event") of the first video frame can be determined based on the specific content of the playback scene corresponding to the first video frame, that is, the content displayed on the live broadcast interface of the first video frame during the live broadcast. This application does not limit this. Taking the above live video stream used to broadcast the game "Game 1" as an example, the event information of the playback information corresponding to the first video frame may include: "event=1010001" indicates that "the game starts and enters the BP (ban and pick, i.e. "draft") screen", "event=1010007" indicates that "Game 1 officially starts and all game characters start from the fountain", "event=1020003" indicates that "the game character controlled by the streamer of Game 1 has a 5-kill", etc.

[0119] Based on a preset second correspondence between the event information and recommendation information of the first video frame, after determining the event information of the playback information corresponding to the first video frame, the target recommendation information corresponding to the event information of the first video frame can be determined from at least one preset recommendation information according to the preset second correspondence. For example, taking the above live video stream used to stream the game "Game One" as an example, "ads_type=0" can be set to specifically represent the end of the current game, or specifically represent the recommendation information when entering the game state from the pre-game introduction (relative to the state of "idle time" in the game). "ads_type=1" can specifically represent idle time recommendation information, such as the recommendation information when a game character "temporarily dies". "ads_type=p" can specifically represent the recommendation information corresponding to "entering the BP screen at the beginning of the game", corresponding to the above "event=1010001". "ads_type=q" can specifically represent the recommendation information corresponding to "Game One officially starts, all game characters start from the fountain", corresponding to the above "event=1010007". "ads_type=w" can specifically represent the recommendation information corresponding to "the game character controlled by the streamer in Game 1 has a 5-kill streak", which corresponds to "event=1020003" above.

[0120] Based on the aforementioned preset first and second correspondences, after determining the type information of the live video stream and the event information of the playback information corresponding to the first video frame, target recommendation information corresponding to the event information of the playback information corresponding to the first video frame and the type information of the live video stream can be determined from at least one preset recommendation information. That is, as mentioned above, "ads_type=p" can correspond to "game_type=1" and "event=1010001", representing recommendation information such as "the game starts and enters the BP screen".

[0121] By determining the corresponding target recommendation information based on the type information of the live video stream, determining the corresponding target recommendation information based on the event information of the playback information corresponding to the first video frame, and determining the corresponding target recommendation information based on the type information of the live video stream and the event information of the playback information corresponding to the first video frame, the target recommendation information can be efficiently pushed and displayed by fully combining the video scene corresponding to the live video stream.

[0122] Optionally, the time corresponding to the first video frame matches the trigger time of the preset recommendation information, including at least one of the following:

[0123] The time corresponding to the first video frame is the same as the preset trigger time of the recommendation information;

[0124] The event information corresponding to the first video frame is the event information of any preset scene.

[0125] As described above, the event information corresponding to the first video frame is the event information of the playback scene corresponding to that first video frame. Preset scenes can include, but are not limited to, scenes with specific meaning in a live video stream. For example, in a live video stream for shopping, the preset scene could be a scene displaying the outline of the target product, a scene displaying the price tag of the target product, etc. In a live video stream for gaming, the preset scene could be a scene corresponding to a "highlight moment," such as "the streamer-controlled character in Game 1 achieves a 5-kill streak," or "the streamer-controlled character in Game 2 is performing a target action," etc.

[0126] By determining that the time corresponding to the first video frame matches the trigger time of the preset recommendation information when the event information corresponding to the first video frame is any preset scene event information, the trigger time of the recommendation information can be effectively set in the live video stream by combining the scene information in the live video stream. This allows the recommendation information to be played during the playback of the live video stream by combining the scene information in the live video stream, thus achieving a good triggering effect for the recommendation information.

[0127] It is understandable that when the event information corresponding to the first video frame is the event information of any preset scene, the time corresponding to the first video frame can be the trigger time of the preset recommendation information.

[0128] By setting the trigger time of recommendation information in the live video stream, the system can effectively set the trigger time of recommendation information. This is achieved by considering the trigger time of the first video frame as matching the trigger time of the preset recommendation information when the time corresponding to the first video frame is the same as the trigger time of the preset recommendation information; by considering the event information corresponding to the first video frame as the event information of any preset scene; and by considering the trigger time of the first video frame as the same as the trigger time of the preset recommendation information and the event information of any preset scene.

[0129] Optionally, the matching of the time corresponding to the first video frame with the preset trigger time of the recommendation information further includes:

[0130] The trigger time of the preset recommendation information is corrected according to the preset time offset, and the corrected time is determined.

[0131] If the corrected time is the same as the time corresponding to the first video frame, then the time corresponding to the first video frame is determined to match the trigger time of the preset recommendation information.

[0132] In this implementation, the preset time offset can be determined based on actual conditions (experimental or empirical values), and this application does not impose any restrictions on this. For example, the preset time offset can be determined based on the lifespan of the execution entity of the live video stream processing method, its processing efficiency, and the transmission environment of the live video stream. It is understood that when the lifespan of the execution entity of the live video stream processing method is short, the processing efficiency is high, and the transmission environment of the live video stream is free of noise, a smaller preset time offset can be set.

[0133] The relationship between the time corresponding to the first video frame and the preset trigger time of the recommendation information can be expressed as: Preset trigger time of the recommendation information = Time corresponding to the first video frame - Preset time offset. The preset time offset can be positive or negative. When the preset time offset is positive, it means the preset trigger time of the recommendation information is less than the time corresponding to the first video frame; correspondingly, during actual video playback, the first video frame is played first, followed by the target recommendation information. When the preset time offset is negative, it means the preset trigger time of the recommendation information is greater than the time corresponding to the first video frame; correspondingly, during actual video playback, the target recommendation information is played first, followed by the first video frame.

[0134] Therefore, in order to ensure that the target recommendation information can be played synchronously with the first video frame, the trigger time of the preset recommendation information can be corrected according to a preset time offset. The corrected time is then determined as: Corrected time = Preset trigger time of recommendation information + Preset time offset. This ensures that the corrected time is the same as the preset trigger time of the recommendation information. This allows for synchronized playback of the target recommendation information with the first video frame during the live video stream.

[0135] By correcting the preset trigger time of the recommendation information according to the preset time offset, the corrected time is determined so that it is the same as the time corresponding to the first video frame. This can effectively set the trigger time of the recommendation information in the live video stream, so that the playback of the recommendation information can be triggered in combination with scene information in the live video stream during the playback of the live video stream, so as to achieve synchronous playback of the target recommendation information and the first video frame and achieve a good triggering effect of the recommendation information.

[0136] As an example, when using a live video stream for game streaming, the following JSON-formatted data protocol can be used to set the trigger time for preset recommendation information and at least one preset recommendation message within the live video stream. The key content of the specific data protocol can be:

[0137]

[0138] In this data protocol, the field

"version": 2

"ads": {……}

"id": 12345

"video_pts(Presentation Time Stamp): 10124180

"pts_offset": 1200

"ads_type": 1

"game_type": 2

"event": 1010002

[0139] Specifically, the content of each field in the above data protocol can be set through the following Table 1:

[0140] Table 1

[0141]

[0142]

[0143]

[0144] Optionally, the target recommendation information, converted to a preset format, is inserted into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, including:

[0145] The target recommendation information, converted to a preset format, and the corresponding identifier of the recommendation information are inserted into the SEI unit of the first video frame to form a target SEI frame, so that the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information.

[0146] As described above, the specific format of the target recommendation information differs from that of the live video stream. When inserting the target recommendation information, converted to a preset format, into the SEI unit of the first video frame, both the target recommendation information and the live video stream can be converted into data in the target format. Therefore, during the conversion of the target recommendation information and the live video stream to the target format, an identifier for the recommendation information corresponding to the target recommendation information can be set, and the identifier for the recommendation information can also be converted to the target format. Thus, the target recommendation information converted to the preset format and its corresponding identifier can be inserted into the SEI unit of the first video frame. The identifier for the recommendation information can be set to a fixed identifier, i.e., the same identifier can be set for all recommendation information. Alternatively, identifiers corresponding to different recommendation information can be set; this application does not limit this. Different UUIDs (universal unique identifiers) can be used to represent the identifiers of the recommendation information. Specifically, the identifiers for specific recommendation information can be set according to the setting standard in Annex A of ISO / IEC 11578:1996, and the set recommendation identifier can be represented as uuid_iso_iec_11578 ("the name of the identifier for the recommendation information").

[0147] Optionally, since the target recommendation information is in plaintext, it may be tampered with during processing. Therefore, when inserting the target recommendation information into the SEI unit of the first video frame, an encryption algorithm can be used to encrypt the target recommendation information, add an identifier bit corresponding to the encryption method to the encrypted target recommendation information, and insert the encrypted target recommendation information into the SEI unit of the first video frame to form a target SEI frame. The corresponding decoding method is selected according to the identifier bit to complete the parsing of the target recommendation information in the target SEI frame. In this embodiment, the specific settings of the encryption algorithm and the corresponding identifier bit are not limited. Optionally, the correspondence between the encryption algorithm and the corresponding identifier bit may include: identifier bit 0 indicates no encryption; identifier bit 1 indicates the encryption algorithm is TEA (Tiny Encryption Algorithm); identifier bit 2 indicates the encryption algorithm is DES (Data Encryption Standard); identifier bit 3 indicates the encryption algorithm is AES (Advanced Encryption Standard).

[0148] It is understandable that if the identifier of the recommended information in the target SEI frame is not the preset identifier of the recommended information, or if the flag bit in the target SEI frame corresponding to the encryption method is not the preset flag bit, the target SEI frame may not be processed.

[0149] Figure 5 This illustration shows a schematic diagram of an SEI unit corresponding to a live video stream in H.264 format, as shown in an embodiment of this application. Figure 5 As shown, field type(5) represents the type of SEI unit, i.e., SEI payload = 5, indicating that unregistered user data is used in the NAL layer. The field size indicates the size of the SEI unit; when SEI payload = 5, size should be greater than 16, and the specific value can be determined according to the actual situation. The field uuid indicates the identifier of the corresponding recommendation information. Figure 5 In this context, `uuid_iso_iec_11578` (“AdsCtrlSEIV00002”) is used to indicate the identifier corresponding to the recommendation information for “Game 3”. `enctype` (which can also be represented as `method`) is used to indicate the flag bit corresponding to the encryption method, and can be 0, 1, 2, 3, etc. `ads_ctrl_data` is used to indicate the actual data information of the recommendation. `end` (0x80) is used to indicate the end byte of this SEI unit.

[0150] When setting ads_ctrl_data, if 0x000000, 0x000001, 0x000002, or 0x000003 appear in the SEI cell, 0x03 needs to be inserted to prevent contention. Details are as follows:

[0151] 0x00 00 00<---->0x00 00 03 00

[0152] 0x00 00 01<---->0x00 00 03 01

[0153] 0x00 00 02<---->0x00 00 03 02

[0154] 0x00 00 03<---->0x00 00 03 03

[0155] Correspondingly, when parsing the target SEI frame, if the sequence 0x000003 is detected, 0x03 can be discarded, and the remaining data can be identified as the real data corresponding to the SEI frame.

[0156] It should be noted that if the live video stream is in H.265 format, each SEI unit can be represented as SEI NALUUNIT. In the SEI unit, type(39) or type(40) needs to be set, and the other fields except type(5) in an SEI unit corresponding to the live video stream in H.264 format should be set in the same way as described above.

[0157] By configuring the SEI unit as described above, depending on whether "0x03" is inserted into the SEI unit, the target recommendation information converted to a preset format and the corresponding identifier of the recommendation information can be inserted into the SEI unit of the first video frame to form a target SEI frame called SEI_RBSP ("specific information of the first video frame") or SEI_EBSP ("specific information of the first video frame"). Here, RBSP stands for "Raw Byte Sequence Payload" and RBSP stands for "Encapsulated Byte Sequence Payload". Specifically, if "0x03" is not inserted into the SEI unit, the target SEI frame can be called SEI_RBSP ("specific information of the first video frame"). If "0x03" is inserted into the SEI unit, the target SEI frame can be called SEI_EBSP ("specific information of the first video frame").

[0158] Based on the above, by converting the target recommendation information into a preset format and inserting the corresponding recommendation information identifier into the SEI unit of the first video frame to form a target SEI frame, the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information, thereby realizing the parsing of the target SEI frame, determining the target recommendation information and the first video frame corresponding to the target SEI frame, and realizing the synchronous playback of the target recommendation information and the first video frame.

[0159] To facilitate a clearer understanding of the application value of the live video stream processing method provided in this application embodiment, a specific application scenario embodiment is described below. The solution provided in this application can be applied to application scenarios where recommendation information is inserted into live video streams. This solution can be implemented as a terminal device, an application on the terminal device for video processing, a plugin for the application, or a server.

[0160] Figure 6 This diagram illustrates a framework of a method for processing live video streams provided in an embodiment of this application. Figure 6 As shown, the processing method for this live video stream can be implemented through the origin server. Figure 7A flowchart illustrating a method for processing live video streams according to an embodiment of this application is shown. Figure 6 and Figure 7 As shown, the processing method for this live video stream can specifically include the following steps.

[0161] Step S11: Start (i.e., the origin server obtains the live video stream sent by the broadcast control station).

[0162] Step S12: Receive the configured recommendation data (i.e., at least one recommendation) and trigger time (i.e., the preset trigger time of the recommendation) sent by the console.

[0163] Step S13: Encapsulate the target recommendation data (i.e., target recommendation information) into JSON data format.

[0164] Step S13 may include: determining the recommended data corresponding to the live video stream based on the recommended data and trigger time configured in the console, and encapsulating the target recommended data into JSON data format.

[0165] Step S14: Insert the target recommendation data into the live video stream, i.e., step S130 above (insert the target recommendation information converted to a preset format into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, so as to obtain the processed live video stream).

[0166] Specifically, the method for inserting recommendation data into the live video stream can be determined based on the format of the live video stream, as shown in steps S131 and S132 below. It should be understood that steps S141 and S142 are optional steps.

[0167] Step S141: When the format of the live video stream is H.264, the SEI payload is 5 when inserting the recommended data in the first video frame.

[0168] Step S142: When the format of the live video stream is H.265, the corresponding SEI NALU type = 39 when inserting the recommendation data in the first video frame.

[0169] Step S15: After inserting the recommendation data into the live video stream, the live video stream with the inserted recommendation data can be pushed to the CDN so that the CDN stores the live video stream with the inserted recommendation data and distributes the live video stream with the inserted recommendation data.

[0170] Step S16: End the above live video stream processing operation.

[0171] This application also provides a method for playing live video streams. Figure 8A flowchart illustrating a live video stream playback method provided in this application embodiment is shown. The execution entity of this live video stream playback method can be a live video stream playback device. Optionally, the live video stream playback device can include, but is not limited to, a terminal device or a server; optionally, the server can be a cloud server. The terminal device can be a video playback device, video recommendation device, user device, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant, handheld device, computing device, or wearable device, etc. This method can be implemented as an application or as a plugin or functional module of an existing application with video playback functionality. For example, it can be a new functional module of a video application with live video streaming capabilities. Of course, the live streaming application can also have other functions, such as shopping, music, social networking, and gaming functions. The live video stream playback method can be implemented by a processor calling computer-readable instructions stored in memory.

[0172] Optionally, the method can be executed by a user terminal, such as a mobile phone, computer, smart voice interaction device, smart home appliance, vehicle terminal, wearable electronic device, AR / VR device, etc.

[0173] For the same content involved in the live video stream playback method provided in this application embodiment and the live video stream processing method described above, please refer to the corresponding description above, and it will not be repeated here.

[0174] like Figure 8 As shown, the method includes steps S210 to S220.

[0175] Step S210: Obtain the live video stream to be played.

[0176] Step S220: Analyze the target SEI frame in the live video stream to be played to obtain the first video frame and target recommendation information.

[0177] Step S230: If it is determined that the target recommendation information matches the attribute information of the first video frame, play the target recommendation information and the first video frame together.

[0178] After parsing the target SEI frame in the live video stream to be played, and obtaining the first video frame and target recommendation information, the system can determine whether the target recommendation information matches the attribute information of the first video frame by invoking a pre-set correspondence between the attribute information of the first video frame and the target recommendation information. If the target recommendation information matches the attribute information of the first video frame, the target recommendation information and the first video frame are played together. If the target recommendation information does not match the attribute information of the first video frame, only the first video frame is played.

[0179] As an example, if the tag of the target recommendation information is "ads_type=w", the type information of the live video stream is "game_type=1", and the event information of the first video frame is "event=1020003", according to the above description, it can be determined that the attribute information of the target recommendation information matches the attribute information of the first video frame, and the target recommendation information and the first video frame can be played together.

[0180] Optionally, when it is determined that the target recommendation information matches the attribute information of the first video frame, the target recommendation information and the first video frame can be rendered onto the playback interface using a rendering method stored in a preset SDK. This playback interface can be a terminal display, webview, etc., and this application does not impose any limitations on it.

[0181] In order not to affect the user's ability to obtain the event information corresponding to the first video frame based on the video frame, the display area of ​​the event information corresponding to the first video frame in the video frame can be defined as the first area, and the other areas in the video frame corresponding to the first video frame other than the first area can be defined as the display area of ​​the target recommendation information.

[0182] Figure 9 A schematic diagram of the display area for target recommendation information in an embodiment of this application is shown. Figure 9 As shown, a game's draft (BP) screen is displayed. The event information corresponding to the first video frame is "Game 3 is currently on the BP screen." In this video screen, rectangles 11, 12, and 13 contain content related to the first team's BP; rectangles 21, 22, and 23 contain content related to the second team's BP; rectangle 31 displays the current tournament format, date, and BP countdown; rectangle 41 shows the live broadcast; and rectangle 51 is the display area for the target recommendation information. A recommendation information trigger control 52 for the target recommendation information is set within rectangle 51. Clearly, in... Figure 9 In this system, target recommendation information and the first video frame can be played together, and the display area of ​​the target recommendation information will not affect the acquisition of event information corresponding to the first video frame based on the video frame. For example... Figure 9 As shown, after triggering the recommendation information trigger control 52, you can switch from the current playback interface to the interface corresponding to the target recommendation information.

[0183] Since both the first video frame and the target recommendation information are obtained by parsing the target SEI frame, the time corresponding to the first video frame should be the same as the time corresponding to the target recommendation information. Based on the above, after obtaining the live video stream to be played, the first video frame and the target recommendation information are obtained by parsing the target SEI frame in the live video stream. When the attribute information of the target recommendation information matches the attribute information of the first video frame, the target recommendation information and the first video frame are played together. This achieves simultaneous playback of the target recommendation information and the first video frame during the playback of the live video stream, fully combining the attribute information of the first video frame to push and display recommendation information.

[0184] If the execution entity corresponding to the live video stream playback method determines whether the target recommendation information does not match the attribute information of the first video frame, it can generate first feedback information based on the result of the mismatch and send the first feedback information to the execution entity corresponding to the live video stream processing method. This allows the execution entity corresponding to the live video stream processing method to confirm whether there is a problem in the process of determining the target recommendation information, converting the format of the target recommendation information, and inserting the target recommendation information into the SEI unit of the first video frame.

[0185] Optionally, parsing the target SEI frame in the live video stream to be played may include:

[0186] If a video frame containing an identifier with recommendation information is detected in the live video stream, the video frame with the identifier with recommendation information is identified as the target SEI frame, and the target SEI frame is parsed.

[0187] As described above, the target SEI frame is formed by inserting the target recommendation information converted to a preset format and the corresponding recommendation information identifier into the SEI unit of the first video frame. Therefore, it is possible to determine whether each video frame is a target SEI frame by detecting whether each video frame in the live video stream has a recommendation information identifier.

[0188] By detecting whether each video frame in the live video stream has an identifier for recommendation information, it is possible to quickly determine whether the live video stream contains a target SEI frame that includes the target recommendation information.

[0189] Optionally, a trigger control for the target recommendation information can be set in the playback area of ​​the target recommendation information. By clicking the trigger control, the user can jump to the specific interface corresponding to the target recommendation information.

[0190] During the playback of a live video stream, the method may further include:

[0191] After playing the target recommendation information and the first video frame together, the number of times the target recommendation information and the first video frame are played together can be counted, as well as the number of times the trigger control of the target recommendation information is clicked.

[0192] The number of plays and clicks are used to generate a second feedback message, which is then sent to the execution entity corresponding to the live video stream processing method. This allows the execution entity to adjust the triggering time and target recommendation message based on the number of plays and clicks during the subsequent processing of the live video stream.

[0193] By tracking the number of plays and clicks, the processing methods for the aforementioned live video streams can be further optimized, thereby improving the efficiency of pushing and displaying recommended information.

[0194] To facilitate a clearer understanding of the application value of the live video stream playback method provided in this application embodiment, a specific application scenario embodiment is described below. The solution provided in this application can be applied to any application scenario for playing live video streams. This solution can be implemented as a terminal device, an application on the terminal device for video playback, a plugin for the application, or a server.

[0195] Figure 10 A flowchart illustrating a method for playing a live video stream according to an embodiment of this application is shown. Figure 10 As shown, the playback method of this live video stream can be implemented through a playback terminal, which can be a terminal or a processor. When the playback terminal is a terminal, it can specifically be a webpage, application, plugin, or mini-program within the application that can play video. For example... Figure 10 As shown, the method for playing this live video stream can specifically include the following steps.

[0196] Step S21: Begin.

[0197] Step S22: The consumer (i.e., the playback end) pulls the live video stream to be played. This live video stream can be a live video stream distributed by CDN.

[0198] Step S23: Play the live video stream. In step S23, video frames in the live video stream that are not the target SEI frame can be played directly.

[0199] Step S24: Call the SDK to parse the SEI frame (i.e., the target SEI frame) recommendation data, which is the above step S220, to parse the target SEI frame in the live video stream to be played, and obtain the first video frame and target recommendation information.

[0200] Step S25: Confirm whether the UUID is the default identifier for the recommendation information, AdsCtrlSEIV00002. If the UUID is the identifier for the recommendation information of Game 3, proceed to step S26; if the UUID is not the identifier for the recommendation information of Game 3, proceed to step S28.

[0201] Step S26: Render the recommendation data (i.e., the target recommendation information) onto the webview and display it (i.e., play the target recommendation information together with the first video frame).

[0202] In step S26, the method may further include: if it is determined that the target recommendation information matches the attribute information of the first video frame, rendering the target recommendation information onto the webview corresponding to the first video frame, thereby enabling the joint playback of the target recommendation information and the first video frame.

[0203] Step S27: Report the exposure (i.e., number of plays) and clicks (i.e., number of clicks) data of the recommended data (i.e., generate second feedback information from the number of plays and clicks, and send the second feedback information to the execution entity corresponding to the processing method of the live video stream). After completing step S27, step S28 can be executed directly.

[0204] Step S28: If the UUID is not an identifier for the recommendation information of Game 3, do not render the recommendation data.

[0205] This application also provides a live video stream processing apparatus. Figure 11 A schematic diagram of a live video stream processing apparatus provided in an embodiment of this application is shown. Figure 11 As shown, the processing device 30 includes an acquisition module 301, a target recommendation information determination module 302, and a target recommendation information insertion module 303, wherein...

[0206] Module 301 is used to acquire live video streams;

[0207] If the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, the target recommendation information determination module 302 is used to determine the target recommendation information corresponding to the attribute information of the first video frame from at least one preset recommendation information.

[0208] The target recommendation information insertion module 303 is used to insert the target recommendation information converted to a preset format into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, so as to obtain the processed live video stream.

[0209] Optionally, the attribute information of the first video frame includes the type information of the playback information corresponding to the live video stream and the event information of the playback scene corresponding to the first video frame;

[0210] When the target recommendation information determination module 302 determines the target recommendation information corresponding to the attribute information of the first video frame from at least one preset recommendation information, it is specifically used to perform any one of the following operations:

[0211] Determine the target recommendation information corresponding to the type information of the live video stream from at least one preset recommendation information;

[0212] Target recommendation information corresponding to the event information of the first video frame is determined from at least one preset recommendation information.

[0213] Optionally, the time corresponding to the first video frame matches the trigger time of the preset recommendation information, including at least one of the following:

[0214] The time corresponding to the first video frame is the same as the preset trigger time for the recommendation information;

[0215] The event information corresponding to the first video frame is the event information of any preset scene.

[0216] Optionally, the matching of the time corresponding to the first video frame with the preset trigger time of the recommendation information further includes:

[0217] The trigger time of the preset recommendation information is corrected according to the preset time offset, and the corrected time is determined.

[0218] If the corrected time is the same as the time corresponding to the first video frame, then the time corresponding to the first video frame is determined to match the trigger time of the preset recommendation information.

[0219] Optionally, when the target recommendation information insertion module 303 inserts the target recommendation information converted to a preset format into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, it is specifically used for:

[0220] The target recommendation information, converted to a preset format, and the corresponding identifier of the recommendation information are inserted into the SEI unit of the first video frame to form a target SEI frame, so that the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information.

[0221] The live video stream processing device of this application embodiment can execute the live video stream processing method provided in this application embodiment. The implementation principle is similar. The actions performed by each module in the live video stream processing device of each embodiment of this application correspond to the steps in the live video stream processing method of each embodiment of this application. For detailed functional descriptions of each module of the live video stream processing device, please refer to the descriptions in the corresponding live video stream processing methods shown above, which will not be repeated here.

[0222] This application also provides a device for playing live video streams. Figure 12 This diagram illustrates a live video stream playback device according to an embodiment of this application. Figure 12 As shown, the playback device 40 includes an acquisition module 401, a parsing module 402, and a playback module 403, wherein...

[0223] The acquisition module 401 is used to acquire the live video stream to be played;

[0224] The parsing module 402 is used to parse the target SEI frame in the live video stream to be played, and obtain the first video frame and target recommendation information;

[0225] If it is determined that the target recommendation information matches the attribute information of the first video frame, the playback module 403 is used to play the target recommendation information and the first video frame together.

[0226] Optionally, when parsing the target SEI frame in the live video stream to be played, the parsing module 402 is specifically used for:

[0227] If a video frame containing an identifier with recommendation information is detected in the live video stream, the video frame with the identifier with recommendation information is identified as the target SEI frame, and the target SEI frame is parsed.

[0228] The live video stream playback device of this application embodiment can execute the live video stream playback method provided in this application embodiment. The implementation principle is similar. The actions performed by each module in the live video stream playback device of each embodiment of this application correspond to the steps in the live video stream playback method of each embodiment of this application. For detailed functional descriptions of each module of the live video stream playback device, please refer to the descriptions in the corresponding live video stream playback methods shown above, which will not be repeated here.

[0229] Based on the same principles as the live video stream processing method and apparatus and the live video stream playback method and apparatus provided in the embodiments of this application, the embodiments of this application also provide an electronic device (such as a server). The electronic device may include a memory, a processor and a computer program stored in the memory. The processor executes the computer program to implement the steps and corresponding content of the method provided in any optional embodiment of this application.

[0230] Optionally, Figure 13 A schematic diagram of the structure of an electronic device to which this application embodiment applies is shown, such as... Figure 13 As shown, Figure 13 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0231] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0232] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0233] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0234] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0235] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0236] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0237] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0238] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0239] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A method for processing live video streams, characterized in that, The method includes: Obtain the live video stream; If the time corresponding to the first video frame of the live video stream matches the trigger time of preset recommendation information, target recommendation information is determined from at least one preset recommendation information; the content of the target recommendation information is independent of the attribute information of the first video frame. The target recommendation information, converted to a preset format, is inserted into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, thus obtaining the processed live video stream; The processed live video stream is sent to the playback device; Receive the second feedback information sent by the playback terminal, and adjust the triggering time and target recommendation information in the subsequent live video stream based on the second feedback information; The second feedback information is generated based on the number of plays and the number of clicks. The number of plays includes the number of times the target recommendation information and the first video frame are played together through the playback interface of the playback terminal. The number of clicks includes the number of times the trigger control is clicked by the playback terminal. The trigger control is set in the playback area of ​​the target recommendation information in the playback interface, and when the trigger control is clicked, the playback interface will jump to the interface corresponding to the target recommendation information.

2. The method according to claim 1, characterized in that, The attribute information of the first video frame includes the type information of the playback information corresponding to the live video stream and the event information of the playback scene corresponding to the first video frame.

3. The method according to claim 1 or 2, characterized in that, The time corresponding to the first video frame matches the trigger time of the preset recommendation information, including at least one of the following: The time corresponding to the first video frame is the same as the trigger time of the preset recommendation information; The event information corresponding to the first video frame is the event information of any preset scene.

4. The method according to claim 3, characterized in that, The time corresponding to the first video frame matches the trigger time of the preset recommendation information, and also includes: The trigger time of the preset recommendation information is corrected according to the preset time offset to determine the corrected time; If the corrected time is the same as the time corresponding to the first video frame, it is determined that the time corresponding to the first video frame matches the trigger time of the preset recommendation information.

5. The method according to claim 1, characterized in that, The step of inserting the target recommendation information, converted to a preset format, into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame includes: The target recommendation information, converted to a preset format, and the corresponding identifier of the recommendation information are inserted into the SEI unit of the first video frame to form a target SEI frame, so that the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information.

6. A method for playing a live video stream, characterized in that, The method includes: Get the live video stream to be played; The target SEI frame in the live video stream to be played is parsed to obtain the first video frame and target recommendation information; the target recommendation information is determined by the generator of the live video stream to be played from at least one preset recommendation information when the time corresponding to the first video frame matches the trigger time of preset recommendation information; the content of the target recommendation information is independent of the attribute information of the first video frame. If it is determined that the target recommendation information matches the attribute information of the first video frame, the target recommendation information and the first video frame are played together through the playback interface; The playback count is obtained based on the number of times the target recommendation information and the first video frame are played together through the playback interface; the click count is obtained based on the number of times the trigger control is clicked; second feedback information is generated based on the playback count and the click count; the trigger control is set in the playback area of ​​the target recommendation information in the playback interface, and when the trigger control is clicked, the playback interface will jump to the interface corresponding to the target recommendation information; The second feedback information is sent to the generator of the live video stream to be played, so that the generator of the live video stream to be played can adjust the triggering time and target recommendation information in the subsequent live video stream based on the second feedback information.

7. The method according to claim 6, characterized in that, The step of parsing the target SEI frame in the live video stream to be played includes: If a video frame containing an identifier with recommendation information is detected in the live video stream, the video frame with the identifier with recommendation information is identified as the target SEI frame, and the target SEI frame is parsed.

8. A processing device for live video streams, characterized in that, The device includes an acquisition module, a target recommendation information determination module, and a target recommendation information insertion module, wherein, The acquisition module is used to acquire live video streams; If the time corresponding to the first video frame of the live video stream matches the trigger time of the preset recommendation information, the target recommendation information determination module is used to determine the target recommendation information from at least one preset recommendation information; the content of the target recommendation information is independent of the attribute information of the first video frame; The target recommendation information insertion module is used to insert target recommendation information converted to a preset format into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, so as to obtain the processed live video stream; The transceiver module is used to send the processed live video stream to the playback terminal; and to receive second feedback information sent by the playback terminal, and to adjust the triggering time and the target recommendation information in subsequent live video streams based on the second feedback information. The second feedback information is generated based on the number of plays and the number of clicks. The number of plays includes the number of times the target recommendation information and the first video frame are played together through the playback interface of the playback terminal. The number of clicks includes the number of times the trigger control is clicked by the playback terminal. The trigger control is set in the playback area of ​​the target recommendation information in the playback interface, and when the trigger control is clicked, the playback interface will jump to the interface corresponding to the target recommendation information.

9. The apparatus according to claim 8, characterized in that, The attribute information of the first video frame includes the type information of the playback information corresponding to the live video stream and the event information of the playback scene corresponding to the first video frame.

10. The apparatus according to claim 8 or 9, characterized in that, The time corresponding to the first video frame matches the trigger time of the preset recommendation information, including at least one of the following: The time corresponding to the first video frame is the same as the trigger time of the preset recommendation information; The event information corresponding to the first video frame is the event information of any preset scene.

11. The apparatus according to claim 10, characterized in that, The time corresponding to the first video frame matches the trigger time of the preset recommendation information, and also includes: The trigger time of the preset recommendation information is corrected according to the preset time offset to determine the corrected time; If the corrected time is the same as the time corresponding to the first video frame, it is determined that the time corresponding to the first video frame matches the trigger time of the preset recommendation information.

12. The apparatus according to claim 8, characterized in that, When the target recommendation information insertion module inserts the target recommendation information, converted to a preset format, into the SEI supplementary enhancement information unit of the first video frame to form a target SEI frame, it is used to: The target recommendation information, converted to a preset format, and the corresponding identifier of the recommendation information are inserted into the SEI unit of the first video frame to form a target SEI frame, so that the playback end can determine whether a target SEI frame is set in the live video stream based on the identifier of the recommendation information.

13. A device for playing live video streams, characterized in that, The device includes an acquisition module, a parsing module, and a playback module, wherein... The acquisition module is used to acquire the live video stream to be played. The parsing module is used to parse the target SEI frame in the live video stream to be played, and obtain the first video frame and target recommendation information; the target recommendation information is determined by the generator of the live video stream to be played from at least one preset recommendation information when the time corresponding to the first video frame matches the trigger time of the preset recommendation information; the content of the target recommendation information is independent of the attribute information of the first video frame. If it is determined that the target recommendation information matches the attribute information of the first video frame, the playback module is used to play the target recommendation information and the first video frame together. The playback count is obtained based on the number of times the target recommendation information and the first video frame are played together through the playback interface; the click count is obtained based on the number of times the trigger control is clicked; second feedback information is generated based on the playback count and the click count; the trigger control is set in the playback area of ​​the target recommendation information in the playback interface, and when the trigger control is clicked, the playback interface will jump to the interface corresponding to the target recommendation information; The second feedback information is sent to the generator of the live video stream to be played, so that the generator of the live video stream to be played can adjust the triggering time and target recommendation information in the subsequent live video stream based on the second feedback information.

14. The apparatus according to claim 13, characterized in that, When parsing the target SEI frame in the live video stream to be played, the parsing module is used for: If a video frame containing an identifier with recommendation information is detected in the live video stream, the video frame with the identifier with recommendation information is identified as the target SEI frame, and the target SEI frame is parsed.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5, or to implement the steps of the method according to any one of claims 6-7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5, or the steps of the method according to any one of claims 6-7.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5, or the steps of the method according to any one of claims 6-7.

Images