Video stream processing method, video display method, equipment, storage medium and product

Abstract

The invention discloses a video stream processing method, a video display method, equipment, a storage medium and a product. The video stream processing method comprises the following steps: acquiring first video stream data; performing target object identification on video frames in the first video stream data, and generating structural data representing an identification result; packaging the structural data into the first video stream data to obtain second video stream data; sending the second video stream data to a playing device; wherein after analyzing the second video stream data, the playing device renders a display picture corresponding to the view angle of the target object based on the obtained first video stream data and the structure data. The live broadcast system equipment encapsulates the basic video stream data and the structural data corresponding to the target object into single video stream data and sends the single video stream data to the playing equipment; therefore, the playing device extracts and renders the multi-view live broadcast display picture from the original video stream in real time based on the structural data, and the production and transmission cost of the multi-view live broadcast service is remarkably reduced.

Description

Technical Field

[0001] This application relates to streaming data processing technology, and more particularly to a video streaming processing method, a video display method, an apparatus, a storage medium, and a product. Background Technology

[0002] In current business scenarios such as live sports broadcasts, user needs have expanded from viewing from a single main perspective to multi-dimensional viewing of "main perspective + target object perspective". Taking live football broadcasts as an example, users expect to watch or switch to watch the exclusive perspectives of star players and key players. To meet this viewing requirement, the live broadcast system needs to generate and output multi-perspective video stream data in the production and broadcasting stage.

[0003] In related technologies, there are significant pain points in processing video stream data from multiple perspectives: On the one hand, live streaming systems typically encapsulate the video stream data corresponding to each perspective into a complete live stream for transmission. Each live stream needs to complete the entire chain of processing, including encoding, transcoding, and content delivery network (CDN) distribution, which leads to a significant increase in transcoding computing resources and bandwidth transmission costs as the number of perspectives increases, resulting in a significant increase in operating costs. On the other hand, when users frequently switch perspectives during viewing, in order to avoid misalignment of the picture timing, it is necessary to perform real-time synchronization calculations (such as timestamp calibration and latency compensation) on each independent live stream. This not only increases the production and broadcasting complexity of the live streaming system, but also requires additional computing power to maintain multi-stream synchronization, further increasing the overall live streaming cost. Summary of the Invention

[0004] In view of this, embodiments of this application provide a video stream processing method, a video display method, a device, a storage medium, and a product, aiming to provide a low-cost multi-view live streaming solution.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a video stream processing method applied to a live streaming system device, the method comprising:

[0007] Acquire first video stream data; perform target object identification on video frames in the first video stream data, and generate structured data representing the identification results;

[0008] The structured data is encapsulated into the first video stream data to obtain the second video stream data;

[0009] Send the second video stream data to the playback device;

[0010] Wherein, after the playback device parses the second video stream data, it renders the display screen corresponding to the perspective of the target object based on the obtained first video stream data and the structure data.

[0011] In the above scheme, the step of identifying target objects in the video frames of the first video stream data and generating structured data representing the identification results includes:

[0012] Based on preset detection rules, the target frame is determined from the first video stream data;

[0013] Identify the image region of the target object in the target frame, and generate the structural data corresponding to the target frame based on the identified image region;

[0014] The structural data includes at least: frame number and target object region information.

[0015] In the above scheme, the step of encapsulating the structured data into the first video stream data to obtain the second video stream data includes:

[0016] Based on the frame sequence number in the structure data, establish an association mapping relationship between each structure data and the corresponding video frame in the first video stream data;

[0017] Based on the aforementioned association mapping relationship, the structured data is encapsulated into the first video stream data to obtain the second video stream data.

[0018] Secondly, embodiments of this application provide a video display method applied to a playback device, the method comprising:

[0019] Receive the second video stream data;

[0020] The second video stream data is parsed to obtain first video stream data and structural data; the structural data represents the target object recognition result of the video frame in the first video stream data.

[0021] Based on the structured data, image extraction is performed on the video frames in the first video stream data to obtain image frame data of the target object;

[0022] Based on the image frame data of the target object, render the display screen corresponding to the viewpoint of the target object.

[0023] In the above scheme, the step of extracting images from video frames in the first video stream data based on the structured data to obtain image frame data of the target object includes:

[0024] Based on the frame sequence number in the structure data, determine whether each video frame in the first video stream data is associated with the structure data;

[0025] If the frame to be rendered is associated with the structural data, then based on the valid target object region information in the structural data, the frame to be rendered is image extracted to obtain the image frame data of the target object;

[0026] If the frame to be rendered is not associated with the structural data, then based on the valid target object region information associated with the preceding rendered frame and the subsequent cached frame, the target object region information corresponding to the frame to be rendered is calculated, and the image of the frame to be rendered is extracted based on the calculation result to obtain the image frame data of the target object.

[0027] The method for rendering the display screen corresponding to the viewpoint of the target object based on the image frame data of the target object in the above scheme includes:

[0028] Determine whether the current frame is associated with the image frame data of the target object;

[0029] If the current frame is associated with the image frame data of the target object, then based on the image frame data of the current frame, the display screen of the current frame from the perspective of the target object is rendered;

[0030] The method further includes:

[0031] If the current frame is not associated with the image frame data of the target object, rendering is performed based on the set image to obtain the display screen of the current frame from the perspective of the target object.

[0032] Thirdly, embodiments of this application provide a live streaming system device, the live streaming system device comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method described in the first aspect.

[0033] Fourthly, embodiments of this application provide a playback device, the playback device comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method as described in the second aspect.

[0034] Fifthly, embodiments of this application provide a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method as described in the first or second aspect.

[0035] Sixthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method as described in the first or second aspect.

[0036] The technical solution provided in this application embodiment involves a live streaming system device acquiring first video stream data; identifying target objects in the video frames of the first video stream data and generating structural data representing the identification results; encapsulating the structural data into the first video stream data to obtain second video stream data; and sending the second video stream data to a playback device. The playback device, after parsing the second video stream data, renders the display screen corresponding to the target object's perspective based on the obtained first video stream data and structural data. In this way, the live streaming system device encapsulates the basic video stream data and the structural data corresponding to the target object into a single video stream and sends it to the playback device. After parsing the video stream, the playback device can extract and render multi-view live display screens from the original video stream in real time based on the structural data, eliminating the need for the live streaming system device to transmit multiple independent viewpoint live streams, significantly reducing the production and transmission costs of multi-view live streaming services. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the video stream processing method according to an embodiment of this application;

[0038] Figure 2 This is a schematic diagram of the live streaming architecture as described in an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of a live sports event display screen in an application example;

[0040] Figure 4 This is a flowchart illustrating the video display method according to an embodiment of this application;

[0041] Figure 5 This is a schematic diagram illustrating image extraction as an application example.

[0042] Figure 6 This is a schematic diagram of the structure of the video stream processing device according to an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of the video playback device according to an embodiment of this application;

[0044] Figure 8 This is a schematic diagram of the structure of the live streaming system equipment according to an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the playback device according to an embodiment of this application. Detailed Implementation

[0046] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0048] This application provides a video stream processing method applied to live streaming system equipment, aiming to achieve multi-view live streaming at low cost.

[0049] For example, such as Figure 1 As shown, the video stream processing method applied to live streaming system equipment in this application includes:

[0050] Step 101: Obtain the first video stream data.

[0051] Step 102: Target object identification is performed on the video frames in the first video stream data, and structural data representing the identification results is generated.

[0052] Step 103: Encapsulate the structured data into the first video stream data to obtain the second video stream data.

[0053] Step 104: Send the second video stream data to the playback device.

[0054] In this process, after parsing the second video stream data, the playback device renders the display screen corresponding to the perspective of the target object based on the obtained first video stream data and structural data.

[0055] Here, the live streaming architecture of this application embodiment is as follows: Figure 2 As shown, this live streaming architecture includes live streaming system equipment, network distribution equipment, and playback devices. The live streaming system equipment processes the raw audio and video data of the live streaming service into a live stream that meets live streaming requirements and sends the live stream to the network distribution equipment. The live stream can contain video stream data or audio stream data alone, or it can contain both video stream data and audio stream data simultaneously. Network distribution equipment, such as CDN, distributes the live stream to the corresponding playback device according to the request from the playback device. After receiving the live stream, the playback device parses and decodes the live stream to generate visuals and / or audio that can be viewed and / or listened to by the user.

[0056] Here, sending the second video stream data to the playback device includes: sending the second video stream data to the playback device via a network distribution device.

[0057] It should be noted that for multi-view live streaming services such as sports events, where users expect to watch from multiple perspectives, including the main viewpoint and at least one target object's perspective, the relevant technologies typically encapsulate the video stream data corresponding to each perspective into complete live streams for transmission. Users then operate playback devices to obtain the video stream data corresponding to different perspectives, enabling simultaneous viewing from multiple perspectives or switching between viewing perspectives. However, since the video stream data corresponding to each perspective is independently encapsulated into a live stream, each live stream requires separate encoding, transcoding, CDN distribution, and other end-to-end processing. This leads to a significant increase in the transcoding computing resources and bandwidth transmission costs of the live streaming system equipment as the number of perspectives increases. Furthermore, to avoid timing misalignment when users frequently switch perspectives, the live streaming system equipment needs to perform real-time synchronous calculations such as timestamp calibration and latency compensation for each independent live stream. This not only increases the production and broadcasting complexity of the live streaming system equipment but also requires additional computing power to maintain multi-stream synchronization, further increasing the overall live streaming cost.

[0058] It should be noted that in the video stream processing method of this application embodiment, for multi-view live streaming services, the live streaming system device only needs to send one video stream data to the playback device, which enables the playback device to generate multi-view display images based on the video stream data, and the images from each viewpoint remain synchronized, thereby reducing the cost of multi-view live streaming services.

[0059] Here, the first video stream data in this embodiment is the video stream data corresponding to the main viewpoint of the live streaming service. The main viewpoint refers to the basic viewpoint in a multi-view live streaming scenario that can fully present the main subject of the live stream and cover key information. It is the basic screen viewpoint that is loaded first after the user enters the live stream. For example, in a football match live stream, the main viewpoint presents the overall situation of the game, the overall offensive and defensive situation, the players' running routes, and the referee's decisions, etc., which are the core game process.

[0060] Understandably, the first video stream data serves as the foundational video stream data for the live streaming business.

[0061] In some embodiments, the live streaming system equipment includes production and broadcasting equipment, and the first video stream data is generated by the production and broadcasting equipment.

[0062] Here, multi-view live streaming services, in addition to the main viewpoint, also need to present at least one viewpoint from the target object's perspective. This target object's viewpoint typically includes the target object itself; that is, it's a camera movement perspective used to track the target object. For example, the display of a live sports event might look like this... Figure 3 As shown, users can view the main viewpoint and the viewpoint of the star players and key players who need to be focused on in the game through the playback device.

[0063] For example, the method further includes: determining the target object.

[0064] It is understandable that the target object is used to determine the corresponding target object perspective; one or more target objects can be determined before and / or during the live broadcast.

[0065] Here, a corresponding target object perspective will be generated for each target object.

[0066] In some embodiments, the target object can be determined by the live broadcast operator. After the live broadcast operator determines the target object in this live broadcast, the operation system generates a target object list and pushes the target object list to the live broadcast system devices and the portal system, so that the live broadcast system devices can determine the target objects to be identified based on the target object list, and the playback devices can subscribe to the target object list through the portal system to determine the target object perspective supported by this live broadcast.

[0067] In this embodiment, after the live streaming system device acquires the first video stream data corresponding to the main viewpoint, it collects video frames in the first video stream data and performs target object recognition on the images of the video frames. It should be noted that after the live streaming system device completes the target object recognition, it does not generate target object images based on the recognition results or encapsulate multiple target object images into video stream data corresponding to the target object's viewpoint. Instead, it generates structure (Model, M) data based on the recognition results and encapsulates this structure data as extended data of the first video stream data into the first video stream data. After encapsulation, the second video stream data is obtained.

[0068] Here, structured data refers to structured data generated based on the target object recognition results and used to describe key information of the target object, not image data such as video frames; structured data may include information such as structured data sequence number, frame sequence number, and target object region information.

[0069] Understandably, the live streaming system equipment constructs structural data based on the target object recognition results. This structural data can be transmitted synchronously with the video stream data, without the need to additionally generate and transmit the complete video stream data corresponding to the target object's perspective. After receiving the basic video stream data (i.e., the first video stream data) and the corresponding structural data from the live streaming service, the playback device can directly extract the image containing the target object from the video frames of the basic video stream data based on the structural data, thereby generating image frame data corresponding to the target object's perspective on the playback device side.

[0070] Here, the second video stream data can be sent directly to the playback device as a complete live stream, or it can be encapsulated with the audio stream data of the live streaming service into the same live stream before being sent to the playback device.

[0071] It is understood that, in the embodiments of this application, the live streaming system device encapsulates the basic video stream data and the structural data corresponding to the target object into a single video stream data and sends it to the playback device. After the playback device parses the video stream, it can extract and render multi-view live display images from the original video stream in real time based on the structural data. This eliminates the need for the live streaming system device to transmit multiple independent view live streams, significantly reducing the production and transmission costs of multi-view live streaming services.

[0072] The steps of the video stream processing method are described below with reference to specific embodiments.

[0073] For example, target object recognition is performed on video frames in the first video stream data, and structural data representing the recognition result is generated, including: determining the target frame from the first video stream data based on a preset detection rule; identifying the image region of the target object in the target frame; and generating structural data corresponding to the target frame based on the identified image region; wherein the structural data includes at least: frame number and target object region information.

[0074] It is understandable that the target frame is the video frame in the first video stream data that needs to be identified as the target object, and it is determined by the preset detection rules.

[0075] Here, the detection rules include: frame-by-frame detection rules or periodic detection rules.

[0076] In the frame-by-frame detection rule, all video frames in the first video stream data are target frames. That is, the live streaming system device performs target object identification on all video frames in the first video stream data and generates corresponding structure data based on the identification results of each target frame.

[0077] Among them, the periodic detection rule is also called the frame-skipping detection rule. Considering the computing power consumption of target object recognition on the live streaming system equipment, the periodic detection rule will not perform target object recognition on all video frames in the first video stream data. Instead, it will skip detection on video frames in the first video stream data based on a set interval (e.g., 200ms). That is, the target frame is determined based on the set interval, and the interval between consecutive target frames is set. The live streaming system equipment performs target object recognition on each target frame in the first video stream data and generates corresponding structured data based on the recognition results of each target frame.

[0078] Here, the method further includes: acquiring image sample data of the target object; and training the target detection model based on the image sample data to obtain a trained target detection model.

[0079] Accordingly, identifying the image region of the target object in the target frame includes: identifying the image region of the target object in the target frame based on the trained target detection model and the determined target object.

[0080] Here, an object detection model can be used to detect and identify the image region where the target object is located in the image of the target frame. Specifically, before the live broadcast, the image sample data of each target object collected is input into the object detection model for model training. The target detection model obtained after training can accurately identify the corresponding target object. During the live broadcast, the target frame from the first video stream data is image parsed and input into the trained object detection model. The object detection model detects and identifies whether the image of the target frame contains the image of the target object and the corresponding real-time image region, and generates structural data representing the recognition result.

[0081] In some embodiments, in order to improve the image region recognition accuracy of the target detection model, during the model training phase, after obtaining the image sample data of the target object, the region bounding box is labeled on the image sample data based on the target object.

[0082] In some embodiments, the object detection model includes the YOLO (You Only Look Once) model. The YOLO model is a single-stage object detection model that transforms the detection problem into a regression problem, simultaneously predicting the bounding box and category of the target object using a single neural network. Because the YOLO model has a fast object recognition speed, it can be applied to real-time scenarios such as video stream recognition.

[0083] It should be noted that the object recognition of the target frame based on the object detection model is only one application example provided in this application, and this application does not specifically limit the object recognition method of the target frame; correspondingly, the object detection model used for object recognition is not limited to the YOLO model, and those skilled in the art can choose other forms of object detection models according to the actual needs of the scenario.

[0084] Here, after the live streaming system device identifies the image region of the target object in the target frame, it generates structured data corresponding to the target frame, including at least the frame number and the target object region information. The frame number of the structured data is determined based on the corresponding target frame. Each video frame in the first video stream data carries a frame number, which represents the sequential position of the video frame in the first video stream data. The target object region information is determined based on the position of the target object in the image of the target frame and the region boundary.

[0085] In some embodiments, the frame number is the image display time stamp (PTS); each video frame in the first video stream data carries the PTS, and the sequence position of the video frame in the first video stream data can be determined based on the temporal characteristics of the PTS. When the live streaming system device generates the structure data corresponding to the target frame, it adds the PTS carried by the target frame to the structure data.

[0086] In some embodiments, the target object region information includes: center point position and region size parameters. The center point position is the location of the target object image within the target frame, and the region size parameters are the pixel dimensions of the bounding box of the target object image with the center point position as the positioning reference. Here, since the display interface of the playback device is typically a rectangular display interface, the region size parameters include: region width and region height.

[0087] Understandably, target object region information can instruct the playback device to extract images from a specified region of the target frame's image, thereby obtaining image frame data corresponding to the target object's perspective, including the target object.

[0088] In some embodiments, the structured data further includes a structured data sequence number and / or target object information. If the live broadcast sets multiple target object perspectives, the live broadcast system device needs to simultaneously perform multiple target object recognitions on the target frame image. Accordingly, the structured data sequence number represents the sequence number of each target object recognition result in this round of recognition. For example, in a football match live broadcast, if the set target object perspectives include player A's perspective and player B's perspective, the live broadcast system device will recognize player A and player B on the target frame image, and associate the target object region information of player A with structured data sequence number: 1 and / or target object information: player A, and associate the target object region information of player B with structured data sequence number: 2 and / or target object information: player B. The target objects corresponding to each target object region information in the structured data are distinguished by the structured data sequence number and / or target object information.

[0089] In some embodiments, the structural data may also include the sequence number of the last successfully identified frame.

[0090] In some embodiments, the live streaming system device may not be able to identify the target object in the target frame image due to reasons such as the target frame image not containing the target object; here, after the live streaming system device identifies the target object in the target frame, it indicates that the target object has not been identified by setting the center point position to a first set value (e.g., null value).

[0091] In some embodiments, if the preset detection rule is a periodic detection rule, after determining that no target object is identified from the target frame, a supplementary detection frame interval (e.g., video frames with a duration interval of less than 50ms from the target frame) is determined based on the frame sequence number of the target frame, and target object identification is performed on the video frames in the supplementary detection frame interval to generate target object region information, so as to improve the target object identification accuracy.

[0092] In some embodiments, the structural data further includes a setting image identifier. The setting image identifier, along with target object region information, structural data sequence number, and / or target object information, indicates whether the display screen from the target object's perspective in the current frame uses a setting image.

[0093] Here, the set image is a catch-all image. If the target object is not identified in the current frame, the center point is empty, and the playback device cannot obtain the target object image of the current frame from the first video stream data through structural data, the live streaming system device can instruct the display screen of the current frame from the target object's perspective to use the set image by setting the set image identifier to a second set value. The set image can be a pre-determined fixed image, or it can be the display screen from other perspectives or camera positions in the live streaming service. For example, for a football match live streaming service, the set image can be set to the image of the football, or it can be set to the current view from the stands.

[0094] In one application example of this application, the structured data is as follows:

[0095] {"PTS":9999,"objects":[{"title":"A","seq":1,"center_point":"66,77","weight":192,"height":168,"bak":0," con_pts":9998},{"title":"B","seq":2,"center_point":null,"weight":0,"height":0,"bak":1,"con_pts":9996}]}

[0096] Wherein, PTS is the frame number, title is the target object information, seq is the structure data number, center_point is the center point position, weight is the region width, height is the region height, bak is the set image identifier, and con_pts is the last successfully recognized frame number.

[0097] Where bak=1 is the second setting value, and bak=0 is a non-second setting value.

[0098] For example, encapsulating structural data into first video stream data to obtain second video stream data includes: establishing an association mapping relationship between each structural data and the corresponding video frame in the first video stream data based on the frame sequence number in the structural data; and encapsulating the structural data into the first video stream data based on the association mapping relationship to obtain second video stream data.

[0099] Understandably, after the live streaming system device identifies the target object in the target frame of the first video stream data, it generates a structured dataset (M1, M2, ...). Each structured data point M in the structured dataset... n The structural data is generated based on the recognition results of different video frames. Since the structural data and the corresponding video frames carry the same frame number, an association mapping relationship between the structural data and the corresponding video frames is established based on the frame number when encapsulating the structural data. The structural data and the corresponding video frames are then encapsulated together based on the association mapping relationship. This allows the playback device to determine the association mapping relationship between the structural data and the video frames based on the frame number after parsing the second video stream data, and to extract images in the corresponding video frames based on the target object region information of the structural data.

[0100] In some embodiments, encapsulating structural data into first video stream data includes: encapsulating the structural data into Supplemental Enhancement Information (SEI) data and adding it to the corresponding video frame in the first video stream data.

[0101] In this embodiment, the structured data is serialized and encapsulated in SEI data of the target payload type according to the standard encapsulation format. Then, the SEI data is inserted into the Network Abstraction Layer Unit (NALU) sequence of the corresponding video frame, so that the structured data can be bound and transmitted with the video stream data.

[0102] In some embodiments, encapsulating structural data into first video stream data includes adding structural data to the protocol extension field of the corresponding video frame in the first video stream data.

[0103] In this embodiment, by adding a protocol extension field to the protocol unit corresponding to the video frame of the first video stream data (e.g., the message unit of the RTMP protocol, the TS packet of the HLS protocol), and writing the serialized structure data into the protocol extension field, the structure data can be bound to the video stream data for transmission.

[0104] Here, after the live streaming system device generates the second video stream data, it sends the second video stream data to the network distribution device, and the network distribution device then transmits the second video stream data to the corresponding playback device.

[0105] It is understood that the embodiments of this application can instruct the playback device to generate multi-view display images based on the first video stream data by carrying a small amount of structured data in the second video stream data. Compared with the live broadcast system equipment, which needs to produce and transmit multiple video stream data to present multi-view display images, the production and transmission costs of the multi-view live broadcast service in the embodiments of this application are significantly reduced.

[0106] In some embodiments, the above-described video stream processing method can be executed separately by each sub-device in the live streaming system equipment. In this embodiment, the live streaming system equipment includes not only production and broadcasting equipment, but also target object recognition equipment, streaming media equipment, and transcoding production equipment. Specifically, after generating first video stream data, the production and broadcasting equipment sends the first video stream data to the streaming media equipment used for video stream encapsulation, and simultaneously sends the first video stream data to the target object recognition equipment via live streaming bypass. The target object recognition equipment sets up a trained target detection model, and upon receiving the first video stream data, performs target object recognition on the first video stream data based on preset detection rules, generates a structured dataset, and sends it to the streaming media equipment. The streaming media equipment associates and maps the received structured dataset with the first video stream data, and encapsulates the structured data and the first video stream data together according to the live stream format requirements to obtain second video stream data conforming to the live stream format. The second video stream data is then sent to the transcoding production equipment for format conversion and bitrate matching.

[0107] Based on the aforementioned video stream processing method for application live streaming system devices, this application embodiment also provides a video display method applied to playback devices, which also aims to achieve multi-view live streaming at low cost.

[0108] Here, the playback device can be a mobile phone, computer, tablet computer, television, smart projector, in-vehicle smart display terminal, smart wearable device, portable playback terminal, or smart screen, or other terminal device with network connectivity and display capabilities. This application does not specifically limit the form of the playback device.

[0109] For example, such as Figure 4 As shown, the video display method applied to a playback device according to this application includes:

[0110] Step 401: Receive the second video stream data.

[0111] Step 402: Parse the second video stream data to obtain the first video stream data and structure data.

[0112] Step 403: Based on the structured data, extract images from the video frames in the first video stream data to obtain image frame data of the target object.

[0113] Step 404: Based on the image frame data of the target object, render the display screen corresponding to the viewpoint of the target object.

[0114] Among them, the structural data represents the target object recognition result of the video frame in the first video stream data.

[0115] Here, receiving the second video stream data includes receiving the second video stream data sent by the live streaming system device.

[0116] It is understood that the second video stream data is the video stream data generated after the live streaming system device executes the aforementioned video stream processing method. The second video stream data carries the first video stream data and structural data; the structural data corresponds to the video frames in the first video stream data.

[0117] Here, the first video stream data is the video stream data corresponding to the main perspective of the live streaming business, that is, the basic video stream data of the live streaming business.

[0118] Accordingly, after parsing the second video stream data to obtain the first video stream data and structural data, the method further includes: rendering the display screen corresponding to the main viewpoint based on the first video stream data.

[0119] It is understandable that for multi-view live streaming services, the playback device needs to display the user a main viewpoint and at least one target object's viewpoint. In this embodiment, the display screen corresponding to the main viewpoint can be directly obtained by decoding and rendering the video frames of the first video stream data.

[0120] It is understandable that after the playback device parses the second video stream data, the resulting structure data is the recognition result obtained by the live streaming system device after performing target object recognition on the first video stream data. Based on the structure data, the playback device can determine the image position and region of the target object in the video frames of the first video stream data. Then, it can directly extract the image frame data containing the target object from the video frames of the first video stream data. Finally, based on the image frame data of the target object, the rendering of the target object's perspective image is completed.

[0121] It is understood that the playback device in this application embodiment supports image extraction from the basic video stream data of multi-view live streaming services based on structural data to generate display images corresponding to the perspectives of other target objects. This allows the live streaming system device to transmit only the basic video stream data and the structural data representing the target object recognition results without transmitting multiple independent live streams. By utilizing the collaborative work of the live streaming system side and the playback device side, multi-view presentation of live streaming services is realized, significantly reducing the production and transmission costs of multi-view live streaming services.

[0122] In some embodiments, the method further includes: determining the viewpoint of the target object to be rendered in response to a user operation instruction.

[0123] Here, user operation commands can be generated based on the user's operation of the playback device, such as remote control, button control, or touch control, and are used to instruct the playback device to display the selected target object's view.

[0124] Here, the playback device subscribes to the target object list from the portal system to determine the target object perspectives supported for this live stream; the user selects the target object perspective they wish to watch from the supported target object perspectives for this live stream.

[0125] It is understandable that although the playback terminal supports users to watch multiple viewpoints simultaneously, users may choose not to watch the target object's viewpoint (i.e., only watch the main viewpoint) or choose to watch some of the target object's viewpoints (for example, a live streaming service provides three target object viewpoints, and the user chooses to watch one of them) based on their actual viewing needs and habits. Therefore, in the video display method of the embodiment, before extracting images from the video frames in the first video stream data based on the structure data, it is also necessary to predetermine the target object's viewpoint to be rendered and determine the target object to be extracted based on the target object's viewpoint to be rendered.

[0126] Accordingly, based on the structured data, image extraction is performed on the video frames in the first video stream data to obtain image frame data of the target object, including: based on the structured data, image extraction of the target object to be extracted is performed on the video frames in the first video stream data to obtain image frame data of the target object to be extracted.

[0127] Accordingly, based on the image frame data of the target object, the display screen corresponding to the viewpoint of the target object is rendered, including: based on the image frame data of the target object to be extracted, the display screen corresponding to the viewpoint of the target object to be rendered is rendered.

[0128] The steps of the video display method are described below with reference to specific embodiments.

[0129] For example, based on structural data, image extraction is performed on video frames in the first video stream data to obtain image frame data of the target object, including: determining whether each video frame in the first video stream data is associated with structural data based on the frame sequence number in the structural data; if the video frame is associated with structural data, determining whether the associated structural data includes valid target object region information; if it includes valid target object region information, then image extraction is performed on the video frame based on the valid target object region information to obtain the associated first image frame data.

[0130] For example, based on structural data, image extraction is performed on video frames in the first video stream data to obtain image frame data of the target object. This includes: determining whether each video frame in the first video stream data is associated with structural data based on the frame sequence number in the structural data; if the frame to be rendered is associated with structural data, then image extraction is performed on the frame to be rendered based on the valid target object region information in the structural data to obtain image frame data of the target object; if the frame to be rendered is not associated with structural data, then the target object region information corresponding to the frame to be rendered is calculated based on the valid target object region information associated with the preceding rendered frame and the subsequent buffered frame, and image extraction is performed on the frame to be rendered based on the calculation result to obtain image frame data of the target object. Here, the frame to be rendered is any video frame in the first video stream data used for image rendering.

[0131] Here, the structural data includes at least the frame number and target object region information.

[0132] Here, there is an association mapping relationship between the structure data and the video frames in the first video stream data. The association mapping relationship is constructed based on the frame sequence number of the structure data. Specifically, each video frame in the first video stream data carries a frame sequence number, which represents the sequence position of the video frame in the first video stream data. The structure data and the corresponding video frame carry the same frame sequence number. Based on the frame sequence number in the structure data, the associated video frame can be determined from the first video stream data. That is, the structure data represents the target object recognition result of the associated video frame.

[0133] In some embodiments, the frame number is PTS.

[0134] It should be noted that the second video stream data may only carry structural data associated with some video frames of the first video stream data. For example, when the live streaming system device performs skip detection on the first video stream data based on periodic detection rules, it only generates associated structural data based on the target frame for target object identification. Therefore, before the playback device extracts images from the video frame, it needs to determine whether the video frame has associated structural data based on the structural data and the frame number of the video frame.

[0135] Here, the target object region information represents the position of the target object in the image of the associated video frame and the region boundary; in some embodiments, the target object region information includes: center point position and region size parameters. The center point position is the position of the target object image in the image of the associated video frame, and the region size parameter is the pixel size of the bounding box of the target object image with the center point position as the positioning reference. In some embodiments, the region size parameter includes: region width and region height.

[0136] It should be noted that the target object region information in the structured data includes valid target object region information and invalid target object region information. Valid target object region information is generated after the live streaming system equipment determines that the target object has been identified in the image of the video frame, while invalid target object region information is generated after the live streaming system equipment determines that the target object has not been identified in the image of the video frame.

[0137] In some embodiments, it is determined whether the associated structural data includes valid target object region information: if the center point position in the target object region information is not a first preset value, it is determined whether the associated structural data includes valid target object region information; if the center point position in the target object region information is a first preset value, it is determined whether the associated structural data includes invalid target object region information.

[0138] Here, the first setting value can be set to an empty value.

[0139] It is understandable that if there is associated valid target object region information in the frame to be rendered in the first video stream data, the playback device can determine the image region where the target object is located in the image of the associated video frame based on the valid target object region information, and directly extract the image from the image region to obtain the associated image frame data including the target object.

[0140] It should be noted that if the second video stream data does not carry the structural data associated with the frame to be rendered, the playback device cannot directly obtain the target object region information used for image extraction of the frame to be rendered. In this embodiment, the target object region information of the frame to be rendered is calculated by using the effective target object region information of the preceding rendered frame and the following cached frame of the frame to be rendered through methods such as smoothing calculation.

[0141] Here, the preceding rendering frame is the preceding video frame in the first video stream data that is closest to the frame to be rendered and associated with valid target object region information; if the frame to be rendered is the current frame, then the preceding rendering frame is the video frame that was rendered last based on the image frame data of the target object; wherein, the image frame data of the target object is obtained by image extraction based on the associated valid target object region information.

[0142] Here, the subsequent cached frame is the next video frame in the first video stream data that is closest to the frame to be rendered and associated with valid target object region information; in this embodiment of the application, before rendering the display screen of the current frame from each perspective, the playback device can cache the current frame and a first set number of consecutive frames and associated structural data in the first video stream data, for example, cache the current frame and video frames and associated structural data within 5 seconds; the video frames within 5 seconds can be determined based on the frame rate and frame sequence number.

[0143] It is understandable that, due to the temporal relationship between the preceding rendered frame and the subsequent buffered frame, the target object region information of the frame to be rendered can be calculated based on the target object region information of the preceding rendered frame and the subsequent buffered frame, thus ensuring the continuity and accuracy of image extraction.

[0144] To ensure the accuracy of image extraction, in some embodiments, the subsequent buffered frames are frames that are associated with valid target object region information within a set frame interval; the set frame interval includes a second set number of consecutive frames after the current frame; the second set number is less than or equal to the first set number; for example, the set frame interval includes video frames within 3 seconds after the current frame, and the video frames within the set frame interval can be determined based on the frame rate and frame sequence number.

[0145] Here, if the frame number is PTS, the video frames within the set frame interval can be determined based on the timing range of PTS.

[0146] Accordingly, the calculation of the target object region information of the frame to be rendered based on the valid target object region information associated with the preceding rendered frame and the subsequent cached frame also includes: determining whether there is a subsequent cached frame in the set frame interval; if so, calculating the target object region information of the frame to be rendered based on the valid target object region information associated with the preceding rendered frame and the subsequent cached frame.

[0147] In some embodiments, the target object region information of the frame to be rendered is calculated based on the valid target object region information associated with the preceding rendered frame and the following cached frame. This includes: obtaining a first mapping relationship between the frame number and the center point position based on the frame number of the preceding rendered frame and the following cached frame and the associated center point position; obtaining a second mapping relationship between the frame number and the region size parameter based on the frame number of the preceding rendered frame and the following cached frame and the associated region size parameter; calculating the center point position corresponding to the frame to be processed based on the frame number of the frame to be processed and the first mapping relationship; and calculating the region size parameter corresponding to the frame to be processed based on the frame number of the frame to be processed and the second mapping relationship.

[0148] Here, the first and second mapping relationships are linear mapping relationships.

[0149] It is understood that, based on the continuity of the target object's movement and scaling in consecutive video frames, this application embodiment reasonably calculates and infers the position and coverage of the target object in each video frame between the preceding and subsequent buffer frames after obtaining the target object region information of the preceding and subsequent rendered frames and the subsequent buffer frames. Specifically, based on the frame number of the preceding and subsequent buffer frames, the frame interval between them is determined in combination with the frame rate, and the frame change of the center point position and the frame change of the region size parameter are calculated based on the frame interval, respectively, to obtain the first mapping relationship and the second mapping relationship; when calculating the target object region information corresponding to the frame to be processed, the temporal position of the frame to be processed between the preceding and subsequent buffer frames is determined based on the frame number of the frame to be processed, and then the center point position, region width, and region height corresponding to the frame to be processed are obtained based on the aforementioned mapping relationship.

[0150] It is understood that, in the embodiments of this application, even if the structural information associated with the frame to be processed cannot be directly obtained, the target object region information corresponding to the frame to be processed can be calculated by using a smoothing calculation method, and the image of the frame to be processed can be extracted based on the calculated target object region information to obtain the image frame data of the target object.

[0151] In one application example, such as Figure 5 As shown, after the playback device determines the target object region information of the frame to be processed, it performs image extraction on the frame to be processed based on the target object region information to obtain image frame data of each target object.

[0152] In some embodiments, the structural data further includes a structural data sequence number and / or target object information. The structural data sequence number represents the sequence number of each target object identification result in this round of identification; the target object region information is associated with the structural data sequence number and / or target object information.

[0153] It is understandable that the structural data associated with video frames includes target object region information of one or more target objects; the playback device determines the target object region information of the target object to be extracted from the structural data based on the structural data sequence number and / or target object information.

[0154] In some embodiments, the structural data may also include the sequence number of the last successfully identified frame.

[0155] In some embodiments, the structural data further includes a setting image identifier. The setting image identifier, along with target object region information, structural data sequence number, and / or target object information, indicates whether the display screen from the target object's perspective in the current frame uses a setting image.

[0156] Accordingly, if the frame to be rendered is associated with structural data, then based on the valid target object region information in the structural data, image extraction is performed on the frame to be rendered to obtain the image frame data of the target object, including: if the frame to be rendered is associated with structural data, then determining whether the set image identifier in the associated structural data is a second set value; if the set image identifier is not a second set value, then based on the valid target object region information in the structural data, image extraction is performed on the frame to be rendered to obtain the image frame data of the target object.

[0157] Accordingly, the method further includes: if the set image identifier associated with the current frame is a second set value, then rendering is performed based on the set image to obtain the display screen of the current frame from the perspective of the target object.

[0158] Here, the set image is a catch-all image; the second setting value instructs the playback device not to extract the image from the corresponding video frame, but to directly use the set image as the display screen of the target object.

[0159] In some embodiments, if the live streaming system device determines that no target object is identified in the image of the target frame in the first video stream data, it sets the target object region information in the associated structure data to invalid target object region information by setting the center point position to null, and at the same time sets the set image identifier in the structure data to a second set value to instruct the playback device not to extract the image of the target frame.

[0160] For example, a method for rendering a display screen corresponding to the viewpoint of a target object based on the image frame data of the target object includes: determining whether the current frame is associated with the image frame data of the target object; if the current frame is associated with the image frame data of the target object, then rendering the display screen of the current frame from the viewpoint of the target object based on the image frame data of the current frame; the method further includes: if the current frame is not associated with the image frame data of the target object, rendering based on a set image to obtain the display screen of the current frame from the viewpoint of the target object.

[0161] It should be noted that when the playback device in this application renders the display screen of the current frame from the perspective of the target object, the rendering result can be generated based on the extracted image frame data containing the target object, or it can be generated based on a set image; wherein, the set image is a pre-set catch-all image.

[0162] Understandably, when a playback device renders the display screen from the perspective of the target object in the current frame, if it can extract the image frame data of the target object from the current video frame, it will render the viewpoint based on the image frame data. The display screen from the perspective of the target object will include the target object and the target object will fully fill the display screen, resulting in a viewpoint display effect equivalent to a close-up shot of the target object. If the image frame data of the target object cannot be obtained, in order to ensure the user's viewing experience, a set image will be used to fill the display effect of the current frame from the perspective of the target object.

[0163] The set image can be a pre-determined fixed image, or it can be a display from other perspectives or camera positions in the live broadcast. For example, for a live football match broadcast, the set image can be a picture of the football, or it can be a view from the current stands.

[0164] Here, image frame data for which the current frame is not associated with a target object includes: invalid target object region information associated with the target object in the current frame, or no subsequent buffered frames in the set frame interval corresponding to the current frame.

[0165] It is understandable that when the set image identifier associated with the current frame is the second set value, the current frame is also associated with invalid target object region information of the target object. At this time, the playback device determines that it cannot extract the image of the target object from the image of the current frame, so it directly uses the set image for rendering. In addition, if there are no subsequent buffered frames in the set frame interval, even if there are video frames with valid target object region information associated with the target object in the subsequent video frames of the current frame, the accuracy of the target object region information of the current frame calculated is low due to the long time interval, so the set image is directly used for rendering.

[0166] In order to implement the method of the embodiments of this application, the embodiments of this application also provide a video stream processing device for live streaming system equipment. The video stream processing device corresponds to the video stream processing method described above, and the steps in the embodiments of the video stream processing method described above are also fully applicable to the embodiments of this device.

[0167] like Figure 6 As shown, the video stream processing device includes an acquisition module 601, an identification module 602, an encapsulation module 603, and a transmission module 604. The acquisition module 601 acquires first video stream data; the identification module 602 identifies target objects in the video frames of the first video stream data and generates structural data representing the identification results; the encapsulation module 603 encapsulates the structural data into the first video stream data to obtain second video stream data; and the transmission module 604 transmits the second video stream data to a playback device. The playback device, after parsing the second video stream data, renders the display screen corresponding to the viewpoint of the target object based on the obtained first video stream data and structural data.

[0168] In some embodiments, the identification module 602 is specifically used to: determine the target frame from the first video stream data based on a preset detection rule; identify the image region of the target object in the target frame; and generate structural data corresponding to the target frame based on the identified image region; wherein the structural data includes at least: frame number and target object region information.

[0169] In some embodiments, the encapsulation module 603 is specifically used to: establish an association mapping relationship between each structure data and the corresponding video frame in the first video stream data based on the frame sequence number in the structure data; and encapsulate the structure data into the first video stream data based on the association mapping relationship to obtain the second video stream data.

[0170] It should be noted that the video stream processing apparatus provided in the above embodiments is only illustrated by the division of the above-described program modules when performing video stream processing. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the video stream processing apparatus and the video stream processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0171] In order to implement the method of the embodiments of this application, the embodiments of this application also provide a video display device applied to a playback device. The video display device corresponds to the video display method described above, and the steps in the method embodiments described above are also fully applicable to the device embodiments.

[0172] like Figure 7 As shown, the video display device includes a receiving module 701, a parsing module 702, an extraction module 703, and a rendering module 704. The receiving module 701 receives second video stream data; the parsing module 702 parses the second video stream data to obtain first video stream data and structural data; the structural data represents the target object recognition result of the video frames in the first video stream data; the extraction module 703 extracts images from the video frames in the first video stream data based on the structural data to obtain image frame data of the target object; and the rendering module 704 renders the display screen corresponding to the viewpoint of the target object based on the image frame data of the target object.

[0173] In some embodiments, the extraction module 703 is specifically used to: determine whether each video frame in the first video stream data is associated with the structural data based on the frame sequence number in the structural data; if the frame to be rendered is associated with the structural data, then perform image extraction on the frame to be rendered based on the valid target object region information in the structural data to obtain the image frame data of the target object; if the frame to be rendered is not associated with the structural data, then calculate the target object region information corresponding to the frame to be rendered based on the valid target object region information associated with the preceding rendered frame and the subsequent cached frame, and perform image extraction on the frame to be rendered based on the calculation result to obtain the image frame data of the target object.

[0174] In some embodiments, the rendering module 704 is specifically used to: determine whether the current frame is associated with the image frame data of the target object; if the current frame is associated with the image frame data of the target object, then render the display screen of the current frame from the perspective of the target object based on the image frame data of the current frame;

[0175] In some embodiments, the rendering module 704 is further configured to: if the current frame is not associated with image frame data of the target object, render based on a set image to obtain the display screen of the current frame from the perspective of the target object.

[0176] It should be noted that the video display device provided in the above embodiments is only illustrated by the division of the above-described program modules when displaying video. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the video display device and the video display method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0177] Based on the hardware implementation of the above program modules, and in order to implement the video stream processing method applied to the live streaming system device in this application embodiment, this application embodiment also provides a live streaming system device, such as... Figure 8 As shown, the live streaming system device 800 includes at least one processor 801, a memory 802, a user interface 803, and at least one network interface 804. The various components in the live streaming system device 800 are coupled together via a bus system 805. It can be understood that the bus system 805 is used to implement communication between these components. In addition to a data bus, the bus system 805 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general labeled all buses as Bus System 805.

[0178] The user interface 803 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0179] The memory 802 in this embodiment is used to store various types of data to support the operation of the live streaming system device 800. Examples of such data include any computer program used to operate on the live streaming system device 800.

[0180] The video stream processing method for live streaming system equipment disclosed in this application can be applied to or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the video stream processing method can be completed by the integrated logic circuits in the hardware of the processor 801 or by instructions in software form. The processor 801 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically in memory 802. The processor 801 reads information from memory 802 and, in conjunction with its hardware, completes the steps of the video stream processing method for live streaming system equipment provided in the embodiments of this application.

[0181] In an exemplary embodiment, the live streaming system device 800 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned video stream processing method.

[0182] It is understood that memory 802 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), EEPROM, ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memory 802 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0183] Based on the hardware implementation of the above program modules, and in order to implement the video display method applied to the playback device in this application embodiment, this application embodiment also provides a playback device, such as... Figure 9 As shown, the playback device 900 includes at least one processor 901, a memory 902, a user interface 903, and at least one network interface 904. The various components in the playback device 900 are coupled together via a bus system 905. It can be understood that the bus system 905 is used to implement communication between these components. In addition to a data bus, the bus system 905 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 9 The general labeled all buses as Bus System 905.

[0184] The user interface 903 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0185] The memory 902 in this embodiment is used to store various types of data to support the operation of the playback device 900. Examples of such data include any computer program used to operate on the playback device 900.

[0186] The video display method for a playback device disclosed in this application can be applied to, or implemented by, a processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the video display method can be completed by integrated logic circuits in the hardware of the processor 901 or by instructions in software form. The processor 901 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, specifically a memory 902. The processor 901 reads information from the memory 902 and, in conjunction with its hardware, completes the steps of the video display method for a playback device provided in the embodiments of this application.

[0187] In an exemplary embodiment, the playback device 900 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned video display method.

[0188] It is understood that memory 902 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be ROM, PROM, EPROM, EEPROM, FRAM, Flash Memory, magnetic surface memory, optical disk, or read-only optical disk; magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as SRAM, SSRAM, DRAM, SDRAM, DDRSDRAM, ESDRAM, SLDRAM, and DRRAM. The memory 902 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0189] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 802 including a computer program, which can be executed by the processor 801 of the live streaming system device 800 to complete the steps described in the video stream processing method applied to the live streaming system device according to this application embodiment; and a memory 902 including a computer program, which can be executed by the processor 901 of the playback device 900 to complete the steps described in the video display method applied to the playback device according to this application embodiment. The computer-readable storage medium can be a ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0190] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 801 of a live streaming system device 800 and by a processor 901 of a playback device 900 to complete the steps described in the method of this application embodiment.

[0191] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0192] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video stream processing method, characterized in that, The method, applied to live streaming system equipment, includes: Obtain the first video stream data; Target object identification is performed on video frames in the first video stream data, and structural data representing the identification results is generated; The structured data is encapsulated into the first video stream data to obtain the second video stream data; Send the second video stream data to the playback device; Wherein, after the playback device parses the second video stream data, it renders the display screen corresponding to the perspective of the target object based on the obtained first video stream data and the structure data.

2. The method according to claim 1, characterized in that, The step of identifying target objects in video frames of the first video stream data and generating structured data representing the identification results includes: Based on preset detection rules, the target frame is determined from the first video stream data; Identify the image region of the target object in the target frame, and generate the structural data corresponding to the target frame based on the identified image region; The structural data includes at least: frame number and target object region information.

3. The method according to claim 2, characterized in that, The step of encapsulating the structured data into the first video stream data to obtain the second video stream data includes: Based on the frame sequence number in the structure data, establish an association mapping relationship between each structure data and the corresponding video frame in the first video stream data; Based on the aforementioned association mapping relationship, the structured data is encapsulated into the first video stream data to obtain the second video stream data.

4. A video display method, characterized in that, Applied to a playback device, the method includes: Receive the second video stream data; The second video stream data is parsed to obtain first video stream data and structural data; the structural data represents the target object recognition result of the video frame in the first video stream data. Based on the structured data, image extraction is performed on the video frames in the first video stream data to obtain image frame data of the target object; Based on the image frame data of the target object, render the display screen corresponding to the viewpoint of the target object.

5. The method according to claim 4, characterized in that, The step of extracting images from video frames in the first video stream data based on the structured data to obtain image frame data of the target object includes: Based on the frame sequence number in the structure data, determine whether each video frame in the first video stream data is associated with the structure data; If the frame to be rendered is associated with the structural data, then based on the valid target object region information in the structural data, the frame to be rendered is image extracted to obtain the image frame data of the target object; If the frame to be rendered is not associated with the structural data, then based on the valid target object region information associated with the preceding rendered frame and the subsequent cached frame, the target object region information corresponding to the frame to be rendered is calculated, and the image of the frame to be rendered is extracted based on the calculation result to obtain the image frame data of the target object.

6. The method according to claim 5, characterized in that, The method for rendering the display screen corresponding to the viewpoint of the target object based on the image frame data of the target object includes: Determine whether the current frame is associated with the image frame data of the target object; If the current frame is associated with the image frame data of the target object, then based on the image frame data of the current frame, the display screen of the current frame from the perspective of the target object is rendered; The method further includes: If the current frame is not associated with the image frame data of the target object, rendering is performed based on the set image to obtain the display screen of the current frame from the perspective of the target object.

7. A live streaming system device, characterized in that, The live streaming system device includes: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method according to any one of claims 1-3.

8. A playback device, characterized in that, The playback device includes: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the steps of the method according to any one of claims 4-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images