Progress content preview processing method, device and system

Abstract

Embodiments of the present application disclose a progress content preview processing method, device and system, the method comprising: acquiring live video data, generating each content preview image according to the live video data, and splicing each content preview image to obtain a spliced image corresponding to the live video data; recording a mapping relationship between each content preview image and each video time point; in the process of pulling the live video data at the client side, sending the spliced image corresponding to the live video data and image time mapping data corresponding to the spliced image to the client side, so that the client side analyzes the spliced image and the image time mapping data corresponding to the spliced image, determines and displays a content preview image corresponding to a historical video time point, so that the user can see the content preview image of different historical video time points of the live video data, the user can quickly know the picture content at the live time shift, and the user can quickly locate to the actual desired time shift position.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of Internet, in particular to a progress content preview processing method, device and system. BACKGROUND

[0002] The HLS protocol is used for real-time audio and video stream transmission. Currently, the HLS protocol is widely used in the fields of video on demand and live broadcast. Based on the HLS protocol system, a live broadcast time shift function can be provided. The live broadcast time shift function allows a user watching a live broadcast to backtrack to a historical time point from a current time point and continue watching from the historical time point. In the live broadcast time shift, the user usually positions to the content to be watched by using the time of a progress bar.

[0003] However, in the live broadcast time shift, the user does not know the specific picture content of the time shift when clicking the progress bar, and can only position to the specific position by using the time of the progress bar, which causes the user to need to click multiple times to find the content to be watched, and greatly affects the user experience of the live broadcast time shift. SUMMARY

[0004] In view of the above problems, the present application provides a progress content preview processing method, device, system, computing device and computer storage medium, which are used to solve the following problem: the existing live broadcast time shift can only position to the specific position by using the time of the progress bar, and cannot display the specific picture content of the time shift, which causes the user to be unable to quickly position to the content to be watched.

[0005] According to an aspect of an embodiment of the present application, a progress content preview processing method is provided, comprising:

[0006] Live broadcast video data is obtained, each content preview image is generated according to the live broadcast video data, and each content preview image is spliced to obtain a spliced image corresponding to the live broadcast video data;

[0007] A mapping relationship between each content preview image and each video time point is recorded to form image time mapping data corresponding to the spliced image;

[0008] In the process of pulling the live broadcast video data by the client, the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image are sent to the client, so that the client analyzes the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image in response to a progress preview request of the user for a historical video time point of the live broadcast video data, and determines and displays a content preview image corresponding to the historical video time point.

[0009] Further, generating each content preview image according to the live broadcast video data further comprises:

[0010] extracting key frames from the live video data, to obtain each key frame;

[0011] generating each content preview image using each key frame.

[0012] Further, the live video data comprises: a full-frame compressed encoding frame, a forward prediction encoding frame, and a bidirectional prediction interpolation encoding frame.

[0013] extracting key frames from the live video data, to obtain each key frame further comprises:

[0014] detecting whether a current frame in the live video data is a full-frame compressed encoding frame;

[0015] if the current frame is a full-frame compressed encoding frame, extracting the current frame from the live video data, and determining the current frame as a key frame.

[0016] Further, generating each content preview image using each key frame further comprises:

[0017] for each key frame in each key frame, performing resolution compression processing on a frame image of the key frame, to compress the resolution of the frame image to a preset resolution, to obtain a content preview image corresponding to the key frame.

[0018] Further, the method further comprises:

[0019] writing the image time mapping data corresponding to the spliced image and the preset resolution into a description file.

[0020] Further, splicing each content preview image to obtain a spliced image corresponding to the live video data further comprises:

[0021] adding each content preview image into a preset image container in sequence according to the generation order of the content preview images, and setting position information of each content preview image in the preset image container, to obtain a spliced image corresponding to the live video data.

[0022] Further, recording the mapping relationship between each content preview image and each video time point to form image time mapping data corresponding to the spliced image further comprises:

[0023] obtaining a video time point of each key frame in the live video data, recording a mapping relationship between an image identifier of a content preview image corresponding to each key frame and the video time point of each key frame in the live video data, to form image time mapping data corresponding to the spliced image.

[0024] Further, in the process of pulling the live video data by the client, sending the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client further comprises:

[0025] In response to the current live broadcast play request for the live room sent by the client, a request time point corresponding to the current live broadcast play request is determined;

[0026] The live video data of the live room acquired after the request time point, the spliced image corresponding to the live video data generated after the request time point, and the image time mapping data corresponding to the spliced image are sent to the client in real time.

[0027] According to another aspect of the embodiment of the present application, a progress content preview processing method is provided, comprising:

[0028] Pulling and storing live video data, spliced images corresponding to the live video data, and image time mapping data corresponding to the spliced images from a server;

[0029] Displaying the live video data;

[0030] In response to a progress preview request of a user for a historical video time point of the live video data, analyzing the spliced images corresponding to the live video data and the image time mapping data corresponding to the spliced images, and determining a content preview image corresponding to the historical video time point;

[0031] Displaying the content preview image corresponding to the historical video time point.

[0032] Further, pulling and storing live video data, spliced images corresponding to the live video data, and image time mapping data corresponding to the spliced images from a server further comprises:

[0033] In response to a live room entry operation of a user, sending a current live broadcast play request for the live room to the server, so that the server determines a request time point corresponding to the current live broadcast play request in response to the current live broadcast play request;

[0034] Receiving and storing live video data of the live room acquired after the request time point, spliced images corresponding to the live video data generated after the request time point, and image time mapping data corresponding to the spliced images sent by the server.

[0035] Further, analyzing the spliced images corresponding to the live video data and the image time mapping data corresponding to the spliced images, and determining a content preview image corresponding to the historical video time point further comprises:

[0036] Searching for an image identifier having a mapping relationship with the historical video time point from the image time mapping data corresponding to the spliced image;

[0037] Extracting a content preview image corresponding to the image identifier from the spliced image corresponding to the live video data to determine the content preview image corresponding to the historical video time point.

[0038] Further, before parsing the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, and determining the content preview image corresponding to the historical video time point, the method further comprises:

[0039] judging whether the historical video time point is located before the request time point corresponding to the current live play request;

[0040] If the historical video time point is located before the request time point corresponding to the current live play request, the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image are re-pulled from other clients or a server in the same client domain.

[0041] Further, re-pulling the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image from other clients or a server in the same client domain further comprises:

[0042] requesting, by a point-to-point communication mode, other clients in the same client domain to obtain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image;

[0043] If the other clients in the same client domain do not contain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image, the server is requested to obtain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image.

[0044] Further, the same client domain refers to an organizational unit formed by clients served by the same edge service node.

[0045] Further, displaying the content preview image corresponding to the historical video time point further comprises:

[0046] displaying the content preview image corresponding to the historical video time point at a preset position of a progress display element in a live room display page.

[0047] According to another aspect of the embodiments of the present application, a progress content preview processing apparatus is provided, comprising:

[0048] a splicing module configured to obtain live video data, generate each content preview image according to the live video data, and splice each content preview image to obtain a spliced image corresponding to the live video data;

[0049] a mapping module configured to record a mapping relationship between each content preview image and each video time point to form image time mapping data corresponding to the spliced image.

[0050] The processing module is configured to, in the process of the client pulling the live video data, send the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client, so that the client, in response to a progress preview request of a user for a historical video time point of the live video data, parses the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, determines and displays a content preview image corresponding to the historical video time point.

[0051] According to still another aspect of the embodiments of the present application, a progress content preview processing apparatus is provided, comprising:

[0052] The pulling module is configured to pull and store, from the server, the live video data, the spliced image corresponding to the live video data, and the image time mapping data corresponding to the spliced image.

[0053] The live video display module is configured to display the live video data.

[0054] The parsing module is configured to, in response to a progress preview request of a user for a historical video time point of the live video data, parse the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, and determine a content preview image corresponding to the historical video time point.

[0055] The preview module is configured to display the content preview image corresponding to the historical video time point.

[0056] According to another aspect of the embodiments of the present application, a progress content preview processing system is provided, comprising the progress content preview processing apparatus described above.

[0057] According to still another aspect of the embodiments of the present application, a computing device is provided, comprising a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface being in communication with each other through the communication bus.

[0058] The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the progress content preview processing method described above.

[0059] According to still another aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the progress content preview processing method described above.

[0060] According to the progress content preview processing method, device and system provided by the embodiment of the application, the live video data is acquired, each content preview image is generated according to the live video data, and each content preview image is spliced to obtain a spliced image corresponding to the live video data; a mapping relationship between each content preview image and each video time point is recorded to form image time mapping data corresponding to the spliced image; and in the process of pulling the live video data at the client side, the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image are sent to the client side. In the HLS slice production process, the content preview image is produced in real time, and the plurality of content preview images are spliced in the form of a spliced image in the form of a sprite, the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image are provided to the client side together, the client side analyzes the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image in response to the progress preview request of the user for the historical video time point of the live video data, determines and displays the content preview image corresponding to the historical video time point, and the live time shift content preview image is provided to the user, the user can see the content preview image of the different historical video time points of the live video data, the user can quickly know the picture content of the live time shift, and the user can quickly locate to the time shift position actually wanted to see.

[0061] The above description is only a summary of the technical solutions of the embodiments of the application, in order to more clearly understand the technical means of the embodiments of the application, the embodiments of the application can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the embodiments of the application more obvious and easy to understand, the following specific embodiments of the embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to further aid the full and complete understanding of the embodiments of the application. Throughout the drawings, like reference numerals refer to like parts. In the drawings:

[0063] Figure 1 A flowchart of a progress content preview processing method according to an embodiment of the application is shown;

[0064] Figure 2a A flowchart of a progress content preview processing method according to another embodiment of the application is shown;

[0065] Figure 2b A page display schematic diagram is shown;

[0066] Figure 3A structural block diagram of a progress content preview processing apparatus according to an embodiment of the present application is shown;

[0067] Figure 4 A structural block diagram of a progress content preview processing apparatus according to another embodiment of the present application is shown;

[0068] Figure 5 A structural schematic diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] Exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0070] First, the nomenclature related to one or more embodiments of the present application is explained.

[0071] Live broadcast: generally refers to network video live broadcast, and live broadcast in a broad sense also includes live broadcast of television stations. Live audio and video are pushed to the server in the form of a media stream (i.e., push stream). If there are viewers watching the live broadcast, the server will transmit the video to the website, the application (APP) of the client, or the player of the client, etc., to play the video in real time upon receiving the request of the user.

[0072] Video on demand: a video on demand system that plays programs according to the requirements of viewers. The video content selected or clicked by the user is transmitted to the user who makes the request.

[0073] HTTP Live Streaming (HLS): a dynamic code rate adaptive technology; used for transmission of real-time audio and video streams. The HLS protocol is widely used in the fields of video on demand and live broadcast. The working principle of the HLS protocol is to divide the entire stream into small HTTP-based files for download, and only download some of them each time. The HLS protocol includes an m3u8 index file.

[0074] m3u8 file: an index file. An audio and video file is usually divided into several small files, and then the addresses and playing order of each file are recorded by m3u8.

[0075] Live time shift: a technology that allows viewers to watch past live content at any time during the live broadcast. It is a technology that combines live broadcast and on demand. Viewers can backtrack to a certain time point and watch from that time point until the end of the video content.

[0076] Sprite: also known as a snow picture, a spliced picture, many large web pages need to load many small pictures at the first time of loading, and considering the congestion of the server at the same time, in order to solve this problem, the sprite technology is adopted to alleviate the problem of too long loading time affecting user experience. In computer graphics, when a two-dimensional image is integrated into a scene and becomes part of the entire display image, the image is called a sprite. In this embodiment, the image obtained by splicing each content preview image is called a spliced image.

[0077] JS object score (JavaScript Object Notation, JSON): a lightweight data exchange format, its content is composed of attributes and values, so it also has the advantages of easy reading and processing. JSON is a data format independent of programming language, which is not only a subset of JavaScript, but also uses the habits of the C language family, and its widespread use makes it a general data format.

[0078] Peer to Peer (P2P): belongs to the category of overlay network (Overlay Network), is a network information exchange method compared with client / server (C / S) mode, in C / S mode, the data distribution adopts a special server, and multiple clients obtain data from the server.

[0079] Figure 1 A flowchart of a progress content preview processing method according to an embodiment of the application is shown, which is suitable for a server, as shown in the figure, the method comprises the following steps: Figure 1

[0080] Step S101: obtaining live video data, and generating each content preview image according to the live video data.

[0081] In this embodiment, in the process of applying the HLS protocol to transmit live video data, since the working principle of the HLS protocol is to divide the entire stream into small slices (i.e. HTTP-based files) for downloading, the live video data can be obtained by the server in the process of HLS producing slices, and each content preview image can be generated in real time according to the live video data.

[0082] In an optional implementation, step S101 further comprises: performing key frame extraction on the live video data to obtain each key frame; and generating each content preview image by using each key frame.

[0083] ​In the embodiment, the server acquires live video data, extracts key frames from the live video data, and generates each content preview image using each key frame.

[0084] In an optional embodiment, the live video data includes an intra picture (I-frame), a predictive-frame (P-frame), and a bi-directional interpolated prediction frame (B-frame). Specifically, in video compression, each frame represents a still image. In actual video compression encoding, various algorithms are used to reduce the data capacity. The IPB frame is the most common algorithm. The IPB frame includes an I-frame, a P-frame, and a B-frame. The I-frame is usually the first frame of each group of pictures (GOP). After moderate compression, the I-frame can be used as a random access reference point and can be regarded as a static image. The P-frame removes the temporal redundancy of the previously encoded frames in the image sequence to compress the data volume of the encoded image. The B-frame considers the temporal redundancy between the previously encoded frames and the subsequently encoded frames in the source image sequence to compress the data volume of the encoded image.

[0085] From the perspective of coding and decoding, the I-frame can be decoded into a single complete video frame by a video decompression algorithm. Therefore, the I-frame removes the spatial redundancy of the video frame. The P-frame needs to refer to the previous I-frame or P-frame to decode a complete video frame. The B-frame needs to refer to the previous I-frame or P-frame and the subsequent P-frame to generate a complete video frame. Therefore, the P-frame and the B-frame remove the temporal redundancy of the video frame.

[0086] In an optional embodiment, the step S101 further includes detecting whether the current frame in the live video data is an I-frame. If the current frame is an I-frame, the current frame is extracted from the live video data, and the current frame is determined as a key frame.

[0087] Specifically, the encoder encodes multiple images to produce a segment of GOPs. The decoder reads the segment of GOPs to decode and read the picture and then renders and displays the picture. Since the I-frame can be decoded into a single complete video frame by a video decompression algorithm, that is, the I-frame is a complete picture, and the P-frame and the B-frame record changes relative to the I-frame. If there is no I-frame, the P-frame and the B-frame cannot be decoded. Therefore, in the embodiment, the I-frame is determined as a key frame.

[0088] In an optional implementation, step S101 further includes: for each key frame in each key frame, performing resolution compression processing on the frame image of the key frame, compressing the resolution of the frame image to a preset resolution, and obtaining the content preview image corresponding to the key frame.

[0089] Specifically, resolution affects image size. Resolution is directly proportional to image size; that is, the higher the resolution, the larger the image, and the lower the resolution, the smaller the image. In this embodiment, in order to facilitate loading each content preview image, for each keyframe, the frame image of the keyframe needs to be compressed in resolution. For example, the resolution can be preset to 24*24 and written to the description file, so that the resolution of the frame image can be compressed to 24*24, and the compressed image can be used as the content preview image corresponding to the keyframe.

[0090] Step S102: Stitch together the preview images of each content to obtain the stitched image corresponding to the live video data.

[0091] In this process, each content preview image can be added to a preset image container in the order in which they are generated, and the position information of each content preview image in the preset image container can be set to obtain the stitched image corresponding to the live video data.

[0092] In this embodiment, the preset image container can be the background canvas of sprite sheet technology. When a user moves the desired content during live streaming through the client, the user can display the part they want to show by moving the small images in the background canvas, thereby relieving processing pressure. The principle of sprite sheet technology is to merge many small images into one larger image. Therefore, when the page is loaded for the first time, only the large image that merges the small images, i.e., the sprite sheet, needs to be loaded. This reduces the page loading speed to a certain extent and also alleviates the image processing pressure to a certain extent.

[0093] Step S103: Record the mapping relationship between each content preview image and each video time point to form the image time mapping data corresponding to the stitched image.

[0094] Specifically, the video time points of each keyframe in the live video data are obtained, and the mapping relationship between the image identifier of the content preview image corresponding to each keyframe and the video time points of each keyframe in the live video data is recorded to form the image time mapping data corresponding to the stitched image.

[0095] In addition, image time mapping data corresponding to the spliced image and a preset resolution can also be written into the description file. The description file can be a description file in JSON format. Based on the advantages of easy reading and processing, the image time mapping data corresponding to the spliced image and the preset resolution and other data are written in. The image time mapping data corresponding to the spliced image refers to mapping data between each content preview image in the spliced image and corresponding playing time in the entire live video data (i.e., time corresponding to a progress display element such as a progress bar).

[0096] Specifically, a mapping relationship between an image identifier of each content preview image corresponding to a key frame and a video time point of each key frame in the live video data can be recorded to form image time mapping data corresponding to the spliced image, and the image time mapping data corresponding to the spliced image is written into a description file (such as a JSON file). When the client needs to pull live time-shifted content, the server can send the spliced image and the JSON file to the client together.

[0097] Step S104: In the process of pulling the live video data by the client, the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image are sent to the client.

[0098] In this step, in the process of watching the live broadcast, the server can send the live video data, the spliced image corresponding to the live video data, and the image time mapping data corresponding to the spliced image to the client. If a user initiates a progress preview request for a historical video time point of the live video data, the client can analyze the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, determine and display the content preview image corresponding to the historical video time point. The progress preview request can be an operation performed by the user on a progress display element of the live video.

[0099] In an optional implementation, step S104 further includes: in response to a current live broadcast playing request for the live room sent by the client, determining a request time point corresponding to the current live broadcast playing request; and sending the live video data of the live room acquired after the request time point, the spliced image corresponding to the live video data of the live room generated after the request time point, and the image time mapping data corresponding to the spliced image to the client in real time.

[0100] If the user enters the live broadcast room when the live broadcast has not started, the client obtains the live video data starting from the live broadcast start time, the spliced image corresponding to the live video data, and the image time mapping data corresponding to the spliced image after the live broadcast starts. If the user enters the live broadcast room when the live broadcast has started, the live video data, the spliced image, and the image time mapping data generated before can be transmitted to the client, thereby reducing the data transmission amount, relieving the server pressure, and transmitting the live video data of the live broadcast room obtained after the request time point, the spliced image generated after the request time point, and the image time mapping data corresponding to the spliced image to the client in real time.

[0101] According to the progress content preview processing method provided in the embodiments of the present application, the content preview image can be produced in real time in the HLS slice production process, and the plurality of content preview images are spliced together in the form of a spliced image in the form of a sprite. The spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image are provided to the client, so that the client can analyze the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image in response to the progress preview request of the user for the historical video time point of the live video data, determine and display the content preview image corresponding to the historical video time point, and provide the live time shift content preview image to the user. The user can see the content preview image at different historical video time points of the live video data, quickly know the picture content during the live time shift, and quickly locate the time shift position that the user actually wants to see.

[0102] Figure 2a A flowchart of a progress content preview processing method according to another embodiment of the present application is shown, which is suitable for a client. As shown in the figure, the method includes the following steps: Figure 2a

[0103] Step S201: pulling and storing the live video data, the spliced image corresponding to the live video data, and the image time mapping data corresponding to the spliced image from the server.

[0104] In this step, when the user watches the live broadcast through the client, the client can pull and store the live video data from the server to the local storage device. The server sends the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client during the process of pulling the live video data from the client.

[0105] ​In order to reduce the data transmission amount and relieve the pressure of the server, in an optional implementation, step S201 further includes: in response to the user's live room entering operation, sending a current live play request for the live room to the server, so that the server determines a request time point corresponding to the current live play request in response to the current live play request; receiving and storing the live video data of the live room acquired after the request time point, the spliced image corresponding to the live video data generated after the request time point, and the image time mapping data corresponding to the spliced image sent by the server.

[0106] Step S202: displaying the live video data.

[0107] Specifically, the user can enter the live room through a player or the like of the client, the server sends a current live play request for the live room to the server in response to the user's live room entering operation, receives and stores the live video data of the live room acquired after the request time point, the spliced image corresponding to the live video data generated after the request time point, and the image time mapping data corresponding to the spliced image sent by the server, and displays the live video data to the user in the live room display page through the player or the like according to the received live video data.

[0108] Step S203: in response to the user's progress preview request for a historical video time point of the live video data, determining whether the historical video time point is before the request time point corresponding to the current live play request; if yes, performing step S204; if no, performing step S205.

[0109] Specifically, if the historical video time point is after the request time point corresponding to the current live play request, it means that the user wants to see the historical live video data that has been loaded after entering the live room, and the content preview image corresponding to the historical video time point has been included in the spliced image pulled by the client from the server, so it is not necessary to pull again, and the content preview image can be directly parsed from the spliced image and the image time mapping data corresponding to the spliced image stored locally; if the request time point is after the historical video time point, it means that the user wants to see the historical live content that has been live broadcast before entering the live room but has not been watched by the user, and since the spliced image pulled by the client does not include the content preview image corresponding to the historical video time point, it is necessary to pull the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image again.

[0110] Step S204: pulling the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image from other clients in the same client domain or the server.

[0111] In an optional implementation, step S204 further includes: requesting, by a point-to-point communication mode, other clients in the same client domain to obtain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image; and if the other clients in the same client domain do not contain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image, requesting the server to obtain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image.

[0112] In this step, the point-to-point communication mode can include a P2P communication mode, and if the historical video time point is before the request time point corresponding to the current live broadcast playing request, it indicates that the user wants to watch the historical live broadcast content that has been live broadcast before the user enters the live broadcast room but has not been watched by the user. Since the spliced image pulled by the client does not contain the content preview image corresponding to the historical video time point, the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image need to be pulled again. The spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image can be pulled again from other clients in the same client domain by the P2P communication mode, or the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image can be pulled again from the server.

[0113] The same client domain refers to an organizational unit formed by clients served by the same edge service node. Other clients in the same client domain as the client can also be referred to as neighboring clients of the client, and the live broadcast video data corresponding to the spliced image and the image time mapping data corresponding to the spliced image can be more quickly obtained by querying other clients served by the same edge service node. If the other clients in the same client domain do not contain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image, the server can be requested to obtain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image. This mode can improve the rate and accuracy of pulling the spliced image and the image time mapping data corresponding to the spliced image. The spliced image containing the content preview image corresponding to the historical video time point can be a spliced image obtained by continuously splicing each content preview image generated from real-time live broadcast video data from a live broadcast starting time point.

[0114] Step S205: Analyzing the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image to determine the content preview image corresponding to the historical video time point.

[0115] The image identifier corresponding to the historical video time point is found from the image time mapping data corresponding to the spliced image; and the content preview image corresponding to the image identifier is extracted from the spliced image corresponding to the live video data to determine the content preview image corresponding to the historical video time point.

[0116] After pulling the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, the client needs to parse the spliced image and the image time mapping data corresponding to the spliced image to obtain the content preview image corresponding to the historical video time point. Specifically, each content preview image is represented by an image identifier thereof, and the mapping relationship between the image identifier of each content preview image and each video time point is recorded in the image time mapping data. The image identifier of the content preview image corresponding to the historical video time point is found by combining the video time point recorded in the image time mapping data corresponding to the spliced image, and the content preview image corresponding to the image identifier is found from the spliced image.

[0117] Step S206: The content preview image corresponding to the historical video time point is displayed.

[0118] The client can display the content preview image corresponding to the historical video time point at a preset position of a progress display element such as a progress bar in the live room display page through a player. The preset position can be a position above the progress display element. Figure 2b A page display schematic diagram is shown, as shown in Figure 2b The content preview image corresponding to the historical video time point can be displayed in a small window above the progress bar of the live room display page.

[0119] According to the progress content preview processing method provided by the embodiment of the present application, the client analyzes the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image provided by the server in response to the progress preview request of the user for the historical video time point of the live video data, determines and displays the content preview image corresponding to the historical video time point, and provides the live time shift content preview image to the user, so that the user can see the content preview image of different historical video time points of the live video data, and quickly know the picture content when the live time shift is performed, thereby facilitating the user to quickly locate to the time shift position actually intended to watch.

[0120] Figure 3 A structural block diagram of a progress content preview processing device according to an embodiment of the present application is shown, which is suitable for a server, as shown in Figure 3 The device includes a splicing module 301, a mapping module 302, and a processing module 303.

[0121] The splicing module 301 is configured to acquire live video data, generate each content preview image according to the live video data, splice the content preview images to obtain a spliced image corresponding to the live video data.

[0122] In an optional implementation, the splicing module 301 is further configured to perform key frame extraction on the live video data to obtain each key frame, and generate each content preview image by using the key frame.

[0123] In an optional implementation, the live video data includes a full-frame compression encoded frame, a forward prediction encoded frame and a bidirectional prediction interpolation encoded frame, and the splicing module 301 is further configured to detect whether a current frame in the live video data is the full-frame compression encoded frame, and if the current frame is the full-frame compression encoded frame, extract the current frame from the live video data and determine the current frame as a key frame.

[0124] In an optional implementation, the splicing module 301 is further configured to, for each key frame in the key frames, perform resolution compression processing on a frame image of the key frame to compress the resolution of the frame image to a preset resolution, and obtain a content preview image corresponding to the key frame.

[0125] In an optional implementation, the splicing module 301 is further configured to write image time mapping data corresponding to the spliced image and the preset resolution into a description file.

[0126] In an optional implementation, the splicing module 301 is further configured to add each content preview image into a preset image container in sequence according to a generation order of the content preview images, and set position information of each content preview image in the preset image container to obtain the spliced image corresponding to the live video data.

[0127] The mapping module 302 is configured to record a mapping relationship between each content preview image and each video time point to form image time mapping data corresponding to the spliced image.

[0128] In an optional implementation, the mapping module 302 is further configured to acquire a video time point of each key frame in the live video data, record a mapping relationship between an image identifier of each content preview image corresponding to the key frame and the video time point of the key frame in the live video data, and form the image time mapping data corresponding to the spliced image.

[0129] The processing module 303 is configured to, in the process of pulling the live video data by the client, send the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client, so that the client, in response to a progress preview request of a user for a historical video time point of the live video data, parses the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, determines and displays a content preview image corresponding to the historical video time point.

[0130] In an optional implementation, the processing module 303 is further configured to, in response to a current live broadcast play request of the client for the live room, determine a request time point corresponding to the current live broadcast play request, and send, to the client in real time, live video data of the live room acquired after the request time point, a spliced image corresponding to the live video data generated after the request time point, and image time mapping data corresponding to the spliced image.

[0131] The above modules are described with reference to the corresponding descriptions in the method embodiments, and thus will not be described herein again.

[0132] According to the progress content preview processing apparatus provided in the embodiments of the present application, the content preview image can be produced in real time in the HLS slice production process, and a plurality of content preview images can be spliced together in the form of a spliced image in the form of a sprite, so that the client is provided with the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, so that the client, in response to a progress preview request of a user for a historical video time point of the live video data, can parse the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, determine and display a content preview image corresponding to the historical video time point, thereby providing the user with a live time shift content preview image, allowing the user to see the content preview image at different historical video time points of the live video data, allowing the user to quickly know the screen content at the live time shift, and facilitating the user to quickly locate to the time shift position actually intended to watch.

[0133] Figure 4 A structural block diagram of a progress content preview processing apparatus according to another embodiment of the present application is shown, which is suitable for a client, as shown in the figure, the apparatus comprises a pulling module 401, a live video display module 402, an analysis module 403, and a preview module 404. Figure 4

[0134] The pulling module 401 is configured to pull and store, from a server, live video data, a spliced image corresponding to the live video data, and image time mapping data corresponding to the spliced image.

[0135] ​In an optional implementation, the pulling module 401 is further configured to: in response to a live room entry operation of the user, send a current live play request for the live room to the server, so that the server determines a request time point corresponding to the current live play request in response to the current live play request; and receive and store live video data of the live room acquired after the request time point, a spliced image corresponding to the live video data generated after the request time point, and image time mapping data corresponding to the spliced image sent by the server.

[0136] The live video display module 402 is configured to display the live video data.

[0137] The parsing module 403 is configured to, in response to a progress preview request of the user for a historical video time point of the live video data, parse the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image, and determine a content preview image corresponding to the historical video time point.

[0138] In an optional implementation, the parsing module 403 is further configured to: find an image identifier having a mapping relationship with the historical video time point from the image time mapping data corresponding to the spliced image; and determine the content preview image corresponding to the historical video time point as an image corresponding to the image identifier extracted from the spliced image corresponding to the live video data.

[0139] In an optional implementation, the parsing module 403 is further configured to: determine whether the historical video time point is before a request time point corresponding to the current live play request; and if the historical video time point is before the request time point corresponding to the current live play request, re-pull the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image from other clients in a same client domain or the server.

[0140] In an optional implementation, the parsing module 403 is further configured to: request, by a point-to-point communication mode, other clients in a same client domain to acquire the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image; and if the other clients in the same client domain do not contain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image, request the server to acquire the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image.

[0141] In an optional implementation, the same client domain refers to an organizational unit formed by clients served by a same edge service node.

[0142] The preview module 404 is configured to display the content preview image corresponding to the historical video time point.

[0143] In an optional implementation, the preview module 404 is further configured to: display a preview image of the content corresponding to a historical video time point at a preset position of the progress display element in the live broadcast room display page.

[0144] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0145] According to an embodiment of this application, a progress content preview processing device is provided. In response to a user's progress preview request for historical video time points of live video data, the client parses the stitched image corresponding to the live video data and the image time mapping data corresponding to the stitched image provided by the server, determines and displays the content preview image corresponding to the historical video time point, and realizes the provision of live time-shifted content preview images to users. This allows users to see the content preview images of different historical video time points of the live video data, enabling users to quickly know the screen content during live time-shifting and making it easy for users to quickly locate the time-shifted position they actually want to see.

[0146] This application also provides a progress content preview processing system, which includes the progress content preview processing device applied to the server and the progress content preview processing device applied to the client shown in the above embodiments.

[0147] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can execute a progress content preview processing method in any of the above method embodiments.

[0148] Figure 5 The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0149] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0150] in:

[0151] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0152] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0153] The processor 502 is configured to execute the program 510, and particularly configured to execute the steps in the progress content preview processing method embodiments.

[0154] The program 510 can include program codes including computer operation instructions.

[0155] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the computing device can be of the same type, such as one or more CPUs, or can be of different types, such as one or more CPUs and one or more ASICs.

[0156] The memory 506 is configured to store the program 510. The memory 506 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0157] The program 510 can be specifically configured to enable the processor 502 to execute the progress content preview processing method in any of the method embodiments described above. The specific implementation of each step in the program 510 can refer to the corresponding description in the corresponding step and unit in the progress content preview processing method embodiments described above, and will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, which will not be described here.

[0158] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, based on the description as set forth above. In terms of structure, those skilled in the art will appreciate that the required structure for a variety of these general purpose systems follows from the description above. In addition, embodiments of the present application are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present application as described herein, and any references below to specific languages are provided for disclosure of the best mode of the present application.

[0159] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not described in detail in order not to obscure the understanding of the specification.

[0160] Similarly, it is to be understood that the embodiments of the application can sometimes alternately be termed, discussed, or described in the specification in terms of a process, an apparatus, a system, or a method. Moreover, where a process claim is involved, it will be understood that such claim can be embodied strictly as a process, strictly as an apparatus or system, strictly as a machine or device, strictly as an article of manufacture, or strictly as a composition of matter.

[0161] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and all the processes or units of any method or apparatus disclosed thus can be adopted. Unless explicitly stated, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar function.

[0162] Further, those skilled in the art will appreciate that a combination of features of different embodiments means within the scope of the embodiments of the application and forms different embodiments, although some embodiments herein include certain features rather than others included in other embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0163] The various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Skilled persons should appreciate that a microprocessor or a digital signal processor (DSP) can be used to implement some or all of the functions of some or all of the components according to some or all of the embodiments of the present application in practice. The embodiments of the present application can also be implemented as a device or apparatus program (for example, a computer program and a computer program product) for performing part or all of the methods described herein. The program implementing the embodiments of the present application can be stored in a computer readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0164] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', but it is to be understood that each of these devices can be implemented by its own hardware item. The use of the word 'at least' followed by a list of one or more members does not preclude the existence of additional such members. The word 'first' or'second' does not preclude the existence of additional such members.

Claims

1. A progress content preview processing method, comprising: obtaining live video data, generating each content preview image according to the live video data, and splicing the each content preview image to obtain a spliced image corresponding to the live video data; each content preview image is generated in real time according to the live video data in the process of HLS production slicing; recording a mapping relationship between the each content preview image and each video time point to form image time mapping data corresponding to the spliced image; in the process of pulling the live video data at the client side, sending the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client side, so that the client side determines and displays a content preview image corresponding to a historical video time point of the live video data in response to a progress preview request of a user for the historical video time point of the live video data by analyzing the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image; wherein, in the process of pulling the live video data at the client side, sending the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to the client side further comprises: determining a request time point corresponding to a current live broadcast play request of a live room in response to the current live broadcast play request sent by the client side; sending live video data of the live room obtained after the request time point, a spliced image corresponding to the live video data generated after the request time point, and image time mapping data corresponding to the spliced image to the client side in real time; the method further comprises: if the historical video time point is located before the request time point corresponding to the current live broadcast play request, the client side preferentially re-pulls the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image from other client sides in the same client domain through a point-to-point communication mode, wherein the same client domain refers to an organizational unit formed by client sides served by the same edge service node.

2. The method of claim 1, wherein, the generating each content preview image according to the live video data further comprises: extracting each key frame from the live video data; generating the each content preview image by using the each key frame.

3. The method of claim 2, wherein, the live video data comprises full-frame compression encoding frames, forward prediction encoding frames, and bidirectional prediction interpolation encoding frames; the extracting each key frame from the live video data further comprises: detecting whether a current frame in the live video data is a full-frame compression encoding frame; if the current frame is the full-frame compression encoding frame, extracting the current frame from the live video data and determining the current frame as the key frame.

4. The method of claim 2, wherein, the generating the each content preview image by using the each key frame further comprises: For each of the key frames, a frame image of the key frame is subjected to resolution compression processing to compress the resolution of the frame image to a preset resolution, to obtain a content preview image corresponding to the key frame.

5. The method of claim 4, wherein, The method further comprises: writing the image time mapping data corresponding to the spliced image and the preset resolution into a description file.

6. The method according to any one of claims 1 to 5, wherein, The splicing of the content preview images to obtain the spliced image corresponding to the live video data further comprises: adding the content preview images in sequence into a preset image container according to the generation order of the content preview images, and setting the position information of the content preview images in the preset image container, to obtain the spliced image corresponding to the live video data.

7. The method according to any one of claims 2-5, wherein, The recording of the mapping relationship between the content preview images and the video time points to form the image time mapping data corresponding to the spliced image further comprises: obtaining the video time points of the key frames in the live video data, and recording the mapping relationship between the image identifiers of the content preview images corresponding to the key frames and the video time points of the key frames in the live video data, to form the image time mapping data corresponding to the spliced image.

8. A progress content preview processing method, comprising: pulling and storing live video data, a spliced image corresponding to the live video data, and image time mapping data corresponding to the spliced image from a server; each content preview image in the spliced image is generated in real time according to the live video data in the process of HLS production slicing; displaying the live video data; in response to a progress preview request of a user for a historical video time point of the live video data, analyzing the spliced image corresponding to the live video data and the image time mapping data corresponding to the spliced image to determine a content preview image corresponding to the historical video time point; displaying the content preview image corresponding to the historical video time point; wherein the pulling and storing of the live video data, the spliced image corresponding to the live video data, and the image time mapping data corresponding to the spliced image from the server further comprises: in response to a live room entering operation of a user, sending a current live play request for the live room to the server, so that the server determines a request time point corresponding to the current live play request in response to the current live play request; receiving and storing the live video data of the live room obtained by the server after the request time point, the spliced image corresponding to the live video data generated after the request time point, and the image time mapping data corresponding to the spliced image; the method further comprises: If the historical video time point is located before a request time point corresponding to the current live broadcast playing request, the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image are preferentially pulled from other clients in a same client domain through a point-to-point communication mode, wherein the same client domain refers to an organizational unit formed by clients served by a same edge service node.

9. The method of claim 8, wherein, The analyzing the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image to determine the content preview image corresponding to the historical video time point further includes: finding, from the image time mapping data corresponding to the spliced image, an image identifier having a mapping relationship with the historical video time point; extracting, from the spliced image corresponding to the live broadcast video data, the content preview image corresponding to the image identifier as the content preview image corresponding to the historical video time point.

10. The method of claim 8, wherein, The preferentially pulling the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image from other clients in a same client domain through a point-to-point communication mode further includes: requesting, through the point-to-point communication mode, the other clients in the same client domain to acquire the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image; if the other clients in the same client domain do not contain the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image, requesting the server to acquire the spliced image containing the content preview image corresponding to the historical video time point and the image time mapping data corresponding to the spliced image.

11. The method according to any one of claims 8-10, wherein, The displaying the content preview image corresponding to the historical video time point further includes: displaying the content preview image corresponding to the historical video time point at a preset position of a progress display element in a live broadcast room display page.

12. A progress content preview processing apparatus, comprising: a splicing module configured to acquire live broadcast video data, generate each content preview image according to the live broadcast video data, and splice the each content preview image to obtain a spliced image corresponding to the live broadcast video data; each content preview image is generated in real time according to the live broadcast video data in a process of HLS production slicing; a mapping module configured to record a mapping relationship between the each content preview image and each video time point to form image time mapping data corresponding to the spliced image; a processing module configured to, in a process of pulling the live broadcast video data by a client, send the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image to the client, so that the client analyzes the spliced image corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced image to determine and display a content preview image corresponding to a historical video time point in response to a progress preview request of a user for the historical video time point of the live broadcast video data. The processing module is further configured to: in response to a current live broadcast play request for a live broadcast room sent by the client, determine a request time point corresponding to the current live broadcast play request; and send live broadcast video data of the live broadcast room acquired after the request time point, spliced images corresponding to the live broadcast video data generated after the request time point, and image time mapping data corresponding to the spliced images to the client in real time. If the historical video time point is before a request time point corresponding to the current live broadcast play request, the client preferentially re-pulls the spliced images corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced images from other clients in a same client domain through a point-to-point communication mode, wherein the same client domain refers to an organizational unit formed by clients served by a same edge service node.

13. A progress content preview processing apparatus, comprising: a pulling module configured to pull and store live broadcast video data, spliced images corresponding to the live broadcast video data, and image time mapping data corresponding to the spliced images from a server; each content preview image in the spliced images is generated in real time during HLS production slicing according to the live broadcast video data; a live broadcast video display module configured to display the live broadcast video data; an analysis module configured to, in response to a progress preview request of a user for a historical video time point of the live broadcast video data, analyze the spliced images corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced images, and determine a content preview image corresponding to the historical video time point; a preview module configured to display the content preview image corresponding to the historical video time point; The pulling module is further configured to: in response to a live broadcast room entry operation of the user, send a current live broadcast play request for the live broadcast room to the server, so that the server determines a request time point corresponding to the current live broadcast play request in response to the current live broadcast play request; and receive and store live broadcast video data of the live broadcast room acquired after the request time point, spliced images corresponding to the live broadcast video data generated after the request time point, and image time mapping data corresponding to the spliced images sent by the server. If the historical video time point is before a request time point corresponding to the current live broadcast play request, the client preferentially re-pulls the spliced images corresponding to the live broadcast video data and the image time mapping data corresponding to the spliced images from other clients in a same client domain through a point-to-point communication mode, wherein the same client domain refers to an organizational unit formed by clients served by a same edge service node.

14. A progress content preview processing system, comprising: The progress content preview processing apparatus of claim 12 and the progress content preview processing apparatus of claim 13.

15. A computing device comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface being in communication with each other through the communication bus; The memory is configured to store at least one executable instruction, and the executable instruction is configured to enable the processor to perform operations corresponding to the progress content preview processing method according to any one of claims 1-7.

16. A computer storage medium, having stored therein at least one executable instruction, and the executable instruction is configured to enable a processor to perform operations corresponding to the progress content preview processing method according to any one of claims 1-7.

17. A computing device comprising: The processor, the memory, the communication interface, and the communication bus are configured to perform communication among each other through the communication bus. The memory is configured to store at least one executable instruction, and the executable instruction is configured to enable the processor to perform operations corresponding to the progress content preview processing method according to any one of claims 8-11.

18. A computer storage medium, having stored therein at least one executable instruction, and the executable instruction is configured to enable a processor to perform operations corresponding to the progress content preview processing method according to any one of claims 8-11.

19. A computer program product, comprising at least one executable instruction, and the executable instruction is configured to enable a processor to perform operations corresponding to the progress content preview processing method according to any one of claims 1-7 or perform operations corresponding to the progress content preview processing method according to any one of claims 8-11.

Images