High-definition playing method based on video background separation

Abstract

The application discloses a high-definition playing method based on video background separation, comprising the following steps: S1, a streaming module pulls original data and sends the original data to a transcoding device; S2, the transcoding device performs secondary compression on the received original stream, extracts background content in the original stream and makes the background content into SEI segments; S3, the transcoding device transmits the secondary compressed data to the streaming module, at this time, the streaming module comprises the original stream and the compressed stream; S4, if a user watches a real-time stream, at this time, the compressed stream is transmitted to a client through a streaming media device, the client detects SEI, re-integrates the SEI segments, restores clear YUV background data, and superimposes the clear YUV background data on each frame of data of the compressed stream, so that the compressed stream is played in a clear state; the compressed stream enters a recording module and is stored, and in the recording process, an index module detects SEI frames carrying background data in real time and records indexes in disk storage, and the indexes correspond to recording segments.

Description

Technical Field

[0001] This invention belongs to the field of video encoding and decoding technology, and specifically relates to a high-definition playback method based on video background extraction. Background Technology

[0002] In the field of security video surveillance, whether it's live streaming or video-on-demand, for the videos being viewed, if the original video quality is high and there are many details, the bitrate is high, requiring more memory resources for storage and more bandwidth for transmission. Conversely, if the video is heavily compressed, the recording storage uses less memory, saving storage and bandwidth during live viewing, but the picture is blurry, affecting the viewing of details. Furthermore, video surveillance footage generally doesn't move frequently, so always transmitting clear, high-quality images actually wastes a lot of bandwidth; in weak network environments, data packet loss and latency can also occur. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention provides a high-definition playback method based on video background extraction, which is used to present a clearer live or on-demand video viewing experience with a smaller amount of video data storage.

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] A high-definition playback method based on video background extraction is applied to a high-definition playback system based on video background extraction, comprising a client device, a media server device, and a transcoding device. The client device includes a playback logic control module, the media server device includes a streaming module, a recording module, an index recording module, and a video-on-demand control module, and the transcoding device includes a transcoding module and a background extraction module. The method includes the following steps:

[0006] S1, the streaming module pulls the raw data and sends it to the transcoding device;

[0007] S2, the transcoding device performs secondary compression on the received raw stream, extracts the background content from the raw stream and creates an SEI segment;

[0008] S3, the transcoding device transmits the secondary compressed data to the streaming module, which at this time includes the original stream and the compressed stream;

[0009] S4. If the user is watching a live stream, the compressed stream is transmitted to the client through the streaming media device. The client detects the SEI, reassembles the SEI segments to restore clear YUV background data, and overlays it onto each frame of the compressed stream to restore the compressed stream to a clear state for playback.

[0010] The compressed stream enters the recording module for storage. During the recording process, the indexing module detects SEI frames carrying background data in real time and records the index in disk storage, corresponding to the recording segments. When a user initiates a video-on-demand session, the video-on-demand module first finds the start time of the video-on-demand session, accesses the background frame SEI position recorded by the indexing module to find the index, extracts the SEI separately as frame data from the video segment with the recorded SEI position, and outputs it together with the video-on-demand data.

[0011] In one possible implementation, the transcoding device in S3 performs secondary compression on the received raw stream, extracts the background content from the raw stream and creates SEI segments, including: encoding the YUV raw data of the video background in the raw stream into multiple different SEI segments separately; after the clear video frames are compressed, the output is a video frame sequence in which a clear background YUV data is distributed across multiple SEI segments.

[0012] In one possible implementation, encoding the raw YUV data of the video background in the original stream into multiple different SEI segments includes: extracting the foreground from the raw YUV data through dynamic key regions, with the remaining data being the essentially unchanged background; saving the background image as a separate SEI; then, for each frame in the video stream, compressing the background area with a higher compression ratio and the foreground with a lower compression ratio to obtain the video bitstream; inserting the generated SEI into the video bitstream; and performing the above processing once every first time interval.

[0013] In one possible implementation, the record index includes: establishing an index table to record forward SEI index information, wherein the record information in the index table includes video clips and the video clips corresponding to their SEI indexes.

[0014] In one possible implementation, the background frame SEI location lookup index recorded by the access index module includes: when on-demand playback begins, the video segment corresponding to the on-demand start time is confirmed. The on-demand module first searches the index table recorded by the index module to determine the video segment corresponding to its SEI index. If the video segment to be played has no SEI data, the video segment where the SEI index is located needs to be borrowed from the previous one. The on-demand module first reads the video segment where the SEI index is located, parses the SEI segment inside, and then sends it together with the data of the video segment corresponding to the on-demand start time.

[0015] In one possible implementation, the raw data is high-definition, high-bitrate, and low-compression audio and video data.

[0016] In one possible implementation, the source of the raw data includes a network camera (IPC) or a network video recorder (NVR).

[0017] The present invention has the following advantages: by extracting the background of video data through transcoding and combining it with the mechanism of streaming media storage index, a solution is achieved that produces a video display with a small amount of transmitted data but a clear picture. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of the high-definition playback method based on video background extraction according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the high-definition playback system based on video background extraction used in the high-definition playback method based on video background extraction according to an embodiment of the present invention.

[0020] Figure 3 This is an example of index representation in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the access index in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Reference Figure 1 The above is a flowchart of the steps of a high-definition playback method based on video background extraction according to an embodiment of the present invention, which is applied to, for example... Figure 2 The system shown is a high-definition playback system based on video background extraction, comprising a client device, a media server device, and a transcoding device. The client device includes a playback logic control module, the media server device includes a streaming module, a recording module, an index recording module, and a video-on-demand control module, and the transcoding device includes a transcoding module and a background extraction module. The system includes the following steps:

[0024] S1, the streaming module retrieves raw data and sends it to the transcoding device; the raw data is high-definition, high-bitrate, and low-compression audio and video data, for example, the bitrate of 1080P video data is generally 4Mbps. The raw data source includes network cameras (IPCs) or network video recorders (NVRs).

[0025] S2, the transcoding device performs secondary compression on the received raw stream, extracts the background content from the raw stream and creates an SEI segment;

[0026] S3, the transcoding device transmits the secondary compressed data to the streaming module, which at this time includes the original stream and the compressed stream;

[0027] S4. If the user is watching a live stream, the compressed stream is transmitted to the client through the streaming media device. The client detects the SEI, reassembles the SEI segments to restore clear YUV background data, and overlays it onto each frame of the compressed stream to restore the compressed stream to a clear state for playback.

[0028] The compressed stream enters the recording module for storage. During the recording process, the indexing module detects SEI frames carrying background data in real time and records the index in disk storage, corresponding to the recording segments. When a user initiates a video-on-demand session, the video-on-demand module first finds the start time of the video-on-demand session, accesses the background frame SEI position recorded by the indexing module to find the index, extracts the SEI separately as frame data from the video segment with the recorded SEI position, and outputs it together with the video-on-demand data.

[0029] Specifically, in another embodiment of the present invention, a high-definition playback method based on video background extraction includes the following steps: In step S3, the transcoding device performs secondary compression on the received raw stream, extracting the background content from the raw stream and creating SEI segments. This includes: separately encoding the YUV raw data of the video background in the raw stream into multiple different SEI segments; after the clear video frames are compressed, the output is a video frame sequence in which clear background YUV data is distributed across multiple SEI segments. General security video recordings, due to their relatively fixed locations, have unchanging background environments (such as people, vehicles, and flying objects) except for changing factors.

[0030] Furthermore, the raw YUV data of the video background in the original stream is separately encoded into multiple different SEI segments. This includes: extracting the foreground (removing frequently changing information such as people and vehicles) from the raw YUV data through dynamic key regions, while the remaining data is the largely unchanged background (occupying a significant portion of the image). The background image is saved separately as an SEI (the background data with a clear SEI can be uncompressed or compressed at a lower compression ratio, which can reduce the bitrate to some extent while maintaining clarity). Then, for each frame in the video stream, the background area is compressed at a high compression ratio (because the background has already been saved to the SEI, the background in each frame of the bitstream can be compressed at a high compression ratio), while the foreground is compressed at a lower compression ratio to obtain the video bitstream. For example, the background can be compressed by 4-10 times, and 1080P (assuming an original input bitrate of 4Mbps) can be compressed to 400kbps-1Mbps, while the compression ratio of the foreground remains unchanged. The generated SEI is then inserted into the video bitstream. This process is performed every few minutes or tens of minutes. This is equivalent to extracting and saving the background separately to compress the video stream to the maximum extent.

[0031] After the above operations, media data with a high compression ratio is obtained, which can then be transmitted to the subsequent recording and live streaming modules; in short, the background image at one moment is used for the next few minutes; it is equivalent to the bitstream only clearly saving the background once, and the content of the changing parts of each frame.

[0032] In another embodiment of the present invention, a high-definition playback method based on video background extraction is provided. In step S4, the index recording includes: establishing an index table to record forward SEI index information. The information recorded in the index table includes video segments and the video segments corresponding to their SEI indexes.

[0033] like Figure 3 As shown, the real-time stream, after passing through the recording module, is already re-encoded data with background extraction. The clear background image data is distributed across three SEI data segments (SEI1, 2, 3), transmitted along with the audio and video data. To reduce the bitrate, the SEI changes infrequently (once every few minutes to tens of minutes). The SEI is stored in the recording file along with the audio and video data. Some recording segments may lack an SEI, resulting in a blurry image when playing these segments alone. To ensure a clear image is displayed immediately regardless of which recording segment is started from during on-demand playback, an index file is needed to record the "forward SEI index." The information recorded in the index table is as follows:

[0034] Video clip 1: SEI index (this file, video clip 1); Video clip 2: SEI index (video clip 1); Video clip 3: SEI index (video clip 1); Video clip 4: SEI index (video clip 4); Video clip 5: SEI index (video clip 4).

[0035] Specifically, accessing the background frame SEI position lookup index recorded by the index module includes: when on-demand playback starts, confirming the video segment corresponding to the on-demand start time, the on-demand module will first search the index table recorded by the index module to determine the video segment corresponding to its SEI index. If the video segment to be played has no SEI data, it needs to borrow the video segment where the SEI index is located. The on-demand module first reads the video segment where the SEI index is located, parses the SEI segment inside, and then sends it together with the data of the video segment corresponding to the on-demand start time.

[0036] Continue as Figure 4 As shown, when on-demand playback begins, if the start time is the time of recording segment 3, in order to display the recording content of segment 3 clearly and at a low bitrate, the on-demand module will first search the records of the index module. It will find that the SEI index of segment 3 is recording segment 1, which means that there is no SEI data for the time period of segment 3 and it needs to borrow from the past. Therefore, the on-demand module will first read recording segment 1, parse the SEI segment inside it, and then send it out together with the data of recording segment 3. In this way, the user can see the clear recording segment 3.

[0037] Since the video background in the security field does not change frequently, SEI background extraction is not generated frequently, and a single SEI transmission can restore high definition within tens of minutes; this allows us to compress the video by tens of times, reduce the transmission bit rate and bandwidth usage, and still see a clear video image through SEI frames.

[0038] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A high-definition playback method based on video background extraction, characterized in that, A high-definition playback system based on video background extraction is applied to a client device, a media server device, and a transcoding device. The client device includes a playback logic control module, the media server device includes a streaming module, a recording module, an index recording module, and a video-on-demand control module, and the transcoding device includes a transcoding module and a background extraction module. The system includes the following steps: S1, the streaming module pulls the raw data and sends it to the transcoding device; S2, the transcoding device performs secondary compression on the received raw stream, extracts the background content from the raw stream and creates SEI segments, specifically including: extracting the foreground from the YUV raw data through dynamic key regions, and the remaining is the basically unchanged background image; saving the background image separately as an SEI, and then for each frame in the video stream, compressing the background area with a higher compression ratio and the foreground with a lower compression ratio to obtain the video bitstream; inserting the generated SEI into the video bitstream; performing the above processing once every first time interval, thereby encoding the YUV raw data of the video image background in the raw stream into multiple different SEI segments separately, and after the clear video frame is compressed, outputting a video frame sequence that disperses a clear background YUV data into multiple SEI segments; S3, the transcoding device transmits the secondary compressed data to the streaming module, which at this time includes the original stream and the compressed stream; S4. If the user is watching a live stream, the compressed stream is transmitted to the client device through the media server device. The client device detects the SEI, reassembles the SEI segments, restores the clear YUV background data, and overlays it onto each frame of the compressed stream, so that the compressed stream can be played in a clear state. The compressed stream is stored in the recording module. During recording, the index recording module detects SEI frames carrying background data in real time and records the index in disk storage, corresponding to the recording segments. When a user initiates on-demand playback, the on-demand control module first finds the on-demand start time, accesses the lookup index of the background frame SEI position recorded by the index recording module, extracts the SEI from the video segment with the recorded SEI position as frame data, and outputs it together with the on-demand data. The access to the lookup index of the background frame SEI position recorded by the index recording module includes: when on-demand playback starts, confirming the recording segment corresponding to the on-demand start time, the on-demand control module first searches the index table recorded by the index recording module to determine the recording segment corresponding to its SEI index. If the recording segment to be played has no SEI data, it borrows the recording segment where the SEI index is located. The on-demand control module first reads the recording segment where the SEI index is located, parses the SEI segment inside, and then sends it together with the data of the recording segment corresponding to the on-demand start time.

2. The high-definition playback method based on video background extraction as described in claim 1, characterized in that, The record index includes: establishing an index table to record forward SEI index information. The record information in the index table includes video clips and the video clips corresponding to their SEI indexes.

3. The high-definition playback method based on video background extraction as described in claim 1 or 2, characterized in that, The raw data is high-definition, high-bitrate, and low-compression audio and video data.

4. The high-definition playback method based on video background extraction as described in claim 3, characterized in that, The raw data sources include network cameras (IPCs) or network video recorders (NVRs).

Images