A method and device for decoding byte stream of network offline video file

Abstract

The present application discloses a method and device for decoding a byte stream of an offline network video file, including: storing the network video file in a main memory buffer after receiving it; directly decapsulating based on the byte information in the memory buffer to obtain the metadata of the video file; and using the hardware decoding unit of the graphics processor to hard-decode the video based on the metadata to obtain video frame data. The present application provides a decapsulator directly based on memory byte information, eliminating the redundant operation of decapsulating the byte stream after it is stored as a file on the disk. Using the graphics processor hard decoding unit greatly improves the decoding speed compared to using the central processing unit for soft decoding. Downstream computing-intensive tasks usually need to be performed in the graphics processing unit to increase the speed. The frame data after hard decoding is already in the video memory, and the downstream computing task data object is directly constructed through zero memory copy technology.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for decoding a byte stream of a network offline video file. Background Art

[0002] With the rapid development of computer hardware, deep learning, cloud computing and network technology, multimedia-based artificial intelligence technology has been widely used. The typical scenario of artificial intelligence video analysis application adopts the client-server architecture. After the server receives the offline video file fragments from the client, it performs corresponding artificial intelligence analysis. After the server receives the offline video file, the existing technology does not provide the decapsulation function directly from the memory data. For video decoding, it is necessary to first store the offline video byte stream on the disk as a hard disk file, and use the file path interface decapsulator of the existing solution to decapsulate the offline video in the hard disk to obtain the metadata and then perform software decoding on the central processor. The decoding performed by the existing technology has redundant operations and takes a long time.

[0003] In addition, in the decoding process provided by the prior art, there is a high data transmission load between the central processing unit and the image processing unit; if the video frame data is obtained using the decoding method of the prior art, it takes a long time and occupies more resources in downstream applications. Summary of the invention

[0004] Based on the above problems, the present application provides a method and device for decoding a byte stream of a network offline video file.

[0005] The present application discloses a method for decoding a byte stream of a network offline video file, the method comprising:

[0006] Receive the video file byte stream transmitted by the network side and store it in the main memory buffer of the central processing unit;

[0007] Decapsulating the offline video file byte stream in the main memory buffer using a decapsulator in the central processing unit to obtain meta information of the offline video file, wherein the meta information includes basic attributes of the video content body, and the basic attributes include video frame number, frame rate, packet number and resolution;

[0008] The content body part and the meta information are sent to a graphics processor, so that the graphics processor uses its hardware decoding unit to perform hardware decoding on the content body based on the meta information to obtain video frame data.

[0009] Optionally, the offline video byte stream decoding method further comprises:

[0010] Determining the location of the meta information by using the decapsulator when receiving the byte stream of the offline video file;

[0011] If the meta-information is located at the header of the video file, upon receiving a predetermined number of byte streams, the central processing unit continues to receive the remaining video file byte stream information in an asynchronous manner and simultaneously asynchronously transmits the received byte stream information to the graphics processor so that the graphics processor performs hardware decoding using its hardware decoding unit.

[0012] Optionally, the offline video byte stream decoding method further comprises:

[0013] Using the video frame data, construct a downstream task data object in the graphics processor, wherein the task data object includes data meta information and a data content body, wherein the meta information includes a shape and a number of channels;

[0014] The downstream task processing application processes the task data object to obtain a processing result;

[0015] The processing result is returned to the client.

[0016] The present application discloses a decoding device for a network offline video file byte stream, the device comprising:

[0017] Receiving module: receives the video file byte stream transmitted by the network side and stores it in the main memory buffer of the CPU;

[0018] A decapsulation module, using a decapsulator in a central processing unit to directly decapsulate the byte stream of the offline video file in the main memory buffer from the memory to obtain the meta information of the offline video file;

[0019] The sending module sends the content body and the meta information to a graphics processor, so that the graphics processor uses its hardware decoding unit to perform hardware decoding on the content body based on the meta information to obtain video frame data.

[0020] Optionally, the network offline video file byte stream decoding device further comprises:

[0021] A determination module, used for determining the location of the meta information, and determining the location of the meta information by using the decapsulator when receiving the byte stream of the offline video file;

[0022] If the meta-information is located at the header of the video file, upon receiving a predetermined number of byte streams, the central processing unit continues to receive the remaining video file byte stream information in an asynchronous manner and simultaneously asynchronously transmits the received byte stream information to the graphics processor so that the graphics processor performs hardware decoding using its hardware decoding unit.

[0023] Optionally, the network offline video file byte stream decoding device further comprises:

[0024] A downstream task data object generation module, which constructs a downstream task data object in the graphics processor using the video frame data, wherein the task data object includes data metadata and data content body;

[0025] A downstream task processing module, wherein the downstream task processing application processes the task data object to obtain a processing result;

[0026] The downstream task result returning module returns the processing result to the client.

[0027] The present application also provides an electronic device, including a memory and a processor, wherein:

[0028] The memory is used to store the computer program;

[0029] The processor is used to execute the computer program to implement the above-mentioned method for decoding the byte stream of the network offline video file.

[0030] The present application also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-mentioned method for decoding the byte stream of an offline network video file.

[0031] Compared with the prior art, the method disclosed in the present application does not need to store the byte stream information of the received network video file in the hard disk file first, and can directly decapsulate and decode the byte stream of the received offline video file from the memory in the central processing unit, thereby improving the decoding efficiency. Secondly, the present application directly hands over the main part of the decapsulated video content to the graphics processing unit, and uses its hard decoding unit to implement high-performance decoding operations. Compared with the soft decoding of the central processing unit, the decoding performance is greatly improved; furthermore, the video byte data before decoding has undergone inter-frame timing and intra-frame space compression, and its data volume is much smaller than the decoded frame data. Compared with decoding on the central processing unit side and then transmitting it to the video memory, the transmission load is much smaller.

[0032] The video frames decoded by the method provided in the present application are located in the storage space of the graphics processor, namely, the video memory. In practical applications, most downstream computing-intensive tasks need to be performed in the graphics processing unit to improve computing performance. The data decoded by the present application is already in the video memory. When constructing downstream computing task data objects, zero memory copy technology can be used. By shallow copying, the pointer of the data object content body is directly pointed to the address of the decoded video frame, thereby saving video memory space consumption and computing delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 A flowchart of a method for decoding a byte stream of a network offline video file provided by this application;

[0035] Figure 2 A flowchart of another method for decoding a byte stream of an offline network video file provided by the present application;

[0036] Figure 3 A schematic diagram of the structure of a decoding device for a network offline video file byte stream provided in the present application. DETAILED DESCRIPTION

[0037] As mentioned above, in the current process of decoding the byte stream of an offline video file, after the central processing unit receives the byte stream of the offline video file, it is necessary to save the byte stream of the offline video file as a file and store it in the hard disk, and use a decapsulator to decapsulate the video file in the hard disk to obtain the offline video metadata, and then decode the video file after the main part of the offline video file content and the metadata are sent to the central processing unit or the graphics processor. First, from the perspective of downstream applications after decoding, the process of receiving the video byte stream and then saving it as a file is a non-essential operation, which is redundant and time-consuming. Secondly, the soft decoding of the CPU has lower performance than the hard decoding of the GPU. Furthermore, after the CPU decodes the video byte information into video frames and then transmits the video frames to the GPU, the data transmission load between the CPU and the GPU is much higher than that of transmitting the byte information to the GPU and then decoding it. Finally, downstream computing-intensive tasks often need to be performed on the GPU to improve computing efficiency. The data after hard decoding on the GPU is already in the GPU memory. The downstream task data objects can be directly constructed through zero memory copy technology, saving the video memory and time consumption for constructing data objects.

[0038] In order to solve the above problems, the present application provides a method and device for decoding the byte stream of an offline network video file. The method receives the byte stream of the offline video file and stores it in the main memory buffer of the central processing unit, directly decapsulates the memory byte information, obtains the meta information of the offline video file, and the meta information is used to represent the basic attributes of the offline video; sends the offline video file content body and the meta information to the graphics processor, so that the graphics processor uses its hardware decoding unit to decode the video based on the meta information to obtain video frame data. Compared with the prior art, the method disclosed in the present application does not need to store the byte stream information of the received network video file in a hard disk file first, and can directly decapsulate and decode the byte information of the received offline video file from the memory in the central processing unit, thereby improving the decoding efficiency. Secondly, the present application directly hands over the main part of the decapsulated video content to the graphics processing unit, and uses its hard decoding unit to implement high-performance decoding operations. Compared with the soft decoding of the central processing unit, the decoding performance is greatly improved; thirdly, the video byte data before decoding has undergone inter-frame timing and intra-frame space compression, so its data volume is much smaller than the decoded frame data. Compared with decoding on the central processing unit side and then transmitting it to the video memory, the transmission load is much smaller when the byte data before decoding is first transmitted to the video memory.

[0039] The video frames decoded by the method provided in the present application are located in the storage space of the graphics processor, namely, the video memory. In practical applications, most downstream computing-intensive tasks need to be performed in the graphics processing unit to improve computing performance. The data decoded by the present application is already in the video memory. When constructing downstream computing task data objects, zero memory copy technology can be used. By shallow copying, the pointer of the data object content body is directly pointed to the address of the decoded video frame, thereby saving video memory space consumption and computing delay.

[0040] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described in detail and clearly in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0041] Figure 1 A flowchart of a method for decoding a byte stream of a network offline video file provided by the present application, the method comprising the following steps:

[0042] S101: Receive a video file byte stream transmitted by a network side and store it in a main memory buffer of a central processing unit.

[0043] The provider of the byte stream of the offline video file can be a client that needs video analysis services. The client sends the video file segment to the server of the video analysis service provider through the network. When the server receives the byte stream of the offline video file, it directly stores the byte stream in the main memory without saving it to the disk as a video file.

[0044] S102: Decapsulate the offline video file byte stream in the main memory buffer using a decapsulator in the central processing unit to obtain meta information of the offline video file, wherein the meta information includes basic attributes of the video content body, and the basic attributes include video frame number, frame rate, packet number and resolution.

[0045] The byte stream of the offline video file is stored in the main memory of the CPU in the form of a binary file, and the meta information is usually located at the head or tail of the video file. The meta information is required in the process of decoding the video file, and the content of the meta information mainly includes video resolution, video duration, video frame rate and encoding format.

[0046] S103: Sending the content subject body and the meta information to a graphics processor, so that the graphics processor uses its hardware decoding unit to perform hardware decoding on the content subject based on the meta information to obtain video frame data.

[0047] The obtained meta information and the decapsulated binary file may be sent to a graphics processor, and a hardware decoding unit in the graphics processor may use the meta information to perform hardware decoding on the decapsulated binary file, and the video frames generated after decoding are stored in a storage space of the graphics processor.

[0048] In the above method, after the central processing unit receives the byte stream of the offline video file, it can directly decapsulate it using the decapsulator in the central processing unit. The method provided in the prior art does not provide a decapsulator for the memory address or data object, so it is necessary to first save the byte stream of the offline video file from the memory to the hard disk, and then use the file path interface to decapsulate the video file in the hard disk. The decapsulator used in the prior art cannot directly decapsulate the byte data in the memory. Compared with the prior art, the method provided in this application does not require redundant operations, thereby improving the process efficiency. The byte stream of the offline video file is decoded in the graphics processor using metadata to obtain a video frame. There is a hardware decoding unit in the graphics processor, and the use of hardware decoding improves the decoding speed.

[0049] When receiving the byte stream of an offline video file, the location of the meta information will be different. Figure 2 A flowchart of another method for decoding a byte stream of an offline network video file provided by the present application mainly includes the following steps:

[0050] S201: Receive a byte stream of an offline video file.

[0051] The video application service provider server host uses an asynchronous non-blocking method to receive the video file segments sent by the client through the network and store them in a memory buffer.

[0052] S202: Determine whether the meta information is located in the header.

[0053] The decapsulator attempts to decapsulate in real time according to the existing content in the buffer, and determines whether the meta information is located in the header of the video information. If so, it proceeds to S203. If not, it proceeds to S206.

[0054] S203: After the decapsulator understands the decapsulation and obtains the header meta information, it sends it to the image processing unit.

[0055] Usually, the meta information of a video file is much smaller than the video file content itself. If the decapsulator finds that the original video information is located in the file header, the decapsulator directly decapsulates the header meta information and sends it to the image processing unit.

[0056] S204 The image processing unit decodes asynchronously.

[0057] When the video file is not completely received, the metadata of the video file has been obtained in advance. At this time, the receiving and decoding can be asynchronously performed. While the central processor receives the remaining byte stream of the offline video file, the image processing unit decodes the received file information.

[0058] S205: Use zero memory copy technology to construct downstream computing task data objects.

[0059] Downstream computing-intensive tasks usually need to be performed on the image processing unit to increase the computing speed, and the video frame data after hard decoding by the image processing unit is already stored in the video memory of the image processing unit. The input data of downstream computing tasks is usually represented in a specified format, such as tensors. In addition to the content data of the tensor itself, its content also includes some attribute information of the tensor, such as shape, number of channels, data location, etc. The attribute information is usually small and only takes a small amount of storage units and construction time. Through the zero memory copy technology, the main part of the tensor data content directly points to the decoded video frame data address, which reduces the memory space occupied and saves the construction time of the data tensor object, thereby saving the delay of the entire end-to-end application.

[0060] S206: After synchronously blocking and receiving the complete byte stream data, decapsulation is performed to obtain the video data metadata.

[0061] S207: Distribute the video meta information and the video content part to the image processing unit.

[0062] S208: The image processing unit starts hard decoding of the video packet to obtain video frame data.

[0063] S209: The downstream computing task performs corresponding computing and data processing through the constructed data objects.

[0064] S210: The server returns the processing result to the client.

[0065] This application also discloses Figure 3 The decoding device 300 of an offline video byte stream shown includes:

[0066] The receiving module 301 is used to receive the byte stream of the offline video file and store it in the central processing unit;

[0067] The decapsulation module 302 is used for decapsulation, and uses the decapsulator in the central processing unit to decapsulate the byte stream of the offline video file to obtain meta information in the byte stream of the offline video file, where the meta information is used to represent the video information of the offline video file corresponding to the byte stream of the offline video file;

[0068] The sending module 303 is used to send the byte stream of the decapsulated offline video file and the meta information, and send the byte stream of the decapsulated offline video file and the meta information to the graphics processor, so that the graphics processor uses the meta information to decode the byte stream of the decapsulated offline video file to obtain a video frame.

[0069] The schematic diagram of the decoding device of the offline video byte stream provided in the present application may also include the following modules:

[0070] A determination module, used for determining the location of the meta information, and determining the location of the meta information by using the decapsulator when receiving the byte stream of the offline video file;

[0071] If the meta-information is located at the header of the video file, upon receiving a predetermined number of byte streams, the central processing unit continues to receive the remaining video file byte stream information in an asynchronous manner and simultaneously asynchronously transmits the received byte stream information to the graphics processor so that the graphics processor performs hardware decoding using its hardware decoding unit.

[0072] A downstream task data object generation module, which constructs a downstream task data object in the graphics processor using the video frame data, wherein the task data object includes data metadata and data content body;

[0073] A downstream task processing module, wherein the downstream task processing application processes the task data object to obtain a processing result;

[0074] The downstream task result returning module returns the processing result to the client.

[0075] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned offline video byte stream decoding method steps are implemented.

[0076] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, all of which fall within the scope of protection of this application.

[0077] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0078] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for decoding a byte stream of an offline network video file, applied to a central processing unit, characterized in that: include: Receive the video file byte stream transmitted by the network side and store it in the main memory buffer of the central processing unit; Decapsulating the offline video file byte stream in the main memory buffer using a decapsulator in the central processing unit to obtain meta information of the offline video file, wherein the meta information includes basic attributes of the video content body, and the basic attributes include video frame number, frame rate, packet number and resolution; Sending the content body and the meta information to a graphics processor, so that the graphics processor uses its hardware decoding unit to perform hardware decoding on the content body based on the meta information to obtain video frame data; Determining the location of the meta information by using the decapsulator when receiving the byte stream of the offline video file; If the meta information is located at the header of the video file, upon receiving a predetermined number of byte streams, the central processing unit continues to receive the remaining video file byte stream information in an asynchronous manner and simultaneously asynchronously transmits the received byte stream information to the graphics processor so that the graphics processor performs hardware decoding using its hardware decoding unit; If the meta information is located at the end of the offline video file, the step of receiving the video file byte stream transmitted by the network side and storing it in the main memory buffer of the central processing unit includes: The video file byte stream transmitted by the network side is completely received and stored in the main memory buffer of the central processing unit so that the graphics processor can be used for decoding in subsequent steps.

2. The method according to claim 1, characterized in that The method further comprises: Using the video frame data, construct a downstream task data object in the graphics processor, wherein the task data object includes data meta information and a data content body, wherein the meta information includes a shape and a number of channels; The downstream task processing application processes the task data object to obtain a processing result; The processing result is returned to the client.

3. The method according to claim 1, characterized in that The receiving the byte stream of the offline video file and storing it in the central processing unit includes: asynchronously receiving the byte stream of the offline video file and storing it in the main memory buffer of the central processing unit in the form of a binary file.

4. A decoding device for an offline network video byte stream, applied to a central processing unit, characterized in that: include: Receiving module: receives the video file byte stream transmitted by the network side and stores it in the main memory buffer of the CPU; A decapsulation module, using a decapsulator in a central processing unit to directly decapsulate the byte stream of the offline video file in the main memory buffer from the memory to obtain the meta information of the offline video file; A sending module, which sends the content body and the meta information to a graphics processor, so that the graphics processor uses its hardware decoding unit to perform hardware decoding on the content body based on the meta information to obtain video frame data; A determination module, used for determining the location of the meta information, and determining the location of the meta information by using the decapsulator when receiving the byte stream of the offline video file; If the meta information is located at the header of the video file, upon receiving a predetermined number of byte streams, the central processing unit continues to receive the remaining video file byte stream information in an asynchronous manner and simultaneously asynchronously transmits the received byte stream information to the graphics processor so that the graphics processor performs hardware decoding using its hardware decoding unit; If the meta information is located at the end of the offline video file, the step of receiving the video file byte stream transmitted by the network side and storing it in the main memory buffer of the central processing unit includes: The video file byte stream transmitted by the network side is completely received and stored in the main memory buffer of the central processing unit so that the graphics processor can be used for decoding in subsequent steps.

5. The device according to claim 4, characterized in that The device further comprises: A downstream task data object generation module, which constructs a downstream task data object in the graphics processor using the video frame data, wherein the task data object includes data metadata and data content body; A downstream task processing module, wherein the downstream task processing application processes the task data object to obtain a processing result; The downstream task result returning module returns the processing result to the client.

6. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the method for decoding the network offline video file byte stream as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method for decoding a byte stream of a network offline video file as described in any one of claims 1 to 3 is implemented.

Images