A video stream compression and decompression method and system based on OpenMP thread nesting

Abstract

The application discloses a video stream compression and decompression method and system based on OpenMP thread nesting, adopts parallel pipeline to process video frame data, simplifies original JPEG coding, compresses RGB data into YCbCr data for transmission, adopts OpenMP thread nesting to process the compression and decompression of the video frame data in a multithreading mode under the condition that the outer thread processes the video frame data by using the parallel pipeline, effectively improves the support resolution and fluency of remote playing video, saves the engineering cost of client computing, and improves user experience.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a video stream compression and decompression method and system based on OpenMP thread nesting. BACKGROUND

[0002] In recent years, due to the advantages of low local hardware investment cost and high cloud resource utilization, the use scenarios of remote virtual desktop applications are increasing. However, due to network bandwidth limitations, remote desktop systems have to use high compression ratio data compression algorithms to reduce the amount of data transmission and avoid network congestion as much as possible, thereby ensuring smooth experience. High compression ratio algorithms take longer and have greater data distortion, especially in high data transmission scenarios such as video, real-time rendering and games, and the experience of the current remote virtual desktop is poor.

[0003] The current serial video data processing method of the remote virtual desktop is to perform the steps of "grabbing video frames", "compression" and "transmission" on the remote server side, and "receiving", "decompression" and "rendering" on the local client side. The six steps are performed in series. In the compression step, MJPEG uses JPEG encoding to serially process each frame. JPEG encoding mainly involves hue transformation, sampling, DCT, quantization, RLE encoding, canonical Huffman encoding and DC (direct current component) encoding, which takes a long time and often cannot meet the real-time display requirements. The JPEG encoding process is shown in Figure 1 .

[0004] Currently, in the park network, building network and local area network, the network bandwidth can be improved to 10 Gbps through all-optical fiber networking. In particular, with the development of FTTR technology, the terminal bandwidth in the local area network will break through 50 Gbps. There is currently no remote virtual desktop video compression and decompression algorithm suitable for high bandwidth scenarios.

[0005] In high bandwidth scenarios, video frame data can be compressed lightly to minimize compression time while ensuring video quality after decompression.

[0006] Parallelization is also an effective method to improve compression and decompression efficiency. OpenMP is a set of guiding compilation processing solutions for multi-processor program design of shared memory parallel systems. It provides a high-level abstract description of parallel algorithms, is particularly suitable for parallel program design on multi-core CPU machines, and is an API that can easily parallelize and multithread programs through high-level instructions. In the simplest case, OpenMP can even parallelize the program by adding only one line of instructions. SUMMARY

[0007] The technical problems to be solved by the present application are to provide a video stream compression and decompression method and system based on OpenMP thread nesting to solve the technical problem of long processing time in the video stream processing process.

[0008] The present application adopts the following technical solutions:

[0009] A video stream compression and decompression method based on OpenMP thread nesting comprises the following steps:

[0010] S1, parallel pipeline processing video frame data is adopted;

[0011] S2, the original JPEG encoding is simplified, and the RGB data is compressed into YCbCr data for transmission, and in the case that the outer thread uses parallel pipeline to process the video frame data obtained in step S1, OpenMP thread nesting is adopted to process the compression and decompression of the video frame data in a multi-threaded manner.

[0012] Specifically, in step S1, parallel pipeline processing is realized by using OpenMP multi-thread parallel technology.

[0013] Further, the architecture of the parallel pipeline includes a server end and a client end, and the server end and the client end communicate through a Socket.

[0014] Further, the server end has three threads of frame rate simulation, multi-thread compression and Socket transmission; the client end has three threads of Socket receiving, multi-thread decompression and OpenGL rendering, and the six threads are executed by judging whether the input queue is empty, and if not, they are executed in parallel, and if empty, they are looped and waited.

[0015] Specifically, in step S2, the original JPEG encoding is simplified, and the specific steps are as follows:

[0016] The encoding steps after DCT are removed, and only the two steps of color phase transformation and sampling are retained.

[0017] Specifically, in step S2, the number of compression parallel threads thread_num is the number of block processing, each thread has a corresponding compression starting line number line_on and a compression ending line number up_limit, the color phase transformation formula for compression adopts BT601 standard, and YCbCr or YUV data is down-sampled, and the conversion formula is calculated by using shift operation.

[0018] The number of threads thread_num of parallel decompression is the number of block processing, each thread has corresponding decompression starting line number line_on and decompression end line number up_limit, using SIMD and using BT601 standard for color phase conversion formula of decompression, sampling YCbCr or YUV data, and conversion formula uses shift operation to calculate.

[0019] Further, in the parallel decompression process, after R1, G1 and B1 are calculated, Y1 is subtracted from R2, G2 and B2 and Y2 is added.

[0020] Further, YCbCr or YUV data is down-sampled or up-sampled using the 4:2:0 mode.

[0021] Further, a frame is vertically divided into multiple equal-sized regions, and multi-thread parallel execution is performed on each region; if the video frame cannot be evenly divided, the last two regions are merged into one region, and the regions do not affect each other, and each region uses a thread to calculate and process using the color phase conversion and sampling algorithm.

[0022] In the second aspect, the embodiment of the present application provides a video stream compression and decompression system based on OpenMP thread nesting, comprising:

[0023] The video processing module processes video frame data using parallel pipeline processing;

[0024] The compression and decompression module simplifies the original JPEG encoding, compresses RGB data into YCbCr data for transmission, and uses OpenMP thread nesting to process the compression and decompression of video frame data in a multi-threaded manner under the condition that the outer thread processes the video frame data obtained by the video processing module 1 using parallel pipeline.

[0025] Compared with the prior art, the present application has at least the following beneficial effects:

[0026] The application discloses a video stream compression and decompression method based on OpenMP thread nesting, and comprises the following steps: S1, parallel pipeline processing of video frame data is adopted to improve the efficiency and speed of video processing; specifically, the video frame data is divided into multiple blocks, each block is given to a different processing unit for processing, parallel calculation is realized, and the processing speed of the video frame is accelerated; S2, RGB data is compressed into YCbCr or YUV data for transmission, and OpenMP thread nesting is adopted to process the compression and decompression of the video frame data in a multi-thread mode, so that the efficiency of video transmission is improved and the amount of transmission data is reduced; specifically, since the human eye is more sensitive to the brightness signal and less sensitive to the chroma signal, only the brightness signal and part of the chroma signal need to be transmitted during video transmission, and unnecessary RGB data is compressed, so that the amount of transmission data is reduced. In addition, the OpenMP thread nesting mode can simultaneously utilize multiple CPU cores for video frame compression and decompression, thereby improving the processing speed.

[0027] Further, OpenMP is a shared memory-based parallel programming model, which distributes tasks to multiple threads for simultaneous execution, fully utilizing the parallelism of multi-core processors; by using the characteristics of OpenMP thread nesting, video compression and decompression tasks are distributed to multiple threads for simultaneous execution, thereby maximizing the use of system resources and further improving the coding speed.

[0028] Further, in order to realize the communication between the server side and the client side, the method adopts Socket technology to transmit data to the server side or obtain the processing result from the server side through the network; the Socket communication can realize distributed processing, that is, the data is dispersed to different computing nodes for processing, thereby improving the parallelism and efficiency of data processing; in addition, the Socket communication can also realize real-time transmission and asynchronous processing of data, thereby improving the real-time performance and response speed of data processing; at the same time, the Socket communication can also connect computers of different platforms and different systems, realize cross-platform data processing, has good compatibility and expansibility, and can improve the efficiency, parallelism, real-time performance and compatibility of data processing.

[0029] Further, by using the multi-thread and queue structure, the computing capacity of multi-core CPU can be fully utilized to realize fast processing and transmission of video streams; at the same time, the queue mode is adopted to coordinate the execution of different threads, so that the competition condition and deadlock are avoided, and the efficiency of parallel processing is improved.

[0030] Further, in the traditional video encoding process, DCT (Discrete Cosine Transform) is a necessary step, but also the most computationally intensive step; therefore, the encoding steps including DCT are eliminated, and the hue transformation and sampling steps are retained, thereby greatly reducing the computational complexity and delay of encoding, and being able to improve the encoding speed and real-time performance of the video. And, the hue transformation and sampling steps are mainly to convert the RGB color space to the YCbCr or YUV color space, and to down-sample the YUV components; not only can be performed at the encoding end, but also can be performed at the decoding end, so that the flexibility of the codec implementation is higher; in addition, these steps have little effect on image quality, so that the encoding efficiency and real-time performance can be guaranteed while maintaining good visual effects.

[0031] Further, step S2 is the core step of compression and decompression, and by setting the number of parallel threads, the efficiency of processing the video stream can be improved and the processing time can be reduced. At the same time, by using the BT601 standard and sampling on YCbCr or YUV data, the processing of video data can be ensured to comply with widely accepted standards, making the processing result more versatile; in the conversion of color space, shift operation is used for calculation, which can greatly improve the operation speed and avoid the use of multiplication and division operations, thereby reducing the calculation time. Shift operation is a fast bit operation that can move the binary representation of a number left or right by a certain number of bits, which is equivalent to multiplication or division operation, but the operation speed is faster. Therefore, using shift operation for calculation in video processing can greatly improve the processing efficiency and speed up the processing of video data.

[0032] Further, the sampling method is YUV420, and the Y component of each pixel corresponds to a color component, while the U and V components are sampled in horizontal and vertical directions with each row or column being skipped. Therefore, for a group of continuous pixel data, the UV values of each pixel are shared, while the Y value is unique. In this method, after R1, G1 and B1 are calculated, Y1 is subtracted from R2, G2 and B2, and Y2 is added to R2, G2 and B2, where Y1 and Y2 refer to continuous Y component data; since YUV420 method is used, the U and V values of each pixel are sampled in horizontal and vertical directions with each row or column being skipped, and the Y values between adjacent pixels usually do not change much; therefore, subtracting Y1 from R2, G2 and B2 can minimize the influence of Y1 on the values of R2, G2 and B2 between adjacent pixels, thereby reducing the amount of calculation; at the same time, adding Y2 can adjust the values of R2, G2 and B2 according to the change of Y values between adjacent pixels, thereby realizing smoother color transition; therefore, the purpose of subtracting Y1 and adding Y2 from R2, G2 and B2 is to reduce the amount of calculation and realize smoother color transition, while combining the principle of YUV420 sampling method to minimize the influence of Y values between adjacent pixels.

[0033] Further, the YCbCr or YUV data is down-sampled or up-sampled using a 4:2:0 manner, that is, the number of samples of the Cb and Cr components is reduced by half in the horizontal and vertical directions respectively, while the Y component remains unchanged; this sampling manner is widely used in video encoding because it can reduce the amount of video data while retaining the quality and details of the image as much as possible; specifically, for YCbCr or YUV data, the Y component represents luminance information, and the Cb and Cr components represent chrominance information; since the human eye is more sensitive to luminance than to chrominance, in video encoding, the chrominance information can be down-sampled or up-sampled to reduce the amount of data without significantly affecting the quality of the video. At the same time, since the 4:2:0 sampling manner is used in this method, that is, the Cb and Cr components in the horizontal and vertical directions are reduced by half, the amount of video data can be effectively reduced, and the encoding efficiency of 4:2:0 sampling is higher and the compatibility is better than other sampling manners.

[0034] Further, the processing speed of the video frame is improved by using parallel computing, and after a frame is vertically divided into multiple blocks of equal size, parallel computing is performed on each block to improve the processing speed of the entire video frame. By using a multi-threaded manner, multiple blocks can be processed at the same time, thereby realizing parallel computing and improving the computing speed. At the same time, for a video frame that cannot be evenly divided, the last two blocks are merged into one block, which can avoid the situation that the pixel data in the block is incomplete during processing, and ensure the accuracy of the calculation result. In addition, the regions do not affect each other, which can ensure the stability of the calculation process and avoid the calculation interference between different regions.

[0035] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0036] In summary, the present application improves the compression and decompression speed and image quality, reduces the system delay and calculation amount, and has strong stability and scalability.

[0037] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 JPEG compression encoding flowchart;

[0039] Figure 2 Pipelining and serial processing of multiple frames comparison chart;

[0040] Figure 3 Video stream parallelization pipeline overall design architecture;

[0041] Figure 4For the purpose of simplifying JPEG encoding mode;

[0042] Figure 5 For the purpose of comparing the time consumption of the original JPEG and the simplified JPEG in the video encoding stage of playing different resolution videos;

[0043] Figure 6 For the purpose of the compression thread nesting diagram;

[0044] Figure 7 For the purpose of the four-thread parallel block processing diagram;

[0045] Figure 8 For the purpose of comparing the time consumption of the nested parallel compression and the original parallel compression in the video processing stage of playing different resolution videos;

[0046] Figure 9 For the purpose of comparing the time consumption of the nested parallel decompression and the original parallel decompression in the video processing stage of playing different resolution videos;

[0047] Figure 10 For the purpose of comparing the time consumption of the serial processing and the nested parallel processing in the video processing stage of compression-transmission-decompression. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0049] In the description of the present application, it should be understood that the terms “include” and “contain” indicate the existence of described features, whole, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0050] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms “a”, “an” and “the” are intended to include the plural forms.

[0051] It should be further understood that the term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, and includes these combinations, for example, A and / or B can mean only A, or only B, or both A and B. In addition, the character " / " in this paper generally represents that the front and rear associated objects are an "or" relationship.

[0052] It should be understood that although the terms first, second, third, etc. may be used in embodiments of the present application to describe a certain range, etc., these ranges should not be limited to these terms. These terms are only used to distinguish the ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the present application.

[0053] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting". Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0054] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These drawings are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and certain details can be omitted. The shapes of various regions, layers and their relative size and positional relationship shown in the drawings are only exemplary, and in actuality, there can be deviations due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes and relative positions according to actual needs.

[0055] The present application is a video stream compression and decompression method based on OpenMP thread nesting, comprising the following steps:

[0056] S1, processing video frames in a pipeline;

[0057] Please refer to Figure 2 Compared with the existing video frame data serial "frame grabbing, compression, transmission, reception, decompression and rendering" process, the present application uses OpenMP multi-thread parallel technology to realize parallel pipeline processing of the above process, realizes different processing of different video frames at the same time, improves the throughput and resource utilization, and shortens the overall processing delay.

[0058] Please refer to Figure 3The pipeline architecture of the application has two ends, which are server end (Server) and client end (Client), and the two ends communicate through Socket. The Server end has three threads of frame rate simulation, multi-thread compression and Socket transmission; the Client end has three threads of Socket receiving, multi-thread decompression and OpenGL rendering, and the six threads are executed by judging whether the input queue is empty, and if not, they are executed in parallel, and if empty, they are looped and waited.

[0059] S2, compression and decompression strategy;

[0060] S201, simplified JPEG encoding strategy;

[0061] Please refer to Figure 4 The original JPEG encoding is simplified, and the encoding steps after DCT are removed, and the hue transformation and sampling steps are retained, so that the RGB data is simply and lightly compressed into YCbCr data for transmission.

[0062] S202, parallel optimization strategy of thread nesting;

[0063] Please refer to Figure 6 For the transmission and compression pipeline, the compression thread is nested, and for the compression and decompression steps, in the case that the outer thread has processed the video frame using the pipeline, OpenMP thread nesting is used to process the compression and decompression of the video frame data in multiple threads.

[0064] OpenMP parallel compression and decompression strategy;

[0065] A frame is vertically divided into multiple equal regions, and each region is processed in multiple threads in parallel. If the video frame cannot be evenly divided, the last two regions are merged into one, and the regions do not affect each other, and each region is calculated and processed using a thread using the hue transformation and sampling algorithm, such as Figure 7 The 4-thread block is shown.

[0066] S2021, parallel compression strategy

[0067] The number of threads thread_num of compression parallel is the number of block processing, and each thread has its own compression start line number line_on and compression end line number up_limit. For the hue transformation formula of compression, BT601 standard is used, and 4:2:0 method is used for YCbCr (YUV) data downsampling, and the conversion formula is calculated using shift operation, which is faster.

[0068] S2022, parallel decompression strategy

[0069] The thread number thread_num of decompression parallelism is the number of block processing, each thread has its own decompression start line number line_on and decompression end line number up_limit. Further, the application uses SIMD and adopts BT601 standard for the color phase conversion formula of decompression, uses 4:2:0 mode to sample YCbCr (YUV) data, and uses shift operation to calculate the conversion formula, so that the calculation speed is faster. Among them, because two Y components share one UV component, after R1, G1 and B1 are calculated, simple Y1 subtraction and Y2 addition operations are performed on R2, G2 and B2 to reduce the calculation amount and processing time.

[0070] In another embodiment of the application, a video stream compression and decompression system based on OpenMP thread nesting is provided, which can be used to implement the above-mentioned video stream compression and decompression method based on OpenMP thread nesting. Specifically, the video stream compression and decompression system based on OpenMP thread nesting includes a video processing module and a compression and decompression module.

[0071] The video processing module processes video frame data in parallel pipeline.

[0072] The compression and decompression module simplifies the original JPEG encoding, compresses RGB data into YCbCr data for transmission, and uses OpenMP thread nesting to process the compression and decompression of video frame data in a multi-threaded manner when the outer thread processes the video frame data obtained by the video processing module 1 using parallel pipeline.

[0073] In another embodiment of the present application, a terminal device is provided, which comprises a processor and a memory, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiment of the present application can be used for the operation of the video stream compression and decompression method based on OpenMP thread nesting, which comprises:

[0074] The video frame data is processed by using a parallel pipeline; the original JPEG encoding is simplified, and the RGB data is compressed into YCbCr data for transmission; in the case that the video frame data is processed by using a parallel pipeline in an outer thread, OpenMP thread nesting is used to process the compression and decompression of the video frame data in a multi-thread mode.

[0075] In another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, and is used to store programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. Furthermore, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory.

[0076] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the video stream compression and decompression method based on OpenMP thread nesting in the above embodiments. The one or more instructions stored in the computer readable storage medium are loaded and executed by the processor to implement the following steps:

[0077] The video frame data is processed in parallel pipeline, and the original JPEG encoding is simplified, and the RGB data is compressed into YCbCr data for transmission. In the case of processing the video frame data in the outer thread using the parallel pipeline, OpenMP thread nesting is used to process the compression and decompression of the video frame data in a multi-threaded manner.

[0078] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without creative labor based on the embodiments in the present application belong to the scope of protection of the present application.

[0079] The pipeline processing pseudocode is as follows:

[0080]

[0081]

[0082]

[0083] Taking processing four frames as an example, the present application adopts a pipeline processing manner, and the overall time delay is shortened by 62.5% compared with other serial processing manners.

[0084] In the Server, the inter-frame processing time is the sum of the serial frame grabbing, compression and transmission time, to the maximum value of the processing time of one of the three parallel operations of the present application.

[0085] In the Client, the inter-frame processing time is the sum of the serial receiving, decompression and rendering time, to the maximum value of the processing time of one of the three parallel operations of the present application, and the inter-frame processing time is shortened by 66.7%.

[0086] The encoding time of the simplified JPEG and the original JPEG is compared, and the test data is obtained multiple times and averaged.

[0087] Referring to Figure 5 , the encoding time of the encoding strategy of the present application is greatly reduced.

[0088] Specifically, under the conditions of 480P, 1080P, 2K and 4K resolution, the encoding time of the present application is reduced by 56%, 61%, 58% and 52% respectively.

[0089] The pseudo code for parallel processing optimization of compression is as follows:

[0090]

[0091] Referring to Figure 8 , after using outer layer parallel optimization, the present application uses thread nesting for compression step, and performs inner layer parallel processing on the processing algorithm, greatly reducing the compression time. Compared with the original parallel compression and nested parallel compression, the compression time is reduced by 87%, 78%, 83% and 87% under the processing time of 480P, 1080P, 2K and 4K video resolution.

[0092] The pseudo code for parallel processing optimization of decompression is as follows:

[0093]

[0094] Referring to Figure 9 , after using outer layer parallel optimization, the present application uses thread nesting for decompression step, and performs inner layer parallel processing on the processing algorithm, greatly reducing the decompression time. Compared with the original parallel decompression and nested parallel decompression, the decompression time is reduced by 86%, 65%, 78% and 69% under the processing time of 480P, 1080P, 2K and 4K video resolution.

[0095] Referring to Figure 10 , in the compression-transmission-decompression processing flow, the overall time of serial processing and nested parallel processing is tested. Under the conditions of 480P, 1080P, 2K and 4K resolution, the overall time of nested parallel processing is reduced by 81.5%, 71.8%, 81.8% and 82.1% per frame compared with serial processing.

[0096] In summary, the video stream compression and decompression method and system based on OpenMP thread nesting of the application can solve the problems of video transmission display lag in the traditional remote desktop system, inability to play high-resolution video, long time consumption of video stream encoding and decoding processing, inability to guarantee real-time performance, and limited computing power of the client, and propose a video stream compression and decompression method based on OpenMP thread nesting. First, multiple threads are created in the outer layer, and the operations of frame grabbing, compression, transmission, receiving, decompression, and rendering are performed in parallel. Second, the JPEG encoding process is simplified, and thread nesting is used for compression and decompression to solve the problem of long time consumption in the video stream processing process, effectively improve the support resolution and smoothness of the remote video playback, and save the engineering cost of client computing and improve the user experience. The compression and decompression speed is improved: multi-thread parallel computing can fully utilize the computing resources of multi-core CPU and improve the compression and decompression speed. The system delay is reduced: the pipeline architecture and Socket are used for communication, which can make the compression and decompression operations parallel, and reduce the system delay. The calculation amount is reduced: by eliminating the DCT encoding step and using the conversion formula of shift operation, the calculation amount is reduced. The image quality is improved: using the BT601 standard and YUV420 sampling method can better preserve image details and color information and improve image quality. The stability and scalability are strong: a frame of video is vertically divided into multiple blocks, and multi-thread parallel computing is used for each block, which makes the method more stable and can adapt to different sizes of video streams. Through Socket communication, encoding and decoding processing can be performed on different platforms, and the method has good scalability.

[0097] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of the functional units and modules are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0098] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0099] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented in other ways. For example, the apparatus / terminal embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between each set of interfaces can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0101] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place, or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0102] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0103] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0104] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0105] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0106] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 one flowchart or multiple flowcharts and / or blocks Figure 1 one flowchart or multiple flowcharts and / or blocks

[0107] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A method for video stream compression and decompression based on OpenMP thread nesting, characterized in that, The method comprises the following steps: S1, parallel pipeline processing video frame data is adopted, OpenMP multi-thread parallel technology is adopted to realize parallel pipeline processing, the architecture of parallel pipeline comprises a server end and a client end, the server end and the client end communicate through a Socket, the server end has three threads of frame rate simulation, multi-thread compression and Socket transmission; the client end has three threads of Socket receiving, multi-thread decompression and OpenGL rendering, the six threads execute by judging whether an input queue is empty, execute in parallel when the input queue is not empty, and wait in a loop when the input queue is empty; S2, the original JPEG encoding is simplified, RGB data is compressed into YCbCr data for transmission, under the condition that the video frame data obtained in step S1 is processed by the outer thread using parallel pipeline, OpenMP thread nesting is adopted to process the compression and decompression of the video frame data in a multi-thread mode; The number of threads thread_num of compression parallelism is the number of block processing, each thread has a corresponding compression start line number line_on and a compression end line number up_limit, the color phase conversion formula for compression adopts BT601 standard, YCbCr or YUV data is down-sampled, and the conversion formula is calculated using a shift operation; The number of threads thread_num of decompression parallelism is the number of block processing, each thread has a corresponding decompression start line number line_on and a decompression end line number up_limit, YCbCr or YUV data is up-sampled using SIMD and the color phase conversion formula for decompression adopts BT601 standard, and the conversion formula is calculated using a shift operation.

2. The OpenMP thread nest based video stream compression and decompression method according to claim 1, wherein, In step S2, the original JPEG encoding is simplified as follows: The encoding steps after DCT are removed, and the color phase conversion and sampling steps are retained.

3. The OpenMP thread nest based video stream compression and decompression method of claim 1, wherein, In the parallel decompression process, after R1, G1 and B1 are calculated, Y1 is subtracted from R2, G2 and B2, and Y2 is added.

4. The OpenMP thread nest based video stream compression and decompression method of claim 1, wherein, YCbCr or YUV data is down-sampled or up-sampled in a 4:2:0 manner.

5. The OpenMP thread nest based video stream compression and decompression method of claim 1, wherein, A frame is vertically divided into multiple equal-sized regions, and each region is executed in multi-thread parallel mode; if the video frame cannot be evenly divided, the last two regions are merged into one region, and the regions do not affect each other, and each region is calculated and processed by a thread using the color phase conversion and sampling algorithm.

6. A video stream compression and decompression system based on OpenMP thread nesting, characterized in that, The method comprises the following steps: The video processing module adopts parallel pipeline processing of video frame data, OpenMP multi-thread parallel technology is adopted to realize parallel pipeline processing, the architecture of parallel pipeline comprises a server end and a client end, the server end and the client end communicate through a Socket, the server end has three threads of frame rate simulation, multi-thread compression and Socket transmission; the client end has three threads of Socket receiving, multi-thread decompression and OpenGL rendering, the six threads execute by judging whether an input queue is empty, execute in parallel when the input queue is not empty, and wait in a loop when the input queue is empty; The compression and decompression module is used for simplifying the original JPEG coding, compressing RGB data into YCbCr data for transmission, and processing the video frame data obtained by the video processing module 1 in the case of using parallel pipeline in the outer thread, adopting OpenMP thread nesting, and processing the compression and decompression of the video frame data in a multi-thread mode. The thread number thread_num of the compression parallelism is the number of block processing, each thread has a corresponding compression start line number line_on and a compression end line number up_limit, the hue conversion formula for compression adopts the BT601 standard, YCbCr or YUV data is down-sampled, and the conversion formula is calculated by using a shift operation. The thread number thread_num of the decompression parallelism is the number of block processing, each thread has a corresponding decompression start line number line_on and a decompression end line number up_limit, the hue conversion formula for decompression adopts the BT601 standard, YCbCr or YUV data is up-sampled, and the conversion formula is calculated by using a shift operation.

Images