A low-bandwidth and low-latency coding compression method based on neural network

Abstract

The present invention discloses a low-bandwidth and low-latency coding and compression method based on a neural network. The method of the present invention first receives a video data code stream sent by an image acquisition device or sensor, and converts it into a data format that can be processed by the system through an image signal processing module, and continuously sends it in an online data stream mode; then, the original video data is denoised by a neural network noise reduction algorithm, and the original video data is subjected to motion detection by a neural network motion detection algorithm, including mean filtering to reduce the video frame; the pixel values ​​input point by point are binarized and written into DDR as auxiliary information; finally, the noise reduction data of the original video is coded and compressed in combination with the motion detection auxiliary information. The method of the present invention is based on convolutional neural network noise reduction and motion detection, and is further reduced and binarized through mean filtering, which ensures the quality of the video frame while achieving efficient coding compression with low bandwidth and low latency.

Description

Technical Field

[0001] The present invention belongs to the technical field of integrated circuits, and in particular relates to a low-bandwidth and low-latency coding compression method based on a neural network. Background Art

[0002] Video is a continuous sequence of images, consisting of consecutive frames, with each frame forming an image. Due to the human eye's persistence of vision, when a frame sequence is played at a certain rate, we perceive a video of continuous motion. Because consecutive frames are highly similar, to facilitate storage and transmission, the original video must be encoded and compressed to remove redundancy in spatial and temporal dimensions. The digitized video signal has a very high data bandwidth, typically exceeding 20MB / s, making it difficult for computers to store and process it. Compression technology typically reduces the data bandwidth to 1-10MB / s, allowing the video signal to be stored and processed by a computer.

[0003] Symmetry is a key feature of compression coding. Symmetry means that compression and decompression take up the same computing power and time, and symmetric algorithms are suitable for real-time compression and transmission of video. In electronic publishing and other multimedia applications, the video is generally pre-compressed and processed before playing, so asymmetric coding can be used. Asymmetry means that a lot of processing power and time are required for compression, while decompression can be played back in real time, that is, compression and decompression are performed at different speeds. Generally speaking, the time to compress a video is much longer than the time to play back (decompress) the video. Conventional compression coding such as Figure 1 shown.

[0004] A CNN (Convolutional Neural Network) is a deep, feedforward neural network that incorporates convolutional computations. Its artificial neurons can respond to surrounding cells within a limited coverage area, making it an excellent choice for processing large images. CNNs are modeled after biological visual perception mechanisms and are capable of both supervised and unsupervised learning. The shared convolution kernel parameters within the hidden layer and the sparsity of inter-layer connections enable CNNs to learn grid-like topology features, such as pixels and audio, with minimal computational effort and consistent results. Summary of the Invention

[0005] The purpose of the present invention is to propose a low-bandwidth and low-latency coding compression method based on neural network.

[0006] At present, the use of neural networks for noise reduction has become a common and mature step in video encoding. The method of the present invention is based on the video data after neural network noise reduction, and adds auxiliary information after neural network motion detection. By taking advantage of the neural network, it is possible to distinguish foreground and background data, find the degree of matching between the previous frame and the current frame, and improve the reliability of encoding; the auxiliary information is first binarized before being written into the DDR, which greatly reduces the bandwidth occupancy rate and achieves low-bandwidth compression; in addition, the entire framework is in an online data stream mode except for reading data from the DDR during video encoding. When writing 1 / 4 frame, 1 / 4 of the Skip information is available for encoding; if additional hardware is used, the Skip information can only be calculated after the information of an entire frame is completely output, and encoding can only begin after the calculation is completed. Compared with conventional compression encoding methods, the encoding compression method proposed in the present invention greatly improves the encoding compression efficiency and achieves low-latency transmission.

[0007] The method of the present invention includes an online video data acquisition module, a neural network noise reduction module, a neural network motion detection module, a pixel value binarization module, and a coding compression module. The details are as follows:

[0008] Step (1) receiving and processing the video signal from the sensor through the image signal processing (ISP) module, and sending the data to the subsequent modules in an online data stream mode;

[0009] Step (2) denoising the original video data using a neural network noise reduction module, and writing the denoised video data into a double data rate synchronous dynamic random access memory (DDR);

[0010] Step (3) performs motion detection on the original video data through a neural network motion detection module, and then performs image reduction processing through mean filtering;

[0011] Step (4) binarizing the pixel value of each pixel in the reduced image and writing the binary value into a double data rate synchronous dynamic random access memory (DDR) as auxiliary information;

[0012] Step (5) uses the noise-reduced video data from step (2) and the auxiliary information from step (4) to encode and compress the video.

[0013] Step (1) is as follows: first, the complementary metal oxide semiconductor sensor CMOS (Complementary Metal Oxide Semiconductor) converts the captured light source signal into raw RAW data; then, the RAW format of the video stream is converted into YUV data that the system can process, and sent to the subsequent modules in an online data stream mode.

[0014] The online data flow mode is specifically as follows: a data enable is sent in each clock cycle, corresponding to a pixel value; according to the data enable, each pixel value of each frame of video data is sent point by point in a top-to-bottom and left-to-right order; except for reading data from DDR during video encoding, the entire framework is in online data flow mode to achieve low-latency transmission.

[0015] Step (2) specifically involves performing a convolution operation on each frame of the online input data using a convolutional neural network (CNN) to train a convolutional neural network model, thereby obtaining denoised video data. Due to noise, video frame detection will fluctuate, and the accuracy of matching blocks cannot be guaranteed. After denoising using the neural network, the video frames are relatively stable, making it easier to distinguish between background and foreground, allowing the encoder to perform skip encoding.

[0016] Step (3) is as follows: motion detection is performed through the convolutional neural network CNN to obtain the mask of each video frame, and the mask of the video frame is reduced by the mean filter, which specifically refers to performing N×N mean filtering on each frame of the video, that is, adding the pixel values ​​of each N×N block and then dividing it by N. 2 , get the pixel mean of each block, N=4,8.

[0017] The raw video data after motion detection is input in an online data stream mode, and one line of data is cached in the cache memory to make up the number of pixels required for mean filtering. To save hardware area overhead, mean filtering and motion detection share the same cache memory, and only one line of data needs to be cached for mean filtering.

[0018] The specific mean filter is:

[0019] (3-1) When the data of row 0 is input, every N pixel values ​​are added and written into the cache memory;

[0020] (3-2) When the first row of data is input, after adding every N pixel values, the data in the cache memory of the previous row is taken out, added to it, and then written to the cache memory;

[0021] (3-3) When the second row of data is input, after adding every N pixel values, the data in the cache memory of the previous row is taken out, added to it, and then written to the cache memory;

[0022] (3-4) This process continues until the N-1th row of data is input. After every N pixel values ​​are added, the data in the cache memory of the previous row is taken out, added to it, and directly passed to the subsequent module.

[0023] (3-5) Repeat the above operation for each N×N block until the last pixel of the last row of the image, and obtain the reduced image.

[0024] Step (4) is specifically as follows: the pixel value P of each pixel of the reduced image is judged according to the threshold T. If P ≥ T, the pixel is marked as 1, otherwise it is marked as 0. The area marked as 1 represents the foreground area, and the area marked as 0 represents the background area. In this way, the original 8-bit pixel value is converted to 1 bit, which greatly reduces the bandwidth occupancy and realizes low-bandwidth compression. The threshold T of binarization is set in the range of [64,128].

[0025] Step (5) specifically comprises: reading the denoised video data and the binarized auxiliary information in the DDR respectively, taking the denoised video data of the previous frame image as the reference frame, adopting the conventional coding mode for the current frame area corresponding to the foreground area of ​​the reference frame, and adopting the skip coding mode for the current frame area corresponding to the background area of ​​the reference frame, thereby reducing the bit rate while minimizing the image quality loss; encoding the denoised video data of the first frame image in the conventional coding mode.

[0026] The present invention proposes a low-bandwidth and low-latency coding and compression method based on a neural network. It is based on convolutional neural network noise reduction and motion detection, and is further reduced and binarized through mean filtering, and auxiliary information is used to reduce the bit rate. While ensuring the quality of video frames, it achieves efficient coding compression with low bandwidth and low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of conventional compression coding;

[0028] Figure 2 Schematic diagram of compression coding of the present invention;

[0029] Figure 3 It is a timing diagram of the online data flow mode;

[0030] Figure 4 4×4 mean filtering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described below with reference to the accompanying drawings. A low-bandwidth, low-latency coding and compression method based on a neural network is described below, with low bandwidth and low latency as the goals.

[0032] like Figure 2 As shown in FIG, a low-bandwidth and low-latency coding and compression method based on a neural network includes an online video data acquisition module, a neural network noise reduction module, a neural network motion detection module, a pixel value binarization module, and a coding and compression module. The details are as follows:

[0033] Step (1) receives and processes the video signal from the sensor through the image signal processing (ISP) module, and sends the data to the subsequent modules in an online data stream mode. Specifically, first, the complementary metal oxide semiconductor (CMOS) sensor converts the captured light source signal into raw RAW data; then, the RAW format of the video code stream is converted into YUV data that can be processed by the system, and sent to the subsequent modules in an online data stream mode.

[0034] like Figure 3 As shown in the figure, the online data streaming mode specifically sends a data enable per clock cycle, corresponding to a pixel value. Based on the data enable, each pixel value of each frame of video data is sent point by point in a top-to-bottom and left-to-right order. Except for reading data from DDR during video encoding, the entire framework uses the online data streaming mode to achieve low-latency transmission.

[0035] YUV data can be in various formats:

[0036] Case a. Single Y format;

[0037] Case b. YUV420 non-interlaced format;

[0038] Case c. YUV420 UV semi-interleaved format;

[0039] Case d. YUV422 non-interleaved format;

[0040] Case e.YUV422 UV semi-interleaved format.

[0041] Step (2) reduces the noise of the original video data through the neural network noise reduction module, and writes the denoised video data into the double-rate synchronous dynamic random access memory DDR; specifically, each frame of the online input data is convolved by the convolutional neural network CNN (Convolutional Neural Network), and a convolutional neural network model is trained to obtain the denoised video data.

[0042] Step (3) performs motion detection on the original video data through the neural network motion detection module, and then reduces the image through mean filtering; specifically, motion detection is performed through the convolutional neural network CNN to obtain the mask of each video frame, and the mask of the video frame is reduced through mean filtering, specifically referring to performing N×N mean filtering on each frame of the video, that is, adding the pixel values ​​of each N×N block and then dividing it by N 2 , get the pixel mean of each block, N=4,8, in this embodiment N=4.

[0043] The raw video data after motion detection is input in an online data stream mode, and one line of data is cached in the cache memory to make up the number of pixels required for mean filtering. To save hardware area overhead, mean filtering and motion detection share the same cache memory, and only one line of data needs to be cached for mean filtering.

[0044] like Figure 4 As shown, taking N=4 as an example, the steps are as follows:

[0045] Step (3-1). When the 0th row of data is input, every 4 pixel values ​​are added and written into the cache memory;

[0046] Step (3-2). When the first line of data is input, after every 4 pixel values ​​are added, the data in the cache memory of the previous line is taken out, added thereto, and then written to the cache memory;

[0047] Step (3-3). When the second line of data is input, after every 4 pixel values ​​are added, the data in the cache memory of the previous line is taken out, added thereto, and then written to the cache memory;

[0048] Step (3-4). When the third line of data is input, after every 4 pixel values ​​are added, the data in the cache memory of the previous line is taken out, added thereto, and directly passed to the subsequent module;

[0049] Step (3-5). Repeat the above operation for each 4×4 block until the last pixel of the last row of the image to obtain the reduced image.

[0050] Step (4) binarizes the pixel value of each pixel in the reduced image and writes it into the double-bit synchronous dynamic random access memory (DDR) as auxiliary information. Specifically, the pixel value P of each pixel in the reduced image is judged according to the threshold T. If P ≥ T, the pixel is marked as 1, otherwise it is marked as 0. The area marked as 1 represents the foreground area, and the area marked as 0 represents the background area. In this way, the original 8-bit pixel value is converted to 1 bit, which greatly reduces the bandwidth usage and achieves low-bandwidth compression. The binarization threshold T is set in the range of [64, 128], with a default value of 100.

[0051] Step (5) uses the denoised video data from step (2) and the auxiliary information from step (4) to encode and compress the video. Specifically, the denoised video data and the binarized auxiliary information in the DDR are read separately, the denoised video data of the previous frame is used as a reference frame, the current frame area corresponding to the foreground area of ​​the reference frame adopts a conventional encoding mode, and the current frame area corresponding to the background area of ​​the reference frame adopts a skip encoding mode, thereby reducing the bit rate while minimizing the image quality loss; the denoised video data of the first frame is encoded in the conventional encoding mode.

[0052] The low-bandwidth and low-latency coding and compression method based on neural networks proposed in the present invention is based on convolutional neural network noise reduction processing, which ensures the quality of video frames while also facilitating the distinction between foreground and background information. The advantages of neural networks are then used for motion detection to obtain auxiliary information; the auxiliary information is first binarized in pixel value before being written into DDR, which greatly reduces bandwidth occupancy and achieves low-bandwidth compression. In addition, the entire framework is in online data stream mode except for reading data from DDR during video encoding. When writing 1 / 4 frame, 1 / 4 of the Skip information is available for encoding; the use of additional hardware requires that the information of an entire frame be completely output before the Skip information can be calculated, and encoding can only begin after the calculation is completed. Compared with this, the coding and compression method proposed in the present invention greatly improves the coding and compression efficiency and achieves low-latency transmission.

Claims

1. A low-bandwidth and low-latency coding compression method based on a neural network, characterized in that: The coding and compression method is implemented by an online video data acquisition module, a neural network noise reduction module, a neural network motion detection module, a pixel value binarization module, and a coding and compression module. The specific steps are as follows: Step (1) receiving and processing the video signal from the sensor through the image signal processing (ISP) module, and sending the data to the neural network noise reduction module in an online data stream mode; Step (2) performing a convolution operation on each frame of the original video data through a neural network noise reduction module, training a convolutional neural network model, thereby obtaining denoised video data, and writing the denoised video data into a double data rate synchronous dynamic random access memory DDR; Step (3) performs motion detection on the original video data through a neural network motion detection module, and then performs image reduction processing through mean filtering; Step (4) binarizing the pixel value of each pixel in the reduced image and writing the binary value into a double data rate synchronous dynamic random access memory (DDR) as auxiliary information; Step (5) uses the denoised video data of step (2) and the auxiliary information of step (4) to encode and compress the video; specifically, the denoised video data and the binarized auxiliary information in the DDR are read respectively, the denoised video data of the previous frame image is used as the reference frame, the current frame area corresponding to the foreground area of ​​the reference frame adopts the conventional encoding mode, the current frame area corresponding to the background area of ​​the reference frame adopts the skip encoding mode, and the denoised video data of the first frame image is encoded in the conventional encoding mode.

2. The low-bandwidth and low-latency coding compression method based on a neural network according to claim 1, characterized in that: Step (1) is specifically as follows: first, the complementary metal oxide semiconductor sensor CMOS converts the captured light source signal into raw RAW data; then, the RAW format of the video stream is converted into YUV data that the system can process, and sent to the neural network noise reduction module in an online data stream mode.

3. The low-bandwidth and low-latency coding compression method based on a neural network as claimed in claim 2, characterized in that: The online data flow mode is specifically as follows: one data enable is sent in each clock cycle, corresponding to one pixel value; according to the data enable, each pixel value of each frame of video data is sent point by point in order from top to bottom and from left to right; except for reading data from DDR during video encoding, the entire framework is in online data flow mode to achieve low-latency transmission.

4. A neural network-based low-bandwidth and low-latency coding and compression method according to claim 1, 2 or 3, characterized in that: Step (3) is as follows: motion detection is performed through convolutional neural network CNN to obtain the mask of each video frame, and the mask of the video frame is reduced by mean filtering, specifically, each frame of the video is subjected to a motion detection. Mean filtering, that is, for each The pixel values ​​of the block are added and then divided by , get the pixel mean of each block; The original video data after motion detection is input in the form of online data stream mode, and a line of data is cached in the cache memory to make up the number of pixels required for mean filtering.

5. The low-bandwidth and low-latency coding compression method based on a neural network according to claim 4, characterized in that: Mean filtering and motion detection share the same cache memory, and only one row of cache is required for mean filtering.

6. The low-bandwidth and low-latency coding compression method based on a neural network according to claim 4, characterized in that: The mean filtering is specifically: (3-1) When the data of row 0 is input, every N pixel values ​​are added and written into the cache memory; (3-2) When the first row of data is input, after adding every N pixel values, the data in the cache memory of the previous row is taken out, added to it, and then written to the cache memory; (3-3) When the second row of data is input, after adding every N pixel values, the data in the cache memory of the previous row is taken out, added to it, and then written to the cache memory; (3-4) This process is repeated until the N-1th row of data is input. After every N pixel values ​​are added, the data in the cache memory of the previous row is retrieved and added to it, and then directly transmitted to the neural network motion detection module. (3-5) For each Repeat the above operation until the last pixel of the last row of the image to obtain the reduced image.

7. The low-bandwidth and low-latency coding and compression method based on a neural network according to claim 4, characterized in that: 。 8. The low-bandwidth and low-latency coding compression method based on a neural network according to claim 4, characterized in that: Step (4) is specifically: the pixel value of each pixel of the reduced image is According to the threshold T, if , then the pixel is marked as 1, otherwise it is marked as 0. The area marked as 1 represents the foreground area, and the area marked as 0 represents the background area.

9. The neural network-based low-bandwidth and low-latency coding and compression method according to claim 8, characterized in that: The setting interval of the binarization threshold T is .

Images