A DVS data generation method based on AVS motion estimation coding

Abstract

A DVS data generation method based on AVS motion estimation coding, S1. Reading a video; S2. Obtaining motion estimation: obtaining motion estimation of adjacent prediction frames of the same reference frame, and calculating the residual of two adjacent prediction frames; and S3. Generating DVS data: utilizing the similarity of AVS and DVS motion estimation, and generating DVS data according to the residual of adjacent prediction frames. The method of the invention can effectively generate DVS simulation data with a low amount of calculation, thereby solving the problem of large amount of calculation for generating DVS data due to motion estimation of an optical flow algorithm when using a video to generate DVS simulation data, and quickly generating DVS data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a DVS data generation method based on AVS motion estimation coding. Background Art

[0002] With the popularization of smart terminal devices, the sensors installed in the devices are becoming more and more diversified. DVS (Dynamic Vision Sensor, also known as event camera) is a new computer vision sensor in recent years. DVS is mainly used to collect pixel changes within a region, especially pixel changes caused by object movement, and obtain information about object movement by counting these changes. Compared with ordinary cameras, DVS does not return color images, but returns event graphs, that is, it returns the position, direction and timestamp of object movement within the region. It is mainly used for data collection of movement and change. It is favored and concerned by the industry because of its fast speed, good privacy protection and small amount of data. However, since DVS has not yet been widely used in commercial applications, the data collected by DVS is still relatively small, and the neural network algorithm based on deep learning requires a large amount of training data sets when designing and training DVS related algorithms. At present, most of them use optical flow algorithms to simulate and generate DVS data, but the calculation of optical flow method requires a lot of computing power. At the same time, the digital audio and video coding standard (Audio Video Coding Standard, referred to as AVS) led by China is gradually being promoted and used, and the third generation AVS3 standard has been launched. In the AVS standard, video encoding has the function of motion estimation. The motion vectors of pixel blocks that change between video frames will be calculated through the motion estimation function module.

[0003] In the prior art, video stream data is generally decoded into complete image frames, and then the optical flow algorithm is used to estimate the motion of the video. This will result in a large amount of redundant data being decoded, and the optical flow algorithm consumes a huge amount of computation, especially the optical flow method based on deep neural networks. Summary of the invention

[0004] The present invention provides a DVS data generation method based on AVS motion estimation coding, utilizing the characteristics of AVS motion estimation coding to simulate and generate DVS data, encoding RGB video using an AVS encoder to obtain motion vectors, and encoding the motion vectors using a DVS encoder to generate DVS simulation data, thereby realizing DVS data generation with low computational complexity. The method of the present invention can effectively generate DVS simulation data with low computational complexity, thereby solving the problem of large computational complexity of generating DVS data due to motion estimation of an optical flow algorithm when using video to generate DVS simulation data, and quickly generating DVS data.

[0005] The technical solution of the present invention is as follows:

[0006] A DVS data generation method based on AVS motion estimation coding comprises the following steps: S1. reading a video; S2. obtaining motion estimation: obtaining motion estimation of adjacent prediction frames of the same reference frame, and calculating the residuals of two adjacent prediction frames; and S3. generating DVS data: utilizing the similarity of AVS and DVS motion estimation, and generating DVS data according to the residuals of adjacent prediction frames.

[0007] Preferably, in the above-mentioned DVS data generation method based on AVS motion estimation encoding, in step S1, the AVS codec is used to encode and decode the video, the video in other formats is format converted using the AVS encoder, and the video is decoded using the AVS decoder to obtain the AVS decoded video data stream.

[0008] Preferably, in the above-mentioned DVS data generation method based on AVS motion estimation coding, in step S2, the function of the AVS decoder to read the motion estimation vector is utilized to obtain the motion estimation vector of the video frame and calculate the residual of two adjacent predicted frames.

[0009] Preferably, in the above-mentioned DVS data generation method based on AVS motion estimation coding, in step S2, the position where the pixel block of the current video frame changes relative to the adjacent previous frame of video and the motion direction and timestamp of the corresponding pixel block are determined.

[0010] Preferably, in the above-mentioned DVS data generation method based on AVS motion estimation coding, in step S2, when calculating the data, a timestamp is simulated and generated according to the frame rate of the video, and the formula is:

[0011]

[0012] Among them, t n is the timestamp of the nth frame, n is the nth frame, and F is the video frame rate.

[0013] Preferably, in the above-mentioned DVS data generation method based on AVS motion estimation encoding, in step S3, the position of the pixel block generated by step S2, the direction of motion of the corresponding pixel block, and the simulated timestamp are used as input data of the DVS encoder, encoded by the DVS encoder, and DVS data is output.

[0014] According to the technical solution of the present invention, the beneficial effects produced are:

[0015] The present invention provides a DVS data generation method based on AVS motion estimation coding. In the DVS data generation process, the optical flow method is no longer used for data generation. Instead, according to the motion estimation characteristics of AVS codec, a block coding motion estimation method in a video encoder is used, that is, a motion estimation coding function based on the AVS coding standard is used to quickly obtain a motion estimation vector from a video stream, and the motion estimation vector obtained by the AVS decoder is used as the motion estimation vector required for DVS coding. Finally, DVS data is generated by DVS coding through a DVS encoder under extremely low computational complexity, which effectively solves the problem of large computational complexity of generating DVS data by optical flow motion estimation, and finally realizes the function of generating DVS data under low computational complexity.

[0016] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0018] Figure 1 is a flow chart of a DVS data generation method based on AVS motion estimation coding of the present invention;

[0019] Figure 2 is a schematic diagram of AVS motion estimation coding involved in the DVS data generation method based on AVS motion estimation coding of the present invention; and

[0020] Figure 3 It is a schematic diagram of simulating DVS data of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0022] The principle of the present invention is: using the AVS motion estimation coding characteristics to obtain motion estimation from the AVS video stream, thereby realizing the function of generating DVS data; the present invention uses the AVS codec to encode and decode the video stream to obtain the inter-frame motion estimation of the video, and uses these motion estimations to generate DVS data, thereby achieving the effect of low computational complexity in generating DVS data

[0023] like Figure 1 As shown, the specific steps of the DVS data generation method based on AVS motion estimation coding of the present invention are as follows:

[0024] S1. Read video: Use the AVS codec standard technology to uniformly encode the video into the AVS video encoding format, and use the AVS decoder to decode it to obtain the AVS decoded video data stream.

[0025] At present, there are many commonly used video encodings, the most common ones are MPEG-4, H.264, AVS, etc. In the video inter-frame prediction block coding, the motion estimation coding of each encoder may be different. In the present invention, a unified motion estimation coding specification is required. On the other hand, the AVS codec standard has a mature motion estimation coding specification. At the same time, because it is a newer coding standard, it has absorbed a lot of experience and has its latecomer advantage in the field of coding. Therefore, the present invention uses the AVS codec to encode and decode the video, uses the AVS encoder to convert the format of videos in other formats, and uses the AVS decoder to decode the video to obtain the video data stream after AVS decoding.

[0026] S2. Obtain motion estimation: obtain motion estimation of adjacent prediction frames of the same reference frame, and calculate the residual of two adjacent prediction frames. Use the function of the AVS decoder to read the motion estimation vector, obtain the motion estimation vector of the video frame, and calculate the residual of two adjacent prediction frames.

[0027] In the AVS coding standard, block coding technology is used for inter-frame compression coding, and the block coding technology includes a motion estimation function. Figure 2 Schematic diagram of AVS motion estimation coding involved in the present invention. Figure 2 As shown, the reference frame has a 4*4 pixel block. The difference between the predicted frame and the reference frame is that the pixel block moves from the upper left corner of the image to the lower right corner, and the other pixel areas remain unchanged. Estimating the movement direction and distance of such a pixel block is usually called motion estimation. Obviously, motion estimation can describe the difference between two frames (predicted frame and reference frame) and the specific location where the change occurs. The goal of the present invention is to determine which pixel blocks have changed (i.e., the location where the pixel blocks have changed) in the current video frame relative to the adjacent previous frame of the video, as well as the movement direction and timestamp of the corresponding pixel blocks. AVS motion estimation includes motion vectors, which include movement direction and movement distance. Therefore, it is easy to calculate which pixel blocks have changed in the current frame relative to the previous adjacent frame: when the reference frame is the same, the sum of the motion vectors is the change; when the reference frames are different, the sum of the motion vectors plus the difference between the two reference frames is the change. The corresponding timestamp can be simulated and generated according to the frame rate of the video during data calculation, and the formula is:

[0028]

[0029] t nis the timestamp of the nth frame, n is the nth frame, and F is the video frame rate. In addition, the size of the relevant pixel block is adaptive in the AVS encoder and can be 4*4, 8*8, 16*16, etc.

[0030] S3. Generate DVS data: Utilize the similarity between AVS and DVS motion estimation and generate DVS data based on the residuals of adjacent prediction frames.

[0031] Because DVS data is composed of position, direction of movement, and timestamp, where position is a required element, and direction of movement and timestamp are non-required elements. In the old method, the video frame is decoded, and then the video frame is input into the optical flow algorithm model. After the algorithm model predicts and calculates, the position and direction of the object's movement are predicted. The optical flow algorithm has a relatively large amount of calculation, especially the optical flow calculation based on deep convolutional neural network, which consumes a lot of computing power. In the method of the present invention, through step S2, the position of the pixel block where the change occurs and the direction of movement of the corresponding pixel block and the simulated timestamp are obtained with extremely low calculation amount. It is only necessary to use the position of the pixel block generated by step S2, the direction of movement of the corresponding pixel block, and the simulated timestamp as the input data of the DVS encoder, encode it through the DVS encoder, and output the DVS data, so as to realize the simulation of DVS data (such as Figure 3 So far, all operations of DVS data generation have been completed.

[0032] The optical flow method is computationally complex and time-consuming. It predicts the motion of an object by calculating the residuals of two frames of images. The residual calculation is extremely time-consuming. The method of the present invention utilizes the characteristics of motion estimation coding in the AVS encoding process, and performs residual calculation on the motion estimation of adjacent predicted frames of the same reference frame. The residual results are used to simulate the generation of DVS data. In this method, the residuals in the video encoding are directly used. When a coded video is input, the optical flow method must be thoroughly decoded into an image, and then the residual calculation is performed. That is, the method of the present invention directly uses the already calculated residuals in the video encoding, and there is no need to perform residual calculation. Therefore, compared with the optical flow method, the method of the present invention lacks the residual calculation step with the largest amount of calculation. Therefore, the amount of calculation is much smaller than that of the optical flow method, thereby solving the problem of large amount of calculation caused by the prediction calculation of the optical flow method in the existing methods.

[0033] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A DVS data generation method based on AVS motion estimation coding, characterized in that: The following steps are involved: S1 reads the video, uses the AVS codec, encodes and decodes the video, converts the video in other formats using the AVS encoder, decodes the video using the AVS decoder, and obtains the decoded video data stream AVS; S2. Obtaining motion estimation: obtaining motion estimation of adjacent prediction frames of the same reference frame, calculating the residual of two adjacent prediction frames, using the function of the AVS decoder to read the motion estimation vector, obtaining the motion estimation vector of the video frame, and calculating the residual of the two adjacent prediction frames, Determine the position where the pixel block of the current video frame changes relative to the adjacent previous video frame and the movement direction and time stamp of the corresponding pixel block; and S3. Generate DVS data: Utilize the similarity between AVS and DVS motion estimation and generate DVS data based on the residuals of adjacent prediction frames.

2. The DVS data generation method based on AVS motion estimation coding according to claim 1, characterized in that: In step S2, when calculating data, a timestamp is simulated and generated according to the frame rate of the video, and the formula is: in, is the timestamp of the nth frame, n is the nth frame, and F is the video frame rate.

3. The DVS data generation method based on AVS motion estimation coding according to claim 1, characterized in that: In step S3, the position of the pixel block generated in step S2, the direction of movement of the corresponding pixel block, and the simulated timestamp are used as input data of the DVS encoder, and are encoded by the DVS encoder to output DVS data.

Images