Lightweight video frame corner point extraction device, method and computer device

Abstract

The present application relates to the field of video frame corner point extraction, and particularly relates to a lightweight video frame corner point extraction device, method and computer device, which improves the performance of corner point extraction and reduces the cache capacity. The device comprises a grayscale conversion module, a sliding window module, a line cache module, a corner point extraction module and a vector discrimination module. The grayscale conversion module converts received video data into grayscale image data and sends the grayscale image data to the sliding window module and the corner point extraction module. The grayscale image data is processed by the sliding window and output to the line cache module. The data at the corresponding address in the line cache module is output to the sliding window module. The grayscale image data is output as a vector of a corresponding length after being processed by the corner point extraction module and sent to the vector discrimination module. The vector discrimination module performs corner point discrimination on the input vector. The entire device adopts a multi-stage pipeline structure and is calculated by a pipeline and in parallel. The present application is suitable for corner point extraction of small-capacity embedded platforms.

Description

Technical Field

[0001] This invention relates to the field of video frame corner point extraction, and specifically to a lightweight video frame corner point extraction device, method, and computer device. Background Technology

[0002] Corner point extraction is a crucial technique in computer vision, used to extract key corner points or feature points from image data. These corner points can be considered as part of image sparsity reduction. Corners play a vital role in image processing, representing salient structures and providing excellent descriptions of object edges, textures, and other features. Corner point extraction enables functions such as image stitching, image matching and recognition, and target tracking. With the rapid development of fields like artificial intelligence, autonomous driving, and intelligent surveillance, the demand for video content analysis is constantly increasing, driving the research and application of corner point extraction techniques for video frames.

[0003] When extracting corner points from video frames, the large data volume and high frame rate of video frames mean that corner extraction algorithms typically require significant computing resources and memory. Therefore, high demands are placed on the cache capacity and processing speed of the processing device. Small-capacity embedded platforms, on the other hand, face challenges such as insufficient storage and low performance.

[0004] Therefore, with the continuous improvement of video resolution and high frame rate applications in recent years, research on high-performance, low-capacity corner extraction methods has become more important. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a lightweight video frame corner extraction device, method and computer device, which improves the performance of video frame corner extraction and reduces the buffer capacity during video frame corner extraction.

[0006] The present invention achieves the above-mentioned objectives by adopting the following technical solutions. In the first aspect, the present invention provides a lightweight video frame corner extraction device. The device includes a grayscale conversion module, a sliding window module, a line buffer module, a corner extraction module, and a vector discrimination module. The grayscale conversion module converts the received video data into grayscale image data and sends it to the sliding window module. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module. The corner extraction module processes the input data and outputs a vector of the corresponding length and sends it to the vector discrimination module. The vector discrimination module performs corner determination on the input vector. The device adopts a multi-stage pipeline structure and has pipeline and parallel calculation functions.

[0007] Furthermore, if the video data has a width of W and a height of H, the grayscale image data has a bit width of Pw, and the sliding window has a size of N*N, then the row buffer module consists of N-1 buffer units, each buffer unit consists of a storage unit with a depth of W and a bit width of Pw, and the sliding window consists of N*N register storage units with a bit width of Pw.

[0008] Furthermore, the sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module, specifically including:

[0009] When grayscale image data is input, its coordinates in the image are (x, y). The data is written to the register storage unit at coordinates (N, N) of the sliding window. At the same time, the data stored in the sliding window is shifted to the left, that is, the data in column a is written to the storage unit in column a-1. The data in column 1 is output by sliding, where the data from row 1 to row N-1 is discarded. The data in row N is output to the row buffer module. The data at address x in the row buffer module is output to the sliding window module. When the data (x, y) is output from the sliding window, the data is written to the row buffer module, specifically to the storage unit at address x in the (N-1)th buffer unit of the row buffer module. At the same time, the data at address x in the row buffer module is shifted upwards, and the data at address x in the first buffer unit is discarded.

[0010] Furthermore, the corner extraction module processes the input data and outputs a vector of corresponding length, specifically including:

[0011] The data input through the sliding window is processed by the corner extraction module, which outputs two 16-bit vectors. The first vector is calculated by adding the decision threshold Th to the center corner pixel value C to obtain Tc1, and then adding the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc1. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

[0012] Furthermore, the vector discrimination module performs corner point determination on the input vector, specifically including:

[0013] Each input vector is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner point and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner point and the process exits. Otherwise, if 0 exists and Ptr<=16, set the position of 0 in U2 to Pos1, set Ptr=Ptr+Pos1, and repeat step 1.

[0014] Secondly, the present invention provides a lightweight video frame corner extraction method, applied to the lightweight video frame corner extraction device described above, the method comprising:

[0015] S1. The grayscale conversion module converts the received video data into grayscale image data and sends it to the sliding window module.

[0016] S2. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module.

[0017] S3. The corner extraction module processes the input data and outputs a vector of corresponding length, which is then sent to the vector discrimination module. The vector discrimination module performs corner detection on the input vector.

[0018] Furthermore, step S2 specifically includes:

[0019] When grayscale image data is input, its coordinates in the image are (x, y). The data is written to the register storage unit at coordinates (N, N) of the sliding window. At the same time, the data stored in the sliding window is shifted to the left, that is, the data in column a is written to the storage unit in column a-1. The data in column 1 is output by sliding, where the data from row 1 to row N-1 is discarded. The data in row N is output to the row buffer module. The data at address x in the row buffer module is output to the sliding window module. When the data (x, y) is output from the sliding window, the data is written to the row buffer module, specifically to the storage unit at address x in the (N-1)th buffer unit of the row buffer module. At the same time, the data at address x in the row buffer module is shifted upwards, and the data at address x in the first buffer unit is discarded.

[0020] Furthermore, the specific method by which the corner extraction module processes the input data and outputs a vector of corresponding length includes: the data input through the sliding window passes through the corner extraction module and outputs two 16-bit vectors. The first vector is calculated by adding the decision threshold Th to the pixel value of the center corner point C to obtain Tc1, and pixels P1 to P... 16The calculation is performed in parallel with Tc1. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

[0021] Furthermore, the specific methods used by the vector discrimination module to determine the corner points of the input vector include:

[0022] Each input vector is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner point and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner point and the process exits. Otherwise, if 0 exists and Ptr<=16, set the position of 0 in U2 to Pos1, set Ptr=Ptr+Pos1, and repeat step 1.

[0023] Thirdly, the present invention provides a computer device including a memory storing program instructions, wherein when the program instructions are executed, the lightweight video frame corner extraction method described above is performed.

[0024] The beneficial effects of this invention are as follows:

[0025] This invention improves the performance and efficiency of corner extraction through pipelined and parallel operations. A row-buffered, sliding-window storage structure is used to reduce cache size. Vector discrimination is performed using a state machine and bit shifting, which not only reduces hardware overhead but also improves corner vector decision performance. The overall multi-stage pipelined architecture is suitable for high real-time applications. This device can be used as a peripheral for a SoC chip or for image processing on an FPGA. It features high real-time performance, low capacity, and configurability to meet different image resolution requirements. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the pipeline structure of a lightweight video frame corner extraction device provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of the workflow of the row cache module and the sliding window module provided in this embodiment of the invention;

[0028] Figure 3 This is a schematic diagram of the workflow of the corner extraction module provided in an embodiment of the present invention;

[0029] Figure 4This is a schematic diagram of the workflow of the vector decision module provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, this invention provides a lightweight video frame corner extraction device, which includes: a grayscale conversion module, a line buffer module, a sliding window module, a corner extraction module, and a vector discrimination module. The entire device adopts a four-stage pipeline structure, improving data computation efficiency through pipelined and parallel processing. The grayscale conversion module converts the RGB or YUV data of the received video frame into grayscale image data.

[0032] like Figure 2 The diagram illustrates the workflow of the row buffer module and sliding window module provided in this embodiment of the invention. If the width of the video image is W, the height is H, the grayscale data bit width is Pw, and the sliding window size is N*N, then the row buffer module consists of N-1 buffer units, each consisting of a storage unit with a depth of W and a bit width of Pw. The sliding window consists of N*N register storage units with a bit width of Pw. When grayscale image data is input, the coordinates of this data in the image are (x, y), and the data is written to the register storage unit at coordinates (N, N) in the sliding window. Simultaneously, the data stored in the sliding window is shifted left, i.e., the data in column a is written to the storage unit in column a-1. The data in column 1 is then output via sliding, where data from row 1 to row N-1 is discarded, and the data in row N is output to the row buffer module; the data at address x in the row buffer module is output to the sliding window module. When data (x, y) is output from the sliding window, the data is written to the row cache module, specifically to the storage unit at address x in the (N-1)th cache unit of the row cache module. At the same time, the data at address x in the row cache module moves up, and the data at address x in the first cache unit is discarded.

[0033] Specifically, such as Figure 2 As shown, if N=5, when grayscale image data is input, the data (x, y) is written to the storage unit in the 5th row and 5th column of the sliding window. The data in the 1st to 4th rows of the 5th column are input by the row buffer module. When the data (x, y) is written to the sliding window module, the data with the horizontal coordinate (x-5) slides out, where the first four rows are discarded. The data with the coordinate (x-5, y) is written to the row buffer module, and the write address is the storage unit with the address x-5 in the 4th buffer unit of the row buffer module. At the same time, the data with the address x-5 is passed upwards.

[0034] like Figure 3 The diagram shows the workflow of the corner extraction module provided in this embodiment of the invention. In this embodiment, the corner extraction module uses the FAST corner extraction module. The module input is a sliding window module, and the output is two vectors of length 16 bits. The first vector is calculated by adding the decision threshold Th to the pixel value of the center corner point C to obtain Tc1, and pixels P1 to P... 16 The calculation compares the pixel value with Tc1 in parallel. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the vectors from pixel P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

[0035] like Figure 4 The diagram shows the workflow of the vector decision module provided in this embodiment of the invention. The input is the output vector of the FAST corner extraction module. After each vector is input, it is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner and the process exits. Otherwise, if 0 exists and Ptr<=16, let the position of 0 in U2 be Pos1, let Ptr=Ptr+Pos1, and repeat step 1.

[0036] Based on the lightweight video frame corner extraction device of the present invention, the present invention also provides a lightweight video frame corner extraction method, the method comprising:

[0037] S1. The grayscale conversion module converts the received video data into grayscale image data and sends it to the sliding window module.

[0038] S2. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module.

[0039] S3. The corner extraction module processes the input data and outputs a vector of corresponding length, which is then sent to the vector discrimination module. The vector discrimination module performs corner detection on the input vector.

[0040] In one embodiment of the present invention, step S2 specifically includes:

[0041] When grayscale image data is input, its coordinates in the image are (x, y). The data is written to the register storage unit at coordinates (N, N) of the sliding window. At the same time, the data stored in the sliding window is shifted to the left, that is, the data in column a is written to the storage unit in column a-1. The data in column 1 is output by sliding, where the data from row 1 to row N-1 is discarded. The data in row N is output to the row buffer module. The data at address x in the row buffer module is output to the sliding window module. When the data (x, y) is output from the sliding window, the data is written to the row buffer module, specifically to the storage unit at address x in the (N-1)th buffer unit of the row buffer module. At the same time, the data at address x in the row buffer module is shifted upwards, and the data at address x in the first buffer unit is discarded.

[0042] In one embodiment of the present invention, the specific method for outputting a vector of corresponding length after the grayscale image data passes through the corner extraction module includes:

[0043] The data input through the sliding window is processed by the corner extraction module, which outputs two 16-bit vectors. The first vector is calculated by adding the decision threshold Th to the center corner pixel value C to obtain Tc1, and then adding the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc1. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

[0044] In one embodiment of the present invention, the specific method for the vector discrimination module to determine the corner points of the input vector includes:

[0045] Each input vector is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner point and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner point and the process exits. Otherwise, if 0 exists and Ptr<=16, set the position of 0 in U2 to Pos1, set Ptr=Ptr+Pos1, and repeat step 1.

[0046] The present invention also provides a computer device, including a memory storing program instructions, which, when executed, perform the lightweight video frame corner extraction method described in this invention.

[0047] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A lightweight video frame corner extraction device, characterized in that, The device includes a grayscale conversion module, a sliding window module, a line buffer module, a corner extraction module, and a vector discrimination module. The grayscale conversion module converts the received video data into grayscale image data and sends it to the sliding window module. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module. The corner extraction module processes the input data and outputs a vector of the corresponding length, which is then sent to the vector discrimination module. The vector discrimination module determines the corners of the input vector. The device adopts a multi-stage pipeline structure and has pipeline and parallel calculation functions. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module, specifically including: When grayscale image data is input, the coordinates of the grayscale image data in the image are (x, y). The data is written to the register storage unit at coordinates (N, N) of the sliding window. At the same time, the data stored in the sliding window is shifted to the left, that is, the data in column a is written to the storage unit in column a-1. The data in column 1 is output by sliding, where the data from row 1 to row N-1 is discarded. The data in row N is output to the row buffer module. The data at address x in the row buffer module is output to the sliding window module. When the data (x, y) is output from the sliding window, the data is written to the row buffer module, to the storage unit at address x in the (N-1)th buffer unit of the row buffer module. At the same time, the data at address x in the row buffer module is shifted upward, and the data at address x in the first buffer unit is discarded. The vector discrimination module performs corner point determination on the input vector, specifically including: Each input vector is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner point and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner point and the process exits. Otherwise, if 0 exists and Ptr<=16, set the position of 0 in U2 to Pos1, set Ptr=Ptr+Pos1, and repeat step 1.

2. The lightweight video frame corner extraction device according to claim 1, characterized in that, If the video data has a width of W and a height of H, the grayscale image data has a bit width of Pw, and the sliding window has a size of N*N, then the row buffer module consists of N-1 buffer units, each buffer unit consists of a storage unit with a depth of W and a bit width of Pw, and the sliding window consists of N*N register storage units with a bit width of Pw.

3. The lightweight video frame corner extraction device according to claim 1, characterized in that, The corner extraction module processes the input data and outputs a vector of corresponding length, specifically including: The data input through the sliding window is processed by the corner extraction module, which outputs two 16-bit vectors. The first vector is calculated by adding the decision threshold Th to the center corner pixel value C to obtain Tc1, and then adding the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc1. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

4. A lightweight video frame corner extraction method, applied to the lightweight video frame corner extraction device as described in any one of claims 1-3, characterized in that, The method includes: S1. The grayscale conversion module converts the received video data into grayscale image data and sends it to the sliding window module. S2. The sliding window module processes the grayscale image data and outputs it to the line buffer module and the corner extraction module. The line buffer module processes the input data and outputs the data at the corresponding address to the sliding window module. S3. The corner extraction module processes the input data and outputs a vector of corresponding length, which is then sent to the vector discrimination module. The vector discrimination module performs corner detection on the input vector.

5. A lightweight video frame corner extraction method according to claim 4, characterized in that, Step S2 specifically includes: When grayscale image data is input, its coordinates in the image are (x, y). The data is written to the register storage unit at coordinates (N, N) of the sliding window. At the same time, the data stored in the sliding window is shifted to the left, that is, the data in column a is written to the storage unit in column a-1. The data in column 1 is output by sliding, where the data from row 1 to row N-1 is discarded. The data in row N is output to the row buffer module. The data at address x in the row buffer module is output to the sliding window module. When the data (x, y) is output from the sliding window, the data is written to the row buffer module, specifically to the storage unit at address x in the (N-1)th buffer unit of the row buffer module. At the same time, the data at address x in the row buffer module is shifted upwards, and the data at address x in the first buffer unit is discarded.

6. The lightweight video frame corner extraction method according to claim 5, characterized in that, The specific methods by which the corner extraction module processes the input data and outputs a vector of corresponding length include: The data input through the sliding window is processed by the corner extraction module, which outputs two 16-bit vectors. The first vector is calculated by adding the decision threshold Th to the center corner pixel value C to obtain Tc1, and then adding the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc1. If the pixel value is greater than Tc1, the output is 1; otherwise, the output is 0, resulting in a 16-bit vector. The second vector is calculated by subtracting the decision threshold Th from the pixel value at the center corner C to obtain Tc2, and then calculating the values ​​of pixels P1 to P... 16 The calculation is performed in parallel with Tc2. If the pixel value is less than Tc2, the output is 1; otherwise, the output is 0, resulting in a vector of length 16 bits.

7. A lightweight video frame corner extraction method according to claim 6, characterized in that, The specific methods used by the vector discrimination module to determine corner points of the input vector include: Each input vector is copied and expanded into a 32-bit vector U1. The pointer Ptr is set to 0. Step 1: Extract 12 bits of data from address Ptr to Ptr+11 of U1 into vector U2. Detect the position of the rightmost position 0 in U2. If 0 is not present, it is determined to be a corner point and the process exits. If 0 exists and Ptr>16, it is determined to be a non-corner point and the process exits. Otherwise, if 0 exists and Ptr<=16, let the position of 0 in U2 be Pos1, let Ptr=Ptr+Pos1, and repeat step 1.

8. A computer device comprising a memory storing program instructions, characterized in that, When the program instructions are executed, the lightweight video frame corner extraction method described in any one of claims 4-7 is performed.

Images