A real-time video splicing method based on DSP

Abstract

The present invention belongs to the field of image processing, and discloses a real-time video splicing method based on DSP. The select operation mode is used to select data in a direction, and the data in the discontinuous RAM address is loaded into the vector register by means of gather and scatter instructions; the image registration algorithm part: the threshold is set before matching to pre-screen feature points, and the feature points are matched based on the Hamming distance. The algorithms all use the SIMD instruction set to process multiple data in parallel to improve the calculation efficiency; the PING-PONG mode is used to transmit data, aiming to hide the cycle of external memory access during the video splicing process, thereby effectively improving the data throughput. Finally, the algorithm scheduling table is generated based on the DAG, and the ORB feature extraction algorithm is executed in parallel on the dual DSP platform. The present invention reduces the calculation complexity; and provides a reference for the research of computer vision-related algorithms on the DSP platform.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a real-time video splicing method based on DSP. Background Art

[0002] The goal of video stitching is to stitch multiple videos acquired from different cameras to generate a panoramic video. Compared with a single video, the stitched video can obtain a wider field of view (FOV) and higher video resolution. Real-time video stitching technology involves a variety of technologies, including camera calibration, distortion correction, feature extraction, image matching, and image fusion. Among them, the key technologies of video stitching, feature extraction and image matching, play a vital role in the visual effect of the stitched panoramic video.

[0003] At present, the research on video splicing is mainly to improve the algorithm to achieve higher splicing accuracy, but the speed of algorithm processing cannot meet the requirements of real-time processing. Moreover, the research on real-time acceleration optimization is based on the pipeline processing of instructions by GPU, ignoring the parallel optimization of the algorithm itself. Therefore, the existing video splicing methods have the following disadvantages:

[0004] (1) The traditional ORB feature extraction and image matching algorithms have high computational complexity, and video stitching is difficult to process in real time;

[0005] (2) The data transmission delay between the system memory and DSP is serious, which consumes a lot of time;

[0006] (3) For two images, the ORB feature extraction algorithm is partially repeated, and the time period is extended. Summary of the invention

[0007] The present invention aims to provide a real-time video splicing method based on DSP to solve the above technical problems.

[0008] To solve the above technical problems, a specific technical solution of a real-time video splicing method based on DSP of the present invention is as follows:

[0009] A real-time video splicing method based on DSP comprises the following steps:

[0010] Step 1: ORB feature extraction: Use the select operation mode to select data in a directional manner, and use the gather and scatter instructions to load the data in the discontinuous RAM addresses into the vector register;

[0011] Step 2: Feature matching: Set a threshold to pre-screen feature points before matching, and match feature points based on Hamming distance. The algorithm uses SIMD instruction set to process multiple data in parallel.

[0012] Step 3: System-level optimization: Use the PING-PONG method to transmit data and hide the external memory access cycle during the video stitching process; generate an algorithm scheduling table based on DAG and execute the ORB feature extraction algorithm in parallel on the dual DSP platform;

[0013] Step 4: Image stitching: Perform perspective transformation on the image to be matched, use the homography matrix, and the matching points are calculated after being screened by the RANSAC algorithm. Finally, the two images to be stitched are directly stitched to obtain a panoramic image.

[0014] Furthermore, the step 1 first uses the FAST operator to detect feature points and determine the main direction of the feature points, and finally creates a BRIEF descriptor. The specific steps are as follows:

[0015] Step 1.1: FAST feature detection;

[0016] Step 1.2: Determine the direction of the feature points;

[0017] The grayscale centroid method is used in the ORB algorithm to determine the direction of feature points;

[0018] Step 1.3 BRIEF feature description;

[0019] After the feature points with directions are detected using the FAST operator, these feature points are described. Further, the FAST in step 1.1 is a fast feature point detection algorithm, and the specific steps are as follows:

[0020] Step 1.1.1: Create an image pyramid;

[0021] In the part of establishing the image pyramid, the scale invariance of the feature points is achieved by image scaling, using the Q21.10 data format and a fixed-point method with a 10-bit left shift.

[0022] Step 1.1.3: Fine extraction of feature points;

[0023] FAST-9 feature point detection is used, that is, the absolute value of the difference between the feature point P and 16 points is calculated. If there are 9 consecutive absolute values ​​greater than or less than the threshold TH, it is determined to be a FAST feature point. In addition, the feature point extraction calculation process is optimized based on DSP;

[0024] Step 1.1.4: Non-maximum suppression of feature points;

[0025] In order to solve the problem of excessive density of FAST feature points after extraction, the non-maximum suppression algorithm is adopted. The parallel optimization of the non-maximum suppression algorithm and the parallel implementation of FAST feature point detection are the same. The processing is performed in a 3×3 neighborhood, and the feature points are compared with the surrounding 8 corresponding points in parallel to retain the feature points with the largest response value.

[0026] Furthermore, the specific steps of performing rough extraction of pixels in the image and filtering out most of the pixels in step 1.1.2 are as follows:

[0027] Step 1.1.2.1: Calculate the absolute value of the difference between the point P to be detected and the surrounding 16 points;

[0028] Step 1.1.2.2: Compare the absolute values ​​between symmetrical points. The rules are as follows:

[0029] ξ i =MAX(|PP i |,|PP i+8 |)

[0030] Among them, i∈[0,7], ξ i is the maximum of two absolute values;

[0031] Step 1.1.2.3: If ξ in step 2 i If all of them are greater than the threshold T, point P is retained as a candidate feature point, and then the fine extraction process is carried out;

[0032] The bold extraction process is to traverse every point in the image, and the pixel data is continuously distributed. In order to achieve 64-way parallel calculation, first analyze the point to be detected P and the surrounding pixel points P i The position relationship is that the P6 pixel corresponding to P is also the P I Corresponding to the center Pixel point, according to this rule, infer any P i The next pixel and the next center point P j Correspondingly, in this way, the adjacent pixels and the corresponding center point P j Perform 64-way SIMD comparison;

[0033] The ROTATE_RIGHT mode in the select operation of the DSP platform is used to send 64B data to the DSP for SIMD comparison. The select operation extracts 2N elements from the 4N elements of the two input vectors, and the elements of the two groups of concatenated 64-bit input vectors are shifted right by a fixed amount, and the result is the rightmost vector after the shift.

[0034] Furthermore, the detailed steps of optimizing the feature point extraction calculation process based on DSP in step 1.1.3 are as follows:

[0035] Step 1.1.3.1: In the vector registers [P0,P1], [P2,P3]…[P 14 ,P 15 ] Compare these 8 groups of adjacent pixels and get the maximum value Max0,1 , Max 2,3 …Max 14,15 ;

[0036] Step 1.1.3.2: Compare the eight groups of maximum values ​​in step 1.1.3.1 one by one to get the maximum value of four consecutive pixels Max 0,1,2,3 , Max 2,3,4,5 …Max 14,15,0,1 ;

[0037] Step 1.1.3.3: Compare the two maximum values ​​of the consecutive pixels in step 1.1.3.2 to obtain the maximum value Max of the eight consecutive pixels. 0,1,2,3,4,5,6,7 , Max 2,3,4,5,6,7,8,9 …Max 14,15,0,1,2,3,4,5 ; Step 1.1.3.4: Compare the 8 maximum values ​​in step 1.1.3.3 with the adjacent pixels at both ends to obtain the maximum value of nine consecutive pixels. Use data multiplexing to obtain P8 and P 15 The maximum value, and then 0,1,2,3,4,5,6,7 By comparison, the amount of calculation is further reduced, and finally the maximum values ​​of 8 groups of 9 consecutive pixels are obtained;

[0038] Step 1.1.3.5: If the following formula is satisfied, the candidate feature point is judged as a FAST feature point:

[0039] |P-Max min | <TH

[0040] Where P is the candidate feature point, Max min is the minimum value among the maximum values ​​of 9 consecutive pixels, TH is the threshold, |P-Max min -1|as the response value of FAST feature point;

[0041] The gather / scatter operation mode is used to realize the continuity of scattered data. The gather operation is to load data from a set of discontinuous DRAM addresses into a vector register; scatter is the corresponding reverse operation, which saves the data in the vector register into discrete DRAM addresses; the 64-bit scattered candidate feature points in the DRAM buffer are transferred to the vector register through the gather operation, and the surrounding pixel points corresponding to the candidate feature points are obtained through the same offset address, wherein the offset address is obtained by the load alignment vector LAV operation in DSP, and finally, parallel calculation is performed according to the SIMD characteristics in DSP.

[0042] Furthermore, the step 1.2 includes the following specific steps:

[0043] For any feature point, the grayscale centroid is calculated in a circle with a radius of 15 pixels and the direction from the feature point to the centroid is the direction of the feature point. The pixel moment of the feature point is defined as:

[0044]

[0045] Where p,q∈(0,1).x 2 +y 2 ≤15 2 x,y∈[-15,15], and are integers, I(x,y) is the grayscale value of the pixel (x,y);

[0046] The grayscale centroid of the image block is:

[0047]

[0048] The direction of the feature point in the neighborhood is:

[0049]

[0050] The inverse tangent function will generate a large number of floating-point calculations during the solution process. The fixed-point method of left-shifting to retain precision is used, combined with the lookup table method LUT and the linear approximation method to achieve accelerated calculation. The specific steps are as follows:

[0051] S1: Create the LUT of arctan(x), take the interval [0,1] in x∈(-∞,+∞), divide the interval into 64 equal segments, assign an initial value to each sub-interval to find the corresponding radian value, and the value of the LUT is:

[0052]

[0053] For values ​​outside the [0,1] interval, convert them;

[0054] S2: According to the absolute value of the pixel moment of the feature point |m 10 | and |m 01 |, calculate the value of temporary variable d:

[0055]

[0056] LUT index value index:

[0057] index=d>>(12-6)

[0058] S3: Find the corresponding radian value in the LUT according to the index value: LUT (index). For radian values ​​exceeding the 1 / 64 precision, an approximate linear method is used to compensate. The specific operations are as follows:

[0059] a) Calculate the value beyond the 1 / 64 precision:

[0060] frac=((d-(index<<(12-6)))<<(6+3))

[0061] b) Calculate the difference between index and index+1 in the LUT:

[0062] diff=LUT(index+1)-LUT(index)

[0063] c) Approximate the lookup table interval data corresponding to diff to linearity, and calculate the radian value corresponding to frac that exceeds the 1 / 64 precision:

[0064] rad diff =(frac*diff+(1<<14))>>15

[0065] d) Final radian value:

[0066] rad=LUT(index)+rad diff

[0067] S4: Finally, according to |m 10 | and |m 01 |Relationship adjustment rad:

[0068] a) If |m 10 |<|m 01 |, indicating that |x|>1 in step 1, and steps S1-S3 calculate the arc tangent of 1 / |x|, and the radians are converted to:

[0069]

[0070] b) If m 10 If <0, you need to subtract rad from π, and the radian conversion is:

[0071] rad=(π*(1<<12)+0.5)-rad

[0072] c) If m 01 <0, you need to subtract rad from 2π, and the radian conversion is:

[0073] rad=(2π*(1<<12)+0.5)-rad

[0074] S5: Convert radians to degrees to get the final angle value:

[0075]

[0076] Furthermore, the step 1.3 includes the following specific steps:

[0077] The ORB algorithm uses the BRIEF feature descriptor and takes the direction of the feature point as the main direction of BRIEF. The principle is: select 256 point pairs around the feature point P in a certain pattern, compare the results 1 or 0 and combine them as descriptors, use the gather operation to transfer discontinuous point pairs to a set of 64-way vector registers for parallel comparison. The rules are as follows:

[0078]

[0079] where i∈[0,64], x i and i is the gray value of a random point pair, R1={x0x1…x 63}, R2={y0y1…y 63} is a set of vector registers, △ is an unsigned type less than comparison operator, and the result is output in a 64byte-1bit vector register. Among them, generating a 256-bit ORB descriptor only requires 4 parallel calculations.

[0080] Furthermore, the step 2 includes the following specific steps:

[0081] Based on the Hamming distance, any point pair corresponding to a Hamming distance less than the threshold is determined as a matching point. The Hamming distance is the number of different characters in the corresponding positions of the two strings.

[0082] Based on DSP, ORB feature points are pre-screened and optimized in parallel during the matching process. The specific process is as follows:

[0083] Step 2.1: Filter out feature points in the domain of interest to reduce the calculation of the Hamming distance. The domain of interest is any two feature points in a pair of images that simultaneously meet the following conditions:

[0084]

[0085] Among them, X and Y are the horizontal and vertical coordinates of the feature point, A is the angle of the feature point, and T X ,T Y ,T A are the thresholds of horizontal and vertical coordinates and angles respectively, i and j represent all feature points in the two images;

[0086] Step 2.2: Use the PING-PONG operation to transfer the descriptors in the array to the 64-way, 8-bit register. For the descriptors of size 256-bit, use the XOR operation to calculate the Hamming distance in parallel. Step 2.3: Use the table lookup method to obtain the Hamming distance for the result in step 2.2. The table lookup process is as follows:

[0087] a) Construct a table Table[i] with 256 elements, where i is the decimal number corresponding to the binary (8 bits) and Table[i] is the number of "1" in this binary;

[0088] b) Perform an AND operation on each 8-bit binary in the descriptor and 0xff to obtain the value of i in step a);

[0089] c) Use the i value in b) to look up the number of "1"s in the table, and add them up to get the descriptor Hamming distance;

[0090] d) determining that the point pair corresponding to the Hamming distance less than the threshold is a pair of matching points;

[0091] Step 2.4: Reverse matching, eliminate the wrong matching points, and only re-judge the matching points obtained by forward matching.

[0092] Furthermore, the system-level optimization on the DSP in step 3 is divided into two parts: in the data transmission part, the PING-PONG buffer is used to transmit tiles, which has improved the efficiency of data calculation; in the algorithm processing scheduling part, the entire video splicing process is highly parallelized on the dual DSP platform based on DAG. The specific steps are as follows:

[0093] Step 3.1: PING-PONG transmission of image data;

[0094] Based on DSP and iDMA, a PING-PONG dual buffer mode is designed to transmit tiles in parallel. These buffers can store the information to be processed in the tile respectively. When the DSP core processes the data of tile0 (in 0) in the PING buffer, tile (in1) can be transmitted through iDMA in the PONG buffer. Then the iDMA transmission delay is hidden by the tile processing process.

[0095] Step 3.2: The algorithm is implemented in parallel on dual DSPs;

[0096] Process FAST-9, NMS, and BRIEF algorithms in parallel on two DSPs;

[0097] DAG is a data structure and an effective tool for describing the process of a system. It is expressed as G = (V, E), where G represents a directed acyclic graph; V is the set of nodes in G, representing a specific algorithm; E is the set of edges in G, representing the logical relationship between algorithms. First, the algorithms are mapped to a two-dimensional plane according to the dependency relationship, and the algorithms are numbered in sequence; then DAG is used to represent the execution flow of the splicing algorithm and the constraint relationship between them; according to the logical relationship, corresponding DAGs are created for feature detection, feature point matching, image fusion, and the data flow between them;

[0098] After generating the corresponding algorithm scheduling table based on DAG, the algorithm correlation and dual DSP structure are analyzed, and the dual DSP is used to convert the serial operation of the FAST feature point extraction algorithm of the two images into parallel operation. In order to realize data sharing between DSPs, a shared memory method is adopted.

[0099] The real-time video stitching process is implemented in DSP serial mode in five steps: tile data initialization, ORB feature extraction, feature point matching, image fusion, and the results are written into tiles and saved in memory. Since two images are used for video stitching, the FAST-9, NMS, and BRIEF algorithm processes are calculated twice in a loop and implemented in parallel using dual DSPs.

[0100] A real-time video splicing method based on DSP of the present invention has the following advantages:

[0101] 1. Algorithm optimization: In the ORB feature extraction algorithm, the select operation mode is used to select data in a directional manner, and the data in the discontinuous RAM address is loaded into the vector register with the help of gather and scatter instructions; in the image registration algorithm, a threshold is set before matching to pre-screen feature points, and feature point matching is performed based on the Hamming distance. The algorithms all use the SIMD instruction set to process multiple data in parallel to improve computing efficiency; 2. System-level optimization: The PING-PONG method is used to transmit data, aiming to hide the cycle of external memory access during the video splicing process, thereby effectively improving data throughput. Finally, based on the DAG generation algorithm scheduling table, the ORB feature extraction algorithm is executed in parallel on the dual DSP platform.

[0102] (1) The present invention reduces the computational complexity while ensuring good results;

[0103] (2) Compared with some existing algorithm acceleration methods, the present invention performs real-time video splicing on DSP, which is more innovative and provides a reference for the research of computer vision-related algorithms on the DSP platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is a framework diagram of the real-time video splicing method based on DSP of the present invention;

[0105] Figure 2 It is the FAST feature extraction flow chart of the present invention;

[0106] FIG3( a ) is a schematic diagram of FAST feature detection of the present invention;

[0107] FIG3( b ) is a schematic diagram of FAST feature detection of the present invention;

[0108] FIG4( a ) is a schematic diagram of the Select operation ROTATE_RIGHT mode of the present invention;

[0109] FIG4( b ) is a schematic diagram of the Select operation ROTATE_RIGHT mode of the present invention;

[0110] Figure 5 A schematic diagram of obtaining feature points for the Gather operation of the present invention;

[0111] Figure 6 It is a schematic diagram of the parallel optimization of the BRIEF algorithm of the present invention;

[0112] Figure 7 A schematic diagram of parallel optimization of the image registration algorithm of the present invention;

[0113] Figure 8 It is a schematic diagram of the tile processing sequence of the present invention;

[0114] FIG9( a ) is a schematic diagram of the image stitching algorithm of the present invention mapped to a point in a two-dimensional plane;

[0115] FIG9( b ) is a schematic diagram of the dependency relationship between algorithms of the present invention;

[0116] FIG9( c ) is a schematic diagram of a DAG-based parallel optimization command of the present invention;

[0117] FIG. 10( a ) is a schematic diagram of an algorithm of the present invention being executed serially in sequence on a single DSP;

[0118] FIG. 10( b ) is a schematic diagram of the algorithm of the present invention being executed in parallel on two DSPs. DETAILED DESCRIPTION

[0119] In order to better understand the purpose, structure and function of the present invention, a real-time video splicing method based on DSP of the present invention is further described in detail below with reference to the accompanying drawings.

[0120] 1. ORB feature extraction: Use the select operation mode to select data in a directional manner, and use the gather and scatter instructions to load data from discontinuous RAM addresses into vector registers;

[0121] ORB is an algorithm that quickly extracts feature points and creates feature vectors. First, the FAST operator is used to detect feature points and determine the main direction of the feature points, and finally the BRIEF descriptor is created.

[0122] 1.1 FAST feature detection

[0123] FAST is a fast feature point detection algorithm. The specific process is as follows Figure 2 shown.

[0124] 1.1.1 Building an image pyramid

[0125] In the part of building the image pyramid, the scale invariance of feature points is achieved by image scaling. In order to speed up the calculation and retain the accuracy, we use the Q21.10 data format and calculate it by the fixed-point method with a 10-bit left shift.

[0126] 1.1.2 Rough extraction of feature points

[0127] In order to reduce the amount of calculation for FAST feature point detection, the pixels in the image are first roughly extracted and most of the pixels are filtered out. The specific steps are as follows:

[0128] 1. Calculate the absolute value of the difference between the point to be detected P and the surrounding 16 points. The pixel distribution is shown in Figure 3(a).

[0129] 2. Compare the absolute values ​​between symmetrical points. The rules are as follows:

[0130] ξ i =MAX(|PP i |,|PP i+8 |)

[0131] where i∈[0,7], ξ i is the maximum of two absolute values.

[0132] 3. If ξ in step 2 i If all of them are greater than the threshold T, point P is retained as a candidate feature point, and then the fine extraction process is carried out.

[0133] The bold extraction process is to traverse every point in the image, and the pixel data is continuously distributed. In order to achieve 64-way parallel computing, we first analyze the relationship between the point to be detected P and the surrounding pixel points P i As shown in Figure 3(a), the pixel P6 corresponding to P is also the pixel P. I Corresponding to the center As shown in Figure 3(b), according to this rule, we can infer that any P i The next pixel and the next center point P j In this way, adjacent pixels can be and the corresponding center point P j Perform a 64-way SIMD compare.

[0134] As for the problem of how to send 64B data to the DSP for SIMD comparison, we use the ROTATE_RIGHT mode in the select operation of the DSP platform, where the select operation extracts 2N elements from the 4N elements of the two input vectors. As shown in Figure 4(a), the elements of the two sets of concatenated 64-bit input vectors are shifted right by a fixed amount, and the result is the rightmost vector after the shift. For a pixel, the first 64-bit elements of the pixel are used as the second input vector, and the subsequent 64-bit elements are used as the first input vector. Two units away from the starting address, that is, the number of elements to the right, and the result is saved in the vector register.

[0135] 1.1.3 Feature point extraction

[0136] The present invention adopts FAST-9 feature point detection, that is, the absolute value of the difference between the feature point P and 16 points is calculated, and 9 consecutive absolute values ​​are greater than or less than the threshold TH, which is determined as a FAST feature point. In addition, the feature point extraction calculation process is optimized based on DSP. The detailed steps are as follows:

[0137] 1. In the vector registers [P0,P1], [P2,P3]…[P 14 ,P 15 ] Compare these 8 groups of adjacent pixels and get the maximum value Max 0,1 , Max 2,3 …Max 14,15 ;

[0138] 2. Compare the adjacent two groups of 8 maximum values ​​in step 1 in turn to obtain the maximum value Max of the four consecutive pixels in Figure 3(a) 0,1,2,3 , Max 2,3,4,5 …Max 14,15,0,1 ;

[0139] 3. Compare the two maximum values ​​of the consecutive pixels in step 2 to obtain the maximum value Max of the eight consecutive pixels in Figure 3(a) 0,1,2,3,4,5,6,7 , Max 2,3,4,5,6,7,8,9 …Max 14,15,0,1,2,3,4,5 ;

[0140] 4. Compare the 8 maximum values ​​in step 3 with the adjacent pixels at both ends to get the maximum value of nine consecutive pixels, such as Max 0,1,2,3,4,5,6,7 Respectively with P8 and P 15 Compare and take Max 0,1,2,3,4,5,6,7,8 and Max 15,0,1,2,3,4,5,6,7 Here, the present invention adopts data multiplexing to first obtain P8 and P 15 The maximum value, and then0,1,2,3,4,5,6,7 Comparison can further reduce the amount of calculation. Finally, the maximum value of 8 groups of 9 consecutive pixels is obtained;

[0141] 5. If the following formula is satisfied, the candidate feature point is judged as a FAST feature point:

[0142] |P-Max min | <TH

[0143] Where P is the candidate feature point, Max min It is the minimum value among the maximum values ​​of 9 consecutive pixels, and TH is the threshold. min -1| is used as the response value of the FAST feature point.

[0144] Since the candidate feature points in the DRAM buffer will be discontinuous when the bold candidate feature points are extracted, the load instruction cannot be used directly. In order to facilitate parallel computing, we propose to use the gather / scatter operation to make the scattered data continuous. The gather operation is to load data from a set of discontinuous DRAM addresses into a vector register; scatter is the corresponding reverse operation, which saves the data in the vector register to discrete DRAM addresses.

[0145] like Figure 5 As shown in the figure, the 64-bit scattered candidate feature points in the DRAM buffer are transferred to the vector register through the gather operation. The surrounding pixel points corresponding to the candidate feature points are obtained through the same offset address, where the offset address is obtained by the Load Align Vector (LAV) operation in the DSP. Finally, parallel calculation is performed based on the SIMD feature in the DSP.

[0146] 1.1.4 Non-maximum suppression of feature points

[0147] In view of the excessive density of the extracted FAST feature points, we use the non-maximum suppression algorithm to solve it. The parallel optimization of the non-maximum suppression algorithm and the parallel implementation of FAST feature point detection are the same. The present invention processes in a 3×3 neighborhood, compares the feature point with the surrounding 8 corresponding points in parallel, and retains the feature point with the largest response value.

[0148] 1.2 Determine the direction of feature points

[0149] In order to make the FAST feature points have rotation invariance, the grayscale centroid method is used in the ORB algorithm to determine the direction of the feature points. For any feature point, the grayscale centroid is calculated in a circle with a radius of 15 pixels and the direction from the feature point to the centroid is the direction of the feature point. The pixel moment of the feature point is defined as:

[0150]

[0151] Where p,q∈(0,1).x 2 +y 2 ≤15 2 x,y∈[-15,15] and are integers. I(x,y) is the grayscale value of the pixel (x,y).

[0152] The grayscale centroid of the image block is:

[0153]

[0154] The direction of the feature point in the neighborhood is:

[0155]

[0156] In the process of solving the inverse tangent function, a large number of floating-point calculations will occur. In order to reduce the complexity of the calculation, the present invention adopts a fixed-point method of left-shifting to retain precision, combined with a look-up table (LUT) and a linear approximation method to achieve accelerated calculation. The specific steps are as follows:

[0157] 1. Establish the LUT of arctan(x). In order to reduce the Flash memory occupied by the lookup table, the present invention takes the interval [0,1] in x∈(-∞,+∞). The interval is divided into 64 segments, and an initial value is assigned to each sub-interval to find the corresponding radian value. The value of the LUT is:

[0158]

[0159] For values ​​outside the [0,1] interval, we can convert them. The specific process is shown in step 4.

[0160] 2. According to the absolute value of the pixel moment of the feature point |m 10 | and |m 01 |, calculate the value of temporary variable d:

[0161]

[0162] LUT index value index:

[0163] index=d>>(12-6)

[0164] 3. Find the corresponding radian value in the LUT according to the index value: LUT (index). For radian values ​​exceeding the 1 / 64 precision, the present invention adopts an approximately linear method to compensate. The specific operation is as follows:

[0165] a) Calculate the value beyond the 1 / 64 precision:

[0166] frac=((d-(index<<(12-6)))<<(6+3))

[0167] b) Calculate the difference between index and index+1 in the LUT:

[0168] diff=LUT(index+1)-LUT(index)

[0169] c) Approximate the lookup table interval data corresponding to diff to linearity, and calculate the radian value corresponding to frac that exceeds the 1 / 64 precision:

[0170] rad diff =(frac*diff+(1<<14))>>15

[0171] d) Final radian value:

[0172] rad=LUT(index)+rad diff

[0173] 4. Finally, according to |m 10 | and |m 01 |Relationship adjustment rad:

[0174] a) If | m 10 |<|m 01 |, indicating that |x|>1 in step 1, and steps 1-3 calculate the inverse tangent of 1 / |x|, and the radians are converted to:

[0175]

[0176] b) If m 10 If <0, you need to subtract rad from π, and the radian conversion is:

[0177] rad=(π*(1<<12)+0.5)-rad

[0178] c) If m 01 <0, you need to subtract rad from 2π, and the radian conversion is:

[0179] rad=(2π*(1<<12)+0.5)-rad

[0180] 5. Convert radians to degrees to get the final angle value:

[0181]

[0182] 1.3 BRIEF Feature Description

[0183] After using the FAST operator to detect feature points with directions, these feature points need to be described. The ORB algorithm uses the BRIEF feature descriptor and uses the direction of the feature point as the main direction of BRIEF. The principle is: select 256 point pairs around the feature point P in a certain pattern, compare the results 1 or 0 and combine them as descriptors. Figure 6 As shown, the two points connected by the line segment represent a pair of random points selected in the Gaussian mode, and the black arrow is the direction of the feature point. Since the point pairs are randomly distributed, the present invention uses the gather operation to transfer the discontinuous point pairs to a set of 64-way vector registers for parallel comparison, and the rules are as follows:

[0184]

[0185] where i∈[0,64], x i and i is the gray value of a random point pair, R1={x0x1…x 63}, R2={y0y1…y 63} is a set of vector registers, △ is an unsigned type less than comparison (Less Than Unsigned, LTU) operator. The result is output in a 64byte-1bit vector register. Among them, generating a 256-bit ORB descriptor only requires 4 parallel calculations, which reduces the calculation time compared to the traditional loop calculation method.

[0186] 2. Feature matching: Set a threshold to pre-screen feature points before matching, and match feature points based on Hamming distance. The algorithm uses SIMD instruction set to process multiple data in parallel;

[0187] Feature matching is to match the feature points with descriptors in two images according to certain criteria. The algorithm of the present invention is based on the Hamming distance to make judgments. Any point pair corresponding to the Hamming distance less than the threshold is determined as a matching point. The Hamming distance is the number of different characters in the corresponding positions of two strings.

[0188] The feature matching efficiency based on Hamming distance in the ORB algorithm is very high, but the calculation process is complicated. To solve this problem, the present invention pre-screens the ORB feature points based on DSP and optimizes them in parallel during the matching process. The specific process is as follows:

[0189] 1. Filter out the feature points in the domain of interest to reduce the calculation of the Hamming distance. The domain of interest is any two feature points in a pair of images that simultaneously meet the following conditions:

[0190]

[0191] Among them, X and Y are the horizontal and vertical coordinates of the feature point, A is the angle of the feature point, and T X ,TY ,T A are the thresholds of horizontal and vertical coordinates and angles respectively, and i and j represent all feature points in the two images.

[0192] 2. If Figure 7 As shown, the descriptors in the array are transferred to the 64-way, 8-bit registers through the PING-PONG operation. For the descriptors of size 256-bit, the XOR operation is used to calculate the Hamming distance in parallel.

[0193] 3. Use the table lookup method to obtain the Hamming distance for the result in step 2. The table lookup process is as follows:

[0194] a) Construct a table Table[i] with 256 elements, where i is the decimal number corresponding to the binary (8 bits), and Table[i] is the number of "1"s in this binary. For example, 0000 0111, i=7, Table[7]=3;

[0195] b) Perform an AND operation on each 8-bit binary in the descriptor and 0xff to obtain the value of i in step a).

[0196] c) Use the i value in b) to look up the table to get the number of "1"s, and add them up to get the descriptor Hamming distance.

[0197] d) Determine the point pair corresponding to the Hamming distance less than the threshold as a pair of matching points.

[0198] 4. Reverse matching to eliminate mismatched points. Only the matching points obtained by forward matching (steps 1-3) are re-evaluated.

[0199] The improved algorithm reduces the calculation of the Hamming distance between feature points through the pre-screening step, quickly obtains the value of the Hamming distance through the table lookup method, and reduces the number of loop calculations through parallel processing of data in the 64-way vector register, thereby improving the matching speed of feature points. In addition, after reverse matching, the matching accuracy is improved.

[0200] 3. System-level optimization: Use PING-PONG mode to transmit data and hide the cycle of external memory access during video splicing; generate algorithm scheduling table based on DAG and execute ORB feature extraction algorithm in parallel on dual DSP platform;

[0201] The system-level optimization on DSP is divided into two parts: in the data transmission part, the PING-PONG buffer is used to transmit tiles to improve the efficiency of data calculation; in the algorithm processing scheduling part, the entire video stitching process is highly parallelized on the dual DSP platform based on DAG.

[0202] 3.1 Image Data PING-PONG Transmission

[0203] The resolution of video stitching images is generally above 720P. Due to the limited DRAM capacity in the DSP, it cannot accommodate the entire image. The image needs to be divided into image sub-blocks (tiles) for transmission and processing by operators. In addition, for each frame of the image, the execution process of the Tile Manager in the DSP is: tile data request input, waiting, data processing, and result output. Since there is only one cache area, the data in the tile can only be processed by the operator, and the processing results can be written back to the tile, and then the next one can be transmitted. Serial transmission makes the DSP core frequently wait, resulting in low efficiency.

[0204] In order to solve the problem of low DSP Core utilization caused by data transmission delay, the present invention designs a PING-PONG double buffer mode parallel transmission tile based on DSP and iDMA. These buffer areas can store the information to be processed in the tile respectively. Figure 8 As shown in the figure, when the DSP core processes the data of tile 0 (in 0) in the PING buffer, tile (in1) can be transferred through iDMA in the PONG buffer. Then the iDMA transfer delay is hidden by the tile processing process, and the performance is greatly improved.

[0205] 3.2 Algorithm parallel implementation on dual DSP

[0206] After the data is transmitted in parallel during the video splicing process, considering Figure 1 The ORB feature extraction process processes two images in sequence. In order to eliminate the time complexity brought by the same algorithm calculation process, it is necessary to parallelize the FAST-9, NMS, and BRIEF algorithms in this process on two DSPs.

[0207] The data dependency between adjacent algorithms in video stitching is caused by the image serial execution algorithm. In addition, when executing the image matching algorithm, the feature point descriptors in the two images are required. After analyzing the data dependency relationship of each part, we need a DAG to describe the dependency relationship between algorithms. DAG is a data structure and an effective tool for describing the process of the system. It is generally expressed as G = (V, E). Among them, G represents a directed acyclic graph; V is the set of nodes in G, representing the specific algorithm; E is the set of edges in G, representing the logical relationship between each algorithm. As shown in Figure 9 (a), the algorithm is first mapped to a two-dimensional plane according to the dependency relationship, and the algorithms are numbered in sequence. Then DAG is used to represent the execution flow of the stitching algorithm and the constraint relationship between them. As shown in Figure 9 (b), the corresponding DAG is created for feature detection, feature point matching, image fusion and the data flow between them according to the logical relationship. Finally, Figure 9 (c) shows the parallel execution algorithm based on the DAG command. For the convenience of analysis, the square here represents the same processing time for each part of the algorithm.

[0208] After generating the corresponding algorithm scheduling table based on DAG, the present invention analyzes the algorithm correlation and dual DSP structure, and uses dual DSP to convert the serial operation of the FAST feature point extraction algorithm of two images into parallel operation, thereby improving the utilization rate of multi-core and the computing power of DSP chips. In addition, in order to realize data sharing between DSPs, a shared memory method is adopted.

[0209] As shown in Figure 10(a), the real-time video stitching process is implemented in DSP serial mode in five steps: tile data initialization, ORB feature extraction, feature point matching, image fusion, and the results are written into tiles and saved in memory. Since two images are used for video stitching, the FAST-9, NMS, and BRIEF algorithm processes are calculated twice in a loop. For this problem, the present invention adopts dual DSP parallel implementation. As shown in Figure 10(b), DSP 2 is used to detect the feature points of an image alone, achieving the effect of parallel computing, eliminating repeated feature extraction processes, and optimizing time complexity.

[0210] 4. Image stitching

[0211] If you want to complete image stitching, you also need to perform perspective transformation on the matching images. This requires the use of a homography matrix, which is calculated after the matching points are screened by the RANSAC algorithm. Finally, the two images to be stitched are directly stitched to obtain a panoramic image.

[0212] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A real-time video splicing method based on DSP, characterized in that: The steps include: Step 1: ORB feature extraction: Use the select operation mode to select data in a directional manner, and use the gather and scatter instructions to load the data in the discontinuous RAM addresses into the vector register; First, use the FAST operator to detect feature points and determine the main direction of the feature points, and finally create the BRIEF descriptor. The specific steps are as follows: Step 1.1: FAST feature detection; Step 1.2: Determine the direction of the feature points; The grayscale centroid method is used in the ORB algorithm to determine the direction of feature points; Step 1.3 BRIEF feature description; After the FAST operator is used to detect the feature points with directions, these feature points are described; Step 2: Feature matching: Set a threshold to pre-screen feature points before matching, and match feature points based on Hamming distance. The algorithm uses SIMD instruction set to process multiple data in parallel. Based on the Hamming distance, any point pair corresponding to a Hamming distance less than the threshold is determined as a matching point. The Hamming distance is the number of different characters in the corresponding positions of the two strings. Based on DSP, ORB feature points are pre-screened and optimized in parallel during the matching process. The specific process is as follows: Step 2.1: Filter out feature points in the domain of interest to reduce the calculation of the Hamming distance. The domain of interest is any two feature points in a pair of images that simultaneously meet the following conditions: Among them, X and Y are the horizontal and vertical coordinates of the feature point, A is the angle of the feature point, and T X ,T Y ,T A are the thresholds of horizontal and vertical coordinates and angles respectively, i and j represent all feature points in the two images; Step 2.2: The descriptors in the array are transferred to the 64-way, 8-bit register through the PING-PONG operation. For the descriptors of size 256-bit, the Hamming distance is calculated in parallel using the XOR operation; Step 2.3: The Hamming distance is obtained by using the table lookup method for the result in step 2.

2. The table lookup process is as follows: a) Construct a table Table[i] with 256 elements, where i is the decimal number corresponding to 8 bits of binary and Table[i] is the number of "1" in this binary; b) Perform an AND operation on each 8-bit binary in the descriptor and 0xff to obtain the value of i in step a); c) Use the i value in b) to look up the number of "1"s in the table, and add them up to get the descriptor Hamming distance; d) determining that the point pair corresponding to the Hamming distance less than the threshold is a pair of matching points; Step 2.4: Reverse matching, eliminate the wrong matching points, and re-judge only the matching points obtained by forward matching; Step 3: System-level optimization: Use the PING-PONG method to transmit data and hide the external memory access cycle during the video stitching process; generate an algorithm scheduling table based on DAG and execute the ORB feature extraction algorithm in parallel on the dual DSP platform; The system-level optimization on DSP in step 3 is divided into two parts: in the data transmission part, the PING-PONG buffer is used to transmit tiles, which has improved the efficiency of data calculation; in the algorithm processing scheduling part, the entire video splicing process is highly parallelized on the dual DSP platform based on DAG. The specific steps are as follows: Step 3.1: PING-PONG transmission of image data; Based on DSP and iDMA, a PING-PONG dual buffer mode is designed to transmit tiles in parallel. These buffers store the information to be processed in the tiles respectively. When the DSP core processes the data of tile 0 (in 0) in the PING buffer, tile (in1) is transmitted through iDMA in the PONG buffer. Then the iDMA transmission delay is hidden by the tile processing process. Step 3.2: The algorithm is implemented in parallel on dual DSPs; Process FAST-9, NMS, and BRIEF algorithms in parallel on two DSPs; DAG is a data structure and an effective tool for describing the process of a system. It is expressed as G = (V, E), where G represents a directed acyclic graph; V is the set of nodes in G, representing a specific algorithm; E is the set of edges in G, representing the logical relationship between algorithms. First, the algorithms are mapped to a two-dimensional plane according to the dependency relationship, and the algorithms are numbered in sequence; then DAG is used to represent the execution flow of the splicing algorithm and the constraint relationship between them; according to the logical relationship, corresponding DAGs are created for feature detection, feature point matching, image fusion, and the data flow between them; After generating the corresponding algorithm scheduling table based on DAG, the algorithm correlation and dual DSP structure are analyzed, and the dual DSP is used to convert the serial operation of the FAST feature point extraction algorithm of the two images into parallel operation. In order to realize data sharing between DSPs, a shared memory method is adopted. The real-time video stitching process is implemented in DSP serial mode in five steps: tile data initialization, ORB feature extraction, feature point matching, image fusion, and the results are written into tiles and saved in memory. Since two images are used for video stitching, the FAST-9, NMS, and BRIEF algorithm processes are calculated twice in a loop and implemented in parallel using dual DSPs. Step 4: Image stitching: Perform perspective transformation on the image to be matched, use the homography matrix, and the matching points are calculated after being screened by the RANSAC algorithm. Finally, the two images to be stitched are directly stitched to obtain a panoramic image.

2. The real-time video splicing method based on DSP according to claim 1, characterized in that: The FAST in step 1.1 is a fast feature point detection algorithm, and the specific steps are as follows: Step 1.1.1: Create an image pyramid; In the part of establishing the image pyramid, the scale invariance of the feature points is achieved by image scaling, using the Q21.10 data format and a fixed-point method with a 10-bit left shift. Step 1.1.2: Rough extraction of feature points; Roughly extract the pixels in the image and filter out most of the pixels; Step 1.1.3: Fine extraction of feature points; FAST-9 feature point detection is used, that is, the absolute value of the difference between the feature point P and 16 points is calculated. If there are 9 consecutive absolute values ​​greater than or less than the threshold TH, it is determined to be a FAST feature point. In addition, the feature point extraction calculation process is optimized based on DSP; Step 1.1.4: Non-maximum suppression of feature points; In order to solve the problem of excessive density of FAST feature points after extraction, the non-maximum suppression algorithm is adopted. The parallel optimization of the non-maximum suppression algorithm and the parallel implementation of FAST feature point detection are the same. The processing is performed in a 3×3 neighborhood, and the feature points are compared with the surrounding 8 corresponding points in parallel to retain the feature points with the largest response value.

3. The real-time video splicing method based on DSP according to claim 2, characterized in that: The specific steps of performing rough extraction of pixels in the image and filtering out most of the pixels in step 1.1.2 are as follows: Step 1.1.2.1: Calculate the absolute value of the difference between the point P to be detected and the surrounding 16 points; Step 1.1.2.2: Compare the absolute values ​​between symmetrical points. The rules are as follows: ξ i =MAX(|P-P i |,|P-P i+8 |) where i∈[0,7], ξ i is the maximum of two absolute values; Step 1.1.2.3: If ξ in step 2 i If all of them are greater than the threshold T, point P is retained as a candidate feature point, and then the fine extraction process is carried out; The bold extraction process is to traverse every point in the image, and the pixel data is continuously distributed. In order to achieve 64-way parallel calculation, first analyze the point to be detected P and the surrounding pixel points P i The position relationship is that the P6 pixel corresponding to P is also the P I Corresponding to the center Pixel point, according to this rule, infer any P i The next pixel and the next center point P j Correspondingly, in this way, the adjacent pixel P i j and the corresponding center point P j Perform 64-way SIMD comparison; The ROTATE_RIGHT mode in the select operation of the DSP platform is used to send 64B data to the DSP for SIMD comparison. The select operation extracts 2N elements from the 4N elements of the two input vectors, and the elements of the two groups of concatenated 64-bit input vectors are shifted right by a fixed amount, and the result is the rightmost vector after the shift.

4. The real-time video splicing method based on DSP according to claim 2 is characterized in that: The detailed steps of optimizing the feature point extraction calculation process based on DSP in step 1.1.3 are as follows: Step 1.1.3.1: In the vector registers [P0,P1], [P2,P3]…[P 14 ,P 15 ] Compare these 8 groups of adjacent pixels and get the maximum value Max 0,1 , Max 2,3 …Max 14,15 ; Step 1.1.3.2: Compare the eight groups of maximum values ​​in step 1.1.3.1 one by one to get the maximum value of four consecutive pixels Max 0,1,2,3 , Max 2,3,4,5 …Max 14,15,0,1 ; Step 1.1.3.3: Compare the two maximum values ​​of the consecutive pixels in step 1.1.3.2 to obtain the maximum value Max of the eight consecutive pixels. 0,1,2,3,4,5,6,7 , Max 2,3,4,5,6,7,8,9 …Max 14,15,0,1,2,3,4,5 ; Step 1.1.3.4: Compare the eight groups of maximum values ​​in step 1.1.3.3 with the adjacent pixels at both ends to obtain the maximum value of nine consecutive pixels. Use data multiplexing to obtain P8 and P 15 The maximum value, and then Max 0,1,2,3,4,5,6,7 By comparison, the amount of calculation is further reduced, and finally the maximum values ​​of 8 groups of 9 consecutive pixels are obtained; Step 1.1.3.5: If the following formula is satisfied, the candidate feature point is judged as a FAST feature point: |P-Max min |<TH Where P is the candidate feature point, Max min is the minimum value among the maximum values ​​of 9 consecutive pixels, TH is the threshold, |P-Max min -1|as the response value of FAST feature point; Use gather / scatter operation mode to realize the continuity of scattered data. Gather operation is to load data from a set of discontinuous DRAM addresses into a vector register; scatter is the corresponding reverse operation, which saves the data in the vector register into discrete DRAM addresses. The 64-bit scattered candidate feature points in the DRAM buffer are transferred to the vector register through the gather operation, and the surrounding pixel points corresponding to the candidate feature points are obtained through the same offset address. The offset address is obtained by the load alignment vector LAV operation in the DSP. Finally, parallel calculation is performed according to the SIMD characteristics in the DSP.

5. The real-time video splicing method based on DSP according to claim 1, characterized in that: The step 1.2 includes the following specific steps: For any feature point, the grayscale centroid is calculated in a circle with a radius of 15 pixels and the direction from the feature point to the centroid is the direction of the feature point. The pixel moment of the feature point is defined as: Where p,q∈(0,1).x 2 +y 2 ≤15 2 x,y∈[-15,15], and are integers, I(x,y) is the grayscale value of the pixel (x,y); The grayscale centroid of the image block is: The direction of the feature point in the neighborhood is: The inverse tangent function will generate a large number of floating-point calculations during the solution process. The fixed-point method of left-shifting to retain precision is used, combined with the lookup table method LUT and the linear approximation method to achieve accelerated calculation. The specific steps are as follows: S1: Create the LUT of arctan(x), take the interval [0,1] in x∈(-∞,+∞), divide the interval into 64 equal segments, assign an initial value to each sub-interval to find the corresponding radian value, and the value of the LUT is: For values ​​outside the [0,1] interval, convert them; S2: According to the absolute value of the pixel moment of the feature point |m 10 | and |m 01 |, calculate the value of temporary variable d: LUT index value inedx: inedx=d>>(12-6) S3: Find the corresponding radian value in the LUT according to the index value: LUT (index). For radian values ​​exceeding the 1 / 64 precision, an approximate linear method is used to compensate. The specific operations are as follows: a) Calculate the value beyond the 1 / 64 precision: frac=((d-(index<<(12-6)))<<(6+3)) b) Calculate the difference between index and index+1 in the LUT: diff=LUT(index+1)-LUT(index) c) Approximate the lookup table interval data corresponding to diff to linearity, and calculate the radian value corresponding to frac that exceeds the 1 / 64 precision: rad diff =(frac*diff+(1<<14))>>15 d) Final radian value: row=LUT(index)+row diff S4: Finally, according to |m 10 | and |m 01 |Relationship adjustment rad: a) If |m 10 |<|m 01 |, indicating that |x|>1 in step 1, and steps S1-S3 calculate the inverse tangent of 1 / |x|, and the radians are converted to: b) If m 10 If <0, you need to subtract rad from π, and the radian conversion is: rad=(π*(1<<12)+0.5)-rad c) If m 01 <0, you need to subtract rad from 2π, and the radian conversion is: rad=(2π*(1<<12)+0.5)-rad S5: Convert radians to degrees to get the final angle value:

6. The real-time video splicing method based on DSP according to claim 1 is characterized in that: The step 1.3 includes the following specific steps: The ORB algorithm uses the BRIEF feature descriptor and takes the direction of the feature point as the main direction of BRIEF. The principle is: select 256 point pairs around the feature point P in a certain pattern, compare the results 1 or 0 and combine them as descriptors, use the gather operation to transfer discontinuous point pairs to a set of 64-way vector registers for parallel comparison. The rules are as follows: where i∈[0,64], x i and i is the gray value of a random point pair, R1={x0x1…x 63 }, R2={y0y1…y 63 } is a set of vector registers, △ is an unsigned type less than comparison operator, and the result is output in a 64byte-1bit vector register. Among them, generating a 256-bit ORB descriptor only requires 4 parallel calculations.

Images