Motion estimation method and device for vehicle-mounted image suitable for straight driving

Abstract

The invention discloses a motion estimation method and equipment suitable for a vehicle-mounted image of linear driving, and the method comprises the steps: firstly, preliminarily determining an estimated value of a displacement based on the displacement rule of different position pixel points between two frames represented by a set motion model when a camera and an automobile perform synchronous driving motion; and block matching is carried out on the basis of the coordinate estimation value, so that an accurate matching result can be obtained in a relatively small range, namely, block matching calculation is carried out by taking the coordinate in the reference frame as the center. According to the method, aiming at the unique use scene of the vehicle-mounted image and the requirement of motion estimation in a high-speed driving scene, the position relation between adjacent frame matching blocks under straight driving is considered, and an accurate motion estimation effect is realized by adopting an overall thought of combining a motion model and an iteration method. The method is low in hardware resource consumption, easy to implement and convenient to implement at low cost.

Description

Technical Field

[0001] The present invention relates to an image processing method and device, and particularly to an image motion estimation method and device. Background Art

[0002] Digital image acquisition uses sensors such as cameras to convert optical signals into electrical signals and stores, transmits, and displays them in digital form. Digital image processing optimizes the acquired digital images for specific usage purposes and scenarios. Common methods include image enhancement and restoration, image coding and compression, image description, etc.

[0003] Motion estimation is a digital image processing method widely used in the fields of video coding and computer vision. Its core idea is to analyze the image information between consecutive frames to determine the motion trajectories of pixels or image blocks in the image sequence, thereby predicting the next frame of the image to achieve the purpose of compressing the data volume or tracking the target. Motion estimation utilizes the inter-frame correlation of videos to reduce redundant information, can improve the coding efficiency, save storage and transmission space, and plays a crucial role in fields such as video compression, target tracking, and 3D reconstruction.

[0004] Motion estimation often adopts the block matching algorithm. The video frame is divided into many non-overlapping blocks, and then within the search range of the reference frame, according to a certain block matching criterion, the most similar matching block of the current block is found. The relative displacement between the matching block and the current block is the motion vector.

[0005] The main task of motion estimation is to find an optimal matching block in the historical reference frame. Usually, the displacement of an object between two adjacent frames in time is not very large. Therefore, starting from the position of the current block, a search can be conducted within a small area around the block at the same position in the historical frame.

[0006] With the development of the intelligent vehicle industry, in-vehicle images will play an increasingly important role in human-vehicle interaction and assisted driving. In-vehicle images refer to multiple frames of video images collected by in-vehicle cameras during driving, including reverse images, driving record images, 360-degree images, etc. Motion estimation, as an important step in the video processing and storage of in-vehicle images, is of great value for improving the subsequent processing effect of images and reducing the resource consumption of operations.

[0007] During driving, in many scenarios, the vehicle runs very fast. That is, in in-vehicle images, the displacement of an object between two adjacent frames in time may change greatly. During the motion estimation using the block matching algorithm, the most similar matching block of the current block often exceeds the general search range of the block matching algorithm, resulting in the inability to accurately achieve block matching or the need to consume additional computing resources to expand the search range of block matching.

[0008] Considering that vehicles often drive in a straight line at high speeds in many scenarios, optimizing the motion estimation method for this scenario can improve the accuracy of block matching, thereby reducing the resource consumption of the matching algorithm and enhancing the performance in aspects such as in-vehicle image compression and noise reduction. Therefore, it is of practical value to propose a motion estimation method suitable for high-speed straight-line driving scenarios. Summary of the Invention

[0009] Object of the Invention: In view of the above-mentioned prior art, a motion estimation method and device for in-vehicle images suitable for straight-line driving are proposed to achieve accurate motion estimation results and have the characteristic of low hardware resource consumption.

[0010] Technical Solution: A motion estimation method for in-vehicle images suitable for straight-line driving includes: Step 1: Perform motion estimation according to prior parameters, including: For two adjacent frames of images, the previous frame is the reference frame and the latter frame is the matching frame; First, for any image block in the matching frame, calculate the coordinate estimation value (x c , y c ) of the center of the match of this image block in the reference frame, x c = x + k(x - x 0 ), y c = y + k(y - y 0 ); where, (x, y) is the center point coordinate of this image block, (x 0 , y 0 ) is the center point coordinate of the matching frame image, k is a proportionality coefficient, k = qv, q is a constant related to the image acquisition device, and v is the driving speed when the in-vehicle camera captures this matching frame; Then, perform block matching calculation within the matching range centered on (x c , y c ) to obtain the matching target of this image block and the corresponding displacement vector; Step 2: Correct the center point coordinates and the coefficient k, including: First, for any successfully matched image block P (x,y) , calculate the difference degree R of the displacement vectors of this image block and other successfully matched image blocks within the set surrounding range; Then, form a set I with the center point coordinates and displacement vectors (x, y, V 1 (x, y), V 2 (x, y)) of each image block in the matching frame that is successfully matched in Step 1 and has a difference degree R less than the preset threshold θ, where V 1 (x, y) and V 2 (x, y) are the abscissa and ordinate of the displacement vector respectively, and the coordinates (x 01 , y 01) is the coordinate of the point with the smallest sum of the squares of the distances from all points in set I to the line determined by the center of the image patch and the corresponding displacement vector; finally, according to the corrected center point coordinates (x 01 , y 01 ), the mean value of the ratio of the displacement vector corresponding to each image patch in set I to the distance from the image patch to the corrected center point is used as the corrected coefficient k; Step 3: When performing motion estimation on the next frame of the current matching frame, the corrected center point coordinates and the corrected coefficient k obtained in Step 2 are used as prior parameters, and Step 1 is re-executed to perform motion estimation, and the center point coordinates and coefficient k are iteratively corrected by executing Step 2, and the motion estimation of each frame is completed through continuous iteration.

[0011] Further, in Step 2, when calculating the difference degree R, for the successfully matched image patch P (x,y) , take several image patches arranged in the form of a k*k matrix centered on the image patch P (x,y) . The successfully matched image patches among them form a set J, and the number of elements in J is N. The specific calculation formula for the difference degree R is: ; where, V 1 (xi, yj) and V 2 (xi, yj) are respectively the abscissa and ordinate of the displacement vector of the image patch P (xi,yj) in the i-th row and j-th column of the k*k matrix.

[0012] Further, in Step 2, the objective function for obtaining the coordinates of the corrected center point is: ; The coordinates obtained by solving this objective function using a method based on mathematical derivation or an enumeration method based on application examples are the coordinates of the corrected center point.

[0013] Further, the correction coefficient is calculated using the following formula: ; where, c is a sign judgment parameter.

[0014] Further, the enumeration method based on application examples includes: searching within a set range around the corrected center point (x’ 0 , y’ 0 ) of the previous frame of the current matching frame, substituting the coordinates of each point within the set range into the objective function respectively, and the coordinates with the minimum value of the objective function are the corrected center point coordinates (x 01 , y 01 ) of the current matching frame.

[0015] Further, the size of the set range is (x’ 0+i,y’ 0 +j), -d ≤ i ≤ d, -d ≤ j ≤ d, where the range value d is taken from 5 to 10.

[0016] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the motion estimation method for on-vehicle images suitable for straight-line driving.

[0017] Beneficial effects: The method of the present invention addresses the unique usage scenarios of on-vehicle images and the requirements for motion estimation in high-speed driving scenarios. Considering the positional relationship between adjacent frame matching blocks during straight-line driving, it adopts an overall idea combining a motion model and an iterative method to achieve accurate motion estimation results. This method consumes less hardware resources in calculation, is easy to implement, and is convenient for low-cost implementation.

[0018] Specifically, the method first determines the basic positional relationship through a motion model. For the scenario of straight-line driving, the relationship between adjacent frames can be simplified to a basically stable center point and other points with larger displacements the farther away from the center point. Based on this basic positional relationship, a search for block matching can be carried out, which can significantly improve the accuracy and reduce the consumption of calculation resources.

[0019] Secondly, prior parameters are calculated using the vehicle speed. The displacement between adjacent frames is strongly correlated with the driving speed of the vehicle. By determining prior displacement parameters based on the driving speed, the convergence speed in the subsequent iterative process can be accelerated.

[0020] Finally, actual parameters are calculated through iteration for motion estimation. Due to the complexity of the actual environment, there will be certain deviations in the displacement data obtained only through the motion model and speed estimation. By continuously correcting through the iterative method until convergence, more accurate results can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the block matching process of this method; Figure 2 It is an enlarged view of the matching range area in the matching frame; Figure 3 It is a schematic diagram of coordinates during the matching operation in step 2; Figure 4 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] The following further explains the present invention with reference to the accompanying drawings.

[0023] A motion estimation method for vehicle-mounted images applicable to high-speed straight driving. When using this method for motion estimation, the following motion model is adopted: There is and only one "fixed point" that does not undergo displacement between two adjacent frames, that is, the center point of the image. The basic displacement amount of the remaining positions outside the center point between two adjacent frames is proportional to the distance from this position to the center point of the image. Denote the proportionality coefficient as k. On the basis of this basic displacement, block matching estimation is performed to obtain an accurate motion estimation result. The specific steps are as follows: Step 1: Calculate the prior parameters as the initial parameters.

[0024] The proportionality coefficient k is positively correlated with the driving speed v of the vehicle. In this embodiment, a linear relationship model is adopted, that is, k = qv, where q is a constant related to the image acquisition device and can be determined through testing. It should be noted that when the vehicle moves forward, k < 0, and when the vehicle moves backward, k > 0. The initial value of the proportionality coefficient is obtained according to the real-time driving speed v of the vehicle for which the vehicle-mounted image is acquired.

[0025] Step 2: Perform motion estimation according to the initial parameters. The specific method is as follows: For two adjacent frames of images, the previous frame is the reference frame and the latter frame is the matching frame. First, divide the matching frame into several image blocks, and use the center coordinates (x, y) of the image block to label any image block, denoted as P (x,y) . In the reference frame, the image block with center coordinates (x, y) is denoted as P' (x,y) . For each image block of the matching frame, according to the above motion model, calculate the coordinate estimation value (x c , y c ) of the matching center of the image block located in the reference frame according to the following formula: x c = x + k(x - x 0 ) y c = y + k(y - y 0 ) Among them, (x 0 , y 0 ) is the center point coordinate of the matching frame image, and it is used as the initial value of the coordinate of the "fixed point". In this embodiment, the matching frame is divided into several image blocks with a pixel size of n * n, and n takes 8 or 16.

[0026] In the method of the present invention, the meaning of the matching center is that when the camera moves synchronously with the vehicle, based on the displacement law of pixel points at different positions between two frames represented by the above motion model, an estimated value of the displacement is initially determined, that is, the coordinate estimation value (x c , y c), and then performing block matching on this basis can obtain accurate matching results within a smaller range, that is, performing block matching calculation with the coordinates (x c , y c ) in the reference frame as the center.

[0027] In this embodiment, according to the given matching range size r*r, the image block P' in the reference frame is sequentially taken out ( x c +i, y c +j ) , -r ≤ i ≤ r, -r ≤ j ≤ r, and the value of r is generally taken as 2 to 3 times the side length of the image block, and the matching difference D(i, j) is calculated according to the following formula.

[0028]

[0029] Among them, P' ( x, y ) (k, l) represents the point with coordinates (k, l) in the image block P' ( x, y ) , P' ( x c +i, y c +j ) (k, l) represents the point with coordinates (k, l) in the image block P' ( x c +i, y c +j ) , as shown in Figure 3 . Find the minimum value of D(i, j) in the set matching range, denoted as D(i', j'), and compare this minimum value D(i', j') with the preset threshold. If it is less than the threshold, the obtained image block P' ( x c +i', y c +j' ) is the matching target, otherwise it is considered that the matching fails, and this image block is marked as an image block that fails to be successfully matched. The reason for the matching failure is generally that the scenery within the range of this image block also moves at a high speed synchronously during the vehicle driving process, that is, it violates the motion model basis set by the present invention. In this case, the motion matching algorithm of the present invention is terminated for this image block.

[0030] For the image block P that successfully obtains the matching target (x,y) , record the corresponding displacement vector as V(x, y) = (x c +i' - x, y c +j' - y), and record the abscissa and ordinate of this displacement vector as: V 1 (x, y), V 2 (x, y), that is, V(x, y) = (V1 (x,y), V 2 (x,y)), V 1 (x,y) = x c + i’ - x, V 2 (x,y) = y c + j’ - y。 Figure 1 and Figure 2 represents the process of block matching, Figure 1 where the left side is the matching frame with one image block marked, the right side is the reference frame, the small dashed box in the reference frame is the image block at the corresponding position to the matching frame, the dashed arrow indicates the matching center estimated based on the motion model, and the large dashed box represents the further given matching range; the solid image block is the finally matched target image block, and the solid arrow is the displacement vector V(x,y). Figure 2 is Figure 1 an enlarged view of the matching range area in the matching frame, which can more clearly show the process of matching and determining the displacement vector.

[0031] Step 3: Correct the center point coordinates and the coefficient k.

[0032] In the motion model of the present invention, the displacement part between two frames is caused by the overall movement of the image brought about by the linear movement of the camera synchronously during the straight-line driving of the vehicle. Based on this principle, the image block of the matching target in Step 2 is obtained. If the scenery in the image itself also moves during the process, the influence of the movement of the scenery itself on the matching needs to be excluded.

[0033] Specifically, for each successfully matched image block P (x,y) , take a total of 25 image blocks arranged in a 5*5 matrix form centered on it. After these image blocks pass through Step 2, some may be successfully matched and some may not. The successfully matched image blocks are grouped into a set J, and the number of elements in J is N. Calculate the difference degree R between the displacement vector of the image block P (x,y) and the displacement vectors of the surrounding image blocks and compare it with a preset threshold θ. The specific calculation method of the difference degree R is as follows:

[0034] where, V 1 (xi,yj) and V 2 (xi,yj) are respectively the abscissa and ordinate of the displacement vector of the image block P (xi,yj) at the i-th row and j-th column in the 5*5 matrix. R is a physical quantity representing the difference degree between the displacement of the image block P (x,y) and the displacements of the surrounding image blocks. Since the displacement in the horizontal / vertical direction is only related to the horizontal / vertical distance from the image block to the center point respectively, the horizontal and vertical directions are calculated separately, and V 1 (x,y) and V2 (x, y) is normalized to eliminate the influence of the distance to the center point on the difference. R is compared with the preset threshold θ. When R < θ, the matching result obtained in step 2 is retained. When R ≥ θ, it is considered that the scene itself has also moved synchronously, and the matching result of the corresponding image block is not involved in the subsequent calculation. Among them, the value of θ is generally [0.1, 0.2].

[0035] Then, all vectors (x, y, V 1 (x,y),V 2 (x, y)) form a set I, then the corrected center point is the point among all points where the sum of the squares of the distances to the center of the image block and the straight line determined by the displacement vector corresponding to the image block is the smallest. The coordinates of the corrected center point are recorded as (x 01 ,y 01 ),but: .

[0036] According to the motion model of the present invention, the corrected center point should be the intersection of all vector extension lines. Under actual conditions, due to the discreteness of the data and some deviations, they may not strictly intersect at one point. However, from a statistical point of view, the point with the smallest sum of square distances from these straight lines can be used as the estimated center point. The practical significance of the above formula is to use the point-to-straight line distance formula to calculate the distance from each straight line to the point (x 0 ,y 0 ) and then find the sum of squares. The objective function can be solved by a method based on mathematical deduction or an enumeration method based on application examples according to the hardware computing resources for executing the method.

[0037] 1. Based on the mathematical derivation method, the above formula is converted into (x 0 ,y 0 ) and simplify it to:

[0038] For x 0 and 0 Find partial derivatives and solve simultaneous equations:

[0039] The solution is:

[0040] 2. In the enumeration method based on application examples, considering that the displacement of the center point in each iteration is usually small, the search is performed within a certain range d centered on the center point determined in the previous frame. d is generally set to 5~10. For the corrected center point (x' 0 ,y' 0)The surrounding points (x’ 0 + i, y’ 0 + j), where -d ≤ i ≤ d, -d ≤ j ≤ d. Substitute these points into the above objective function to find the point that minimizes the value of the objective function.

[0041] The coordinate values obtained by solving through the above method based on mathematical derivation or the enumeration method based on application examples are the corrected center point coordinates.

[0042] When the driving speed of the vehicle is slow, a smaller range d can meet the requirements, and at this time, the enumeration method based on application examples is more resource-saving in terms of calculation. When the driving speed of the vehicle is fast, a larger range d is required. The enumeration method consumes more computing resources, and the method based on mathematical derivation is more resource-saving. Specifically, the corresponding method can be adaptively selected according to the driving situation of the vehicle.

[0043] After recording the coordinates of the corrected center point as (x 01 , y 01 ), the coefficient k is corrected using the following formula:

[0044] The corrected k is obtained. Its meaning is to calculate the ratio of the displacement vector corresponding to the image block to the distance from the image block to the corrected center point. And this formula calculates this ratio for the image blocks in set I, that is, all the image blocks in this frame whose displacements satisfy the motion model of the present invention, and then calculates the average value. The k obtained thereby is used as the estimate of this ratio for the next frame. Among them, c is a sign judgment parameter, which calculates the dot product of each displacement vector and distance vector, and sums them after normalization. When c > 0, the displacement vector and the distance vector are in the same direction (corresponding to the vehicle moving forward), otherwise they are in the opposite direction (corresponding to the vehicle moving backward), and its sign is used as the sign of k. Here, the sign judgment is only a high-accuracy illustration, and in practice, it can be simplified to a certain extent according to needs. For example, only several image blocks with the farthest distances are selected for judgment.

[0045] Step 4: When performing motion estimation on the next frame of the current matching frame, the corrected center point coordinates (x 01 , y 01 ) and the corrected coefficient k obtained in Step 3 are used as prior parameters, and Step 2 is re-executed to perform motion estimation, and the center point and coefficient k are iteratively corrected by executing Step 3. The motion estimation of each frame is completed through continuous iteration, and the overall process is as Figure 4 shown.

[0046] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the above motion estimation method for on-vehicle images applicable to straight-line driving.

[0047] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A motion estimation method for vehicle-mounted images suitable for straight-line driving, characterized in that: include: Step 1: Motion estimation based on prior parameters, including: For two adjacent frames, the previous frame is the reference frame and the next frame is the matching frame. First, for any image block in the matching frame, the coordinate estimate (x c ,y c ), x c =x+k(x-x0),y c =y+k(y-y0); where (x,y) is the coordinate of the center point of the image block, (x0,y0) is the coordinate of the center point of the matching frame image, k is the scale factor, k=qv, q is a constant related to the image acquisition device, and v is the driving speed when the vehicle camera captures the matching frame; then, c ,y c ) to perform block matching calculation within the matching range centered at , and obtain the matching target of the image block and the corresponding displacement vector; Step 2: Correct the center point coordinates and coefficient k, including: First, for any successfully matched image block P (x,y) , calculate the difference R of the displacement vector between the image block and other image blocks that are also successfully matched within the set surrounding range; then, the center point coordinates and displacement vectors (x, y, V1(x, y), V2(x, y)) of each image block that is successfully matched in step 1 and whose difference R is less than the preset threshold θ in the matching frame are combined into a set I, where V1(x, y) and V2(x, y) are the horizontal and vertical coordinates of the displacement vector respectively, and the coordinates of the corrected center point (x 01 ,y 01 ) is the coordinate of the point in set I where the sum of the squares of the distances to the center of the image block and the straight line determined by the corresponding displacement vector is the smallest; finally, according to the corrected center point coordinates (x 01 ,y 01 ), taking the average of the ratios of the displacement vectors corresponding to the image blocks in the set I and the distances from the image blocks to the corrected center point as the corrected coefficient k; Step 3: When performing motion estimation on the next frame of the current matching frame, the corrected center point coordinates and the corrected coefficient k obtained in step 2 are used as prior parameters, and step 1 is re-executed to perform motion estimation. The center point coordinates and the coefficient k are iteratively corrected by executing step 2, and the motion estimation of each frame is completed through continuous iteration.

2. The motion estimation method for vehicle-mounted images suitable for straight-line driving according to claim 1, characterized in that: In step 2, when calculating the difference R, for the successfully matched image block P (x,y) , take the image block P (x,y) A number of image blocks are arranged in the form of a k*k matrix with centered on , and the image blocks that are successfully matched are grouped into a set J, where the number of elements in J is N. The specific calculation formula for the difference R is: ； Among them, V1(xi,yj) and V2(xi,yj) are the image blocks P in the i-th row and j-th column of the k*k matrix respectively. (xi,yj) The horizontal and vertical coordinates of the displacement vector.

3. The motion estimation method for vehicle-mounted images suitable for straight-line driving according to claim 2, characterized in that: In step 2, the objective function for obtaining the coordinates of the corrected center point is: The coordinates obtained by solving the objective function using a method based on mathematical deduction or an enumeration method based on application examples are the coordinates of the corrected center point.

4. The motion estimation method for vehicle-mounted images applicable to straight-line driving according to any one of claims 1 to 3, characterized in that: The correction factor is calculated using the following formula: ; Where c is the symbol judgment parameter.

5. The motion estimation method for vehicle-mounted images suitable for straight-line driving according to claim 3, characterized in that: The enumeration method based on application examples includes: searching within a set range around the center point (x'0, y'0) corrected by the previous frame of the current matching frame, substituting the coordinates of each point within the set range into the objective function, and the coordinate with the minimum value of the objective function is the coordinate of the center point (x'0, y'0) corrected by the current matching frame. 01 ,y 01 ).

6. The motion estimation method for vehicle-mounted images suitable for straight-line driving according to claim 5, characterized in that: The size of the setting range is (x'0+i, y'0+j), -d≤i≤d, -d≤j≤d, and the range value d is 5~10.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the motion estimation method of vehicle-mounted images applicable to straight-line driving as described in any one of claims 1-6 is implemented.

Images