A Motion Estimation Method and Device for On-Vehicle Images Suitable for Straight Driving

Abstract

The present invention discloses a motion estimation method and device for on-vehicle images suitable for straight-line driving. When the camera moves synchronously with the vehicle, based on the displacement law of pixel points at different positions represented by a set motion model between two frames, an estimated value of the displacement is initially determined, that is, the coordinate estimated value of the matching center is obtained. Then, block matching is performed on this basis, and an accurate matching result can be obtained within a smaller range, that is, block matching calculation is performed with the coordinates in the reference frame as the center. The method of the present invention aims at the unique usage scenario of on-vehicle images and the requirements for motion estimation in high-speed driving scenarios. Considering the positional relationship between adjacent frame matching blocks under straight-line driving, the overall idea of combining a motion model and an iterative method is adopted to achieve an accurate motion estimation effect. This method consumes less hardware resources in calculation, is easy to implement, and is convenient for low-cost implementation.

Description

Technical Field

[0001] The present invention relates to an image processing method and device, and particularly to an image motion estimation method and device. Background Art

[0002] Digital image acquisition uses sensors such as cameras to convert optical signals into electrical signals and stores, transmits, and displays them in digital form. Digital image processing optimizes the acquired digital images for the intended use and scenario. Common methods include: image enhancement and restoration, image coding and compression, image description, etc.

[0003] Motion estimation is a digital image processing method widely used in the fields of video coding and computer vision. Its core idea is to analyze the image information between consecutive frames to determine the motion trajectories of pixels or image blocks in the image sequence, thereby predicting the next frame of the image to achieve the purpose of compressing the data volume or tracking the target. Motion estimation utilizes the inter-frame correlation of videos to reduce redundant information, can improve the coding efficiency, save storage and transmission space, and plays a crucial role in fields such as video compression, target tracking, and 3D reconstruction.

[0004] Motion estimation often adopts the block matching algorithm. The video frame is divided into many non-overlapping blocks, and then within the search range of the reference frame, according to a certain block matching criterion, the most similar matching block of the current block is found. The relative displacement between the matching block and the current block is the motion vector.

[0005] The main task of motion estimation is to find an optimal matching block in the historical reference frame. Usually, the displacement of the object between the two adjacent frames in time is not very large. Therefore, the search can be carried out in a small area around the block at the same position in the historical frame starting from the position of the current block.

[0006] With the development of the intelligent vehicle industry, in-vehicle images will play an increasingly important role in human-vehicle interaction and assisted driving. In-vehicle images refer to multiple frames of video images collected by in-vehicle cameras during driving, including reverse images, driving record images, 360-degree images, etc. Motion estimation, as an important step in the video processing and storage of in-vehicle images, is of great value for improving the subsequent processing effect of images and reducing the resource consumption of operations.

[0007] During driving, in many scenarios, the vehicle runs very fast. That is, in in-vehicle images, the displacement of the object between two adjacent frames in time may change greatly. During the motion estimation using the block matching algorithm, the most similar matching block of the current block often exceeds the general search range of the block matching algorithm, resulting in the inability to accurately achieve block matching or the need to consume additional computing resources to expand the search range of block matching.

[0008] Considering that the vehicle travels in a straight line at high speed in many scenarios, optimizing the motion estimation method for this scenario can improve the accuracy of block matching, thereby reducing the resource consumption of the matching algorithm and enhancing the performance in aspects such as in-vehicle image compression and noise reduction. Therefore, it has practical value to propose a motion estimation method suitable for the high-speed straight-line driving scenario. Summary of the Invention

[0009] Object of the Invention: Aiming at the above-mentioned existing technologies, a motion estimation method and device for on-vehicle images suitable for straight-line driving are proposed to achieve accurate motion estimation effects and have the characteristics of low hardware resource consumption.

[0010] Technical Solution: A motion estimation method for on-vehicle images suitable for straight-line driving includes:

[0011] Step 1: Perform motion estimation according to prior parameters, including:

[0012] For two adjacent frames of images, the previous frame is the reference frame and the latter frame is the matching frame; first, for any image block in the matching frame, calculate the coordinate estimation value (x c , y c ) of the matching center of this image block located in the reference frame, x c = x + k(x - x0), y c = y + k(y - y0); where, (x, y) is the center point coordinate of this image block, (x0, y0) is the center point coordinate of the matching frame image, k is the proportionality coefficient, k = qv, q is a constant related to the image acquisition device, and v is the driving speed when the vehicle-mounted camera captures this matching frame; then, perform block matching calculation within the matching range centered on (x c , y c ) to obtain the matching target of this image block and the corresponding displacement vector;

[0013] Step 2: Correct the center point coordinates and the coefficient k, including:

[0014] First, for any successfully matched image block P (x,y) , calculate the difference degree R of the displacement vectors of this image block and other successfully matched image blocks within the set surrounding range; then, form a set I with the center point coordinates and displacement vectors (x, y, V1(x, y), V2(x, y)) of each image block in the matching frame that is successfully matched in Step 1 and has a difference degree R less than the preset threshold θ, where V1(x, y) and V2(x, y) are the abscissa and ordinate of the displacement vector respectively. The corrected center point coordinates (x 01 , y 01 ) are the coordinates of the point with the smallest sum of the squares of the distances to the straight line determined by the center of the image block and the corresponding displacement vector among all points in the set I; finally, according to the corrected center point coordinates (x01 , y 01 ), the average value of the ratio of the displacement vector corresponding to each image block in the set I to the distance from the image block to the corrected center point is used as the corrected coefficient k;

[0015] Step 3: When performing motion estimation on the next frame of the current matching frame, the corrected center point coordinates and the corrected coefficient k obtained in Step 2 are used as prior parameters, and Step 1 is re-executed to perform motion estimation, and the center point coordinates and the coefficient k are iteratively corrected by executing Step 2, and the motion estimation of each frame is completed through continuous iteration.

[0016] Further, in Step 2, when calculating the difference degree R, for the successfully matched image block P (x,y) , take several image blocks arranged in the form of a k*k matrix centered on the image block P (x,y) . The successfully matched image blocks among them form a set J, and the number of elements in J is N. Then the specific calculation formula for the difference degree R is:

[0017] ; where V1(xi, yj) and V2(xi, yj) are respectively the abscissa and ordinate of the displacement vector of the image block P (xi,yj) at the i-th row and j-th column in the k*k matrix.

[0018] Further, in Step 2, the objective function for obtaining the coordinates of the corrected center point is:

[0019] ; The coordinates obtained by solving this objective function using a method based on mathematical derivation or an enumeration method based on application examples are the coordinates of the corrected center point.

[0020] Further, the correction coefficient is calculated using the following formula:

[0021] ; where c is a symbol judgment parameter.

[0022] Further, the enumeration method based on application examples includes: searching within a set range around the corrected center point (x’0, y’0) of the previous frame of the current matching frame, substituting the coordinates of each point within the set range into the objective function respectively, and the coordinates with the minimum value of the objective function are the coordinates of the corrected center point (x 01 , y 01 ) of the current matching frame.

[0023] Further, the size of the set range is (x’0 + i, y’0 + j), -d ≤ i ≤ d, -d ≤ j ≤ d, and the range value d takes 5 to 10.

[0024] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the motion estimation method for on-vehicle images applicable to straight-line driving.

[0025] Beneficial effects: The method of the present invention addresses the unique usage scenarios of on-vehicle images and the requirements for motion estimation in high-speed driving scenarios. Considering the positional relationship between adjacent frame matching blocks during straight-line driving, it adopts the overall idea of combining a motion model and an iterative method to achieve accurate motion estimation results. This method consumes little hardware resources for calculation, is easy to implement, and is convenient for low-cost implementation.

[0026] Specifically, this method first determines the basic positional relationship through a motion model. For the scenario of straight-line driving, the relationship between adjacent frames can be simplified to a basically stable center point and other points where the displacement is larger the farther away from the center point. Based on this basic positional relationship, the search for block matching can significantly improve the accuracy and reduce the consumption of computing resources.

[0027] Secondly, prior parameters are calculated using the vehicle speed. The displacement between adjacent frames is strongly correlated with the driving speed of the vehicle. By determining the prior displacement parameters based on the driving speed, the convergence speed in the subsequent iterative process can be accelerated.

[0028] Finally, the actual parameters are calculated through iteration for motion estimation. Due to the complexity of the actual environment, there will be certain deviations in the displacement data obtained only through the motion model and speed estimation. By continuously correcting through the iterative method until convergence, more accurate results can be obtained. Description of the Drawings

[0029] Figure 1 It is a schematic diagram of the block matching process of this method;

[0030] Figure 2 It is an enlarged view of the matching range area in the matching frame;

[0031] Figure 3 It is a coordinate schematic diagram during the matching operation in step 2;

[0032] Figure 4 It is a flowchart of the method of the present invention. Detailed Embodiment

[0033] The following further explains the present invention with reference to the drawings.

[0034] A motion estimation method for vehicle-mounted images applicable to high-speed straight driving. When using this method for motion estimation, the following motion model is adopted: There is and only one "fixed point" that does not undergo displacement between two adjacent frames, that is, the center point of the image. The basic displacement amount of the remaining positions outside the center point between two adjacent frames is proportional to the distance from this position to the center point of the image. Denote the proportionality coefficient as k. On the basis of this basic displacement, block matching estimation is performed to obtain an accurate motion estimation result. The specific steps are as follows:

[0035] Step 1: Calculate the prior parameters as the initial parameters.

[0036] The proportionality coefficient k is positively correlated with the driving speed v of the vehicle. In this embodiment, a linear relationship model is adopted, that is, k = qv, where q is a constant related to the image acquisition device and can be determined through testing. It should be noted that when the vehicle moves forward, k < 0, and when the vehicle moves backward, k > 0. The initial value of the proportionality coefficient is obtained according to the real-time driving speed v of the vehicle in the images collected by the vehicle-mounted camera.

[0037] Step 2: Perform motion estimation according to the initial parameters. The specific method is as follows:

[0038] For two adjacent frames of images, the previous frame is the reference frame and the latter frame is the matching frame. First, divide the matching frame into several image blocks, and mark any image block with the center coordinates (x, y) of the image block, denoted as P (x,y) . In the reference frame, the image block with the center coordinates (x, y) is denoted as P’ (x,y) . For each image block of the matching frame, according to the above motion model, calculate the coordinate estimation value (x c , y c ) of the matching center of the image block located in the reference frame according to the following formula:

[0039] x c = x + k(x - x0)

[0040] y c = y + k(y - y0)

[0041] Among them, (x0, y0) is the center point coordinates of the matching frame image, and it is used as the initial value of the coordinates of the "fixed point". In this embodiment, the matching frame is divided into several image blocks with a pixel size of n*n, and n takes 8 or 16.

[0042] In the method of the present invention, the significance of the matching center is that when the camera moves synchronously with the vehicle, based on the displacement law of pixel points at different positions between two frames represented by the above motion model, a displacement estimation value is initially determined, that is, the coordinate estimation value (x c , y c), and then performing block matching on this basis can obtain accurate matching results within a smaller range, that is, performing block matching calculation with the coordinates (x c , y c ) in the reference frame as the center.

[0043] In this embodiment, according to the given matching range size r*r, the image block P’ in the reference frame is sequentially taken out ( x c +i, y c +j ) , -r ≤ i ≤ r, -r ≤ j ≤ r, and the value of r is generally taken as 2 to 3 times the side length of the image block, and the matching difference D(i, j) is calculated according to the following formula.

[0044]

[0045] Among them, P’ ( x, y ) (k, l) represents the point with coordinates (k, l) in the image block P’ ( x, y ) , P’ ( x c +i, y c +j ) (k, l) represents the point with coordinates (k, l) in the image block P’ ( x c +i, y c +j ) , as Figure 3 shown. Find the minimum value of D(i, j) in the set matching range, denoted as D(i’, j’), and compare this minimum value D(i’, j’) with the preset threshold. If it is less than the threshold, the obtained image block P’ ( x c +i’, y c +j’ ) is the matching target, otherwise it is considered that the matching fails, and this image block is marked as an image block that fails to be successfully matched. The reason for the matching failure is generally that the scenery within this image block range also moves at a high speed synchronously during the vehicle driving process, that is, it violates the motion model basis set by the present invention. In this case, the motion matching algorithm of the present invention is terminated for this image block.

[0046] For the image block P that successfully obtains the matching target (x,y) , record the corresponding displacement vector as V(x, y) = (x c +i’ - x, y c +j’ - y), and record the abscissa and ordinate of this displacement vector as: V1(x, y), V2(x, y), that is, V(x, y) = (V1(x, y), V2(x, y)), V1(x, y) = xc +i’-x, V2(x,y) = y c +j’-y. Figure 1 and Figure 2 represents the process of block matching, Figure 1 On the left side is the matching frame, where one image block is marked. On the right side is the reference frame. The small dashed box in the reference frame is the image block corresponding to the position in the matching frame. The dashed arrow indicates the matching center estimated based on the motion model. The large dashed box represents the further given matching range. The solid image block is the finally matched target image block, and the solid arrow is the displacement vector V(x,y). Figure 2 is Figure 1 an enlarged view of the matching range area in the matching frame of, which can more clearly show the process of matching and determining the displacement vector.

[0047] Step 3: Correct the center point coordinates and the coefficient k.

[0048] In the motion model of the present invention, the displacement part between two frames is generated by the overall movement of the image caused by the synchronous linear movement of the camera during the straight-line driving of the vehicle. Based on this principle, the image block of the matching target in Step 2 is obtained. If the scenery in the image itself also moves during the process, the influence of the movement of the scenery itself on the matching needs to be excluded.

[0049] Specifically, for each successfully matched image block P (x,y) , take a total of 25 image blocks arranged in a 5*5 matrix form centered on it. After these image blocks pass through Step 2, some may be successfully matched and some may not be successfully matched. The successfully matched image blocks are grouped into a set J, and the number of elements in J is N. Calculate the difference degree R between the displacement vectors of the image block P (x,y) and the surrounding image blocks and compare it with the preset threshold θ. The specific calculation method of the difference degree R is as follows:

[0050]

[0051] where V1(xi,yj) and V2(xi,yj) are the abscissa and ordinate of the displacement vector of the image block P (xi,yj) in the i-th row and j-th column of the 5*5 matrix respectively. R represents the image block P (x,y)The physical quantity representing the difference between the displacement of the object and the displacement of surrounding image blocks. Since the horizontal and vertical displacements are only related to the horizontal and vertical distances from the image block to the center point, respectively, they are calculated separately and normalized using V1(x, y) and V2(x, y) to eliminate the influence of the distance from the center point on the difference. R is compared with a preset threshold θ. If R < θ, the matching result from step 2 is retained. If R ≥ θ, the scene itself is assumed to have moved synchronously, and the matching result of the corresponding image block is not included in subsequent calculations. The value of θ is generally [0.1, 0.2].

[0052] Then, all vectors (x, y, V1(x, y), V2(x, y)) in the matching frame that meet the condition R < θ after step 2 are combined into a set I. The corrected center point is the point with the smallest sum of squares of distances to the center of the image block and the straight line determined by the displacement vector corresponding to the image block among all points. The coordinates of the corrected center point are recorded as (x 01 ,y 01 ),but:

[0053] .

[0054] According to the motion model of the present invention, the corrected center point should be the intersection of all vector extensions. In practice, this may not strictly intersect at a single point due to the discrete nature of the data and some deviations. However, from a statistical perspective, the point with the smallest sum of squared distances from these lines can be used as an estimate of the center point. The practical significance of the above formula is to calculate the distance from each line to the point (x0, y0) using the point-to-line distance formula and then calculate the sum of the squares. This objective function can be solved using a mathematical derivation method or an enumeration method based on application examples, depending on the hardware computing resources available to execute this method.

[0055] 1. Based on the mathematical derivation method, the above formula is sorted according to (x0, y0) and simplified as follows:

[0056]

[0057] Find the partial derivatives of x0 and y0 respectively, and solve the simultaneous equations:

[0058]

[0059] The solution is:

[0060]

[0061] 2. In the enumeration method based on application examples, considering that the displacement generated by the center point during each iteration is usually small, a search is performed within a certain range d centered on the center point determined in the previous frame. d is generally set to 5 - 10. For the points (x’0 + i, y’0 + j) around the center point (x’0, y’0) corrected in the previous frame of the current matching frame, -d ≤ i ≤ d, -d ≤ j ≤ d. Substitute these points into the above objective function to find the point that minimizes the value of the objective function.

[0062] The coordinate values obtained by solving through the above method based on mathematical derivation or the enumeration method based on application examples are the corrected center point coordinates.

[0063] When the driving speed of the vehicle is slow, a smaller range d can meet the requirements. At this time, the enumeration method based on application examples is more resource - saving in terms of calculation. When the driving speed of the vehicle is fast, a larger range d is required. The enumeration method consumes more resources, and the method based on mathematical derivation is more resource - saving in terms of calculation. Specifically, the corresponding method can be adaptively selected according to the driving situation of the vehicle.

[0064] After recording the coordinates of the corrected center point as (x 01 , y 01 ), the coefficient k is corrected using the following formula:

[0065]

[0066] The corrected k is obtained. Its meaning is to calculate the ratio of the displacement vector corresponding to the image block to the distance from the image block to the corrected center point. And this formula calculates this ratio for the image blocks in set I, that is, all the image blocks in this frame whose displacements satisfy the motion model of the present invention, and then takes the average value. The k obtained in this way is used as the estimate of this ratio for the next frame. Among them, c is a sign judgment parameter. Calculate the dot product of each displacement vector and distance vector, and sum them after normalization. When c > 0, the displacement vector and the distance vector are in the same direction (corresponding to the vehicle moving forward), otherwise they are in the opposite direction (corresponding to the vehicle moving backward), and its sign is used as the sign of k. This sign judgment is only a high - accuracy illustration, and in practice, it can be simplified to a certain extent according to needs. For example, only select several image blocks with the farthest distance for judgment.

[0067] Step 4: When performing motion estimation on the next frame of the current matching frame, then use the corrected center point coordinates (x 01 , y 01 ) and the corrected coefficient k obtained in Step 3 as prior parameters, and re - execute Step 2 to perform motion estimation, and perform iterative correction on the center point and coefficient k by executing Step 3. Complete the motion estimation for each frame through continuous iteration. The overall process is as Figure 4 shown.

[0068] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned motion estimation method for on-vehicle images applicable to straight driving is implemented.

[0069] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A motion estimation method for vehicle-mounted images applicable to straight driving, characterized in that, Including: Step 1: Perform motion estimation according to prior parameters, including: For two adjacent frames of images, the previous frame is the reference frame and the subsequent frame is the matching frame. First, for any image block in the matching frame, calculate the coordinate estimate value (x c , y c ) of the center of the match of this image block in the reference frame, where x c = x + k(x - x0), y c = y + k(y - y0); here, (x, y) is the center point coordinate of this image block, (x0, y0) is the center point coordinate of the matching frame image, k is the proportionality coefficient, k = qv, q is a constant related to the image acquisition device, and v is the driving speed when the vehicle-mounted camera captures this matching frame. Then, perform block matching calculation within the matching range centered on (x c , y c ) to obtain the matching target of this image block and the corresponding displacement vector. Step 2: Correct the center point coordinates and coefficient k of the matched frame image, including: First, for any successfully matched image block P (x,y) , calculate the difference degree R of the displacement vectors between this image block and other successfully matched image blocks within the set surrounding range; then, form a set I with the center point coordinates and displacement vectors (x, y, V1(x, y), V2(x, y)) of each image block that is successfully matched in the matching frame in step 1 and has a difference degree R less than the preset threshold θ, where V1(x, y) and V2(x, y) are the abscissa and ordinate of the displacement vector respectively. The coordinates (x 01 , y 01 ) of the corrected center point are the coordinates of the point with the minimum sum of the squares of the distances to the straight line determined by the center of the image block and the corresponding displacement vector among all points in the set I; finally, according to the corrected center point coordinates (x 01 , y 01 ), take the mean value of the ratio of the displacement vector corresponding to each image block in the set I to the distance from the image block to the corrected center point as the corrected coefficient k; When performing motion estimation on the next frame of the current matched frame, use the corrected center point coordinates and corrected coefficient k obtained in Step 2 as prior parameters, and re - execute Step 1 to perform motion estimation, and iteratively correct the center point coordinates and coefficient k by executing Step 2, and complete the motion estimation of each frame through continuous iteration; In the said Step 2, the objective function for obtaining the coordinates of the corrected center point is: The coordinates obtained by solving this objective function using a method based on mathematical derivation or an enumeration method based on application examples are the coordinates of the corrected center point.

2. The motion estimation method for on-vehicle images applicable to straight driving according to claim 1, wherein In step 2, when calculating the difference degree R, for the successfully matched image patch P (x,y) , take several image patches arranged in the form of a k*k matrix centered on the image patch P (x,y) . Form a set J with the successfully matched image patches among them. If the number of elements in J is N, the specific calculation formula for the difference degree R is as follows: Among them, V1(xi, yj) and V2(xi, yj) are respectively the abscissa and ordinate of the displacement vector of the image block P at the i-th row and j-th column in the k*k matrix. (xi,yj) ​ 3. The motion estimation method for on-vehicle images applicable to straight driving according to claim 1 or 2, characterized in that, Calculate the correction coefficient using the following formula: Among them, c is a symbol judgment parameter.

4. The motion estimation method for in-vehicle images applicable to straight driving according to claim 1 or 2, characterized in that, The enumeration method based on application instances includes: searching within a set range around the corrected center point (x'0, y'0) of the previous frame of the current matching frame, substituting the coordinates of each point within the set range into the objective function respectively, and the coordinates with the minimum value of the objective function are the corrected center point coordinates (x 01 , y 01 ) of the current matching frame.

5. The motion estimation method for on-vehicle images applicable to straight driving according to claim 4, characterized in that, The size of the set range is (x’0 + i, y’0 + j), -d ≤ i ≤ d, -d ≤ j ≤ d, and the range value d takes 5 - 10.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the motion estimation method for on - vehicle images applicable to straight - line driving according to any one of claims 1 - 5.

Images