A motion estimation method fusing deep learning feature optical flow and binocular vision
By integrating deep learning feature optical flow with binocular vision, the problem of weak optical flow estimation capability and accumulated error in low-light environments of autonomous driving technology is solved, achieving high-precision, real-time motion estimation, which is applicable to autonomous driving technology.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DALIAN UNIV
- Filing Date
- 2022-09-21
- Publication Date
- 2026-05-22
AI Technical Summary
Existing autonomous driving technologies have weak optical flow estimation capabilities and low accuracy in low-light environments, and traditional methods suffer from problems such as accumulated errors and poor real-time performance.
A motion estimation method that integrates deep learning-based optical flow features and binocular vision is proposed. By preprocessing the driving image dataset, a deep learning-based optical flow feature extraction model is constructed, and distance measurement is performed using a binocular camera to obtain the vehicle's speed.
This method improves the reliability and accuracy of optical flow estimation in low-light environments, avoids the problems of accumulated errors and poor real-time performance, and provides a highly reliable and low-cost motion estimation method.
Smart Images
Figure CN115482257B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, specifically to a motion estimation method that integrates deep learning feature optical flow and binocular vision. Background Technology
[0002] Optical flow is the visual relative motion between an observer and the observed scene, encompassing motion information such as the surfaces and edges of objects within the visual scene. Optical flow can be viewed as a projection of three-dimensional motion onto a two-dimensional plane. Because it contains rich motion and three-dimensional structural information about objects, and is characterized by high robustness, high real-time performance, low cost, and no error accumulation, optical flow can be applied to fields such as autonomous driving.
[0003] Current mainstream speed measurement methods mainly rely on inertial sensors, GPS positioning speed measurement, or a combination of both. Inertial sensors offer high real-time performance, but errors accumulate over time; GPS positioning speed measurement is highly accurate but has a low update frequency, poor real-time performance, and is susceptible to signal interference. When it is necessary to measure speed using the ubiquitous optical flow information in situations with no signal and long-term driving, this is a viable option.
[0004] With the continuous development of computer technology, production learning through artificial neural networks (deep learning) can further improve processing accuracy and speed, and extract rich data information. Existing deep learning optical flow algorithms are mainly divided into supervised learning and unsupervised / semi-supervised learning algorithms. Supervised optical flow models typically achieve end-to-end training by improving gradient descent methods; while unsupervised algorithms, because they do not require real optical flow images as training samples, directly utilize real-world scenes for network training. Furthermore, they often use brightness conservation functions, smoothing change functions, etc., to replace the end-to-end error loss function in supervised learning models. Currently, artificial neural networks can achieve fast and accurate optical flow estimation of moving objects, but optical flow estimation remains unstable in low-light environments. Existing visual odometry systems are mostly monocular visual odometry and visual-inertial odometry, but they all suffer from low estimation accuracy, high requirements for ambient light intensity, and poor stability. Summary of the Invention
[0005] The purpose of this invention is to provide a motion estimation method that integrates deep learning feature optical flow and binocular vision, which improves the problem of weak optical flow estimation capability and low accuracy of traditional algorithms in low light environments; and avoids the defects of accumulated error and poor real-time performance of traditional motion estimation methods, thus providing a new estimation method for autonomous driving technology.
[0006] To achieve the above objectives, this application proposes a motion estimation method that integrates deep learning feature optical flow and binocular vision, comprising:
[0007] Preprocessing of driving image datasets based on controllable adaptive histogram equalization;
[0008] Construct a deep learning-based optical flow feature extraction model and train it to identify moving targets;
[0009] The target object's position is determined by ranging using a binocular camera;
[0010] Obtain the vehicle's speed.
[0011] Furthermore, the driving image dataset undergoes preprocessing based on controllable adaptive histogram equalization. Specifically, the original driving images are scaled to a set resolution and subjected to controllable adaptive histogram equalization. Then, the driving images are cropped to a predetermined value to limit the magnification intensity, resulting in the neighborhood cumulative distribution function.
[0012]
[0013] Among them, cdf min The minimum value of the cumulative distribution function of pixel values, M×N is the number of pixels in the driving image, and G... i The grayscale level is denoted by .
[0014] Furthermore, a deep learning-based optical flow feature extraction model is constructed and trained to identify moving targets, specifically as follows:
[0015] Based on the local smoothing assumption of optical flow characteristics, the optical flow equation is obtained as follows:
[0016]
[0017] Where x is the pixel x-coordinate, y is the pixel y-coordinate, and t is time; dx, dy, and dt are the derivatives of x, y, and t, respectively; / represents optical flow image information; α is the differentiation operator; and Δx, Δy, and Δt are the changes in x, y, and t, respectively.
[0018] Two CNN layers with shared weights are constructed to extract features from driving images;
[0019] Calculate the inner product of the feature pairs from the two driving images: feature f1∈R H×W×D and feature f2∈R H×W×D Let I1 and I2 represent the features of the driving images, respectively. The visual similarity is obtained by taking the pairwise inner product of the feature vectors, as follows:
[0020]
[0021] Where C(f1, f2) ∈ R H×W×H×Wij and kl are the location information of the optical flow points in the first and second frames of the image, respectively; d is the specific channel of the image, with a value range of [0, D-1]; C is the four-dimensional vector feature; where H and W are the image resolution and D is the number of channels;
[0022] A pyramid is constructed to perform pooling operations on the four-dimensional vector features;
[0023] Obtaining four-dimensional vector features from high-resolution driving images: Due to the data cost between pyramid levels, we denote the optical flow corresponding point x′ = (u + f) between two frames. 1 (u), v+f 2 (v)), u is the pixel x-coordinate, v is the pixel y-coordinate, f 1 f represents the optical flow features of the first frame image. 2 The optical flow features of the second frame image are represented by the neighborhood grid. m is the number of layers, then through Find the arbitrary position of the optical flow on each layer, where k is an arbitrary real number; based on this correspondence, the four-dimensional vector feature representation of the high-resolution driving image is as follows:
[0024]
[0025] Where m is the m-th layer of the pyramid, and p and q are the information of the optical flow point in the p-th row and q-th column of the pixel matrix in the m-th layer;
[0026] The CNN layer iteratively updates the driving image data: given the current optical flow state f k Each iteration generates a residual optical flow, i.e., an update value f, relative to the output of the previous iteration. 1 If Δf, then the next optical flow prediction value is Δf+f. k =f k+1 The update method is as follows:
[0027]
[0028] Among them, R t To reset the door, Z t For the update gate, σ is a function operation, H t To retain the information from the previous hidden state, H t-1 For the hidden layer, X t The optical flow input value W r W z This is the weight information matrix.
[0029] After obtaining the four-dimensional vector features of the high-resolution driving image (motion-enhanced image of the moving object), the pixel information of optical flow is acquired under the original resolution tracking of the pyramid, thus obtaining the active region of the moving target. The minimum displacement of this region is v = [vx v y ] T The matching error and minimum value ε(v) within the neighborhood of each point are:
[0030]
[0031] Among them, v x v y These represent the horizontal and vertical displacements at the top of the pyramid, p. x w is the x-coordinate of the optical flow point. x p represents the neighborhood of the horizontal axis. y w is the x-coordinate of the optical flow point. y Let A(x, y) be the optical flow feature of the first frame and B(x, y) be the optical flow feature of the second frame.
[0032] Training to identify moving targets within the activity area;
[0033] A supervised algorithm is selected for the model, and its loss function is set as follows:
[0034]
[0035] That is, the L1 norm of the iteration result and the true value, where N is the number of iterations, and γ = 0.8; f gt To estimate optical flow characteristics, f i For actual optical flow characteristics, Δx gt Δy rt To estimate the transverse and longitudinal displacements of the optical flow, Δx i Δy i These are the actual horizontal and vertical displacement coordinates of the optical flow.
[0036] Furthermore, the CNN layer includes two residual layers with 1 / 2 resolution, two 1 / 4 resolution, and two 1 / 8 resolution. For each halving of resolution between residual layers, the number of channels increases. During feature extraction, two consecutive frames are input, resulting in R. H×W×3 →R H×W×D Where H and W are the image resolutions, and D is the number of channels.
[0037] Furthermore, a pyramid is constructed to perform pooling operations on the four-dimensional vector features, specifically as follows:
[0038] A three-layer similarity vehicle image pyramid is constructed, with 1, 2, and 4 kernels respectively from the first to the third layer. The last two dimensions of the four-dimensional vector features are pooled. The vehicle image pyramid is represented as follows:
[0039]
[0040] Where L represents the pyramid level, IL For the image of layer L, x and y represent the pixel position information of the optical flow points.
[0041] Furthermore, the target's location is obtained by ranging using a binocular camera, specifically:
[0042] Based on the parameters and distortion coefficients of the left and right cameras, a distortion description is obtained;
[0043] The left and right images captured simultaneously in the same scene are projected and transformed, that is, a correction mapping table is obtained and the correction mapping table is used for remapping.
[0044] By using stereo matching (SGM) to find corresponding points in the left and right images, the visual difference (disparity) is obtained. l -u r , where u l u r These are the column coordinates of the target point in the left and right images, respectively. After generating the disparity map, a Guided Filter is used for filtering.
[0045] Calculate the depth of the target object Where f is the focal length, b is the baseline length, d is the parallax, and c xr c xl The column coordinates are the principal points of the two cameras.
[0046] Construct a 3D space based on a set of disparity maps to obtain the three-dimensional coordinates of the pixels:
[0047] [XYZW] T =Q*[gh disparity(g, h)1] T
[0048]
[0049] Where X, Y, Z, and W are the four-dimensional information obtained after matrix transformation; g and h are the position information of each pixel; Q is the perspective transformation matrix; and 3DImage(a, c, d) is the (x, y, z) coordinate information in the viewing coordinate system.
[0050] Furthermore, the distortion disparity(g, h) is described as follows:
[0051]
[0052] Where s1, s2, and s3 are the thin prism distortion coefficients, k1, k2, k3, k4, k5, and k6 are the radial distortion coefficients, p1 and p2 are the tangential distortion coefficients, and r is the distortion radius;
[0053] Furthermore, two images captured simultaneously in the same scene are projected and transformed, specifically as follows:
[0054] Align the two image planes with the baseline, and ensure that the same target point is on the same horizontal line in both images, i.e., coplanar alignment.
[0055] The projection matrix equations of point X in space, through the left and right lenses C1 and C2 of the camera, are as follows:
[0056]
[0057]
[0058] Where (u1, v1, 1) and (u2, v2, 1) are the homogeneous coordinates of x1 and x2 in their respective images, respectively; (X, Y, Z, 1) are the homogeneous coordinates of point P in world coordinates; m ij Let i be the element in the i-th row and j-th column of the projection matrix M;
[0059] When the point becomes a linear element, let c1 and c2 be the lines S corresponding to the left and right cameras in the same space, respectively. The equations of the lines in the spatial coordinate system are:
[0060]
[0061]
[0062] Substituting the above equation into the projection matrix, we obtain the mapping equations of the left and right cameras on the projection plane:
[0063]
[0064]
[0065] Furthermore, the vehicle's speed is obtained, specifically:
[0066]
[0067] Where, d k d k-1 This represents the Z-axis value of the three-dimensional coordinates of the pixels in the current frame and the previous frame.
[0068] Compared with existing technologies, the technical solution adopted in this invention has the following advantages: Based on deep learning optical flow and binocular imaging principles, this invention estimates the motion parameters of vehicle displacement and velocity based on video data, enabling motion estimation even during nighttime driving, thus further improving reliability. Simultaneously, this method avoids the accumulated errors inherent in traditional inertial sensors and the shortcomings of poor anti-interference capability and low update frequency encountered with GPS positioning and speed measurement. This method features high portability, high real-time performance, excellent robustness, no accumulated errors, and low cost, providing a new motion estimation method for autonomous driving technology. Attached Figure Description
[0069] Figure 1 This is a schematic diagram of the hardware involved in the embodiment;
[0070] Figure 2 Here is a block diagram illustrating the principle of the motion estimation method;
[0071] Figure 3 This is a schematic diagram of the motion estimation method.
[0072] Figure 4 This is a structural diagram of the optical flow model;
[0073] Figure 5 Diagram showing the implementation method of the optical flow model;
[0074] Figure 6 This is a schematic diagram illustrating the formation of tangential distortion during calibration and correction.
[0075] Figure 7 This is a schematic diagram of the coplanar alignment principle. Detailed Implementation
[0076] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the application; that is, the described embodiments are only a part of the embodiments of this application, and not all of them.
[0077] Example 1
[0078] This invention utilizes optical flow enhancement information based on artificial neural networks to determine road environment features, identify stationary objects on both sides of the road, determine the distance between the carrier and the target object, and then obtain the speed based on the distance difference between the carrier and the target object and the time between two frames. This method is integrated into a hardware carrier and placed in front of the rearview mirror inside the vehicle to measure vehicle speed based on video. The camera and video speedometer are physically independent, such as... Figure 1 As shown. Its principle block diagram is as follows. Figure 2As shown, the motion estimation system implementing this method includes: a video acquisition unit, a frame-by-frame recording unit, a storage unit, a deep learning intelligent motion enhancement unit, an optical flow feature target recognition unit, and a motion analysis unit. The motion enhancement unit and the target recognition unit are encapsulated together as a moving target calibration unit. Images captured by the binocular camera are connected to the storage unit via a bus through the video acquisition unit to perform the recording function. The computer reads the acquired image data and sequentially identifies roadside objects through the frame-by-frame recording unit, the moving target calibration unit, and the motion analysis unit, constructing three-dimensional coordinate information. It then determines the motion state of the vehicle to which the device is located based on the relative motion speed of the objects, and finally returns the information to the storage unit to perform the motion analysis function.
[0079] like Figure 3 As shown, a motion estimation method that integrates deep learning feature optical flow and binocular vision includes:
[0080] S1. Perform controllable adaptive histogram equalization preprocessing on the vehicle image dataset;
[0081] Specifically, driving videos at different speeds and under different road conditions are collected to create a driving image dataset X = X[n]1, X[n]2, X[n]3…X[n] fo}, where n is the video sequence number and fo is the number of video frames.
[0082] The original driving image is scaled to a set resolution (e.g., 1088×436) and subjected to controlled adaptive histogram equalization (CLAHE). Then, the driving image is cropped to a predetermined value to limit the magnification intensity, resulting in the neighborhood cumulative distribution function:
[0083]
[0084] Among them, cdf min The minimum value of the cumulative distribution function of pixel values, M×N is the number of pixels in the driving image, and G... i This refers to the number of gray levels (which can be set to 256 in this embodiment). It can improve image contrast, enhance color intensity in low-light environments, and facilitate subsequent analysis.
[0085] S2. Construct an optical flow model and train it to recognize moving targets; such as... Figure 4 As shown:
[0086] S2.1 Based on optical flow characteristics, the following assumptions are made for local smoothing: Assumption 1: Constant brightness; the brightness of image pixels does not change during inter-frame motion. Assumption 2: Small motion; the inter-frame motion of pixels is relatively small, meaning the relative motion of the image is small over time. Assumption 3: Spatial consistency; adjacent points on the same surface in the same scene exhibit similar motion. Based on these assumptions, the optical flow equation is obtained:
[0087]
[0088] S2.2 Construct two CNN layers with shared weights to extract features from the driving image. The CNN layer architecture is as follows: Figure 5 As shown;
[0089] Specifically, the CNN layer includes two 1 / 2 resolution layers, two 1 / 4 resolution layers, and two 1 / 8 resolution layers, for a total of six residual layers. The number of channels increases for each halving of resolution between residual layers. During feature extraction, two consecutive frames are input, resulting in R. H×W×3 →R H×W×D Where D is the number of channels, which can be set to 256.
[0090] S2.3 Calculate the inner product of the feature pairs from the two driving images;
[0091] Specifically, feature f1∈R H×W×D and feature f2∈R H×W×D Let I1 and I2 represent the features of the driving images, respectively. The visual similarity is obtained by taking the pairwise inner product of the feature vectors, as follows:
[0092]
[0093] Where C(f1, f2) ∈ R H×W×H×W ij and kl are the location information of the optical flow points in the first and second frames of the image, respectively; d is the specific channel of the image, with a value range of [0, D-1]; C is the four-dimensional vector feature; where H and W are the image resolution and D is the number of channels;
[0094] Based on the optical flow equation and the constant brightness assumption in step S2.1, the motion state between images is obtained using the least squares method:
[0095]
[0096] Among them, I x (q i ), I y (q i ), I t (q i ) represents the optical flow characteristics of the neighborhood pixels surrounding the optical flow pixel location, where i is 1, 2, 3, ...;
[0097] S2.4 Construct a pyramid to perform pooling operations on the four-dimensional vector features;
[0098] Specifically, a three-layer correlation pyramid is constructed, with kernels 1, 2, and 4 respectively from the first to the third layer. The last two dimensions of the four-dimensional vector obtained in step 2.3 are pooled to make motion changes more significant. The driving image pyramid representation is as follows:
[0099]
[0100] Where L represents the pyramid level, I L This is an image of layer L.
[0101] S2.5 acquires four-dimensional vector features from high-resolution driving images;
[0102] Specifically, let x′ = (u + f) be the corresponding point of optical flow between two frames. 1 (u), v+f 2 (v)), u is the pixel x-coordinate, v is the pixel y-coordinate, f 1 f represents the optical flow features of the first frame image. 2 The optical flow features of the second frame image are represented by the neighborhood grid. m is the number of layers, then through Find the arbitrary position of the optical flow on each layer, where k is an arbitrary real number; based on this correspondence, the four-dimensional vector feature representation of the high-resolution driving image is as follows:
[0103]
[0104] Where m is the m-th layer of the pyramid, and p and q are the information of the optical flow point in the p-th row and q-th column of the pixel matrix in the m-th layer;
[0105] The CNN layer described in S2.6 iteratively updates the driving image data;
[0106] Specifically, in addition to generating image features through optical flow estimation, the CNN layer also performs data iteration. The update iteration is a sequence of classic gated recurrent units with previous data, which can be trained using shared-weight convolutional layers. The default initial value is 0, given the current optical flow state as f. k Each iteration generates a residual optical flow, i.e., an update value f, relative to the output of the previous iteration. 1 If Δf, then the next optical flow prediction value is Δf+f. k =f k+1 The update gates of the gated loop unit are as follows:
[0107]
[0108] Among them, R t To reset the door, Z t For the update gate, σ is a function operation, H tTo retain the information from the previous hidden state, H t-1 For the hidden layer, X t The optical flow input value W r W z This is the weight information matrix.
[0109] S2.7 After obtaining the four-dimensional vector features of the high-resolution driving image (motion-enhanced image of the moving object), the pixel information of optical flow is acquired under the original resolution tracking of the pyramid, thus obtaining the active region of the moving target. The minimum displacement of this region is v = [v x v y ] T The matching error and minimum value ε(v) within the neighborhood of each point are:
[0110]
[0111] Among them, v x v y These represent the horizontal and vertical displacements at the top of the pyramid, p. x w is the x-coordinate of the optical flow point. x p represents the neighborhood of the horizontal axis. y w is the x-coordinate of the optical flow point. y Let A(x, y) be the optical flow feature of the first frame and B(x, y) be the optical flow feature of the second frame.
[0112] S2.8 Conduct recognition training for moving targets in the activity area; (e.g., common roadside facilities such as roadside trees, signs, and buildings).
[0113] A supervised algorithm is selected for the model, and its loss function is set as follows:
[0114]
[0115] That is, the L1 norm of the iteration result and the true value, where N is the number of iterations and γ = 0.8.
[0116] S3. Use a binocular camera to measure the distance and obtain the position of the target object;
[0117] S3.1 Based on the parameters and distortion coefficients of the left and right cameras, obtain the distortion description;
[0118] Specifically, barrel distortion is caused by the lens construction, and tangential distortion occurs during camera assembly because the lens and imaging plane are not perfectly parallel. Therefore, the distortion vector is obtained using the intrinsic and extrinsic parameters of the left and right cameras and the distortion coefficients, for correction in subsequent steps. The distortion disparity(g,h) is described as follows:
[0119]
[0120] Where s1, s2, and s3 are the thin prism distortion coefficients, k1, k2, k3, k4, k5, and k6 are the radial distortion coefficients, p1 and p2 are the tangential distortion coefficients, and r is the distortion radius.
[0121] S3.2 Project the left and right images captured simultaneously in the same scene and transform them, that is, obtain the correction mapping table and use the correction mapping table to perform remapping;
[0122] Specifically, due to parallax, the images obtained by the left and right lenses of a binocular camera will not completely overlap. Therefore, two views simultaneously captured on the same scene are projected and transformed: the two image planes are parallel to the baseline, and the same target point is on the same horizontal line in both images, i.e., coplanar alignment. Figure 7 As shown.
[0123] The projection matrix equations of point X in space, through the left and right lenses C1 and C2 of the camera, are as follows:
[0124]
[0125]
[0126] Where (u1, v1, 1) and (u2, v2, 1) are the homogeneous coordinates of x1 and x2 in their respective images; (X, Y, Z, 1) are the homogeneous coordinates of point P in world coordinates; m ij Let be the element in the i-th row and j-th column of the projection matrix M.
[0127] When the point becomes a linear element, let c1 and c2 be the lines S corresponding to the left and right cameras in the same space, respectively. The equations of the lines in the spatial coordinate system are:
[0128]
[0129]
[0130] Substituting the above equation into the projection matrix, we obtain the mapping equations of the left and right cameras on the projection plane:
[0131]
[0132]
[0133] S3.3 uses stereo matching (SGM) to find corresponding points in the left and right images, obtaining the visual difference disparity = u. l -u r , where u l u rThese are the column coordinates of the target point in the left and right images, respectively. After generating the disparity map, a Guided Filter is used for filtering to reduce noise.
[0134] S3.4 Calculate the depth of the target object. Where f is the focal length, b is the baseline length, d is the parallax, and c xr c xl The column coordinates of the principal points of the two cameras;
[0135] S3.5 constructs a 3D space based on a set of disparity maps to obtain the three-dimensional coordinates of the pixels:
[0136] [XYZW] T =Q*[gh disparity(g, h)1] T
[0137]
[0138] Where X, Y, Z, and W are the four-dimensional information obtained after matrix transformation; g and h are the position information of each pixel; Q is the perspective transformation matrix; and 3DImage(a, c, d) is the (x, y, z) coordinate information in the view coordinate system. Finally, a three-dimensional matrix is obtained, recording the X, Y, and Z coordinates (with the left camera as the reference coordinate system).
[0139] S4. Obtain the vehicle's speed;
[0140] Specifically, the Z-axis coordinates obtained in step 3.5 are extracted from two adjacent frames. Since the camera's shooting mode is 25fps and based on the principle of relative motion, the vehicle's speed can be calculated as follows: Where, d k d k-1 This represents the Z-axis value of the three-dimensional coordinates of the pixels in the current frame and the previous frame.
[0141] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.
Claims
1. A motion estimation method integrating deep learning feature optical flow and binocular vision, characterized in that, include: Preprocessing of driving image datasets based on controllable adaptive histogram equalization; Construct a deep learning-based optical flow feature extraction model and train it to identify moving targets; The target object's position is determined by ranging using a binocular camera; Obtain the vehicle's speed; Construct a deep learning-based optical flow feature extraction model and train it to identify moving targets, specifically as follows: Based on the local smoothing assumption of optical flow characteristics, the optical flow equation is obtained as follows: in, The x-coordinate is the pixel coordinate. The vertical coordinate is the pixel coordinate. For time; The derivatives of x, y, and t. Optical flow image information; For differentiation operators, , , These are the changes in x, y, and t. Two CNN layers with shared weights are constructed to extract features from driving images; Calculate the inner product of the feature pairs from the two driving images: Features and Representing driving images and The features of visual similarity are obtained by taking the pairwise inner product of the feature vectors, which yields the four-dimensional vector features of visual similarity, expressed as: in ; , These are the location information of the optical flow points in the first and second frames of the image, respectively. This represents a specific channel of the image being captured, with values ranging from [0, D-1]. C is a four-dimensional vector feature; where H and W are image resolutions, and D is the number of channels; A pyramid is constructed to perform pooling operations on the four-dimensional vector features; Obtaining four-dimensional vector features from high-resolution driving images: Due to the data cost between pyramid levels, record the corresponding optical flow points between two frames. , These are the rates of change of displacement in the x and y directions, respectively. Optical flow features of the first frame image The optical flow features of the second frame image are represented by the neighborhood grid. , If it is the number of layers, then through Find the arbitrary position of the optical flow on each layer, where k is an arbitrary real number; based on the correspondence, the four-dimensional vector feature of the high-resolution driving image is represented as: in This is the m-th level of the pyramid. , These represent the information of the optical flow point in the p-th row and q-th column of the pixel matrix at the m-th layer; The CNN layer iteratively updates the driving image data: given the current optical flow state as... Each iteration generates a residual optical flow, i.e., an update value, relative to the output of the previous iteration. The next optical flow prediction value is... The update method is as follows: in, To reset the door, To update the door, For function operations, To retain the amount of information from the hidden state of the previous stage. As a hidden layer, Optical flow input value , This is the weight information matrix; After obtaining the four-dimensional vector features of the high-resolution driving image, the pixel information of optical flow is acquired under the original resolution tracking of the pyramid, thus obtaining the active region of the moving target. The image displacement of this region is minimized as follows: The matching error and minimum value within the neighborhood of each point for: in, The x-coordinate of the optical flow point. The range of the horizontal axis is the neighborhood. The x-coordinate of the optical flow point. The vertical axis represents the neighborhood range. The optical flow features of the first frame, This refers to the optical flow characteristics of the second frame; Training to identify moving targets within the activity area; A supervised algorithm is selected for the model, and its loss function is set as follows: That is, the L1 norm of the iteration result and the true value, where N is the number of iterations. ; To estimate optical flow characteristics, For actual optical flow characteristics, To estimate the transverse and longitudinal displacements of optical flow, These are the actual horizontal and vertical displacement coordinates of the optical flow.
2. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 1, characterized in that, The driving image dataset undergoes preprocessing based on controllable adaptive histogram equalization. Specifically, the original driving images are scaled to a set resolution and subjected to controllable adaptive histogram equalization. Then, the driving images are cropped to a predetermined value to limit the magnification intensity, resulting in the neighborhood cumulative distribution function. in The minimum value of the cumulative distribution function of pixel values. 1 represents the number of pixels in the driving image. The grayscale level is denoted by .
3. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 1, characterized in that, The CNN layer includes two 1 / 2 resolution, two 1 / 4 resolution, and two 1 / 8 resolution residual layers. For each halving of resolution between residual layers, the number of channels increases. Feature extraction is performed by inputting two consecutive frames, resulting in... → Where H and W are the image resolutions, and D is the number of channels.
4. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 1, characterized in that, Constructing a pyramid to perform pooling operations on the four-dimensional vector features, specifically: A three-layer similarity vehicle image pyramid is constructed, with 1, 2, and 4 kernels respectively from the first to the third layer. The last two dimensions of the four-dimensional vector features are pooled. The vehicle image pyramid is represented as follows: , Where L represents the pyramid level, This is an image of layer L.
5. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 1, characterized in that, The target object's position is determined by ranging using a binocular camera, specifically: Based on the parameters and distortion coefficients of the left and right cameras, a distortion description is obtained; The left and right images captured simultaneously in the same scene are projected and transformed, that is, a correction mapping table is obtained and the correction mapping table is used for remapping. By using stereo matching (SGM) to find corresponding points in the left and right images, the visual difference is obtained. ,in These are the column coordinates of the target point in the left and right images, respectively. After generating the disparity map, a Guided Filter is used for filtering. Calculate the depth of the target object Where f is the focal length, b is the baseline length, and d is the parallax. The column coordinates of the principal points of the two cameras; Construct a 3D space based on a set of disparity maps to obtain the three-dimensional coordinates of the pixels: in, This is the four-dimensional information obtained after matrix transformation; , For each pixel, the position information is given, and Q is the perspective transformation matrix. To inspect the three-dimensional coordinate information in the coordinate system.
6. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 5, characterized in that, The distortion Described as: in It is the distortion coefficient of a thin prism. The radial distortion coefficient is... denoted as tangential distortion coefficient, and r as distortion radius.
7. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 5, characterized in that, The projection transformation of two images captured simultaneously in the same scene is as follows: Align the two image planes with the baseline, and ensure that the same target point is on the same horizontal line in both images, i.e., coplanar alignment. Point X in space, viewed through the left and right lenses of the camera. The projection matrix equations are as follows: in and They are respectively and Homogeneous coordinates in their respective images; Let P be the homogeneous coordinates of point P in world coordinates; Let i be the element in the i-th row and j-th column of the projection matrix M; When a point becomes a linear primitive, let... , Let S be the line S corresponding to the left and right cameras in the same space. The equation of the line in the spatial coordinate system is: Substituting the above equation into the projection matrix, we obtain the mapping equations of the left and right cameras on the projection plane: 。 8. The motion estimation method fusing deep learning feature optical flow and binocular vision according to claim 1, characterized in that, To obtain the vehicle's speed, specifically: m / s in, This represents the Z-axis value of the three-dimensional coordinates of the pixels in the current frame and the previous frame.