Vehicle speed estimation method and device

Abstract

The application provides a vehicle speed prediction method and device, the method comprising: obtaining feature points in each target vehicle image and descriptors corresponding to each feature point according to at least two continuous target vehicle images obtained in advance; matching the feature points of adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point to obtain a matching relationship of the feature points between the adjacent two target vehicle images; obtaining a motion rotation matrix and a motion translation vector between the adjacent two target vehicle images according to the matching relationship of the feature points between the adjacent two target vehicle images, combining a target vehicle orientation and a size of each target vehicle image obtained in advance, and using a camera imaging principle; and obtaining a predicted target vehicle speed according to a self-vehicle rotation matrix and a self-vehicle translation vector obtained in advance and combining the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images. The application can significantly improve the speed measurement accuracy of a target vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a vehicle speed estimation method and device. BACKGROUND

[0002] In the automatic driving technology, the system needs to obtain the position and speed of the other vehicle to predict the trajectory of the other vehicle, and then realize the path planning of the ego vehicle. Therefore, vehicle speed estimation is an important function in automatic driving. At present, monocular cameras are mostly used for vehicle speed estimation to obtain accurate vehicle speed estimation values due to their low cost and rich semantic information.

[0003] The vehicle speed estimation method currently used in the industry mainly uses monocular ranging first, and then calculates the target speed through Kalman filtering. However, monocular cameras cannot directly obtain 3D information, which can easily cause large monocular 3D ranging errors, thereby affecting the speed measurement accuracy. In addition, the speed resolution of the above method is insufficient, which makes it difficult to distinguish non-stationary targets and stationary targets in time, thereby affecting driving safety. SUMMARY

[0004] The present application provides a vehicle speed estimation method and device to solve the problem of poor speed measurement accuracy in the prior art, realize partial decoupling of monocular speed measurement and 3D ranging, improve speed measurement accuracy, and ensure driving safety.

[0005] The present application provides a vehicle speed estimation method, comprising: obtaining feature points in each target vehicle image and descriptors corresponding to each feature point according to at least two consecutive target vehicle images obtained in advance; matching the feature points of adjacent two target vehicle images to obtain the matching relationship of the feature points between the adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point; obtaining the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images by using the camera imaging principle in combination with the target vehicle orientation and size of each frame of the target vehicle image obtained in advance according to the matching relationship of the feature points between the adjacent two target vehicle images; and obtaining the predicted target vehicle speed according to the ego vehicle rotation matrix and the ego vehicle translation vector in combination with the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0006] According to the vehicle speed estimation method provided by the present application, the feature points in each target vehicle image and the descriptors corresponding to each feature point are obtained according to at least two consecutive target vehicle images obtained in advance, which comprises: respectively extracting visual corner points from each frame of target vehicle image to obtain the feature points corresponding to each frame of target vehicle image; and obtaining the descriptors corresponding to each feature point according to the feature points.

[0007] According to the vehicle speed estimation method provided by the application, the descriptor corresponding to each feature point is obtained according to the feature point, and the descriptor extraction model is obtained by inputting the surrounding area image of the feature point of the single-frame target vehicle image into the descriptor extraction model, wherein the descriptor extraction model is trained based on the matching relationship between the training points selected from the adjacent two frames of target training images and the similarity between the corresponding descriptors.

[0008] According to the vehicle speed estimation method provided by the application, the descriptor extraction model is trained, including: obtaining at least two continuous frames of target training images and training points of each target training image; inputting the region image around the training point corresponding to the single-frame target training image into the descriptor extraction model to obtain the training descriptor corresponding to each training point output by the descriptor extraction model; selecting any training point in the current frame of target training image, and obtaining the corresponding similarity based on the training descriptor corresponding to the training point and the training descriptor corresponding to the training point in the previous frame of target training image; based on the similarity, the matching relationship between the training point of the current frame of target training image and the training point of the previous frame of target training image is determined, if the matching is correct, the loss function is constructed according to the similarity of the matching training point; otherwise, the loss function is constructed according to the reciprocal of the similarity of the non-matching training point; the training is ended based on the convergence of the loss function.

[0009] According to the vehicle speed estimation method provided by the application, the feature points are obtained by performing visual corner point extraction on each frame of target vehicle image, including: performing 2D detection on the target vehicle image to obtain a 2D detection frame; performing visual corner point extraction on the region inside the 2D detection frame and the region outside the 2D detection frame to obtain the feature points.

[0010] According to the vehicle speed estimation method provided by the application, the feature points of the adjacent two frames of target vehicle images are matched according to the descriptors to obtain the matching relationship between the feature points of the adjacent two frames of target vehicle images, including: selecting any feature point in the current frame of target vehicle image, and obtaining the corresponding feature similarity based on the descriptor corresponding to the feature point and the descriptor corresponding to the feature point in the previous frame of target vehicle image, and constructing a cost matrix according to the feature similarity corresponding to all feature points in the current frame of target vehicle image; the matching relationship between the feature points of the adjacent two frames of target vehicle images is obtained by using the Hungarian matching algorithm to process the cost matrix.

[0011] According to the vehicle speed estimation method provided by the application, the matching relationship of feature points between the adjacent two target vehicle images is obtained, and the target vehicle orientation and size of each frame of the target vehicle image obtained in advance are combined, and the motion rotation matrix and motion translation vector between the adjacent two target vehicle images are obtained by using the camera imaging principle.

[0012] The application further provides a vehicle speed estimation device, which comprises a feature extraction module, a matching module, an offset prediction module and a speed estimation module.

[0013] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor.

[0014] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of the vehicle speed estimation method.

[0015] The application further provides a computer program product, which comprises a computer program, and the computer program is executable on a processor to implement the steps of the vehicle speed estimation method.

[0016] The vehicle speed estimation method and device provided by the application, by extracting the feature points and descriptors of the target vehicle image, the matching relationship between the feature points of the adjacent frame target vehicle image is determined according to the descriptors, so that the non-stationary target and the stationary target are distinguished in time, and the speed resolution is improved; combined with the camera imaging principle, the speed of the target vehicle is predicted to obtain the motion of the target vehicle relative to the ego vehicle; and according to the motion of the ego vehicle relative to the inertial coordinate system and the motion of the target vehicle relative to the ego vehicle, the motion of the target vehicle in the inertial coordinate system is obtained, that is, the speed of the target vehicle is predicted, so that the speed measurement accuracy of the target vehicle is significantly improved, the safety of automatic driving is effectively improved, and the partial decoupling of monocular speed measurement and 3D ranging is realized. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 is a flowchart of the vehicle speed estimation method provided by the application;

[0019] Figure 2 is a schematic diagram of feature points and matching relationship provided by the application;

[0020] Figure 3 is a schematic diagram of the speed measurement effect of the stationary vehicle and the non-stationary vehicle provided by the application;

[0021] Figure 4 is a structural schematic diagram of the vehicle speed estimation device provided by the application;

[0022] Figure 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme and advantages of the application more clear, the technical scheme in the application will be described clearly and completely in the following combined with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0024] Figure 1 shows a flowchart of a vehicle speed estimation method provided by the application, the method comprises:

[0025] S11, obtaining feature points in each target vehicle image and descriptors corresponding to the feature points according to at least two continuous target vehicle images obtained in advance;

[0026] S12, matching the feature points of adjacent two target vehicle images according to the feature points and the descriptors corresponding to the feature points, to obtain a matching relationship of the feature points between the adjacent two target vehicle images;

[0027] S13, obtaining a motion rotation matrix and a motion translation vector between the adjacent two target vehicle images according to the matching relationship of the feature points between the adjacent two target vehicle images, combining a target vehicle orientation and a target vehicle size of each target vehicle image obtained in advance, and using a camera imaging principle;

[0028] S14, obtaining a predicted target vehicle speed according to a self-vehicle rotation matrix and a self-vehicle translation vector, combining the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0029] It should be noted that S1N in the present specification does not represent the order of the vehicle speed estimation method, and the following will be specifically combined Figures 2-3 to describe the vehicle speed estimation method of the present application.

[0030] Step S11, obtaining feature points in each target vehicle image and descriptors corresponding to the feature points according to at least two continuous target vehicle images obtained in advance.

[0031] In the present embodiment, obtaining feature points in each target vehicle image and descriptors corresponding to the feature points according to at least two continuous target vehicle images obtained in advance includes: respectively performing visual corner point extraction on each target vehicle image to obtain feature points corresponding to each target vehicle image; and obtaining descriptors corresponding to each feature point according to the feature points. Specifically:

[0032] First, respectively performing visual corner point extraction on each target vehicle image to obtain feature points corresponding to each target vehicle image. It should be noted that the visual corner point extraction on each target vehicle image to obtain the feature points includes: obtaining a target vehicle image; obtaining a 2D detection frame by performing 2D detection on the target vehicle image; and performing visual corner point extraction on a region in the 2D detection frame and performing visual corner point extraction on a region outside the 2D detection frame to obtain the feature points, such as the points in Figure 2 .

[0033] It should be added that the target vehicle image can be obtained by continuously tracking the target vehicle using an image tracking algorithm to obtain multiple continuous target vehicle images, so as to facilitate subsequent extraction of the 2D detection frame and the visual corner points of the background region in each tracking frame as the feature points.

[0034] Secondly, based on the feature points, descriptors corresponding to each feature point are obtained. In this embodiment, a descriptor extraction model can be used to extract feature points and obtain descriptors. More specifically, obtaining descriptors corresponding to each feature point includes: determining the surrounding region image of the feature points in a single frame of the target vehicle image based on the feature points corresponding to those feature points; inputting the surrounding region image into the descriptor extraction model to obtain the descriptors output by the model, wherein the descriptor extraction model is trained based on the matching relationship between training points selected from two adjacent target training images and the similarity between corresponding descriptors.

[0035] It should be noted that the descriptor extraction model can be an existing model or a redesigned model; no further limitations are imposed here. For example, a descriptor extraction model can use metric learning to design a neural network, taking the image of the region surrounding the feature point as the input to the neural network, and using the neural network model to extract the descriptors of the feature points, obtaining a 256-dimensional vector as the output, i.e., the descriptor.

[0036] It should be added that training the descriptor extraction model includes: acquiring at least two consecutive frames of target training images and training points for each target training image; inputting the region image surrounding the training point corresponding to a single frame of target training image into the descriptor extraction model to obtain the training points output by the descriptor extraction model and the corresponding training descriptors for each training point; selecting any training point in the current frame of target training image and obtaining the corresponding similarity based on its corresponding training descriptor and the training descriptors corresponding to the training points in the previous frame of target training image; based on the similarity, determining the matching relationship between the training points in the current frame of target training image and the training points in the previous frame of target training image; if they match, constructing a loss function based on the similarity of the matching training points; otherwise, constructing a loss function based on the reciprocal of the similarity of the unmatched training points; and ending the training based on the convergence of the loss function.

[0037] In this embodiment, the loss function is expressed as:

[0038]

[0039]

[0040] Where, p i and p j Each represents a feature point randomly selected from two adjacent frames of the target vehicle image, des i p i The descriptor corresponding to the feature point, des j p j The descriptor corresponding to the feature point. It should be noted that when p... i and pj is a pair of matching points, the cosine distance between their descriptors should be close enough; when p i and p j is not a pair of matching points, the cosine distance between their descriptors should be far enough, and the descriptor of a feature point can be efficiently extracted by the above method.

[0041] In step S12, the feature points of the adjacent two target vehicle images are matched according to the feature points and the descriptors corresponding to the feature points, and the matching relationship between the feature points of the adjacent two target vehicle images is obtained.

[0042] In this embodiment, the feature points of the adjacent two target vehicle images are matched according to the descriptors, and the matching relationship between the feature points of the adjacent two target vehicle images is obtained, including: selecting any feature point in the current frame of target vehicle image, and based on the descriptor corresponding to the feature point and the descriptors corresponding to the feature points in the previous frame of target vehicle image, the corresponding feature similarity is obtained, and the cost matrix is constructed according to the feature similarity corresponding to all feature points in the current frame of target vehicle image; the cost matrix is processed by using the Hungarian matching algorithm, and the matching relationship between the feature points of the adjacent two target vehicle images is obtained.

[0043] It should be noted that the cost matrix is represented as:

[0044]

[0045] Wherein, cost represents the cost matrix, c i,j represents the cosine distance between the descriptor of the i th feature point in the current frame and the descriptor of the j th feature point in the previous frame. It should be noted that by processing the cost matrix by using the Hungarian matching algorithm, the matching relationship between the feature points of the previous and subsequent frames can be obtained, as shown in the middle line segment. Figure 2

[0046] In step S13, according to the matching relationship between the feature points of the adjacent two target vehicle images, the target vehicle orientation and size of each frame of target vehicle image obtained in advance are combined, and the motion rotation matrix and motion translation vector between the adjacent two target vehicle images are obtained by using the camera imaging principle.

[0047] ​In the embodiment, according to the matching relationship of feature points between adjacent two target vehicle images, and in combination with the target vehicle orientation and size of each frame of target vehicle image obtained in advance, the motion rotation matrix and motion translation vector between adjacent two target vehicle images are obtained by using the camera imaging principle, including: obtaining the feature point position coordinates of the target vehicle in each frame of target vehicle image according to the target vehicle orientation and size of each frame of target vehicle image obtained in advance; obtaining the pixel coordinates of each feature point in each frame of target vehicle image according to the position coordinates of the feature points of the target vehicle in the target vehicle body coordinate system, and the rotation matrix and translation vector of the target vehicle body coordinate system to the monocular coordinate system obtained in advance, and using the camera imaging principle; obtaining the motion rotation matrix and motion translation vector between adjacent two target vehicle images according to the pixel coordinates of each feature point in each frame of target vehicle image, and in combination with the matching relationship of feature points between adjacent two target vehicle images. Specifically:

[0048] First, the feature point position coordinates of the target vehicle in each frame of target vehicle image are obtained according to the target vehicle orientation and size of each frame of target vehicle image obtained in advance. It should be noted that the target vehicle orientation and size can be obtained by using a 3D vehicle detection neural network to detect the target vehicle image, and the coordinates of the 3D points on the surface of the target vehicle in the target vehicle body coordinate system are restored by using the target vehicle position, the target vehicle orientation and the size. In addition, the target vehicle body coordinate system is with the target vehicle driving direction as the X axis, the target vehicle width direction as the Y axis, and the direction perpendicular to the ground as the Z axis.

[0049] Secondly, the pixel coordinates of each feature point in each frame of target vehicle image are obtained according to the feature point position coordinates of the target vehicle in each frame of target vehicle image, and the rotation matrix and translation vector of the target vehicle body coordinate system to the monocular coordinate system obtained in advance, and by using the camera imaging principle. It should be noted that since the vehicle is a rigid body, the 3D points corresponding to the feature points are static in the preset target vehicle body coordinate system, so the coordinates remain unchanged, and then according to the camera imaging principle, the following projection formula can be obtained:

[0050] U i,t =sK(R t X i +T t )

[0051] U i,t+1 =sK(R t ΔR t X i +T t +ΔT t )

[0052]

[0053] wherein, Ui,t is the pixel coordinate of the i-th feature point at time t; U i,t+1 is the pixel coordinate of the i-th feature point at time t+1; X i is the coordinate of the 3D point corresponding to the i-th feature point in the target vehicle body coordinate system; s is a scale factor, equal to the reciprocal of the depth of the 3D point; K is the camera intrinsic matrix, composed of the camera focal length fx, fy and the focal point pixel coordinates cx, cy; R t , T t are the rotation matrix and translation vector of the target vehicle body coordinate system to the monocular coordinate system at time t, respectively; ΔR t , ΔT t are the motion rotation matrix and motion translation vector of the target vehicle in the camera coordinate system between the two frames.

[0054] Further, according to the above projection formula, each feature point at time t can obtain two equations:

[0055]

[0056] where u i,t , v i,t represent the pixel coordinates of the i-th feature point at time t.

[0057] Similarly, each feature point at time t+1 can also obtain two equations:

[0058]

[0059] where u i,t+1 , v i,t+1 represent the pixel coordinates of the i-th feature point at time t+1.

[0060] Therefore, each feature point in the two frames can obtain four equations, assuming that the target vehicle has n feature points, then 4n equations can be listed. The number of unknowns is 12: the 6-DOF pose transformation of the target vehicle body coordinate system relative to the camera coordinate system at time t and the 6-DOF inter-frame pose transformation; when n≥3, the above projection equation can be solved by using the method of constructing a calculation graph to obtain the motion prediction ΔR t , ΔT t of the target vehicle relative to the ego vehicle.

[0061] Step S14, according to the pre-acquired ego vehicle rotation matrix and ego vehicle translation vector, combined with the motion rotation matrix and motion translation vector between the adjacent two frames of target vehicle images, the predicted target vehicle speed is obtained.

[0062] In an optional embodiment, before obtaining the predicted target vehicle speed, the method further includes: acquiring a vehicle rotation matrix and a vehicle translation vector, wherein the vehicle rotation matrix and vehicle translation vector are the vehicle rotation matrix ΔR between two adjacent frames of the vehicle in the world coordinate system obtained based on the vehicle odometer. ego and the vehicle's translation vector ΔT ego .

[0063] In this embodiment, the motion of the target vehicle in the inertial coordinate system is divided into two parts using relative motion relationships: the motion of the target vehicle relative to the vehicle itself and the motion of the vehicle itself relative to the inertial coordinate system. The motion ΔR of the vehicle itself relative to the inertial coordinate system is obtained before obtaining the velocity of the target vehicle. ego ΔT ego ; and obtained the motion prediction ΔR of the target vehicle relative to its own vehicle. t ΔT t Therefore, the speed of the target vehicle is expressed as:

[0064]

[0065] ΔR=ΔR ego ΔR t

[0066] ΔT=ΔR ego ΔT t +ΔT ego

[0067] Where V represents the target vehicle speed, ΔR ego Let ΔT represent the rotation matrix of the vehicle. ego Let ΔR represent the rotation matrix of the vehicle. t ΔT represents the motion rotation matrix of the target vehicle between two consecutive frames in the camera coordinate system. t This represents the motion translation vector of the target vehicle between two consecutive frames in the camera coordinate system.

[0068] refer to Figure 3 As shown, the vehicles in the left box are stationary vehicles, while the vehicles in the right box are non-stationary vehicles, such as vehicles moving at low speeds. After deducting the influence of the vehicle's own motion, the projection error of stationary vehicles in the preceding and following frames is small, while the projection error of moving vehicles in the preceding and following frames is large. It can be seen that the reprojection error can be used to significantly determine the dynamic and static attributes of vehicles.

[0069] It should be noted that by running the above method on the collected road test data set, the accuracy in the speed direction is improved by 61% compared with the traditional method, and the speed size accuracy is improved by 13%. In the judgment of dynamic and static properties, the resolution of the traditional method is 2m / s, and the resolution of the method is 0.3m / s. As can be seen, the method can significantly improve the speed measurement accuracy of the target vehicle, better judge the dynamic and static properties of the vehicle, and effectively improve the safety of autonomous driving and ensure driving safety.

[0070] In summary, the embodiment of the application extracts the feature points and descriptors of the acquired target vehicle image, so as to determine the matching relationship of the feature points between the adjacent frame target vehicle images according to the descriptors, thereby distinguishing the non-stationary target and the stationary target in time, improving the speed resolution; in combination with the camera imaging principle, the speed of the target vehicle is predicted to obtain the motion of the target vehicle relative to the ego vehicle; and according to the motion of the ego vehicle relative to the inertial coordinate system and the motion of the target vehicle relative to the ego vehicle, the motion of the target vehicle in the inertial coordinate is obtained, that is, the predicted target vehicle speed, thereby significantly improving the speed measurement accuracy of the target vehicle, effectively improving the safety of autonomous driving, and realizing the partial decoupling of monocular speed measurement and 3D ranging.

[0071] The vehicle speed estimation device provided by the application is described below, and the vehicle speed estimation device described below can be correspondingly referred to the vehicle speed estimation method described above.

[0072] Figure 4 The structure of a vehicle speed estimation device according to the application is shown in the schematic diagram, the device comprises:

[0073] The feature extraction module 41 obtains the feature points in each target vehicle image and the descriptors corresponding to each feature point according to at least two consecutive target vehicle images obtained in advance;

[0074] The matching module 42 matches the feature points of the adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point, and obtains the matching relationship of the feature points between the adjacent two target vehicle images;

[0075] The offset prediction module 43 obtains the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images according to the matching relationship of the feature points between the adjacent two target vehicle images, in combination with the target vehicle orientation and size of each target vehicle image obtained in advance, and by using the camera imaging principle;

[0076] The speed estimation module 44 obtains the predicted target vehicle speed according to the ego vehicle rotation matrix and the ego vehicle translation vector, in combination with the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0077] In the embodiment, the feature extraction module 41 comprises: a feature point extraction unit configured to extract visual corner points from each target vehicle image to obtain feature points corresponding to each target vehicle image; and a descriptor extraction unit configured to obtain descriptors corresponding to each feature point based on the feature points.

[0078] In an optional embodiment, the feature point extraction unit comprises: an image acquisition subunit configured to acquire the target vehicle image; a 2D target detection subunit configured to perform 2D detection on the target vehicle image to obtain a 2D detection frame; and a visual point extraction subunit configured to extract visual corner points from a region within the 2D detection frame and extract visual corner points from a region outside the 2D detection frame based on the 2D detection frame to obtain the feature points.

[0079] In addition, the descriptor extraction unit can adopt a descriptor extraction model. In this case, the descriptor extraction unit comprises: an image acquisition subunit configured to determine a surrounding region image of the feature points of a single target vehicle image based on the feature points corresponding to the single target vehicle image; a data input subunit configured to input the surrounding region image into the descriptor extraction model; and a descriptor extraction subunit configured to obtain descriptors output by the descriptor extraction model, wherein the descriptor extraction model is trained based on a matching relationship between training points selected from adjacent two target training images and a similarity between corresponding descriptors.

[0080] In an optional embodiment, the device further comprises a training module configured to train the descriptor extraction model. Specifically, the training module comprises: a training data acquisition unit configured to acquire at least two consecutive target training images and training points of each target training image; a training descriptor acquisition unit configured to input a region image around the training points corresponding to a single target training image into the descriptor extraction model to obtain training points and training descriptors corresponding to each training point output by the descriptor extraction model; a similarity acquisition unit configured to select any training point in a current target training image and obtain a corresponding similarity based on a training descriptor corresponding to the training point and a training descriptor corresponding to a training point in a previous target training image; and a judgment unit configured to judge a matching relationship between the training point of the current target training image and the training point of the previous target training image based on the similarity, construct a loss function based on the similarity of the matching training points if the matching relationship is matched, or construct a loss function based on an inverse of the similarity of the non-matching training points if the matching relationship is not matched, and end the training based on convergence of the loss function.

[0081] The matching module 42 comprises: a pre-distance acquisition unit, which selects any feature point in a current frame target vehicle image, and obtains a corresponding feature similarity based on a descriptor corresponding to the feature point and a descriptor corresponding to a feature point in a previous frame target vehicle image; a matrix construction unit, which constructs a cost matrix according to the feature similarity corresponding to all feature points in the current frame target vehicle image; and a relationship matching unit, which processes the cost matrix by using a Hungarian matching algorithm to obtain a matching relationship between feature points in adjacent two frame target vehicle images.

[0082] The offset prediction module 43 comprises: a first coordinate acquisition unit, which obtains position coordinates of feature points of a target vehicle in a target vehicle coordinate system in each frame target vehicle image according to a target vehicle orientation and size of each frame target vehicle image obtained in advance; a second coordinate acquisition unit, which obtains pixel coordinates of each feature point in each frame target vehicle image according to the position coordinates of the feature points of the target vehicle in each frame target vehicle image and a rotation matrix and a translation vector of the target vehicle coordinate system to a monocular coordinate system obtained in advance, and by using a camera imaging principle; and a motion offset data acquisition unit, which obtains a motion rotation matrix and a motion translation vector between adjacent two frame target vehicle images according to the pixel coordinates of each feature point in each frame target vehicle image and in combination with the matching relationship between the feature points in the adjacent two frame target vehicle images.

[0083] In an optional embodiment, the device further comprises: a self-vehicle data acquisition module, which acquires a self-vehicle rotation matrix and a self-vehicle translation vector before obtaining the predicted target vehicle speed, the self-vehicle rotation matrix and the self-vehicle translation vector being a self-vehicle rotation matrix ΔR ego and a self-vehicle translation vector ΔT ego .

[0084] In summary, the embodiment of the application extracts feature points and descriptors of a target vehicle image obtained by the feature extraction module, so that the matching module determines a matching relationship between feature points in adjacent frame target vehicle images according to the descriptors, thereby distinguishing non-stationary targets and stationary targets in time and improving the speed resolution; the offset prediction module is used to predict the speed of the target vehicle in combination with the camera imaging principle, so as to obtain the motion of the target vehicle relative to the ego vehicle; the speed estimation module obtains the motion of the target vehicle in the inertial coordinate system, i.e., the predicted target vehicle speed, according to the motion of the ego vehicle relative to the inertial coordinate system and the motion of the target vehicle relative to the ego vehicle, thereby significantly improving the speed measurement accuracy of the target vehicle and effectively improving the safety of automatic driving, and realizing partial decoupling of monocular speed measurement and 3D ranging.

[0085] Figure 5 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 5As shown, the electronic device can include a processor 51, a communications interface 52, a memory 53, and a communications bus 54, wherein the processor 51, the communications interface 52, and the memory 53 complete mutual communication through the communications bus 54. The processor 51 can invoke a logical instruction in the memory 53 to execute a vehicle speed estimation method, which includes: obtaining feature points in each target vehicle image and descriptors corresponding to each feature point according to at least two consecutive target vehicle images obtained in advance; matching the feature points of adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point to obtain a matching relationship of the feature points between the adjacent two target vehicle images; obtaining a motion rotation matrix and a motion translation vector between the adjacent two target vehicle images according to the matching relationship of the feature points between the adjacent two target vehicle images, in combination with a target vehicle orientation and a size of each target vehicle image obtained in advance, and by using a camera imaging principle; and obtaining a predicted target vehicle speed according to a self-vehicle rotation matrix and a self-vehicle translation vector obtained in advance, in combination with the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0086] In addition, the logical instruction in the memory 53 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partly, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk, and various media that can store program codes.

[0087] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the vehicle speed prediction method provided by the above method, which comprises: obtaining feature points in each target vehicle image and descriptors corresponding to each feature point according to at least two continuous target vehicle images obtained in advance; matching the feature points of adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point to obtain the matching relationship of the feature points between the adjacent two target vehicle images; obtaining the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images by using the camera imaging principle according to the matching relationship of the feature points between the adjacent two target vehicle images and the target vehicle orientation and size of each target vehicle image obtained in advance; and obtaining the predicted target vehicle speed according to the ego-vehicle rotation matrix and the ego-vehicle translation vector and the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0088] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the vehicle speed prediction method provided by the above method, which comprises: obtaining feature points in each target vehicle image and descriptors corresponding to each feature point according to at least two continuous target vehicle images obtained in advance; matching the feature points of adjacent two target vehicle images according to the feature points and the descriptors corresponding to each feature point to obtain the matching relationship of the feature points between the adjacent two target vehicle images; obtaining the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images by using the camera imaging principle according to the matching relationship of the feature points between the adjacent two target vehicle images and the target vehicle orientation and size of each target vehicle image obtained in advance; and obtaining the predicted target vehicle speed according to the ego-vehicle rotation matrix and the ego-vehicle translation vector and the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images.

[0089] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0090] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0091] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A vehicle speed estimation method characterized by comprising: The method comprises the following steps: According to the pre-acquired at least two consecutive target vehicle images, the feature points in each target vehicle image and the corresponding descriptors of each feature point are obtained; According to the feature points and the corresponding descriptors of each feature point, the feature points of adjacent two target vehicle images are matched to obtain the matching relationship of the feature points between the adjacent two target vehicle images; According to the matching relationship of the feature points between the adjacent two target vehicle images, combined with the target vehicle orientation and size of each frame of the pre-acquired target vehicle image, and by using the camera imaging principle, the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images are obtained; According to the pre-acquired ego-vehicle rotation matrix and ego-vehicle translation vector, combined with the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images, the predicted target vehicle speed is obtained; According to the matching relationship of the feature points between the adjacent two target vehicle images, combined with the target vehicle orientation and size of each frame of the pre-acquired target vehicle image, and by using the camera imaging principle, the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images are obtained, which comprises: According to the target vehicle orientation and size of each frame of the pre-acquired target vehicle image, the position coordinates of the feature points of the target vehicle in the target vehicle body coordinate system in each frame of the target vehicle image are obtained; According to the position coordinates of the feature points of the target vehicle in each frame of the target vehicle image, and the pre-acquired rotation matrix and translation vector from the target vehicle body coordinate system to the monocular coordinate system, and by using the camera imaging principle, the pixel coordinates of each feature point in each frame of the target vehicle image are obtained; According to the pixel coordinates of each feature point in each frame of the target vehicle image, and combined with the matching relationship of the feature points between the adjacent two target vehicle images, the motion rotation matrix and the motion translation vector between the adjacent two target vehicle images are obtained; According to the camera imaging principle, the projection formula is obtained as: ； wherein, is the pixel coordinate of the i-th feature point at time t; is the pixel coordinate of the i-th feature point at time t+1; t is the pixel coordinate of the i-th feature point at time t+1; is the coordinate of the 3D point corresponding to the i-th feature point in the target vehicle coordinate system; s represents a scale factor, which is equal to the reciprocal of the depth of the 3D point; is the camera intrinsic matrix, composed of the camera focal length f, the focal point pixel coordinates (u0, v0), and the principal point pixel coordinates (cx, cy); are the rotation matrix and the translation vector of the target vehicle coordinate system to the monocular coordinate system at time t, respectively; t are the motion rotation matrix and the motion translation vector of the target vehicle between the two frames in the camera coordinate system, respectively;​​​​​​​​​​ According to the projection formula, the first equation of each feature point at t time is obtained: ； wherein, , Xi(t) denotes the pixel coordinate of the i-th feature point at time t; The second equation of each feature point at t+1 time is obtained: ； wherein, , represents the pixel coordinate of the i-th feature point at time t+1. Two first equations and two second equations are obtained for each feature point in the front and back two frames, 4 equations are obtained, the target vehicle has n feature points in total, then 4n projection equations are obtained, and the number of unknowns is 12, including 6 degrees of freedom pose transformation of the target vehicle body coordinate system relative to the camera coordinate system at t time and 6 degrees of freedom inter-frame pose transformation; When n≥3, according to the 4n projection equations, a motion prediction of the target vehicle relative to the ego vehicle is obtained by using a method of constructing a calculation graph , ; The target vehicle speed is represented as: ； wherein, denotes the target vehicle speed, denotes the ego vehicle rotation matrix, denotes the ego vehicle rotation matrix, denotes the motion rotation matrix of the target vehicle between the two consecutive target vehicle images in the camera coordinate system, denotes the motion translation vector of the target vehicle between the two consecutive target vehicle images in the camera coordinate system.

2. The vehicle speed estimation method according to claim 1, characterized by, According to the pre-acquired at least two consecutive target vehicle images, the feature points in each target vehicle image and the corresponding descriptors of each feature point are obtained, which comprises: Visual corner points are extracted from each frame of target vehicle image respectively to obtain the feature points corresponding to each frame of target vehicle image; According to the feature points, the descriptors corresponding to each feature point are obtained.

3. The vehicle speed estimation method according to claim 2, characterized by, According to the feature points, the descriptors corresponding to each feature point are obtained, which comprises: According to the feature points corresponding to a single frame of target vehicle image, the surrounding area image of the feature points of the single frame of target vehicle image is determined; inputting the surrounding area image into a descriptor extraction model to obtain a descriptor output by the descriptor extraction model, wherein the descriptor extraction model is trained based on a matching relationship between training points selected from adjacent two frames of target training images and a similarity between corresponding descriptors.

4. The vehicle speed estimation method according to claim 3, characterized by, training the descriptor extraction model, comprising: obtaining at least two continuous frames of target training images and training points of each of the target training images; inputting an area image around the training point corresponding to a single frame of the target training image into a descriptor extraction model to obtain a training descriptor corresponding to each of the training points output by the descriptor extraction model; selecting any training point in a current frame of target training image, and obtaining a corresponding similarity based on a training descriptor corresponding to the training point and a training descriptor corresponding to a training point in a previous frame of target training image; based on the similarity, determining a matching relationship between the training point of the current frame of target training image and the training point of the previous frame of target training image, if the matching, constructing a loss function according to the similarity of the matching training points; otherwise, constructing a loss function according to the reciprocal of the similarity of the non-matching training points; based on the convergence of the loss function, ending the training.

5. The vehicle speed estimation method according to claim 2, characterized by, the visual corner point extraction of each frame of target vehicle image comprises: performing 2D detection on the target vehicle image to obtain a 2D detection frame; performing visual corner point extraction on the area within the 2D detection frame and performing visual corner point extraction on the area outside the 2D detection frame to obtain feature points.

6. The vehicle speed estimation method according to claim 1, characterized by, the matching of the feature points of the adjacent two frames of target vehicle images based on the descriptors comprises: selecting any feature point in a current frame of target vehicle image, and obtaining a corresponding feature similarity based on a descriptor corresponding to the feature point and a descriptor corresponding to a feature point in a previous frame of target vehicle image, and constructing a cost matrix according to the feature similarity corresponding to all feature points of the current frame of target vehicle image; processing the cost matrix by using the Hungarian matching algorithm to obtain the matching relationship between the feature points of the adjacent two frames of target vehicle images.

7. A vehicle speed estimation device characterized by comprising: comprises: a feature extraction module, which obtains feature points in each of the target vehicle images and descriptors corresponding to each of the feature points based on at least two continuous frames of target vehicle images obtained in advance; a matching module, which matches the feature points of adjacent two frames of target vehicle images based on the feature points and the descriptors corresponding to each of the feature points to obtain a matching relationship between the feature points of the adjacent two frames of target vehicle images; a displacement prediction module, which obtains a motion rotation matrix and a motion translation vector between the adjacent two frames of target vehicle images based on the matching relationship between the feature points of the adjacent two frames of target vehicle images, the orientation and size of the target vehicle of each of the target vehicle images obtained in advance, and the camera imaging principle; a speed estimation module, which obtains a predicted target vehicle speed based on a self-vehicle rotation matrix and a self-vehicle translation vector obtained in advance and the motion rotation matrix and the motion translation vector between the adjacent two frames of target vehicle images. The offset prediction module comprises: A first coordinate acquisition unit obtains position coordinates of feature points of a target vehicle in a target vehicle coordinate system in each frame of the target vehicle image according to a target vehicle orientation and size of each frame of the target vehicle image obtained in advance; A second coordinate acquisition unit obtains pixel coordinates of each feature point in each frame of the target vehicle image according to the position coordinates of the feature points of the target vehicle in each frame of the target vehicle image and a rotation matrix and a translation vector of the target vehicle coordinate system to a monocular coordinate system obtained in advance, and utilizes a camera imaging principle; A motion offset data acquisition unit obtains a motion rotation matrix and a motion translation vector between the adjacent two frames of the target vehicle image according to the pixel coordinates of each feature point in each frame of the target vehicle image and in combination with a matching relationship of the feature points between the adjacent two frames of the target vehicle image; According to the camera imaging principle, a projection formula is obtained as follows: ； wherein, is the pixel coordinate of the i-th feature point at time t; is the pixel coordinate of the i-th feature point at time t+1; is the coordinate of the i-th feature point in the target vehicle coordinate system at time t; is the coordinate of the i-th feature point in the target vehicle coordinate system at time t+1; t is the coordinate of the i-th feature point in the target vehicle coordinate system at time t+1; is the coordinate of the i-th feature point in the target vehicle coordinate system at time t+1; is the coordinate of the i-th feature point in the target vehicle coordinate system at time t+1; K is the camera intrinsic matrix, composed of the camera focal length f, the principal point pixel coordinate (u0, v0), and the focal length ratio k; , , , , , are the rotation matrix and the translation vector of the target vehicle coordinate system to the monocular coordinate system at time t and t+1, respectively; t , , are the motion rotation matrix and the motion translation vector of the target vehicle in the camera coordinate system between the two frames. According to the projection formula, a first equation is obtained for each feature point at t time: ； wherein, , Xi(t) denotes the pixel coordinate of the i-th feature point at time t; A second equation is obtained for each feature point at t+1 time: ； wherein, , represents the pixel coordinate of the i-th feature point at time t+1. Two first equations and two second equations are obtained for each feature point in the front and back two frames, 4 equations are obtained, the target vehicle has n feature points in total, 4n projection equations are obtained, and the number of unknowns is 12, including 6 degrees of freedom pose transformation of the target vehicle coordinate system relative to the camera coordinate system at t time and 6 degrees of freedom inter-frame pose transformation; When n≥3, according to the 4n projection equations, a motion prediction of the target vehicle relative to the ego vehicle is obtained by using a method of constructing a calculation graph 、 ; The target vehicle speed is expressed as: ； wherein, denotes the target vehicle speed, denotes the ego vehicle rotation matrix, denotes the ego vehicle rotation matrix, denotes the motion rotation matrix of the target vehicle between the two consecutive target vehicle images in the camera coordinate system, denotes the motion translation vector of the target vehicle between the two consecutive target vehicle images in the camera coordinate system.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the vehicle speed estimation method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the vehicle speed estimation method according to any one of claims 1 to 6.

Images