A monocular vision-based three-dimensional vehicle tracking method, device and electronic equipment

Abstract

The present disclosure relates to the technical field of target three-dimensional tracking detection, and provides a monocular vision-based three-dimensional vehicle tracking method, device and electronic equipment. The method comprises: acquiring a surrounding scene image of a vehicle, the surrounding scene image being a two-dimensional image and comprising at least one target vehicle; inputting the two-dimensional image into a pre-trained deep learning neural network model to obtain two-dimensional composite information of the target vehicle in the two-dimensional image; determining three-dimensional ranging information of the target vehicle in a vehicle body coordinate system based on parameters of a camera and the two-dimensional composite information of the target vehicle; and determining a three-dimensional tracking state of the target vehicle based on a re-identification feature of the target vehicle, the two-dimensional composite information and the three-dimensional ranging information. The present disclosure recovers three-dimensional information of the target vehicle in the vehicle body coordinate system by using a two-dimensional image of the target vehicle, thereby reducing a model landing difficulty coefficient and being more practical.

Description

Technical Field

[0001] This disclosure relates to the field of target 3D tracking and detection technology, and in particular to a 3D vehicle tracking method, apparatus and electronic device based on monocular vision. Background Technology

[0002] Vision-based 3D vehicle detection is one of the key technologies in autonomous driving environmental perception systems. First, image data containing the target vehicle is acquired through a camera. Then, the data is input into a deep learning neural network. The output of the deep learning neural network directly or indirectly obtains the 3D information of the target vehicle, including its category, 3D coordinates of the center point, orientation angle, and 3D dimensions. The tracking module then performs smoothing filtering to achieve dynamic obstacle 3D perception in a 3D coordinate system, providing data support for subsequent multi-sensor fusion modules.

[0003] However, existing vehicle 3D detection methods are usually based on deep learning, regressing information such as the 2D pixel coordinates of the target's 3D center point projected onto the image, the depth, orientation angle, and 3D dimensions of the target's 3D center point. The advantage of this method is that the overall process model design is relatively simple, and the detection accuracy and stability are relatively high. However, the disadvantages are that the model has poor generalization performance and is highly dependent on the training data. Once the camera model changes or the camera's installation position on the autonomous vehicle changes, the original data will become inapplicable. In addition, this method has extremely high requirements for training data, requiring joint annotation by vision and LiDAR. The 3D information of the target in the LiDAR annotation results is used as the ground truth of the target in the image. Therefore, the spatiotemporal synchronization requirements of the sensors are strict, especially the time synchronization accuracy, which is a major technical challenge in the field of autonomous driving. As a result, the implementation difficulty is high and it lacks practicality. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus and electronic device for three-dimensional vehicle tracking based on monocular vision, in order to solve the technical problem that existing three-dimensional vehicle detection methods have extremely high requirements for training data, resulting in a high degree of difficulty in model deployment and a lack of practicality.

[0005] A first aspect of this disclosure provides a three-dimensional vehicle tracking method based on monocular vision, comprising: acquiring a surrounding scene image captured by a monocular camera on a vehicle, the surrounding scene image including at least one target vehicle driving around the vehicle, and the surrounding scene image being a two-dimensional image; inputting the two-dimensional image into a pre-trained deep learning neural network model, and obtaining two-dimensional composite information of the target vehicle in the two-dimensional image from the output of the deep learning neural network model; determining the three-dimensional ranging information of the target vehicle in the vehicle coordinate system based on the camera parameters and the two-dimensional composite information of the target vehicle; using a pedestrian re-identification algorithm to identify the surrounding scene image of the target vehicle to obtain the re-identification features of the target vehicle; and determining the three-dimensional tracking state of the target vehicle based on the re-identification features of the target vehicle, the two-dimensional composite information, and the three-dimensional ranging information.

[0006] A second aspect of this disclosure provides a three-dimensional vehicle tracking device based on monocular vision, comprising: an image acquisition module configured to acquire surrounding scene images captured by a monocular camera on a vehicle, the surrounding scene images including at least one target vehicle driving around the vehicle, and the surrounding scene images being two-dimensional images; an image recognition module configured to input the two-dimensional images into a pre-trained deep learning neural network model, and obtain two-dimensional composite information of the target vehicle in the two-dimensional image from the output of the deep learning neural network model; an inverse perspective transformation module configured to determine three-dimensional ranging information of the target vehicle in a vehicle coordinate system based on camera parameters and the two-dimensional composite information of the target vehicle; a feature extraction module configured to identify the surrounding scene images of the target vehicle using a pedestrian re-identification algorithm to obtain re-identification features of the target vehicle; and a target tracking module configured to determine the three-dimensional tracking state of the target vehicle based on the re-identification features, the two-dimensional composite information, and the three-dimensional ranging information of the target vehicle.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] The advantages of this disclosure compared to existing technologies are as follows: This monocular vision-based 3D vehicle tracking method acquires surrounding scene images captured by a monocular camera on the vehicle. These surrounding scene images include at least one target vehicle traveling around the vehicle, and the surrounding scene images are two-dimensional images. The two-dimensional images are input into a pre-trained deep learning neural network model, and the output of the deep learning neural network model yields the two-dimensional composite information of the target vehicle in the two-dimensional image. Based on the camera parameters and the two-dimensional composite information of the target vehicle, the three-dimensional ranging information of the target vehicle in the vehicle coordinate system is determined. A pedestrian re-identification algorithm is used to identify the surrounding scene images of the target vehicle, obtaining the re-identification features of the target vehicle. Based on the re-identification features, the two-dimensional composite information, and the three-dimensional ranging information of the target vehicle, the three-dimensional tracking state of the target vehicle is determined. This achieves the effect of recovering the three-dimensional information of the target vehicle in the vehicle coordinate system from the two-dimensional images of the target vehicle. Thus, during the training of the deep learning neural network model, only the two-dimensional information of the images needs to be labeled, reducing the dependence of model training on sample data. Furthermore, there is no requirement for spatiotemporal synchronization, resulting in lower implementation costs and higher feasibility for the deep learning neural network model, making it more widely applicable and practical. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a three-dimensional vehicle tracking method based on monocular vision provided in an embodiment of this disclosure;

[0011] Figure 2 This is a flowchart illustrating step S103 provided in one embodiment of the present disclosure;

[0012] Figure 3 This is a schematic diagram illustrating the principle of calculating three-dimensional ranging information using the inverse perspective transformation principle of a camera, provided in an embodiment of this disclosure.

[0013] Figure 4 This is a flowchart illustrating step S105 provided in one embodiment of the present disclosure;

[0014] Figure 5 This is a schematic diagram of the structure of a three-dimensional vehicle tracking device based on monocular vision provided in an embodiment of this disclosure;

[0015] Figure 6This is a schematic diagram of the structure of an inverse perspective transformation module provided in an embodiment of this disclosure;

[0016] Figure 7 This is a schematic diagram of the structure of a target tracking module provided in an embodiment of this disclosure;

[0017] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0019] Please see Figure 1 This is a flowchart illustrating a monocular vision-based 3D vehicle tracking method provided in an embodiment of this disclosure. In practical applications, this monocular vision-based 3D vehicle tracking method can be executed by a vehicle with autonomous driving capabilities or an electronic device connected to a vehicle network.

[0020] like Figure 1 As shown, this monocular vision-based 3D vehicle tracking method includes:

[0021] S101, acquire the surrounding scene image captured by the monocular camera on the vehicle, the surrounding scene image includes at least one target vehicle driving around the vehicle, and the surrounding scene image is a two-dimensional image;

[0022] S102, input the two-dimensional image into the pre-trained deep learning neural network model, and obtain the two-dimensional composite information of the target vehicle in the two-dimensional image from the output of the deep learning neural network model;

[0023] S103, based on the camera parameters and the two-dimensional composite information of the target vehicle, determines the three-dimensional ranging information of the target vehicle in the vehicle coordinate system.

[0024] S104, Use the pedestrian re-identification algorithm to identify the scene image around the target vehicle and obtain the re-identification features of the target vehicle.

[0025] S105 determines the three-dimensional tracking status of the target vehicle based on the re-identification features, two-dimensional composite information, and three-dimensional ranging information of the target vehicle.

[0026] The working principle of the above-mentioned monocular vision-based 3D vehicle tracking method is as follows: Monocular vision is used to acquire surrounding scene images. A pre-trained deep learning neural network model infers from these images to obtain the two-dimensional composite information of the target vehicle. Then, based on the camera's imaging principle, the camera parameters and this two-dimensional composite information are used to calculate the target's distance information in the vehicle coordinate system, i.e., the target vehicle's three-dimensional distance information. Finally, this two-dimensional composite information is used for target vehicle tracking and matching, and the corresponding three-dimensional distance information is smoothed and filtered to recover the real-time three-dimensional information of the target vehicle, thus achieving 3D tracking. Therefore, this embodiment recovers the target vehicle's three-dimensional information in the vehicle coordinate system by acquiring two-dimensional images of the target vehicle. This reduces the model's dependence on sample data during deep learning neural network model training, as only the two-dimensional information of the images needs to be labeled. Furthermore, there is no requirement for spatiotemporal synchronization, resulting in lower implementation costs and higher feasibility for the deep learning neural network model, enabling its widespread application.

[0027] In step S101 above, monocular vision refers to the technology of acquiring scene information through a single camera. It is an important branch of computer vision with wide applications, including robot vision, autonomous driving, and intelligent monitoring. Monocular vision technology can achieve tasks such as visual perception, target detection, pose estimation, and 3D reconstruction. In this embodiment, a two-dimensional image of the surrounding scene is acquired using monocular vision, which contains one or more target vehicles, and these target vehicles are the objects being tracked.

[0028] In step S102 above, the trained deep learning neural network model can be deployed on the vehicle. For example, the trained deep learning neural network model can be deployed on the vehicle's host computer. When surrounding scene images are acquired, the surrounding scene images are directly input into the deep learning neural network model deployed on the host computer for inference to obtain the two-dimensional composite information of the target vehicle in the surrounding scene images. Alternatively, it can be deployed on an electronic device that is connected to the vehicle. For example, the trained deep learning neural network model can be deployed on a server. The vehicle can communicate with the server via the network. When surrounding scene images are acquired, the surrounding scene images are first sent to the server. The deep learning neural network model deployed on the server performs inference on the surrounding scene images and obtains the corresponding two-dimensional composite information. Then, the server returns the inference result to the vehicle.

[0029] In some embodiments, the training method of the deep learning neural network model includes: offline training of the deep learning neural network model using historical surrounding scene images and annotation results matching the historical surrounding scene images to obtain a weight file of the deep learning neural network model; and updating the pre-trained deep learning neural network model based on the weight file.

[0030] The annotation results of the surrounding scene images of this history include at least one of the following: vehicle category, two-dimensional bounding box, visible surface attributes on the image, vertical edges, wheel contact points, and wheel contact point attributes. Among them, visible surface attributes include whether the four sides of the target vehicle (front / rear / left / right) are visible from the image; vertical edges refer to the boundary lines between two adjacent sides of the vehicle's four sides, and are vertically labeled; wheel contact points refer to the intersections of all visible wheels of the vehicle with the ground; there are a total of 6 types of wheel contact point attributes: left front wheel / right front wheel / left rear wheel / right rear wheel / left middle wheel / right middle wheel.

[0031] Specifically, training a deep learning neural network model using historical surrounding scene data allows for the modification of the weights of each node within the network, resulting in a corresponding weight file. This weight file is not fixed. Once the initially trained deep learning neural network model meets preset requirements, it can be deployed on a vehicle or an electronic device connected to the vehicle for intelligent recognition of surrounding scene images. During the use of this model, the number of historical surrounding scene images can be increased to continue training the initially trained model, resulting in a new weight file. This new weight file can then be used to update the deployed deep learning neural network model, improving its accuracy and optimizing its recognition capabilities.

[0032] Understandably, when a trained deep learning neural network model is used to collect real-time images of the surrounding environment of a vehicle, the two-dimensional composite information obtained from the output of the deep learning neural network model is consistent with the annotation results during the training phase. That is, the two-dimensional composite information obtained from the output of the deep learning neural network model also includes at least one of the following: vehicle category of the target vehicle, two-dimensional bounding box, visible surface attributes on the image, vertical edge, wheel contact point, and wheel contact point attributes.

[0033] The deep learning neural network model can be specifically implemented as various existing neural networks. For example, in this embodiment, the deep learning neural network model is preferably the anchor-free series of CenterNet. CenterNet is an anchor-free target detection algorithm, which is an improvement on the CornerNet algorithm.

[0034] In some embodiments, before inputting the two-dimensional image into the pre-trained deep learning neural network model, the method further includes: sequentially performing image scaling, image normalization, image standardization, and image channel transformation on the surrounding scene image to obtain a pre-processed image of the surrounding scene image.

[0035] Specifically, in this embodiment, image scaling is implemented by adjusting the surrounding scene image to the resolution of the model input; image normalization is implemented by dividing the channel values ​​of the surrounding scene image by 255; image standardization is implemented by subtracting the mean from the channel values ​​of the surrounding scene image and then dividing by the variance, where the mean is the average value of each channel and the variance is the average of the squares of the differences between each channel and the mean; and image channel transformation is implemented by converting the surrounding scene image from BGR mode to RGB mode. This embodiment enhances important image features for subsequent processing by preprocessing the surrounding scene image, while reducing some unnecessary information, thereby improving the inference efficiency of the deep learning neural network model.

[0036] In step S103 above, the camera parameters include, but are not limited to, the camera's intrinsic and extrinsic parameters. For example, the camera parameters include the camera's height above the ground, focal length, and the camera's mounting position on the vehicle.

[0037] In some embodiments, see Figure 2 Based on the camera parameters and the two-dimensional composite information of the target vehicle, the three-dimensional ranging information of the target vehicle in the vehicle body coordinate system is determined, including:

[0038] S201, Based on the size of the surrounding scene image and the coordinates of the four vertices of the two-dimensional bounding box of the target vehicle in the surrounding scene image, determine whether the target vehicle in the surrounding scene image is truncated to the left or right.

[0039] S202, If the target vehicle is cut off to the left and right, perform height measurement on the target vehicle;

[0040] S203, if the target vehicle is not cut off to the left or right, continue to determine whether the target vehicle in the surrounding scene image is cut off to the bottom.

[0041] S204, If the target vehicle is a lower segment, perform width measurement on the target vehicle;

[0042] S205, if the target vehicle is not a lower truncation, then perform three-dimensional ranging on the target vehicle.

[0043] Specifically, in step S201 above, the specific implementation method for identifying whether a target vehicle in the surrounding scene image is truncated horizontally is not unique. For example, in this embodiment, the aspect ratio of the target vehicle's two-dimensional bounding box can be calculated using the coordinates of the four vertices of the bounding box. Since the aspect ratio of the target vehicle is fixed, when the target vehicle in the surrounding scene image is truncated horizontally, the length of the target vehicle's two-dimensional bounding box becomes shorter. Therefore, compared to the untruncated case, the aspect ratio of the target vehicle's two-dimensional bounding box will be smaller when the target vehicle is truncated horizontally. Thus, by calculating the aspect ratio of the target vehicle's two-dimensional bounding box using the coordinates of the four vertices of the bounding box, if the calculated aspect ratio is less than a preset value (preferably the aspect ratio of the target vehicle under normal conditions), it is determined that the target vehicle in the surrounding scene image is truncated horizontally. Of course, other specific implementation methods can also be used in practice to determine whether a target vehicle in the surrounding scene image is truncated horizontally; this embodiment does not limit this.

[0044] In step S202 above, when the target vehicle is determined to be truncated on the left and right, the height of the target vehicle is measured, including: based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, the horizontal and vertical distances and height dimensions of the target vehicle in the vehicle coordinate system are determined by the camera imaging model.

[0045] Specifically, given the camera's position and focal length, the target vehicle's prior dimensions (length and width), and the target vehicle's two-dimensional bounding box in the surrounding images, the width or height of the target vehicle can be calculated by establishing equations using similar triangles based on the pinhole camera imaging principle. Therefore, for cases where the target vehicle is truncated on the left or right, the height can be calculated to overcome the problem of incomplete width of the target vehicle in the surrounding images.

[0046] Furthermore, the specific implementation method for determining whether the target vehicle in the surrounding scene image is truncated downwards in step S203 above is not unique. For example, similar to the principle of determining left and right truncations, the aspect ratio of the target vehicle's two-dimensional bounding box can be calculated using the coordinates of the four vertices. Since the aspect ratio of the target vehicle is fixed, when the target vehicle in the surrounding scene image is truncated downwards, the width of the target vehicle's two-dimensional bounding box becomes shorter. Therefore, compared to the untruncated case, the aspect ratio of the target vehicle's two-dimensional bounding box will be larger when the target vehicle is truncated downwards. Thus, by calculating the aspect ratio of the target vehicle's two-dimensional bounding box using the coordinates of the four vertices of the two-dimensional bounding box, if the calculated aspect ratio is greater than a preset value, it is determined that the target vehicle in the surrounding scene image is truncated downwards. Of course, other specific implementation methods can also be used in practice to determine whether the target vehicle in the surrounding scene image is truncated downwards, and this embodiment does not limit this.

[0047] In step S204 above, when the target vehicle is determined to be a lower truncation, the width of the target vehicle is measured, including: based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, the horizontal and vertical distances and width dimensions of the target vehicle in the vehicle coordinate system are determined by the camera imaging model.

[0048] Similarly, for the case of a truncated section, based on the imaging principle of a pinhole camera, an equation can be established using similar triangles to calculate the width or height of the target vehicle. Therefore, this embodiment can overcome the problem of incomplete height representation of the target vehicle in the surrounding scene image by calculating its width.

[0049] It should be noted that when the target vehicle is truncated horizontally or vertically, only the horizontal and vertical distances and dimensions (length and width) of the target vehicle in the vehicle coordinate system can be calculated, but the orientation angle of the target vehicle cannot be calculated. Furthermore, equations are established using similar triangles to calculate the horizontal and vertical distances and dimensions (length and width) of the target relative to the camera in the image. Then, coordinate system transformations are performed to obtain the horizontal and vertical distances and dimensions (length and width) of the target vehicle in the vehicle coordinate system. These are all conventional calculation methods, and therefore will not be described in detail here.

[0050] Specifically, the three-dimensional ranging information of the target vehicle includes at least one of the following: the horizontal and vertical distances of the target vehicle in the vehicle coordinate system, its length and width dimensions, and its heading angle. In this embodiment, the heading angle is also called the heading angle.

[0051] In step S205 above, if the target vehicle is not truncated to the left or right or to the bottom, three-dimensional ranging of the target vehicle is performed, including: determining whether the number of wheel contact points in the surrounding scene image of the target vehicle is at least two; if the number of wheel contact points is two, the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system are determined by inverse perspective transformation based on the wheel contact points; if the number of wheel contact points is more than two, the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system are determined by inverse perspective transformation based on the horizontal coordinates of the two-dimensional bounding box and vertical edges of the target vehicle, as well as the at least three vehicle contact points of the target vehicle.

[0052] Specifically, based on the principle of camera inverse perspective transformation and the assumption of ground plane, the three-dimensional distance measurement information of the target vehicle in the vehicle's body coordinate system is calculated using the vertical edge of the target wheel, the wheel contact point, and the wheel contact point attributes. Here, the ground plane is the plane where the contact point between the target vehicle's wheel and the ground is located.

[0053] In the case of two wheel contact points, the points on the ground plane (i.e., the wheel contact points) are projected from the image into the vehicle coordinate system. Since two points can determine a straight line, the ranging information is further transformed to the center of the target vehicle using the wheel contact point attributes. This allows the calculation of the target vehicle's lateral and longitudinal distances, length, width, and orientation angle. It should be noted that, given the camera position and points on the ground plane, using the inverse perspective transformation principle to calculate the 3D ranging information such as the lateral and longitudinal distances, length, width, and orientation angle of the wheel contact points in the vehicle coordinate system is a conventional method. Combining this with the target vehicle's prior length and width dimensions and wheel contact point attributes, the target wheel's lateral and longitudinal distances, length, width, and orientation angle in the vehicle coordinate system can be determined. Therefore, this will not be described in detail here.

[0054] In cases where there are two or more wheel contact points, the same principle of camera inverse perspective transformation and ground plane assumption are used. Based on the minimum / maximum x-coordinates of the target vehicle's two-dimensional bounding box in the surrounding scene image, the x-coordinates of the vertical edges, and three points on the ground plane (i.e., wheel contact points), a system of equations is established. By solving the system of equations, the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system can be calculated. Since this ranging method does not depend on the target vehicle's y-coordinate in the surrounding scene image, it has good robustness in situations with bumpy road surfaces.

[0055] For example, see Figure 3 This paper illustrates a schematic diagram of a method for calculating 3D ranging information using the principle of inverse perspective transformation of a camera. Assuming there are three wheel contact points, K is the camera's intrinsic parameter, and R and t are its extrinsic parameters, representing the rotation matrix and offset vector, respectively; C is the camera's position in the vehicle coordinate system, C = -R^(-1)*t. Then, there are three sets of equations (each set representing the camera's perspective transformation formula):

[0056]

[0057] Here, only the first line of each equation system is used, forming a new equation system with three lines. θ represents the orientation angle; lv and wv are the prior width and height; * represents the ordinate of the three points on the image, which is not actually used; X represents the position of the target vehicle in the vehicle coordinate system (i.e., Figure 3 In the equations d_horizontal, d_vertical, and Z_c, it can be seen that by solving the above system of equations, the horizontal and vertical distances and orientation angles of the target vehicle in the vehicle coordinate system can be obtained.

[0058] In some embodiments, see Figure 4 Based on the target vehicle's re-identification features, two-dimensional composite information, and three-dimensional ranging information, the three-dimensional tracking status of the target vehicle is determined, including:

[0059] S401 uses re-identification features and two-dimensional composite information of the target vehicle for joint matching;

[0060] S402, If the current result does not match the existing tracker, a new tracker is created to track the target vehicle, wherein the current result re-identifies the features and the two-dimensional composite information of the target vehicle.

[0061] S403, if the current result matches an existing tracker, update the tracker state; wherein, the tracker includes a filter tracker, and updating the tracker state includes using multiple filter trackers to independently filter different information in the three-dimensional ranging information to obtain the three-dimensional tracking state of the target vehicle.

[0062] In this embodiment, the filter tracker is preferably a Kalman filter. Of course, other filters can also be used in practice, and this embodiment does not limit this.

[0063] Specifically, in step S401 above, the person re-identification algorithm (Re-ID algorithm) is a person recognition technology based on deep learning and computer vision. Its purpose is to identify and match multiple images of the same person in different times and scenarios. Compared with traditional face recognition technology, the person re-identification algorithm has higher accuracy and stability. The person re-identification algorithm typically includes two main steps: feature extraction and matching calculation. Feature extraction is the process of abstracting the appearance information of pedestrians into vectors, i.e., obtaining re-identification features. Commonly used feature extraction methods include local features and global features. Matching calculation involves calculating the similarity of the abstracted vectors and comparing the similarity of these vectors to determine whether they belong to the same pedestrian. Since these are all conventional calculation methods, they will not be elaborated further here.

[0064] The filtering of the three-dimensional ranging information by the filter tracker includes using one filter to filter the lateral and longitudinal distances of the target vehicle, and using another filter to filter the orientation angle. In other words, this embodiment improves the stability of the target vehicle ranging by independently filtering the information in the detected three-dimensional ranging information.

[0065] For example, assuming the target vehicle's lateral and longitudinal distances in the vehicle coordinate system are (x, y), its length and width dimensions are (l, w), and its orientation angle is α, a linear Kalman filter algorithm is used. One independent filter filters the target vehicle's motion state (x, y), while another independent filter filters the orientation angle α. Specifically, the target vehicle's state can be directly obtained using the detection values ​​of l, w, and α. For the target vehicle's motion state, the Kalman filter's state variables are defined as sx, sy, vx, vy, where sx, sy represent the lateral and longitudinal distances of the target center point in the vehicle coordinate system, vx, vy represent the lateral and longitudinal velocities of the target center point in the vehicle coordinate system, and the Kalman filter's observation values ​​are x, y, where x, y represent the lateral and longitudinal distances of the target center point in the vehicle coordinate system. When there is more than one ranging method, there can be multiple observation values, which can be calculated using a weighted average method to obtain the final observation value. Furthermore, another Kalman filter is used to separately filter the detection value of α to update the target vehicle's motion state.

[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0067] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0068] Please see Figure 5 This is a schematic diagram of the structure of a three-dimensional vehicle tracking device based on monocular vision provided in an embodiment of this disclosure, as shown below. Figure 5 As shown, the monocular vision-based 3D vehicle tracking device includes:

[0069] The image acquisition module 510 is configured to acquire surrounding scene images captured by a monocular camera on the vehicle, the surrounding scene images including at least one target vehicle driving around the vehicle, and the surrounding scene images being two-dimensional images.

[0070] The image recognition module 520 is configured to input the two-dimensional image into a pre-trained deep learning neural network model, and obtain two-dimensional composite information of the target vehicle in the two-dimensional image from the output of the deep learning neural network model.

[0071] The inverse perspective transformation module 530 is configured to determine the three-dimensional ranging information of the target vehicle in the vehicle coordinate system based on the parameters of the camera and the two-dimensional composite information of the target vehicle.

[0072] The feature extraction module 540 is configured to use a pedestrian re-identification algorithm to identify the scene image around the target vehicle and obtain the re-identification features of the target vehicle.

[0073] The target tracking module 550 is configured to determine the three-dimensional tracking status of the target vehicle based on the re-identification features, two-dimensional composite information and three-dimensional ranging information of the target vehicle.

[0074] This monocular vision-based 3D vehicle tracking device acquires surrounding scene images from a monocular camera on the vehicle. These images include at least one target vehicle traveling around the vehicle, and are 2D images. The 2D images are input into a pre-trained deep learning neural network model, and the output of the model yields 2D composite information of the target vehicle. Based on the camera parameters and the 2D composite information of the target vehicle, the device determines the 3D ranging information of the target vehicle in the vehicle coordinate system. A pedestrian re-identification algorithm is used to identify the surrounding scene images of the target vehicle, obtaining its re-identification features. Based on the re-identification features, 2D composite information, and 3D ranging information, the device determines the 3D tracking state of the target vehicle. This achieves the effect of recovering the 3D information of the target vehicle in the vehicle coordinate system from its 2D images. During the training of the deep learning neural network model, only the 2D information of the images needs to be labeled, reducing the model's dependence on sample data. Furthermore, there is no requirement for spatiotemporal synchronization. Therefore, the implementation cost of the deep learning neural network model is low, its feasibility is high, and it can be widely applied, making it more practical.

[0075] In some embodiments, combined with Figure 5 As shown, the monocular vision-based 3D vehicle tracking device also includes:

[0076] The image preprocessing module 560 is configured to perform image scaling, image normalization, image standardization, and image channel transformation on the surrounding scene image sequentially before inputting the two-dimensional image into a pre-trained deep learning neural network model, so as to obtain a preprocessed image of the surrounding scene image.

[0077] In some embodiments, combined with Figure 5 As shown, the monocular vision-based 3D vehicle tracking device also includes:

[0078] The model training module 570 is configured to train a deep learning neural network model offline using historical surrounding scene images and annotation results matched with the historical surrounding scene images, to obtain a weight file of the deep learning neural network model.

[0079] The model update module 580 is configured to update the pre-trained deep learning neural network model based on the weight file.

[0080] In some embodiments, the two-dimensional composite information includes at least one of the following information about the target vehicle: vehicle category, two-dimensional bounding box, visible surface attributes on the image, vertical edges, wheel contact points, and wheel contact point attributes.

[0081] In some embodiments, see Figure 6 The inverse perspective transformation module 530 includes: a left-right truncation judgment unit 531, configured to determine whether the target vehicle in the surrounding scene image is left-right truncation based on the size of the surrounding scene image and the coordinates of the four vertices of the two-dimensional bounding box of the target vehicle in the surrounding scene image; a height measurement unit 532, configured to perform height measurement on the target vehicle if it is left-right truncation; a bottom truncation judgment unit 533, configured to continue judging whether the target vehicle in the surrounding scene image is bottom truncation if it is not left-right truncation; a width measurement unit 534, configured to perform width measurement on the target vehicle if it is bottom truncation; and a three-dimensional measurement unit 535, configured to perform three-dimensional measurement on the target vehicle if it is not bottom truncation.

[0082] In some embodiments, Figure 6 The height ranging unit 532 is specifically configured to determine the horizontal and vertical distances and height of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

[0083] In some embodiments, Figure 6 The width ranging unit 534 is specifically configured to determine the horizontal and vertical distances and width of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

[0084] In some embodiments, Figure 6 The three-dimensional ranging unit 535 is specifically configured to determine whether the number of wheel contact points in the surrounding scene image of the target vehicle is at least two: if the number of wheel contact points is two, the target vehicle's horizontal and vertical distances, length and width dimensions, and orientation angle in the vehicle coordinate system are determined by inverse perspective transformation based on the wheel contact points; if the number of wheel contact points is more than two, the target vehicle's horizontal and vertical distances, length and width dimensions, and orientation angle in the vehicle coordinate system are determined by inverse perspective transformation based on the horizontal coordinates of the target vehicle's two-dimensional bounding box and vertical edges, as well as the target vehicle's at least three vehicle contact points.

[0085] In some embodiments, see Figure 7The target tracking module 550 includes: a matching judgment unit 551, configured to perform joint matching using re-identification features and two-dimensional composite information of the target vehicle; a first tracking unit 552, configured to create a new tracker to track the target vehicle if the current result does not match an existing tracker, wherein the current result includes re-identification features and two-dimensional composite information of the target vehicle; and a second tracking unit 553, configured to update the tracker state if the current result matches an existing tracker, wherein the tracker includes a filtering tracker, and updating the tracker state includes using multiple filtering trackers to independently filter different information in the three-dimensional ranging information to obtain the three-dimensional tracking state of the target vehicle.

[0086] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0087] Figure 8 This is a schematic diagram of the electronic device 8 provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module / unit in the various device embodiments described above.

[0088] Electronic device 8 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.

[0089] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0090] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0093] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A monocular vision-based three-dimensional vehicle tracking method, characterized by, include: Acquire surrounding scene images captured by a monocular camera on the vehicle, wherein the surrounding scene images include at least one target vehicle driving around the vehicle, and the surrounding scene images are two-dimensional images; The two-dimensional image is input into a pre-trained deep learning neural network model, and the two-dimensional composite information of the target vehicle in the two-dimensional image is obtained from the output of the deep learning neural network model. Based on the parameters of the camera and the two-dimensional composite information of the target vehicle, the three-dimensional ranging information of the target vehicle in the vehicle coordinate system is determined. The pedestrian re-identification algorithm is used to identify the scene images around the target vehicle to obtain the re-identification features of the target vehicle; Based on the re-identification features, two-dimensional composite information and three-dimensional ranging information of the target vehicle, the three-dimensional tracking status of the target vehicle is determined; The determination of the three-dimensional ranging information of the target vehicle in the vehicle coordinate system based on the camera parameters and the two-dimensional composite information of the target vehicle includes: Based on the size of the surrounding scene image and the two-dimensional bounding box of the target vehicle in the surrounding scene image, it is determined that the target vehicle is not truncated to the left or right, not truncated to the bottom, and the number of wheel contact points in the surrounding scene image of the target vehicle is more than 2. Based on the horizontal coordinates of the target vehicle's two-dimensional bounding box and vertical edges, as well as at least three vehicle grounding points, the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system are determined by inverse perspective transformation.

2. The monocular vision-based three-dimensional vehicle tracking method of claim 1, wherein, Before inputting the two-dimensional image into the pre-trained deep learning neural network model, the method further includes: sequentially performing image scaling, image normalization, image standardization, and image channel transformation on the surrounding scene image to obtain a pre-processed image of the surrounding scene image.

3. The monocular vision-based three-dimensional vehicle tracking method of claim 1, wherein, The training method for the deep learning neural network model includes: The deep learning neural network model is trained offline using historical surrounding scene images and the annotation results matched with the historical surrounding scene images to obtain a weight file of the deep learning neural network model. The pre-trained deep learning neural network model is updated based on the weight file.

4. The monocular vision-based three-dimensional vehicle tracking method according to any one of claims 1-3, characterized in that, The two-dimensional composite information includes at least one of the following information about the target vehicle: vehicle category, two-dimensional bounding box, visible surface attributes on the image, vertical edges, wheel contact points, and wheel contact point attributes.

5. The monocular vision-based three-dimensional vehicle tracking method of claim 4, wherein, The method of determining the vehicle as a three-dimensional ranging information of the target vehicle in the vehicle coordinate system based on the parameters of the camera and the two-dimensional composite information of the target vehicle also includes: Based on the dimensions of the surrounding scene images and the two-dimensional bounding box of the target vehicle in the surrounding scene images, the target vehicle is determined to be truncated on the left and right sides, and the height of the target vehicle is measured. Based on the dimensions of the surrounding scene images and the two-dimensional bounding box of the target vehicle in the surrounding scene images, the target vehicle is determined to be neither truncated to the left nor right and to the right and to the left, and the width of the target vehicle is measured. Based on the dimensions of the surrounding scene images and the two-dimensional bounding box of the target vehicle in the surrounding scene images, it is determined that the target vehicle is not truncated to the left or right, not truncated to the bottom, and that there are two wheel contact points in the surrounding scene images where the target vehicle is located. Then, based on the wheel contact points, the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system are determined by inverse perspective transformation.

6. The monocular vision-based three-dimensional vehicle tracking method of claim 5, wherein, The height measurement of the target vehicle includes: determining the horizontal and vertical distances and height of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

7. The monocular vision-based three-dimensional vehicle tracking method of claim 5, wherein, Width measurement of the target vehicle includes: determining the horizontal and vertical distances and width of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

8. A monocular vision-based three-dimensional vehicle tracking apparatus, characterized by comprising: include: The image acquisition module is configured to acquire surrounding scene images captured by a monocular camera on the vehicle, the surrounding scene images including at least one target vehicle driving around the vehicle, and the surrounding scene images being two-dimensional images; The image recognition module is configured to input the two-dimensional image into a pre-trained deep learning neural network model, and obtain two-dimensional composite information of the target vehicle in the two-dimensional image from the output of the deep learning neural network model. The inverse perspective transformation module is configured to determine the three-dimensional ranging information of the target vehicle in the vehicle coordinate system based on the parameters of the camera and the two-dimensional composite information of the target vehicle. The feature extraction module is configured to use a pedestrian re-identification algorithm to identify the scene image around the target vehicle and obtain the re-identification features of the target vehicle. The target tracking module is configured to determine the three-dimensional tracking status of the target vehicle based on the re-identification features, two-dimensional composite information, and three-dimensional ranging information of the target vehicle. The inverse perspective transformation module includes a three-dimensional ranging unit, configured to: based on the size of the surrounding scene image and the two-dimensional bounding box of the target vehicle in the surrounding scene image, sequentially determine that the target vehicle is neither truncated to the left nor to the right, and that the number of wheel contact points in the surrounding scene image where the target vehicle is located is more than two; based on the two-dimensional bounding box and the horizontal coordinate of the vertical edge of the target vehicle, and the at least three vehicle contact points of the target vehicle, determine the horizontal and vertical distances, length and width dimensions, and orientation angle of the target vehicle in the vehicle coordinate system through inverse perspective transformation.

9. The three-dimensional vehicle tracking device based on monocular vision according to claim 8, characterized in that, Also includes: The image preprocessing module is configured to perform image scaling, image normalization, image standardization, and image channel transformation on the surrounding scene image sequentially before inputting the two-dimensional image into the pre-trained deep learning neural network model, so as to obtain a preprocessed image of the surrounding scene image.

10. The three-dimensional vehicle tracking device based on monocular vision according to claim 8, characterized in that, Also includes: The model training module is configured to train a deep learning neural network model offline using historical surrounding scene images and annotation results that match the historical surrounding scene images, to obtain a weight file of the deep learning neural network model. The model update module is configured to update the pre-trained deep learning neural network model based on the weight file.

11. The three-dimensional vehicle tracking device based on monocular vision according to any one of claims 8-10, characterized in that, The two-dimensional composite information includes at least one of the following information about the target vehicle: vehicle category, two-dimensional bounding box, visible surface attributes on the image, vertical edges, wheel contact points, and wheel contact point attributes.

12. The three-dimensional vehicle tracking device based on monocular vision according to claim 11, characterized in that, The inverse perspective transformation module also includes: The height ranging unit is configured to measure the height of the target vehicle when it is determined that the target vehicle is truncated to the left and right based on the size of the surrounding scene image and the two-dimensional bounding box of the target vehicle in the surrounding scene image. The width ranging unit is configured to perform width ranging on the target vehicle based on the size of the surrounding scene image and the two-dimensional bounding box of the target vehicle in the surrounding scene image, and sequentially determine that the target vehicle is not truncated to the left or right and is truncated to the bottom. The three-dimensional ranging unit is also configured to determine, based on the size of the surrounding scene image and the two-dimensional bounding box of the target vehicle in the surrounding scene image, that the target vehicle is neither truncated to the left nor to the right, and that the number of wheel contact points in the surrounding scene image of the target vehicle is 2. Then, based on the wheel contact points, the horizontal and vertical distances, length and width dimensions and orientation angle of the target vehicle in the vehicle coordinate system are determined by inverse perspective transformation.

13. The three-dimensional vehicle tracking device based on monocular vision according to claim 12, characterized in that, The height ranging unit is specifically configured to determine the horizontal and vertical distances and height of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

14. The three-dimensional vehicle tracking device based on monocular vision according to claim 12, characterized in that, The width ranging unit is specifically configured to determine the horizontal and vertical distances and width of the target vehicle in the vehicle coordinate system based on the prior length and width dimensions of the target vehicle and the width of the two-dimensional bounding box, using a camera imaging model.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

Images