Target tracking method, device and system based on vehicle-end image, vehicle and medium

Abstract

This invention relates to the field of intelligent driving technology and discloses a target tracking method, device, system, vehicle, and medium based on vehicle-mounted images. The invention extracts several detection frames from a preset number of vehicle-mounted images, performs target detection on each detection frame using a target detection model, calculates the current target's trajectory based on its coordinates in each detection frame, and then calculates the current target's coordinates in all interval frames based on its trajectory. Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in adjacent detection frames, and their respective acquisition times, the size of the bounding box of the current target in the current interval frame is calculated. Finally, according to the current target's coordinates and bounding box size in the current interval frame, the bounding box is projected onto the current interval frame, obtaining the coordinates and bounding boxes of all targets in the preset number of vehicle-mounted images. This effectively reduces the computational load required for target tracking.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a target tracking method, device, system, vehicle, and medium based on vehicle-side images. Background Technology

[0002] Multi-Object Tracking (MOT) technology plays a crucial role in fields such as autonomous driving, intelligent transportation, and security monitoring. Vehicle-mounted image-based MOT refers to the process of capturing scene information such as roads, pedestrians, and vehicles during vehicle operation, tracking multiple target objects (such as pedestrians, vehicles, and bicycles) in real time, and predicting their trajectories. This technology provides essential perception capabilities for autonomous driving, helping vehicles to perform path planning and obstacle avoidance safely and intelligently.

[0003] In autonomous driving systems, visual perception is one of the key technical modules. Vehicles can "see" their surroundings through cameras, identifying and tracking dynamic objects on the road, such as other vehicles, pedestrians, bicycles, and obstacles. Multi-target tracking technology analyzes consecutive frame sequences of images to identify targets and track them consistently, ensuring the monitoring and analysis of all important targets in complex traffic environments.

[0004] With the rapid development of deep learning technology, object detection algorithms based on convolutional neural networks (CNNs) (such as YOLO, SSD, and Faster RCNN) have significantly improved detection accuracy and speed, providing more accurate input for multi-object tracking. Simultaneously, deep learning-based tracking methods are also emerging, utilizing deep neural networks to learn the appearance features and motion patterns of targets, enabling better handling of complex scenes and target interactions.

[0005] However, the mounting of vehicle-mounted cameras on moving vehicles results in significant motion and perspective changes between image frames. Secondly, the scenes in vehicle-mounted images are highly complex, including fast-moving targets, complex backgrounds, lighting variations, and dynamic shadows. Finally, real-time performance is crucial; vehicles traveling at high speeds need to process large amounts of image data in real time for target detection and tracking, and make rapid decisions. Therefore, relying too heavily on pre-trained models, such as using various models to detect and track video frames frame by frame, places a significant burden on the vehicle's hardware computing power, leading to high production costs. Summary of the Invention

[0006] In view of this, the present invention provides a target tracking method, device, system, vehicle and medium based on vehicle-side images to solve the problem that using various models to detect and track video frames one by one consumes a large proportion of the vehicle's hardware computing power, resulting in high production costs.

[0007] In a first aspect, the present invention provides a target tracking method based on vehicle-end images. The method includes: acquiring a preset number of vehicle-end images, and extracting several frames of vehicle-end images from the preset number of vehicle-end images as detection frames, wherein the preset number of vehicle-end images are images continuously acquired within a preset period at a preset frame rate; performing target detection on each detection frame using a pre-trained target detection model to obtain annotation results corresponding to all targets in each detection frame, and calculating the motion trajectory of the current target based on the coordinates of the current target in each detection frame, wherein the annotation results include at least the coordinates corresponding to the target and the bounding box used to annotate the target; and calculating the motion trajectory of the current target. The process involves calculating the coordinates of the current target across all interval frames, where each interval frame is a vehicle-end image of the preset number of frames, excluding the detection frames. Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames, and their respective acquisition times, the size of the bounding box of the current target in the current interval frame is calculated. According to the coordinates of the current target in the current interval frame and the size of the bounding box, the bounding box is projected onto the current interval frame. This process is repeated across all interval frames, detection frames, and all targets to obtain the coordinates and bounding boxes of all targets in the preset number of vehicle-end images. The target tracking result is then obtained based on the coordinates and bounding boxes of the targets.

[0008] The target tracking method based on vehicle-end images provided in this embodiment acquires a preset number of vehicle-end images, extracts several frames from these images as detection frames, performs target detection on each detection frame using a target detection model, obtains the annotation results corresponding to all targets in each detection frame, calculates the motion trajectory of the current target based on its coordinates in each detection frame, and then calculates the coordinates of the current target in all interval frames based on its motion trajectory. Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and their respective acquisition times, the size of the bounding box of the current target in the current interval frame is calculated. Finally, according to the coordinates of the current target in the current interval frame and the size of the bounding box, the bounding box is projected onto the current interval frame to obtain the coordinates and bounding boxes of all targets in the preset number of vehicle-end images. By adopting a target detection method that primarily uses mathematical calculations and secondarily uses a model, the computational load required for target tracking can be effectively reduced, and higher tracking accuracy and real-time tracking performance can be achieved.

[0009] In one optional implementation, calculating the motion trajectory of the current target based on its coordinates in each detection frame includes: obtaining the first coordinates of the current target in an initial detection frame, where the initial detection frame is the frame image where the current target is first detected; obtaining the second coordinates of the current target in a current detection frame, where the current detection frame is any frame other than the initial detection frame among all detection frames; calculating the translation and rotation matrices of the current target in the current detection frame relative to the initial detection frame based on the first and second coordinates; calculating the absolute coordinates of the current target in the current detection frame using the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system based on the translation and rotation matrices of the current target in the current detection frame relative to the initial detection frame and the second coordinates; and calculating the motion trajectory of the current target based on the absolute coordinates of the current target in each detection frame.

[0010] This invention takes into account that the coordinate system corresponding to the target is in motion. By calculating the absolute coordinates of the current target in the same coordinate system in each detection frame, and using the absolute coordinates to calculate the motion trajectory of the current target, the practicality and accuracy of motion trajectory calculation can be improved.

[0011] In one optional implementation, the absolute coordinates are three-dimensional coordinates. The motion trajectory of the current target is calculated based on its absolute coordinates in each detection frame, including: fitting a motion trajectory curve based on the absolute three-dimensional coordinates of the current target in all detection frames, including the second detection frame; determining the predicted three-dimensional coordinates of the current target in all detection frames based on the fitted motion trajectory curve; calculating the weighted loss function value corresponding to each three-dimensional coordinate component based on the predicted and absolute three-dimensional coordinates of the current target in all detection frames; summing the weighted loss function values ​​corresponding to each three-dimensional coordinate component to obtain the sum of loss functions; and adjusting the fitted motion trajectory curve until the sum of loss functions is less than a preset loss function threshold to obtain the adjusted motion trajectory of the current target.

[0012] This invention fits the motion trajectory using the least squares method and adjusts the motion trajectory using a weighted loss function to obtain the optimal motion trajectory of the current target, thereby improving the accuracy of coordinate prediction of the current target in subsequent interval frames.

[0013] In one optional implementation, the coordinates of the current target in the current interval frame are absolute coordinates with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system. Projecting the bounding box into the current interval frame according to the coordinates of the current target in the current interval frame and the size of the bounding box includes: calculating the absolute coordinates of the current target in the current interval frame based on the motion trajectory of the current target; calculating the translation and rotation matrices of the current target in the current interval frame relative to the initial detection frame based on the absolute coordinates of the current target in the current interval frame and the first coordinates of the current target in the initial detection frame; calculating the third coordinates of the current target in the current interval frame using the translation and rotation matrices of the current target in the current interval frame relative to the initial detection frame and the first coordinates of the current target in the initial detection frame; calculating the boundary point coordinates of the bounding box based on the size of the bounding box and the third coordinates, according to the rule that the coordinates of the current target in the current interval frame are the midpoint coordinates of the bounding box; and projecting the bounding box into the current interval frame based on the boundary point coordinates of the bounding box.

[0014] This invention calculates the bounding box size of the current target in the current interval frame based on the bounding box size of the current target in the adjacent detection frames of the current interval frame, and projects the bounding box into the current interval frame based on the coordinates of the current target in the current interval frame, thereby achieving target position prediction.

[0015] In an optional implementation, the annotation results corresponding to all targets in each detection frame further include the boundary point coordinates of the bounding box. The method further includes: calculating the fourth coordinate of the current target in the current detection frame and the size of the first bounding box based on the motion trajectory of the current target; calculating the boundary point coordinates of the first bounding box based on the size of the first bounding box, according to the rule that the fourth coordinate of the current target in the current detection frame is the midpoint coordinate of the bounding box; comparing whether the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than a preset difference threshold; if the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than the preset difference threshold, then adjusting the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model using the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model, to obtain the final boundary point coordinates of the first bounding box in the current detection frame.

[0016] If the bounding box position of the current target in the current detection frame predicted by the motion trajectory does not match the bounding box position detected by the target detection model, the bounding box position can be appropriately corrected to obtain the final bounding box position of the current target in the current detection frame, thus avoiding abrupt changes in the bounding box tracking trajectory.

[0017] In an optional implementation, the method further includes: if the current target is not detected in the current detection frame, then calculating the coordinates of the current target in the current detection frame based on the motion trajectory of the current target; calculating the size of the bounding box of the current target in the current detection frame based on the acquisition time corresponding to the current detection frame, the bounding box size of the current target in the adjacent detection frames, and the acquisition time corresponding to each frame; and projecting the bounding box onto the current detection frame according to the coordinates of the current target in the current detection frame and the size of the bounding box, until the duration during which the current target is not detected exceeds a preset duration threshold.

[0018] Even if the current target is not detected in the vehicle-side image of the current detection frame, the present invention still predicts the coordinates of the current target in the current detection frame based on the motion trajectory of the current target, and then draws a bounding box to ensure real-time tracking of the current target.

[0019] In an optional implementation, the annotation results corresponding to all targets in each detection frame further include the target depth and the target category. The method further includes: if the pre-trained target detection model does not detect the depth of the current target, obtaining the actual feature size and bounding box size of the current target based on the detected current target category; and calculating the depth of the current target based on the actual feature size of the current target, the bounding box size, and the shooting parameters of the image acquisition device for acquiring vehicle-side images.

[0020] If the present invention determines that the target detection model has not detected the depth of the current target, it can obtain the actual size of the current target based on the sub-category of the current target, and calculate the depth of the current target based on the bounding box size and camera focal length of the current target in the detection frame, thereby ensuring the integrity of the target information in the detection frame.

[0021] In an optional implementation, the annotation results corresponding to all targets in each detection frame further include the identifiers of the targets. The method further includes: if no identifier information is detected for the current target in the current detection frame, then obtaining all targets and their corresponding bounding boxes in adjacent detection frames; normalizing the bounding box size corresponding to the current target in the current detection frame and the bounding box sizes corresponding to all targets in adjacent detection frames to the same size; calculating the similarity between the image in the bounding box corresponding to the current target and the image in the bounding box of the same size corresponding to all targets in adjacent detection frames; and marking the identifier of the target corresponding to the adjacent detection frame with the highest image similarity and greater than a preset similarity threshold as the identifier of the current target in the current detection frame.

[0022] After no identifier information is detected for the current target, this invention calculates the similarity between the images in each bounding box of adjacent detection frames and the images in the bounding box of the current target. The identifier of the corresponding target in the adjacent detection frames with the highest similarity and greater than a preset similarity threshold is marked as the identifier of the current target in the current detection frame, thereby improving the accuracy of subsequent motion trajectory calculation based on the coordinates of the same target in all detection frames.

[0023] Secondly, the present invention provides a target tracking device based on vehicle-end images. The device includes: a detection frame extraction module, used to acquire a preset number of vehicle-end images and extract several vehicle-end images as detection frames from the preset number of vehicle-end images, wherein the preset number of vehicle-end images are images continuously acquired at a preset frame rate within a preset period; a target detection module, used to perform target detection on each detection frame using a pre-trained target detection model, obtain annotation results corresponding to all targets in each detection frame, and calculate the motion trajectory of the current target based on the coordinates of the current target in each detection frame, wherein the annotation results include at least the coordinates corresponding to the target and the bounding box used to annotate the target; and a target coordinate calculation module, used to calculate the target coordinates based on the coordinates of the current target in each detection frame. The system calculates the target's motion trajectory and coordinates across all interval frames, where each interval frame is a vehicle-end image excluding the detection frames within a preset number of frame-level images. A bounding box determination module calculates the size of the bounding box of the current target in the current interval frame based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames, and their respective acquisition times. A target tracking module projects the bounding box onto the current interval frame according to the target's coordinates and bounding box size, traversing all interval frames, detection frames, and all targets to obtain the coordinates and bounding boxes of all targets in the preset number of frame-level images, and obtains the target tracking result based on the target's coordinates and bounding boxes.

[0024] Thirdly, the present invention provides a target tracking system based on vehicle-end images. The system includes a controller, which includes a memory and a processor. The memory and the processor are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the target tracking method based on vehicle-end images described in the first aspect or any corresponding embodiment.

[0025] Fourthly, the present invention provides a vehicle comprising the target tracking system based on vehicle-side images described in the third aspect above.

[0026] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the target tracking method based on vehicle-end images according to the first aspect or any corresponding embodiment thereof.

[0027] The target tracking method based on vehicle-end images provided by this invention acquires a preset number of vehicle-end images, extracts several frames from these images as detection frames, performs target detection on each detection frame using a target detection model, obtains the annotation results corresponding to all targets in each detection frame, calculates the motion trajectory of the current target based on its coordinates in each detection frame, calculates the coordinates of the current target in all interval frames based on its motion trajectory, calculates the size of the bounding box of the current target in the current interval frame based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the acquisition time of each frame, and finally projects the bounding box onto the current interval frame according to the coordinates of the current target in the current interval frame and the size of the bounding box, to obtain the coordinates and bounding boxes of all targets in the preset number of vehicle-end images. By adopting a target detection method that primarily uses mathematical calculations and secondarily uses a model, the computational load required for target tracking can be effectively reduced, and higher tracking accuracy and real-time tracking can be achieved. Attached Figure Description

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating a target tracking method based on vehicle-end images according to an embodiment of the present invention;

[0030] Figure 2a This is a schematic diagram of the VCS coordinate system of a vehicle according to an embodiment of the present invention;

[0031] Figure 2b This is a schematic diagram of the camera coordinate system of a vehicle according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of a boundary box according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the current target tracking status in a video frame according to an embodiment of the present invention;

[0034] Figure 5 This is a flowchart illustrating another target tracking method based on vehicle-end images according to an embodiment of the present invention;

[0035] Figure 6 This is a flowchart illustrating a specific example of a target tracking method based on vehicle-side images according to an embodiment of the present invention;

[0036] Figure 7 This is a structural block diagram of a target tracking device based on vehicle-end images according to an embodiment of the present invention;

[0037] Figure 8 This is a structural block diagram of a target tracking system based on vehicle-end images according to an embodiment of the present invention;

[0038] Figure 9 This is a structural block diagram of a vehicle according to an embodiment of the present invention;

[0039] Figure 10 This is a schematic diagram of the hardware structure of the controller according to an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Early multi-object tracking methods typically relied on a detection-by-detection framework. This approach first identifies potential targets in an image using an object detector, and then uses filters (such as Kalman filters) or data association algorithms (such as the Hungarian algorithm) to associate these detected targets across consecutive frames, thereby achieving tracking. However, these methods often perform poorly when handling complex scenes such as occlusion, target deformation, lighting variations, and multi-object interactions.

[0042] In recent years, with the rapid development of deep learning technology, object detection algorithms based on convolutional neural networks (CNNs) (such as YOLO, SSD, and Faster RCNN) have significantly improved detection accuracy and speed, providing more accurate input for multi-object tracking. At the same time, deep learning-based tracking methods have also gradually emerged, utilizing deep neural networks to learn the appearance features and motion patterns of targets, enabling better handling of complex scenes and target interactions.

[0043] Multi-target tracking in vehicle-mounted images presents unique challenges compared to traditional video surveillance. First, vehicle-mounted cameras are installed on moving vehicles, resulting in significant motion and viewpoint changes between image frames. Second, vehicle-mounted images exhibit high scene complexity, including fast-moving targets, complex backgrounds, lighting variations, and dynamic shadows. Finally, high real-time performance is required; vehicles traveling at high speeds need to process massive amounts of image data in real time for target detection and tracking, and make rapid decisions. Therefore, over-reliance on pre-trained models, such as using various models to perform frame-by-frame detection and tracking of video footage, places a significant burden on the vehicle's hardware computing power, leading to high production costs.

[0044] According to an embodiment of the present invention, a target tracking method based on vehicle-end images is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] This embodiment provides a target tracking method based on vehicle-side images, which can be used in the aforementioned controller. Figure 1 This is a flowchart of a target tracking method based on vehicle-side images according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0046] Step S101: Obtain a preset number of vehicle-end images, and extract several vehicle-end images from the preset number of vehicle-end images as detection frames.

[0047] Among them, the vehicle-side images with a preset number of frames are images continuously acquired at a preset frame rate within a preset period.

[0048] In this invention, the commonly used vehicle-side detection targets are mainly divided into dynamic targets and static targets. Static targets include obstacles, traffic signs, and traffic cones, etc., which are relatively simple to track, mainly based on the vehicle's movement trajectory. Dynamic targets include vehicles, pedestrians, and other dynamic targets. By detecting and calculating the coordinate information of various targets during the vehicle's movement, the coordinates are reflected in the bounding box information on the screen. This is just an example. The preset number of vehicle-side images can be vehicle-side images continuously acquired at a preset frame rate within a preset period, or they can be road videos acquired during the vehicle's movement. For videos, it is often necessary to detect them frame by frame. For example, a frame rate of 40 frames per second means that 40 vehicle-side images need to be detected per second. The way to acquire the preset number of vehicle-side images can be through sensors installed on the vehicle, such as cameras or video acquisition devices, or through image acquisition devices such as cameras on the road. This is just an example.

[0049] After acquiring a preset number of vehicle-end images, this embodiment of the invention can extract several vehicle-end images from the preset number of vehicle-end images as detection frames. The method of extracting detection frames is not limited; it can be done by randomly extracting several vehicle-end images from the preset number of vehicle-end images, or by extracting several vehicle-end images from the preset number of vehicle-end images based on a preset frame interval. Generally, to ensure the accuracy of the motion trajectory, at least three vehicle-end images can be extracted as detection frames. Furthermore, the setting of the frame interval is not limited; it can be set based on factors such as the accuracy of subsequent target motion trajectory calculation or reducing the cost of model detection. For example, the frame interval can be set to 10 frames. Furthermore, the first frame of the camera video can be set as the starting frame, and one frame can be extracted every 10 frames thereafter as a detection frame, thus extracting 4 vehicle-end images as detection frames. This is merely an example.

[0050] Step S102: Use the pre-trained target detection model to perform target detection on each detection frame, obtain the annotation results corresponding to all targets in each detection frame, and calculate the motion trajectory of the current target based on the coordinates of the current target in each detection frame.

[0051] The annotation results should include at least the coordinates of the target and the bounding box used to annotate the target.

[0052] This invention can utilize a pre-trained multi-task object detection model to perform object detection on each detection frame, obtaining annotation results for all targets in each detection frame. The object results may include the coordinates of the target in each detection frame and the bounding box (BBOX) used to annotate the target. Commonly used coordinates in intelligent driving systems are as follows: Figure 2a The vehicle coordinate system (VCS) shown is as follows: Figure 2bThe camera coordinate system shown is replaced by the vehicle coordinate system in subsequent embodiments for target tracking. The shape of the bounding box (BBOX) is not limited. Figure 3 As shown, for example, a rectangle can be used to enclose the target in an image, with its width and height representing the target's maximum width and maximum height, respectively. This is just an example.

[0053] This invention does not limit the method for calculating the motion trajectory of the current target based on its coordinates in each detection frame. Mathematical methods can be used to estimate intermediate positions from known coordinate points, forming a smooth curve, i.e., the motion trajectory of the current target. Alternatively, the `plot` function in MATLAB can be used to plot the motion trajectory of the current target. The horizontal axis of the current target's motion trajectory can be the acquisition time corresponding to the frame image, and the vertical axis can be the coordinates of the current target. Figure 4 As shown, the dashed curve represents the trajectory of the current target, and t0, t1, t2, and t3 represent the time corresponding to each detection frame. The target detection model can be used to detect the bounding box of the current target in each detection frame. This is just an example.

[0054] Step S103: Based on the current target's motion trajectory, calculate the coordinates of the current target in all interval frames.

[0055] Among them, the vehicle-end images in the vehicle-end images with a preset number of interval frames are the vehicle-end images other than the detection frames.

[0056] In this embodiment of the invention, all frames of vehicle-end images except the detection frame can be identified as interval frames. Then, based on the acquisition time corresponding to the interval frame, the coordinates of the current target in all interval frames can be calculated from the motion trajectory of the current target. This is just an example.

[0057] Step S104: Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the acquisition time corresponding to each frame, calculate the size of the bounding box of the current target in the current interval frame.

[0058] This invention can calculate the size of the bounding box of the current target in the current interval frame based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the corresponding acquisition times, using an interpolation algorithm. Generally, the bounding box sizes in the current interval frame are similar to those in the closer detection frames, such as... Figure 4 As shown, t2.1, t2.2, and t2.3 are the acquisition times corresponding to each interval frame. The bounding box size of the current target in all interval frames from t2 to t3 is calculated by using the bounding box size in the detection frames corresponding to times t2 and t3. This is just an example.

[0059] Step S105: Based on the coordinates of the current target in the current interval frame and the size of the bounding box, project the bounding box into the current interval frame, traverse all interval frames, detection frames and all targets, obtain the coordinates and bounding boxes of all targets in the vehicle-end image of the preset number of frames, and obtain the target tracking result based on the coordinates and bounding boxes of the targets.

[0060] This invention can utilize the coordinates of the current target in the current interval frame and the size of the bounding box to project the bounding box into the current interval frame. For example, the coordinates of the current target in the current interval frame can be used as the coordinates of the bounding box vertices. Then, the bounding box is drawn according to the bounding box size. Finally, all interval frames and detection frames are traversed to obtain the following result: Figure 4 The image shows the coordinates and bounding box of the current target in all vehicle-side images, from which the target tracking result is obtained. This is used as an example to enable autonomous driving on the vehicle side.

[0061] The target tracking method based on vehicle-end images provided in this embodiment acquires a preset number of vehicle-end images, extracts several frames from these images as detection frames, performs target detection on each detection frame using a target detection model, obtains the annotation results corresponding to all targets in each detection frame, calculates the motion trajectory of the current target based on its coordinates in each detection frame, and then calculates the coordinates of the current target in all interval frames based on its motion trajectory. Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and their respective acquisition times, the size of the bounding box of the current target in the current interval frame is calculated. Finally, according to the coordinates of the current target in the current interval frame and the size of the bounding box, the bounding box is projected onto the current interval frame to obtain the coordinates and bounding boxes of all targets in the preset number of vehicle-end images. By adopting a target detection method that primarily uses mathematical calculations and secondarily uses a model, the computational load required for target tracking can be effectively reduced, and higher tracking accuracy and real-time tracking performance can be achieved.

[0062] This embodiment provides a target tracking method based on vehicle-side images, which can be used in the aforementioned controller. Figure 5 This is a flowchart of a target tracking method based on vehicle-side images according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0063] Step S501: Acquire a preset number of vehicle-end images, and extract several frames of vehicle-end images from the preset number of vehicle-end images as detection frames. The preset number of vehicle-end images are images continuously acquired within a preset period at a preset frame rate. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0064] Step S502: Target detection is performed on each detection frame using a pre-trained target detection model to obtain the annotation results for all targets in each detection frame. Based on the coordinates of the current target in each detection frame, the motion trajectory of the current target is calculated. The annotation results include at least the coordinates of the target and the bounding box used to annotate the target.

[0065] In one optional implementation, the annotation results corresponding to all targets in each detection frame also include the target depth and target category. If the pre-trained target detection model does not detect the depth of the current target, the actual feature size and bounding box size of the current target are obtained based on the detected current target category. The depth of the current target is calculated based on the actual feature size of the current target, the bounding box size, and the shooting parameters of the image acquisition device for acquiring vehicle-side images.

[0066] The target detection model used in this embodiment of the invention can be used to identify common target categories in the field of intelligent driving, including vehicles, pedestrians, signs and traffic lights, etc. It also supports the subdivision of the major categories. For example, vehicles can be divided into SUVs, sedans, tricycles, etc. The target detection model performs target detection on each detection frame and records the target information detected therein, which mainly includes target major category, sub-category, identifier (track_id), distance (or depth) and direction information, etc., which are only examples.

[0067] After the object detection model has completed detection of all frames, if it is determined that the object detection model has not detected the depth of the current target, it can first be determined whether there is one or more cameras in the vehicle that are capturing images. If there are multiple cameras, the depth of the current target can be estimated using the binocular ranging principle based on the bounding box positions of the current target in different calibrated cameras. This is just an example. If the vehicle is capturing images using only a single camera, since the actual sizes of targets in the same sub-category are relatively similar, the sub-category of the current target can be obtained, and the actual feature size w of the current sub-category can be determined, including length, width, and height. Then, the bounding box size p of the current target in the detection frame can be obtained, and the depth of the current target can be calculated using the following formula:

[0068]

[0069] Where d represents the depth of the current target, and f represents the shooting parameters of the image acquisition device, such as the camera focal length.

[0070] In this embodiment of the invention, the actual feature size and bounding box size features can be selected according to actual needs. For example, if the vehicle in this sub-category has a small error in the height direction, the feature can be the height in the actual feature size and the height in the bounding box size. Alternatively, multiple feature sizes can be used to calculate the depth separately, and finally, a weighted summation method can be used to calculate the depth of the current target.

[0071] If the present invention determines that the target detection model has not detected the depth of the current target, it can obtain the actual size of the current target based on the sub-category of the current target, and calculate the depth of the current target based on the bounding box size and camera focal length of the current target in the detection frame, thereby ensuring the integrity of the target information in the detection frame.

[0072] In one optional implementation, the annotation results corresponding to all targets in each detection frame also include the identifier of the target. If no identifier information is detected for the current target in the current detection frame, all targets and their corresponding bounding boxes in adjacent detection frames are obtained; the size of the bounding box corresponding to the current target in the current detection frame and the size of the bounding boxes corresponding to all targets in adjacent detection frames are normalized to the same size; the similarity between the image in the bounding box corresponding to the current target and the image in the bounding box of the same size corresponding to all targets in adjacent detection frames is calculated; the identifier of the target corresponding to the adjacent detection frame with the highest image similarity and greater than a preset similarity threshold is marked as the identifier of the current target in the current detection frame.

[0073] In this embodiment of the invention, the identifier track_id is a unique identifier for the target, which can identify the same target in multiple detection frames. If the identifier information of the current target is not detected in the current detection frame, all targets and their corresponding bounding boxes in adjacent detection frames can be obtained. The bounding box size of the current target in the current detection frame and the bounding box sizes of all targets in adjacent detection frames can be normalized to the same size to improve the accuracy of similarity calculation. Then, the similarity between the image in the bounding box corresponding to the current target and the image in the bounding box of the same size corresponding to all targets in adjacent detection frames can be calculated. The identifier of the target in the adjacent detection frame with the highest image similarity and greater than the preset similarity threshold is marked as the identifier of the current target in the current detection frame.

[0074] After no identifier information is detected for the current target, this invention calculates the similarity between the images in each bounding box of adjacent detection frames and the images in the bounding box of the current target. The identifier of the corresponding target in the adjacent detection frames with the highest similarity and greater than a preset similarity threshold is marked as the identifier of the current target in the current detection frame, thereby improving the accuracy of subsequent motion trajectory calculation based on the coordinates of the same target in all detection frames.

[0075] Specifically, the first coordinate of the current target in the initial detection frame is obtained, where the initial detection frame is the frame image where the current target is first detected; the second coordinate of the current target in the current detection frame is obtained, where the current detection frame is any frame other than the initial detection frame among all detection frames; based on the first and second coordinates, the translation and rotation matrices of the current target in the current detection frame compared to the initial detection frame are calculated; based on the translation vector, rotation matrix, and second coordinate of the current target in the current detection frame compared to the initial detection frame, the absolute coordinates of the current target in the current detection frame are calculated, with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system; based on the absolute coordinates of the current target in each detection frame, the motion trajectory of the current target is calculated.

[0076] In this embodiment of the invention, since the vehicle is moving, the origin of the corresponding VCS coordinate system is also moving. Therefore, it is necessary to determine the absolute coordinates of the target in each frame to calculate the target's trajectory. The detection frame in which the current target is first detected can be used as the initial detection frame (wherein, the initial detection frame and the above-mentioned starting frame may be the same frame or different frames). The VCS coordinate system of the current target in the initial detection frame can be used as the absolute coordinate system until the current target disappears. The absolute coordinates of the current target in subsequent frames are all represented by this coordinate system.

[0077] For example, if the initial detection frame is the starting frame, the coordinates of the current target in the VCS coordinate system (absolute coordinate system) at time t0 (initial detection frame) are P0 = (x0, y0, z0), and the coordinates of the current target in the VCS coordinate system (absolute coordinate system) at time t0 are obtained. i The coordinates in the VCS coordinate system of the current detection frame corresponding to the time point are P. i =(x i ,y i ,z i The current target is between t0 and t1. i The trajectory S can be represented by the translation vector T(t) i ) and rotation matrix R(t i This indicates that the target value at time t corresponds to the current target value. i The absolute coordinate of time is P ωi =R(t) i )·P i +T(t i This allows us to obtain the absolute coordinates of the current target in each detection frame under the same coordinate system, and to calculate the motion trajectory of the current target based on the absolute coordinates of the current target in each detection frame.

[0078] This invention takes into account that the coordinate system corresponding to the target is in motion. By calculating the absolute coordinates of the current target in the same coordinate system in each detection frame, and using the absolute coordinates to calculate the motion trajectory of the current target, the practicality and accuracy of motion trajectory calculation can be improved.

[0079] Specifically, the absolute coordinates are three-dimensional coordinates, and step S502 above includes:

[0080] Step S5021: Based on the absolute three-dimensional coordinates of the current target in all detection frames including the second detection frame, fit a motion trajectory curve, and based on the fitted motion trajectory curve, determine the predicted three-dimensional coordinates of the current target in all detection frames.

[0081] Step S5022: Based on the predicted and absolute 3D coordinates of the current target in all detection frames, calculate the weighted loss function value corresponding to each 3D coordinate component.

[0082] Step S5023: Sum the weighted loss function values ​​corresponding to each three-dimensional coordinate component to obtain the sum of the loss functions.

[0083] Step S5024: Adjust the fitted motion trajectory curve until the sum of the loss functions is less than the preset loss function threshold to obtain the adjusted motion trajectory of the current target.

[0084] This invention does not limit the method of calculating the motion trajectory of the current target using its coordinates in each detection frame. A nonlinear regression function can be used to fit the motion trajectory. Due to the uncertainty of the current target's motion, the trajectory should be significantly influenced by the nearest coordinate position during the trajectory fitting process. Earlier positions have less influence on the trajectory equation. Therefore, a weighted least squares polynomial fitting method can be used to reduce the impact of earlier positions on the fitting effect, allowing for more accurate prediction of the current target's coordinates in interval frames. In vehicle operation, the object's motion is relatively simple, and the accuracy requirement for fitting is not high. Furthermore, since weighted least squares fitting is used, older coordinates have little impact on the fitting effect. Considering the high computational cost of higher-order polynomials, a 3rd-4th order polynomial can be chosen to cover most fitting requirements.

[0085] This invention tracks each target appearing in a frame individually. Starting from the second detection frame after a target appears, motion trajectory fitting is performed. First, component matrices, loss weight matrices, and target vectors corresponding to each three-dimensional coordinate component are constructed. For the x-component, the constructed component matrix is ​​as follows:

[0086]

[0087] Among them, t k Let m represent the detection frame at time k, m represent the number of detection frames, and n represent the polynomial degree.

[0088] The loss weight matrix is:

[0089]

[0090] Where, ω k It is the weight corresponding to the coordinate point of the current target in the k-th detection frame; coefficient component a x =(X T WX) - 1 X T The same method is applied to the y and z components of Wx, and will not be repeated here.

[0091] For each of the three-dimensional coordinate components, we define an nth-degree polynomial for fitting. Let the polynomials be:

[0092]

[0093]

[0094] Among them, a ix a iy a iz These are the polynomial coefficients corresponding to the coordinate components; i represents the i-th degree polynomial; t i This represents the parameter variable in a polynomial.

[0095] This invention embodiment is based on the polynomial fitting of the motion trajectory curve described above, and based on the fitted motion trajectory curve, determines the predicted three-dimensional coordinates of the current target in all detection frames. Then, based on the predicted and absolute three-dimensional coordinates of the current target in all detection frames, it calculates the weighted loss function value corresponding to each three-dimensional coordinate component. The weighted loss function for a single coordinate component is defined as follows:

[0096]

[0097] Among them, L x (a x ) represents the weighted loss function value of the x component; a x =[a x0 ,a x1 ,a x2 ,…,a xn ] represents the coefficient vector to be optimized, a y and a z Similar; f x (t k () represents the predicted coordinates of the x-component, x k Represents the actual coordinates of the x-component; ω k The weight represents the coordinate point of the current target in the k-th detection frame, reflecting the influence of the coordinate point of the current target in the k-th detection frame on the fitting process, and is defined as:

[0098] ω k =e -α(k-2)

[0099] Here, α>0 is a decay factor that controls the rate of decrease of the weight, indicating that the loss weight corresponding to different detection frames is negatively correlated with the acquisition order of the detection frames. That is, the earlier the target point of the detection frame is acquired, the smaller the corresponding weight.

[0100] To fit the three coordinate components, the total loss function L(a) can be defined as the sum of the loss functions for each component:

[0101] L(a)=L x (a x )+L y (a y )+L z (a z )

[0102] The embodiments of the present invention can adjust the fitted motion trajectory curve until the sum of the loss functions is less than a preset loss function threshold, thereby obtaining the adjusted motion trajectory of the current target.

[0103] This invention fits the motion trajectory using the least squares method and adjusts the motion trajectory using a weighted loss function to obtain the optimal motion trajectory of the current target, thereby improving the accuracy of coordinate prediction of the current target in subsequent interval frames.

[0104] Step S503: Based on the current target's motion trajectory, calculate the coordinates of the current target across all interval frames. The interval frames are the vehicle-side images within a preset number of frames, excluding the detection frames. For details, please refer to [link to details]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0105] Step S504: Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and their respective acquisition times, calculate the size of the bounding box of the current target in the current interval frame. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0106] Step S505: Based on the coordinates of the current target in the current interval frame and the size of the bounding box, project the bounding box into the current interval frame, traverse all interval frames, detection frames and all targets, obtain the coordinates and bounding boxes of all targets in the vehicle-end image of the preset number of frames, and obtain the target tracking result based on the coordinates and bounding boxes of the targets.

[0107] Specifically, the coordinates of the current target in the current interval frame are absolute coordinates with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system. Step S505 includes:

[0108] Step S5051: Calculate the absolute coordinates of the current target in the current interval frame based on the motion trajectory of the current target, and calculate the translation matrix and rotation matrix of the current target in the current interval frame compared with the initial detection frame based on the absolute coordinates of the current target in the current interval frame and the first coordinate of the current target in the initial detection frame.

[0109] Step S5052: Calculate the third coordinate of the current target in the current interval frame using the translation and rotation matrices of the current target in the current interval frame compared to the initial detection frame, and the first coordinate of the current target in the initial detection frame.

[0110] Step S5053: According to the rule that the coordinates of the current target in the current interval frame are the coordinates of the midpoint of the bounding box, calculate the coordinates of the boundary points of the bounding box based on the size of the bounding box and the third coordinate.

[0111] Step S5054: Based on the coordinates of the boundary points of the bounding box, project the bounding box into the current interval frame.

[0112] This invention utilizes a fitted motion trajectory curve to calculate the absolute coordinates of the current target in the current interval frame. Then, based on the absolute coordinates of the current interval frame and the vehicle's motion trajectory from the initial detection frame to the current interval frame, its coordinates in the VCS coordinate system of the current interval frame can be inversely solved. Since the motion trajectory of the current target is known, the absolute coordinates of the current target in the current interval frame are calculated. Finally, based on the absolute coordinates of the current target in the current interval frame and the first coordinate of the current target in the initial detection frame, the translation matrix R of the current target in the current interval frame relative to the initial detection frame is calculated. 0→j and rotation matrix t 0→j , where j represents the interval frame corresponding to time j:

[0113]

[0114] The position vector of the current target in the initial detection frame is P0 = [X0, Y0, Z0, 1]. T That is, the position vector P in the VCS coordinate system of the current interval frame. j for:

[0115]

[0116] Based on the rule that the coordinates of the current target in the current interval frame are the midpoint coordinates of the bounding box, this embodiment of the invention can further determine the bounding box size and P based on the rule that the current target's coordinates in the current interval frame are the midpoint coordinates of the bounding box. jThe bounding box coordinates are calculated. The shape of the bounding box can be a rectangle, a circle, or an irregular shape. The corresponding boundary points can be the four vertices, the boundary points corresponding to the diameter of the circle, or randomly selected boundary points. Generally, the bounding box is a rectangle. The coordinates of the four vertices of the bounding box are calculated. Finally, based on the coordinates of the four vertices of the bounding box, the bounding box is projected onto the image of the current interval frame using the camera extrinsic and intrinsic parameter matrices.

[0117] This invention calculates the bounding box size of the current target in the current interval frame based on the bounding box size of the current target in the adjacent detection frames of the current interval frame, and projects the bounding box into the current interval frame based on the coordinates of the current target in the current interval frame, thereby achieving target position prediction.

[0118] In one optional implementation, the annotation results corresponding to all targets in each detection frame further include the boundary point coordinates of the bounding box. Based on the motion trajectory of the current target, the fourth coordinate of the current target in the current detection frame and the size of the first bounding box are calculated. According to the rule that the fourth coordinate of the current target in the current detection frame is the midpoint coordinate of the bounding box, the boundary point coordinates of the first bounding box are calculated based on the size of the first bounding box. The difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is compared to see if it is greater than a preset difference threshold. If the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than the preset difference threshold, the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model are adjusted using the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model to obtain the final boundary point coordinates of the first bounding box in the current detection frame.

[0119] In addition to calculating the coordinates and bounding boxes of the target in the interval frame using the above-described mathematical methods, this embodiment of the invention can also calculate the coordinates of the current target in the current detection frame using the motion trajectory curve of the current target when the next detection frame appears according to the acquisition time sequence. It also calculates the size of the predicted bounding box in the current detection frame based on the bounding box size of the current target in adjacent detection frames. Finally, it calculates the boundary point coordinates of the predicted bounding box according to the rule that the coordinates of the current target in the current detection frame are the midpoint coordinates of the bounding box. Then, it compares whether the difference between the boundary point coordinates of the predicted bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the target detection model is greater than a preset difference threshold. If the difference is greater than the preset difference threshold, the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame can be adjusted using the difference between the boundary point coordinates of the predicted bounding box and the corresponding bounding box of the current target in the current detection frame detected by the target detection model, thus obtaining the final bounding box position of the current target in the current detection frame. This is just an example; interpolation or Kalman filtering methods can also be used to correct the bounding box and obtain the final bounding box position of the current target in the current detection frame.

[0120] If the bounding box position of the current target in the current detection frame predicted by the motion trajectory does not match the bounding box position detected by the target detection model, the bounding box position can be appropriately corrected to obtain the final bounding box position of the current target in the current detection frame, thus avoiding abrupt changes in the bounding box tracking trajectory.

[0121] In one optional implementation, if the current target is not detected in the current detection frame, the coordinates of the current target in the current detection frame are still calculated based on the motion trajectory of the current target; the size of the bounding box of the current target in the current detection frame is calculated based on the acquisition time corresponding to the current detection frame, the bounding box size of the current target in the adjacent detection frames, and the acquisition time corresponding to each frame; the bounding box is projected into the current detection frame according to the coordinates of the current target in the current detection frame and the size of the bounding box, until the duration during which the current target is not detected exceeds a preset duration threshold.

[0122] In this embodiment of the invention, if the current target is not detected in the current detection frame, where not detecting the current target could mean that the current target is not detected in the current frame (e.g., the current target is lost, obscured, or exceeds the range of the vehicle's image), the motion trajectory of the current target is still used to calculate the coordinates of the current target in the current detection frame. Based on the acquisition time corresponding to the current detection frame, the bounding box size of the current target in the adjacent detection frames, and their respective acquisition times, the size of the bounding box of the current target in the current detection frame is calculated. According to the coordinates of the current target in the current detection frame and the size of the bounding box, the bounding box is projected onto the current detection frame. The drawing of the bounding box of the current target stops after a time delay set by the system. The time delay set by the system, i.e., the duration during which the current target is not detected, can be set according to factors such as the vehicle's driving status and road conditions.

[0123] Even if the current target is not detected in the vehicle-side image of the current detection frame, the present invention still predicts the coordinates of the current target in the current detection frame based on the motion trajectory of the current target, and then draws a bounding box to ensure real-time tracking of the current target.

[0124] In specific embodiments, such as Figure 6As shown, a preset number of vehicle-side images are acquired, the starting frame and the interval frame number are determined, and the next frame after each interval frame number is determined as the detection frame, resulting in several detection frames. A multi-task object detection model is then used to detect the starting frame and the detection frames, obtaining the annotation results for all targets in all detection frames. If the object detection model cannot identify the depth of the current target, the depth of the current target is calculated based on the actual feature size of the current target, the size of the bounding box, and the shooting parameters of the image acquisition device acquiring the vehicle-side images. The coordinates of the current target are obtained from all detection frames, and the motion trajectory curve of the current target is calculated based on the coordinates of the current target in all detection frames. The motion trajectory curve of the current target can be calculated by fitting the motion trajectory of the current target using the least squares method and a loss function, and then based on the current target's motion... The method predicts the coordinates of the current target across all interval frames based on its motion trajectory. Then, based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in adjacent detection frames, and their respective acquisition times, the size of the bounding box of the current target in the current interval frame is calculated. The bounding box is then projected onto the current interval frame according to the target's coordinates and the bounding box size. This process continues until the next detection frame. In the next detection frame, the target's motion trajectory can also be used to calculate its coordinates. If the target cannot be detected in the current detection frame, the bounding box is still drawn based on the target's coordinates and bounding box size. Tracking stops when a time threshold is reached. This method can be applied to pure vision-based intelligent driving systems, has less reliance on other perception systems, and can be widely adopted. Furthermore, the mathematical scheme can effectively reduce the computational load required for tracking and achieve high tracking accuracy, improving real-time performance. For detailed explanations, please refer to the above embodiments, which will not be repeated here.

[0125] This embodiment also provides a target tracking device based on vehicle-end images, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0126] This embodiment provides a target tracking device based on vehicle-side images, such as... Figure 7As shown, it includes: a detection frame extraction module 701, used to acquire a preset number of vehicle-end images and extract several vehicle-end images from the preset number of vehicle-end images as detection frames, wherein the preset number of vehicle-end images are images continuously acquired within a preset period at a preset frame rate; a target detection module 702, used to perform target detection on each detection frame using a pre-trained target detection model, obtain the annotation results corresponding to all targets in each detection frame, and calculate the motion trajectory of the current target based on the coordinates of the current target in each detection frame, wherein the annotation results include at least the coordinates of the target and the bounding box used to annotate the target; and a target coordinate calculation module 703, used to calculate the current target coordinates based on the motion trajectory of the current target. The coordinates of all interval frames are used to determine the vehicle-end images in the vehicle-end images of the preset number of interval frames, excluding the detection frames. The bounding box determination module 704 is used to calculate the size of the bounding box of the current target in the current interval frame based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the acquisition time of each frame. The target tracking module 705 is used to project the bounding box into the current interval frame according to the coordinates of the current target in the current interval frame and the size of the bounding box, traverse all interval frames, detection frames, and all targets to obtain the coordinates and bounding boxes of all targets in the vehicle-end images of the preset number of frames, and obtain the target tracking result based on the coordinates and bounding boxes of the targets.

[0127] In some optional implementations, the target detection module 702 includes: a first coordinate acquisition unit, used to acquire the first coordinates of the current target in an initial detection frame, the initial detection frame being the frame image where the current target is first detected; a second coordinate acquisition unit, used to acquire the second coordinates of the current target in a current detection frame, the current detection frame being any frame other than the initial detection frame among all detection frames; a matrix calculation unit, used to calculate the translation matrix and rotation matrix of the current target in the current detection frame relative to the initial detection frame based on the first and second coordinates; an absolute coordinate calculation unit, used to calculate the absolute coordinates of the current target in the current detection frame, with the vehicle coordinate system in the initial detection frame as the absolute coordinate system, based on the translation vector, rotation matrix, and second coordinates of the current target in the current detection frame relative to the initial detection frame; and a motion trajectory calculation unit, used to calculate the motion trajectory of the current target based on the absolute coordinates of the current target in each detection frame.

[0128] In some optional implementations, the motion trajectory calculation unit includes: a trajectory curve fitting subunit, used to fit a motion trajectory curve based on the absolute three-dimensional coordinates of the current target in all detection frames including the second detection frame, and to determine the predicted three-dimensional coordinates of the current target in all detection frames based on the fitted motion trajectory curve; a loss function calculation unit, used to calculate the weighted loss function value corresponding to each three-dimensional coordinate component based on the predicted three-dimensional coordinates and absolute three-dimensional coordinates of the current target in all detection frames; to sum the weighted loss function values ​​corresponding to each three-dimensional coordinate component to obtain the sum of loss functions; and a trajectory adjustment unit, used to adjust the fitted motion trajectory curve until the sum of loss functions is less than a preset loss function threshold to obtain the adjusted motion trajectory of the current target.

[0129] In some optional implementations, the coordinates of the current target in the current interval frame are absolute coordinates with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system. The target tracking module 705 includes: a matrix calculation unit, used to calculate the absolute coordinates of the current target in the current interval frame based on the motion trajectory of the current target, and to calculate the translation matrix and rotation matrix of the current target in the current interval frame relative to the initial detection frame based on the absolute coordinates of the current target in the current interval frame and the first coordinate of the current target in the initial detection frame; a third coordinate calculation unit, used to calculate the third coordinates of the current target in the current interval frame using the translation matrix and rotation matrix of the current target in the current interval frame relative to the initial detection frame and the first coordinate of the current target in the initial detection frame; a bounding box coordinate calculation unit, used to calculate the boundary point coordinates of the bounding box based on the size of the bounding box and the third coordinate, according to the rule that the coordinates of the current target in the current interval frame are the midpoint coordinates of the bounding box; and a bounding box projection unit, used to project the bounding box onto the current interval frame based on the boundary point coordinates of the bounding box.

[0130] In some optional implementations, the annotation results corresponding to all targets in each detection frame further include the boundary point coordinates of the bounding box. The device further includes: a bounding box size calculation module, used to calculate the fourth coordinate of the current target in the current detection frame and the size of the first bounding box based on the motion trajectory of the current target; a boundary point coordinate calculation module, used to calculate the boundary point coordinates of the first bounding box based on the size of the first bounding box, according to the rule that the fourth coordinate of the current target in the current detection frame is the midpoint coordinate of the bounding box; and a bounding box comparison module, used to compare the boundary point coordinates of the first bounding box with the coordinates of the current target in the current detection frame detected by the pre-trained target detection model. The module determines whether the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target detected by the pre-trained target detection model in the current detection frame is greater than a preset difference threshold. The bounding box adjustment module adjusts the boundary point coordinates of the corresponding bounding box of the current target detected by the pre-trained target detection model in the current detection frame by using the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target detected by the pre-trained target detection model in the current detection frame, so as to obtain the final boundary point coordinates of the first bounding box in the current detection frame.

[0131] In some optional embodiments, the device further includes: a coordinate calculation module, used to calculate the coordinates of the current target in the current detection frame based on the motion trajectory of the current target if the current target is not detected in the current detection frame; a bounding box size calculation module, used to calculate the size of the bounding box of the current target in the current detection frame based on the acquisition time corresponding to the current detection frame, the bounding box size of the current target in the adjacent detection frames, and the acquisition time corresponding to each frame; and a bounding box projection module, used to project the bounding box onto the current detection frame according to the coordinates of the current target in the current detection frame and the size of the bounding box, until the duration during which the current target is not detected exceeds a preset duration threshold.

[0132] In some optional implementations, the annotation results corresponding to all targets in each detection frame also include the target depth and target category. The device further includes: a target category determination module, used to obtain the actual feature size and bounding box size of the current target based on the detected current target category if the pre-trained target detection model does not detect the depth of the current target; and a target depth calculation module, used to calculate the depth of the current target based on the actual feature size of the current target, the bounding box size, and the shooting parameters of the image acquisition device for acquiring the vehicle-end image.

[0133] In some optional implementations, the annotation results corresponding to all targets in each detection frame also include the target identifiers. The device further includes: a target identifier detection module, used to acquire all targets and their corresponding bounding boxes in adjacent detection frames if no identifier information is detected for the current target in the current detection frame; a bounding box size normalization module, used to normalize the bounding box size corresponding to the current target in the current detection frame and the bounding box sizes corresponding to all targets in adjacent detection frames to the same size; a similarity calculation module, used to calculate the similarity between the image in the bounding box corresponding to the current target and the image in the bounding box of the same size corresponding to all targets in adjacent detection frames; and an identifier marking module, used to mark the identifier of the corresponding target in the adjacent detection frame with the highest image similarity and greater than a preset similarity threshold as the identifier of the current target in the current detection frame.

[0134] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0135] In this embodiment, the target tracking device based on vehicle-end images is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0136] This invention also provides a target tracking system based on vehicle-side images, such as... Figure 8 As shown, it has a controller for executing the above-described target tracking method based on vehicle-end images. For detailed description, please refer to the above embodiments, which will not be repeated here.

[0137] This invention also provides a vehicle, such as... Figure 9 As shown, the vehicle includes the aforementioned target tracking system based on vehicle-mounted images.

[0138] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a controller provided in an optional embodiment of the present invention, such as... Figure 10As shown, the controller includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the controller, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple controllers can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 10 Take a processor 10 as an example.

[0139] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0140] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0141] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the controller. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0142] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0143] The controller also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 10Taking the example of a connection between China and Israel via a bus.

[0144] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the controller, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0145] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0146] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A target tracking method based on vehicle end image, characterized in that, The method includes: Acquire a preset number of vehicle-end images, and extract several vehicle-end images from the preset number of vehicle-end images as detection frames. The preset number of vehicle-end images are images continuously acquired within a preset period at a preset frame rate. The pre-trained target detection model is used to detect targets in each detection frame to obtain the annotation results corresponding to all targets in each detection frame. Based on the coordinates of the current target in each detection frame, the motion trajectory of the current target is calculated. The annotation results include at least the coordinates of the target, the bounding box used to annotate the target, and the coordinates of the boundary points of the bounding box. Based on the motion trajectory of the current target, calculate the coordinates of the current target in all interval frames, where the interval frames are the vehicle-end images in the vehicle-end images of the preset number of frames, excluding the detection frames; Based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the acquisition time corresponding to each frame, the size of the bounding box of the current target in the current interval frame is calculated. Based on the coordinates of the current target in the current interval frame and the size of the bounding box, the bounding box is projected into the current interval frame. All interval frames, detection frames and all targets are traversed to obtain the coordinates and bounding boxes of all targets in the vehicle-end image of a preset number of frames. The target tracking result is obtained based on the coordinates and bounding boxes of the targets. The method further includes: calculating the fourth coordinate of the current target in the current detection frame and the size of the first bounding box based on the motion trajectory of the current target; calculating the boundary point coordinates of the first bounding box based on the size of the first bounding box, according to the rule that the fourth coordinate of the current target in the current detection frame is the midpoint coordinate of the bounding box; comparing whether the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than a preset difference threshold; if the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than the preset difference threshold, then adjusting the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model using the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model, to obtain the final boundary point coordinates of the first bounding box in the current detection frame, so as to avoid abrupt changes in the bounding box tracking trajectory.

2. The method of claim 1, wherein, The step of calculating the motion trajectory of the current target based on its coordinates in each detection frame includes: Obtain the first coordinates of the current target in the initial detection frame, where the initial detection frame is the frame image where the current target is first detected; Obtain the second coordinates of the current target in the current detection frame, where the current detection frame is any frame other than the initial detection frame among all detection frames; Based on the first and second coordinates, calculate the translation and rotation matrices of the current target in the current detection frame compared to the initial detection frame; Based on the translation vector and rotation matrix of the current target in the current detection frame compared to the initial detection frame, and the second coordinate, calculate the absolute coordinates of the current target in the current detection frame with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system; The motion trajectory of the current target is calculated based on the absolute coordinates of the current target in each detection frame.

3. The method of claim 2, wherein, The absolute coordinates are three-dimensional coordinates. Based on the absolute coordinates of the current target in each detection frame, the motion trajectory of the current target is calculated, including: Based on the absolute three-dimensional coordinates of the current target in all detection frames, including the second detection frame, fit a motion trajectory curve, and based on the fitted motion trajectory curve, determine the predicted three-dimensional coordinates of the current target in all detection frames. Based on the predicted and absolute 3D coordinates of the current target in all detection frames, calculate the weighted loss function value corresponding to each 3D coordinate component; The sum of the weighted loss function values ​​corresponding to each three-dimensional coordinate component is obtained by summing the loss functions. The fitted motion trajectory curve is adjusted until the sum of the loss functions is less than a preset loss function threshold, thus obtaining the adjusted motion trajectory of the current target.

4. The method of claim 2, wherein, The coordinates of the current target in the current interval frame are absolute coordinates with the vehicle coordinate system of the current target in the initial detection frame as the absolute coordinate system. The step of projecting the bounding box into the current interval frame according to the coordinates of the current target in the current interval frame and the size of the bounding box includes: Based on the motion trajectory of the current target, calculate the absolute coordinates of the current target in the current interval frame, and based on the absolute coordinates of the current target in the current interval frame and the first coordinates of the current target in the initial detection frame, calculate the translation matrix and rotation matrix of the current target in the current interval frame compared to the initial detection frame. Using the translation and rotation matrices of the current target in the current interval frame compared to the initial detection frame, and the first coordinate of the current target in the initial detection frame, calculate the third coordinate of the current target in the current interval frame; According to the rule that the coordinates of the current target in the current interval frame are the coordinates of the midpoint of the bounding box, the coordinates of the boundary points of the bounding box are calculated based on the size of the bounding box and the third coordinate. Based on the coordinates of the boundary points of the bounding box, the bounding box is projected into the current interval frame.

5. The method according to claim 1, characterized in that, The method further includes: If the current target is not detected in the current detection frame, the coordinates of the current target in the current detection frame are still calculated based on the motion trajectory of the current target. Based on the acquisition time corresponding to the current detection frame, the bounding box size of the current target in the adjacent detection frames, and the corresponding acquisition times, the size of the bounding box of the current target in the current detection frame is calculated. Based on the coordinates of the current target in the current detection frame and the size of the bounding box, the bounding box is projected onto the current detection frame until the duration during which the current target is not detected exceeds a preset duration threshold.

6. The method according to claim 1, characterized in that, The annotation results for each target in each detection frame also include the target's depth and target category. The method further includes: If the pre-trained object detection model fails to detect the depth of the current object, obtain the actual feature size and bounding box size of the current object based on the detected current object category; The depth of the current target is calculated based on the actual feature size of the current target, the size of the bounding box, and the shooting parameters of the image acquisition device that acquires the vehicle-side image.

7. The method according to claim 1, characterized in that, The annotation results corresponding to all targets in each detection frame also include the target identifiers, and the method further includes: If no identifier information is detected for the current target in the current detection frame, then all targets and their corresponding bounding boxes in the adjacent detection frames are obtained; The bounding box size corresponding to the current target in the current detection frame and the bounding box sizes corresponding to all targets in the adjacent detection frames are normalized to the same size; Calculate the similarity between the image in the bounding box corresponding to the current target and the images in the bounding boxes of the same size corresponding to all targets in adjacent detected frames; The identifier of the corresponding target in the adjacent detection frames with the highest image similarity and greater than the preset similarity threshold is marked as the identifier of the current target in the current detection frame.

8. A target tracking device based on vehicle-side images, characterized in that, The device includes: The detection frame extraction module is used to acquire a preset number of vehicle-end images and extract several vehicle-end images from the preset number of vehicle-end images as detection frames. The preset number of vehicle-end images are images continuously acquired at a preset frame rate within a preset period. The target detection module is used to perform target detection on each detection frame using a pre-trained target detection model, obtain the annotation results corresponding to all targets in each detection frame, and calculate the motion trajectory of the current target based on the coordinates of the current target in each detection frame. The annotation results include at least the coordinates of the target and the bounding box and the coordinates of the boundary points of the bounding box used to annotate the target. The target coordinate calculation module is used to calculate the coordinates of the current target in all interval frames based on the motion trajectory of the current target. The interval frames are the vehicle-end images in the vehicle-end images of the preset number of frames, excluding the detection frames. The bounding box determination module is used to calculate the size of the bounding box of the current target in the current interval frame based on the acquisition time corresponding to the current interval frame, the bounding box size of the current target in the adjacent detection frames of the current interval frame, and the acquisition time corresponding to each frame. The target tracking module is used to project the bounding box into the current interval frame according to the coordinates of the current target in the current interval frame and the size of the bounding box, traverse all interval frames, detection frames and all targets, obtain the coordinates and bounding boxes of all targets in the vehicle-end image of a preset number of frames, and obtain the target tracking result based on the coordinates and bounding boxes of the targets. The device further includes: a bounding box size calculation module, used to calculate the fourth coordinate of the current target in the current detection frame and the size of the first bounding box based on the motion trajectory of the current target; a boundary point coordinate calculation module, used to calculate the boundary point coordinates of the first bounding box based on the size of the first bounding box, according to the rule that the fourth coordinate of the current target in the current detection frame is the midpoint coordinate of the bounding box; and a bounding box comparison module, used to compare whether the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than a preset difference threshold; the bounding box... The adjustment module is used to adjust the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame if the difference between the boundary point coordinates of the first bounding box and the boundary point coordinates of the corresponding bounding box of the current target in the current detection frame detected by the pre-trained target detection model is greater than a preset difference threshold, so as to obtain the final boundary point coordinates of the first bounding box in the current detection frame, so as to avoid abrupt changes in the bounding box tracking trajectory.

9. A target tracking system based on vehicle-side images, characterized in that, The system includes a controller, which includes a memory and a processor, and the memory and the processor are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the target tracking method based on vehicle-end images as described in any one of claims 1 to 7.

10. A vehicle, characterized in that, The vehicle includes the target tracking system based on vehicle-side images as described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the target tracking method based on vehicle-end images as described in any one of claims 1 to 7.

Images