Target tracking methods, apparatus, computer equipment, and storage media

Abstract

This application relates to a target tracking method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring two-dimensional scene image processing results and scene point cloud processing results; the two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets; obtaining a target fitting relationship based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, and a pre-constructed Gaussian function; identifying undetected moving targets in the two-dimensional scene image, and determining the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship; updating the two-dimensional scene image processing results according to the bounding box coordinates of the undetected moving targets in the two-dimensional scene image; and tracking the motion trajectories of multiple moving targets based on the updated two-dimensional scene image processing results. This method can improve the accuracy of target tracking.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a target tracking method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of autonomous driving technology, some target tracking technologies have been gradually applied to autonomous vehicles. By detecting and tracking other vehicles, pedestrians and other objects around the autonomous vehicle, the autonomous vehicle can make intelligent driving decisions.

[0003] In traditional technologies, autonomous vehicles are typically equipped with smart surveillance cameras or security cameras to detect targets around the vehicle. By calculating the movement of vehicles or pedestrians in the images, the cameras can track vehicles or pedestrians within their field of view to ensure safe driving and avoid collisions with other vehicles, pedestrians, and obstacles.

[0004] However, current target tracking technologies still have some shortcomings. For example, sensors in autonomous vehicles are prone to missing certain targets, or they may disappear from the sensor's field of view or become difficult to identify due to environmental factors, ultimately resulting in low accuracy of target tracking and detection results. Summary of the Invention

[0005] Therefore, it is necessary to provide a target tracking method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of target tracking and detection results in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a target tracking method. The method includes:

[0007] The process involves obtaining two-dimensional scene image processing results and scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets.

[0008] Based on the bounding box coordinates and projected 2D coordinates of the same moving target, and a pre-constructed Gaussian function, a target fitting relationship is obtained. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the 2D scene image according to the projected 2D coordinates of the moving target.

[0009] Find the undetected moving targets in the two-dimensional scene image, and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship;

[0010] The two-dimensional scene image processing result is updated based on the bounding box coordinates of the missed moving target in the two-dimensional scene image;

[0011] Based on the updated two-dimensional scene image processing results, the motion trajectories of the multiple moving targets are tracked.

[0012] In one embodiment, obtaining the two-dimensional scene image processing result and the scene point cloud processing result includes:

[0013] Acquire a two-dimensional scene image set and scene point cloud data, wherein the two-dimensional scene image dataset includes multiple consecutive two-dimensional scene images;

[0014] Target detection is performed on the two-dimensional scene image and the scene point cloud data respectively to obtain image target detection results and point cloud target detection results. The image target detection results include the bounding box coordinates, bounding box size and first classification label of multiple detected moving targets. The point cloud target detection results include the three-dimensional point cloud coordinates and second classification label of multiple detected moving targets.

[0015] The three-dimensional point cloud coordinates of multiple moving targets are projected onto the pixel coordinate system to obtain the projected two-dimensional coordinates of the multiple moving targets.

[0016] In one embodiment, target detection is performed on the two-dimensional scene image to obtain image target detection results, including:

[0017] Initial target detection is performed on the two-dimensional scene image to obtain initial image target detection results; the initial image target detection results are then subjected to cross-intersection over union (CIU) cascaded matching and posterior estimation processing to obtain the final image target detection result.

[0018] In one embodiment, the image target detection result includes the bounding box coordinates, bounding box size, and first classification label of multiple detected moving targets; the initial target detection of the two-dimensional scene image to obtain the initial image target detection result includes:

[0019] The two-dimensional scene image is input into a trained target detection network to obtain a feature map corresponding to the two-dimensional scene image and a first class classification label. The target detection network includes a convolutional layer, a pooling layer, and a classifier.

[0020] The feature map is subjected to a feature pyramid operation to obtain a multi-scale feature map corresponding to the two-dimensional scene image.

[0021] Based on multi-scale feature maps, the bounding box coordinates and bounding box dimensions of the multiple detected moving targets are determined;

[0022] The trained object detection network is trained based on historical two-dimensional scene images carrying object classification labels.

[0023] In one embodiment, the pre-constructed Gaussian function includes a pre-constructed planar Gaussian function; obtaining the target fitting relationship based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, and the pre-constructed Gaussian function, includes:

[0024] The expression for the plane Gaussian function is determined based on the bounding box coordinates, the projected two-dimensional coordinates, and the pre-constructed plane Gaussian function.

[0025] Calculate the fitting error of the expression for the plane Gaussian function;

[0026] Based on the fitting error, the parameters of the plane Gaussian function expression are adjusted. When the fitting error is minimized, the adjusted plane Gaussian function expression is determined as the target fitting relationship.

[0027] In one embodiment, tracking the motion trajectories of the plurality of moving targets based on the updated two-dimensional scene image processing results includes:

[0028] A posterior estimation is performed on the updated two-dimensional scene image processing result to obtain the posterior estimation result of the updated two-dimensional scene image processing result;

[0029] Based on the posterior estimation results, a binary assignment association is performed to obtain the motion trajectories of multiple moving targets.

[0030] Secondly, this application also provides a target tracking device. The device includes:

[0031] The information acquisition module is used to acquire the two-dimensional scene image processing results and the scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets.

[0032] The fitting relationship confirmation module is used to obtain the target fitting relationship based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, as well as a pre-constructed Gaussian function. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the two-dimensional scene image based on the projected two-dimensional coordinates of the moving target.

[0033] The target fitting module is used to find undetected moving targets in the two-dimensional scene image and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship.

[0034] The result update module is used to update the two-dimensional scene image processing result based on the bounding box coordinates of the missed moving target in the two-dimensional scene image.

[0035] The trajectory generation module is used to track the motion trajectories of the multiple moving targets based on the updated two-dimensional scene image processing results.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0037] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described above.

[0038] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described above.

[0039] The aforementioned target tracking method, apparatus, computer device, storage medium, and computer program product acquire a two-dimensional scene image processing result including the bounding box coordinates of multiple detected moving targets, and a projected two-dimensional coordinate scene point cloud processing result including the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets. Based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, and a pre-constructed Gaussian function, a target fitting relationship is obtained to determine the bounding box coordinates of the moving target in the two-dimensional scene image based on the projected two-dimensional coordinates of the moving target. Then, undetected moving targets in the two-dimensional scene image are identified, and the bounding box coordinates of the undetected moving targets in the two-dimensional scene image are determined based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship. Further, the two-dimensional scene image processing result is updated based on the bounding box coordinates of the undetected moving targets in the two-dimensional scene image, and the motion trajectory of the multiple moving targets is tracked based on the updated two-dimensional scene image processing result. Therefore, the above scheme can make great use of the correlation between point clouds and images, reduce the probability of target misses, and improve the accuracy of target detection and tracking. Attached Figure Description

[0040] Figure 1 This is a diagram illustrating the application environment of a target tracking method in one embodiment;

[0041] Figure 2 This is a flowchart illustrating a target tracking method in one embodiment;

[0042] Figure 3 This is a flowchart illustrating the steps for obtaining image and point cloud processing results in one embodiment;

[0043] Figure 4 This is a schematic diagram of the process of processing image data in a system.

[0044] Figure 5 This is a flowchart illustrating a target tracking method in one embodiment;

[0045] Figure 6 This is a structural block diagram of a target tracking device in one embodiment;

[0046] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The target tracking method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 acquires a two-dimensional scene image processing result including the bounding box coordinates of multiple detected moving targets, and a scene point cloud processing result including projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets. Based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, and a pre-constructed Gaussian function, it obtains a target fitting relationship for determining the bounding box coordinates of the moving target in the two-dimensional scene image based on the projected two-dimensional coordinates of the moving target. Then, server 104 can find undetected moving targets in the two-dimensional scene image and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship. Furthermore, based on the bounding box coordinates of the missed moving targets in the two-dimensional scene image, the two-dimensional scene image processing result is updated, and based on the updated two-dimensional scene image processing result, the motion trajectory of the multiple moving targets is tracked, which can be viewed through the terminal 102.

[0049] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart TVs, smart in-vehicle systems, etc. Portable wearable devices can include smartwatches, smart bracelets, etc. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0050] In one embodiment, such as Figure 2 As shown, a target tracking method is provided, which is applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0051] S200: Obtain the two-dimensional scene image processing results and scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets.

[0052] Two-dimensional image processing results in image information obtained by processing raw image information through various operations and analyses. This can involve outlining or marking targets in the raw image and providing target classification information, bounding box coordinates, bounding box dimensions, etc. For example, detecting and identifying different vehicles, pedestrians, and obstacles in the raw image. Scene point cloud processing results in point cloud information obtained by processing raw point cloud data through various operations and analyses. This can involve detecting and identifying the location, size, and category of targets in the point cloud, such as vehicles, buildings, or pedestrians, or visualizing the point cloud data into a three-dimensional scene or model.

[0053] Furthermore, the scene point cloud processing results also include projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of multiple detected moving targets, to achieve multimodal fusion between image information and point cloud information, and improve the utilization of their correlation. Projecting three-dimensional point cloud coordinates into two-dimensional projected coordinates usually involves the conversion between camera projection or viewpoint projection. A projection matrix can be used, which can project three-dimensional points onto a two-dimensional image. The projection matrix includes the camera intrinsic parameter matrix (such as focal length, principal point, etc.) and the extrinsic parameter matrix (camera position and orientation). For example, assuming the three-dimensional point cloud coordinates are... Located in the world coordinate system, the coordinates of the 3D point cloud can be transformed into the camera coordinate system based on the extrinsic parameter matrix, thus obtaining the coordinates of the 3D point cloud in the camera coordinate system. Then, based on the intrinsic parameter matrix, perspective projection is performed on the coordinates of the 3D point cloud in the camera coordinate system, mapping the 3D point cloud from the camera coordinate system to the image plane. The coordinates obtained from the projection are then normalized to obtain the pixel coordinates on the image. This refers to the projected two-dimensional coordinates corresponding to the three-dimensional point cloud coordinates. The specific implementation of this process depends on the camera parameters and programming environment. Computer vision libraries typically provide functions to perform this transformation. In fields such as autonomous driving, this process can be implemented by specialized software and hardware to handle large amounts of point cloud data.

[0054] Acquiring image and point cloud information typically involves different types of sensors and devices, depending on the application scenario. In the field of autonomous driving, digital cameras (such as RGB cameras) can be used to capture images of the scene, or a camera array can be used to generate stereo or panoramic images. 3D point cloud information can be acquired using LiDAR, which obtains point cloud data by sending laser beams and measuring the return time. Alternatively, depth cameras can be used, providing not only RGB images but also depth information, which can be used for 3D perception and point cloud generation. Acquiring image and point cloud information requires specialized hardware and sensors, and typically necessitates processing and analyzing the captured raw image and point cloud information to meet the specific application requirements.

[0055] S400: Based on the bounding box coordinates and projected 2D coordinates of the same moving target, as well as a pre-constructed Gaussian function, a target fitting relationship is obtained. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the 2D scene image based on the projected 2D coordinates of the moving target.

[0056] The Gaussian function is a function in probability and statistics, commonly used to describe various natural phenomena and data distributions. Its mathematical ease of manipulation makes it convenient for fitting data. Pre-built Gaussian functions have parameters such as mean and standard deviation, which can be adjusted to use different data distributions. Taking the planar Gaussian function as an example, the expression for a pre-built Gaussian function can be shown below:

[0057]

[0058] In the above function expression, This represents the mean of the Gaussian function in the plane. and This represents the standard deviation of the Gaussian function.

[0059] Based on the bounding box coordinates and projected 2D coordinates of the same moving target, and the aforementioned expression for the planar Gaussian function, the parameters in the planar Gaussian function can be determined. For each set of bounding box coordinates and projected 2D coordinates of the moving target, the mean and standard deviation of a set of Gaussian functions can be estimated. Based on these estimates of the mean and standard deviation, a specific expression for a planar Gaussian function can be determined.

[0060] After determining the Gaussian plane function, it can be optimized to better fit the bounding box coordinates and projected 2D coordinates of multiple moving targets. For example, iterative optimization methods, such as the expectation-maximization algorithm or nonlinear least squares method, can be used to adjust the parameters of the Gaussian plane function to best fit the data. The adjusted Gaussian plane function can then serve as the target fitting relationship. Furthermore, the process of determining the target fitting relationship can be performed by a computer to achieve high-precision and high-speed determination of the target fitting relationship.

[0061] S600: Locate undetected moving targets in the 2D scene image and determine the bounding box coordinates of the undetected moving targets in the 2D scene image based on the projected 2D coordinates of the undetected moving targets and the target fitting relationship.

[0062] Among them, missed moving targets refer to targets that are missed during the processing of image information, but can be captured during the processing of 3D point cloud information. In other words, these moving targets have projected 2D coordinates, but do not have bounding box coordinates in the 2D scene image.

[0063] There are many reasons why the aforementioned missed detections occur. For example, 2D images are captured from the viewpoint of a camera, while 3D point clouds are typically obtained from sensors at different angles, such as LiDAR. Some targets may be difficult to detect in 2D images due to occlusion, viewpoint issues, or low contrast, but can be more easily detected in 3D point clouds due to the combined angles from multiple sensors. Furthermore, 2D images and 3D point clouds represent different data modalities; some targets may be more apparent in point clouds because they provide additional depth information, while 2D images typically only provide surface information. Therefore, in fields such as autonomous driving, using multiple sensors to obtain multimodal information—including 2D image information captured by cameras and 3D point cloud information captured by LiDAR—and fusing this multimodal information using appropriate algorithms and techniques can effectively improve the accuracy of target detection.

[0064] For example, by inputting the two-dimensional projected coordinates of the missed moving target into the target fitting relation, the bounding box coordinates of the moving target in the two-dimensional scene image can be obtained. Furthermore, the bounding box size of the target in the two-dimensional scene image can also be obtained. After obtaining the bounding box coordinates of the missed moving target through the target fitting relation, the fitting results can be verified to ensure that the fitted bounding box accurately surrounds the missed moving target, and the bounding box can be adjusted to fit the actual shape and size of the target. The above process can be implemented using a computer combined with various technologies and algorithms to quickly and efficiently process multiple missed moving targets.

[0065] S800: Update the 2D scene image processing results based on the bounding box coordinates of the missed moving targets in the 2D scene image.

[0066] Updating the 2D scene processing results refers to adding missed moving targets to the target list. This addition includes the bounding box coordinates, bounding box size, and classification labels of the missed moving targets. The updated 2D scene processing results include information about targets obtained from processing 2D image information, as well as information about targets missed during 2D image processing but detected during 3D point cloud processing. For example, the updated 2D scene image will have all detected and missed moving targets marked with bounding boxes, along with their coordinate information and classification labels.

[0067] For example, a unique identifier or index can be assigned to each missed moving target, and the bounding box coordinates of the missed moving targets in the 2D scene image obtained based on the target fitting relationship can be collected. These bounding box coordinates are then added to the 2D scene image to update the processing results, such as drawing the bounding box of the missed moving target in the 2D image or marking its location. Furthermore, as new image frames appear, it is still necessary to continuously monitor missed moving targets and update the 2D scene processing results in real time based on their locations.

[0068] S1000: Based on the updated 2D scene image processing results, it tracks the motion trajectories of multiple moving targets.

[0069] The updated 2D scene image processing results contain all targets detected through 2D image information and 3D point cloud information. The next step is to track these targets, which usually involves continuous estimation of the position and motion state of moving targets in multiple frames of images, ultimately obtaining the motion trajectories of multiple moving targets.

[0070] There are various methods for target tracking, and the appropriate method can be selected based on application requirements, available data, and computational resources. In addition, there are target tracking libraries and tools that provide some common target tracking algorithms. For example, the Kalman filter is a commonly used target tracking method that can estimate the target's position and velocity and predict its future position. Besides this, template matching techniques using correlation filters can also be used to track targets. This technique compares the similarity between the current frame and the target template and estimates the target position based on the position with the highest similarity. In this application, the tracking of multiple moving targets is involved. Exemplarily, deep learning methods, such as convolutional neural networks, can be used to learn the motion and appearance features of the targets and achieve end-to-end tracking. Furthermore, multi-target tracking algorithms such as multi-target Kalman filtering and multi-target particle filtering can also be used to estimate the positions and trajectories of multiple targets.

[0071] Based on the updated 2D scene image processing results, the motion trajectories of multiple moving targets are tracked. These trajectories can be displayed in the 2D scene image, for example, by drawing target bounding boxes on the targets, using different colored bounding boxes to represent different types of targets, displaying labels on the targets including their category or other key information, and moving them across video frames to leave tracks for target tracking. In addition, heatmaps can be generated to display the density distribution of targets during multi-target tracking, indicating the degree of target activity within the region.

[0072] The aforementioned target tracking method acquires a 2D scene image processing result including the bounding box coordinates of multiple detected moving targets, and a projected 2D scene point cloud processing result including the projected 2D coordinates of the 3D point cloud coordinates of the multiple detected moving targets. Based on the bounding box coordinates and projected 2D coordinates of the same moving target, and a pre-constructed Gaussian function, a target fitting relationship is obtained to determine the bounding box coordinates of the moving target in the 2D scene image according to the projected 2D coordinates of the moving target. Then, undetected moving targets in the 2D scene image are identified, and their bounding box coordinates in the 2D scene image are determined based on their projected 2D coordinates and the target fitting relationship. Further, the 2D scene image processing result is updated based on the bounding box coordinates of the undetected moving targets in the 2D scene image, and the motion trajectories of the multiple moving targets are tracked based on the updated 2D scene image processing result. Therefore, this scheme can maximize the utilization of the correlation between point clouds and images, reduce the probability of target misses, and improve the accuracy of target detection and tracking.

[0073] like Figure 3 As shown, in one embodiment, S200 includes:

[0074] S220: Acquire a set of two-dimensional scene images and scene point cloud data. The two-dimensional scene image dataset includes multiple consecutive two-dimensional scene images.

[0075] In practical applications, acquiring 2D scene image sets and scene point cloud data typically requires the use of sensors and corresponding software tools. For 2D scene images, standard RGB (Red, Green, Blue) cameras, such as ordinary digital cameras or camcorders, can be used to capture images. For scene point cloud data, depth sensors, such as structured light cameras, time-of-flight cameras, or LiDAR, can be used to obtain point cloud information. In the field of autonomous driving, the aforementioned sensors or devices are usually installed on vehicles to capture target information around the vehicle in real time, typically requiring the acquisition of multiple frames of continuous data to achieve temporal continuity.

[0076] S240: Perform target detection on the two-dimensional scene image to obtain the image target detection result, which includes the bounding box coordinates, bounding box size, and first classification label of multiple detected moving targets.

[0077] Object detection refers to the identification and localization of specific objects in data such as images, videos, and point clouds. For example, in a 2D scene image, bounding boxes are used to mark targets such as vehicles, pedestrians, and obstacles, and the bounding box coordinates and dimensions of moving targets are obtained. In addition, detected moving targets can be classified to obtain classification labels. These labels are denoted as the first-class classification label, representing the classification label of the moving target obtained based on the 2D scene image detection.

[0078] Two-dimensional scene images typically require preprocessing to remove noise, adjust contrast, and scale image size to improve the performance of subsequent object detection algorithms. Feature extraction is then performed on the preprocessed 2D scene images to extract useful information, such as object edges and color histograms. Different features can be extracted for different object detection tasks. Object detection algorithms are the core of the object detection process, and different algorithms can be selected based on different tasks, such as traditional feature-based methods and deep learning algorithms. Traditional feature-based object detection methods mainly use hand-designed features and machine learning algorithms to detect objects in images. For example, the HOG (Histogram of Oriented Gradients) algorithm captures the texture and shape information of objects by calculating the histogram of local gradient directions in the image, and is suitable for tasks such as face detection and pedestrian detection. In addition, commonly used object detection algorithms include deep learning algorithms, such as convolutional neural networks and object detection networks. These algorithms can automatically identify and locate target objects in images by training neural networks, replacing the hand-designed feature extraction process in traditional methods, making object detection more accurate and efficient.

[0079] S262: Perform target detection on scene point cloud data to obtain point cloud target detection results. The point cloud target detection results include the three-dimensional point cloud coordinates and second classification labels of multiple detected moving targets.

[0080] Target detection on scene point cloud data is a technique that analyzes 3D point cloud data to detect moving targets within it.

[0081] For example, the steps for processing scene point cloud data can be as follows: First, perform set abstraction operations on the scene point cloud data to extract local features of the point cloud. This typically involves extracting a subset of the point cloud within a specific radius and calculating the features of these subsets, such as shape descriptors, normals, and colors, for subsequent processing and analysis. After extracting the local features of the point cloud, the point cloud data can be segmented to divide it into different parts or objects, such as separating different objects in the scene or separating ground points from object points. After segmenting the point cloud data, it is usually necessary to assign a classification to each point to indicate whether the point is a foreground point (usually an object) or a background point (usually the ground or other areas of no interest). In this embodiment, the foreground point needs to be saved as the 3D point cloud coordinates of the moving target. After saving the foreground point, it can also be classified, that is, the classification label of the foreground point can be determined. This classification label is denoted as the second classification label, indicating that the classification label is obtained based on the point cloud scene image detection to indicate what type of object the foreground point represents (e.g., vehicle, pedestrian, tree, etc.). The above steps are key steps in point cloud processing, usually implemented by algorithms and machine learning techniques, to transform scene point cloud data into data needed for object detection tasks, such as key information about the location and type of moving targets in the scene, for subsequent use.

[0082] S264: Project the 3D point cloud coordinates of multiple moving targets onto the pixel coordinate system to obtain the projected 2D coordinates of the multiple moving targets.

[0083] In target detection tasks, processing three-dimensional coordinates requires more work than processing two-dimensional coordinates. Therefore, projecting the three-dimensional point cloud coordinates of multiple moving targets to the pixel coordinate system to obtain the projected two-dimensional coordinates of multiple moving targets can reduce computational complexity.

[0084] Projecting the 3D point cloud coordinates of multiple moving targets into a pixel coordinate system can be achieved by first converting the 3D point cloud coordinates from the world coordinate system to the camera coordinate system. This requires camera extrinsic parameters, which describe the camera's position and orientation in the world coordinate system. These parameters can be obtained through camera calibration techniques, such as using parameter points or calibration boards with known positions and orientations. Next, the point cloud coordinates in the camera coordinate system are projected into the pixel coordinate system. This projection process requires camera intrinsic parameters, which describe the internal characteristics of the camera's imaging process, such as focal length, principal point coordinates, and pixel size. These intrinsic parameters can also be obtained through camera calibration techniques. The resulting projected 2D coordinates are typically in pixels and need to be normalized to ensure they lie within the normalized plane of the image, usually within the range of [-1, 1]. Normalization can be achieved by dividing the pixel coordinates by the image width and height. This method, by processing the 3D point cloud coordinates to obtain the projected 2D coordinates, allows for easier fusion with scene image processing data, thus facilitating target detection and tracking.

[0085] In this embodiment, target detection processing is performed on two-dimensional scene image data and scene point cloud data. This includes extracting the bounding box coordinates, bounding box size, and first-class classification label of the moving target from the two-dimensional scene image, and extracting the three-dimensional point cloud coordinates and second-class classification label of the moving target from the scene point cloud data. The three-dimensional point cloud coordinates are then projected onto the pixel coordinate system to obtain the projected two-dimensional coordinates of multiple moving targets. This facilitates the subsequent fusion of image target detection results and point cloud target detection results, thereby improving the efficiency of target detection and tracking.

[0086] In one embodiment, such as Figure 4 As shown, S240 includes:

[0087] S242: Perform initial target detection on the two-dimensional scene image to obtain the initial image target detection result.

[0088] Initial target detection refers to performing a rapid target detection on a two-dimensional scene image to quickly obtain the approximate bounding box coordinates, bounding box size, and first-class classification label of the moving target in the two-dimensional scene image.

[0089] Taking an autonomous vehicle scenario as an example, this vehicle is equipped with cameras to capture images of the road, creating two-dimensional scene images to gather information about the surrounding environment. Initial object detection is performed on the captured two-dimensional scene images, including convolution, pooling, and feature pyramid operations. In initial object detection, the two-dimensional scene image is passed through a series of convolutional layers, each using a convolutional kernel, also called a filter, to extract local features such as edges, textures, and shapes. After convolution, pooling operations, such as max pooling or average pooling, are typically performed to reduce the size of the feature maps without losing important information. After pooling, feature pyramid operations are needed to handle targets of different scales. This is because targets may appear at different sizes in the image, requiring multi-scale feature maps. Feature pyramids detect targets at different scales by constructing feature maps at different levels or resolutions.

[0090] After the above operations, the bounding box coordinates and dimensions of the targets in the 2D scene image can be obtained. Furthermore, by inputting the feature maps obtained through convolution and pooling operations into a classifier, a classification label for each bounding box can be output, such as "vehicle" or "pedestrian," to identify the target category. These steps, through the construction of an object detection model and the computer performing the above processing operations, enable the object detection model to identify targets in 2D scene images and provide their bounding box coordinates, dimensions, and classification information.

[0091] S244, perform cross-intersection over union (CUI) cascaded matching and posterior estimation on the initial image target detection results to obtain the image target detection results.

[0092] Intersection over Union (IoU) is a metric used to evaluate object detection accuracy. It typically measures the degree of overlap between the bounding boxes or regions detected by the model and the ground truth bounding boxes. For example, it measures the degree of overlap between two bounding boxes, with a value between 0 and 1, where 0 indicates no intersection and 1 indicates complete overlap. Cascaded matching is a strategy used in object detection to filter and refine results. This strategy usually involves two or more levels of matching, each with different conditions and methods to determine whether two or more detection results represent the same target. For example, different IoU thresholds can be set to filter bounding boxes. Posterior estimation is used to estimate the probability distribution of an event or parameter. In posterior estimation, to calculate the probability distribution of an event or parameter, some known prior information and data need to be considered. The goal of posterior estimation is to obtain a more accurate estimate of the event or parameter by combining prior information and data.

[0093] Taking the example of performing multiple cascaded matching and intersection-over-union (IoU) filtering on the initial object detection results, a cascaded detector architecture can be built first. This can involve setting up multiple detection stages, each using a different model. For instance, the first stage might use a faster but potentially less accurate model, such as a single lightweight convolutional neural network. The goal of this stage is to quickly generate candidate bounding boxes, but this may result in some noisy or inaccurate bounding boxes in the detection results. The second stage uses a deeper, more computationally expensive model, such as a deep convolutional neural network or a region convolutional neural network. The goal of this stage is to improve accuracy through further feature extraction and classification. After each of these stages, different IoU thresholds can be applied to filter the target bounding boxes. For example, the first stage can use a lower IoU threshold to ensure that most targets are captured, while the second stage can use a higher IoU threshold to improve the accuracy of the bounding boxes. Cascaded matching is a multi-stage matching process in which candidate bounding boxes generated in the first stage are passed to the second stage. The second stage further filters and classifies these candidate bounding boxes. Only bounding boxes confirmed in the second stage are considered as the final detected targets, thereby improving the detection accuracy of the target detection process and reducing the false detection rate and false negative rate of targets.

[0094] In the Cross-Union-Place (CUP) filtering stage, the target detection results undergo CUP filtering. This involves comparing each pair of candidate bounding boxes in the initial target detection results and calculating their CUP threshold. For example, if the CUP threshold of two candidate boxes is higher than a certain threshold, such as 0.5 or 0.7, they are considered two detections of the same target. For instance, multiple target bounding boxes are detected in a 2D scene image using an object detection algorithm. Each bounding box has a corresponding confidence score, and the initial target detection results are sorted according to their confidence scores, for example, from high to low. Then, starting with the bounding box with the highest score, it is considered a reliable target box and retained. The remaining bounding boxes are then detected according to the pre-sorted list. If the detection result of a target bounding box shows that its CUP threshold is lower than the pre-set threshold, such as 0.5 or 0.7, that bounding box is discarded, and no further detection is performed on subsequent bounding boxes. The purpose of the CUP filtering stage is to eliminate detection boxes with low overlap, avoid detecting the same target multiple times, and improve the accuracy and stability of detection.

[0095] In the posterior estimation stage, two consecutive frames of 2D scene images are required. Taking the previous frame and the current frame as an example, the previous frame undergoes multiple cascaded matching and cross-union (CUI) filtering operations to obtain a higher-precision target detection result for that frame. A posterior estimation is then performed on this higher-precision target detection result. Various posterior estimation methods exist, such as maximum a posteriori (MAP), Kalman filtering, and Bayesian linear regression, which are not limited here. After posterior estimation, the higher-precision target detection result from the previous frame can predict the target detection result for the current frame. Simultaneously, the current frame can also undergo the same multiple cascaded matching and CUI filtering operations to obtain a higher-precision target detection result. Finally, a CUI matching operation is performed between the higher-precision target detection result and the predicted target detection result for the current frame, and the final result is used as the target detection result for the current frame. The intersection-union (IU) ratio is used to determine the degree of overlap between two bounding boxes, which can be used to measure the similarity between the predicted box and the ground truth box. For example, the IU ratio between the predicted box and the ground truth box is calculated, and a threshold, such as 0.5 or 0.7, is set to ensure that the matching operation only retains highly overlapping bounding boxes. This reduces the probability of false detections and improves tracking accuracy, because retaining bounding boxes with low matching scores may lead to false detections, such as mistaking different targets for the same target, or mistaking noisy bounding boxes for target bounding boxes.

[0096] In this embodiment, initial target detection is performed on a two-dimensional scene image, and the initial target detection results are sequentially processed by cross-border-union (CUI) concatenated matching and posterior estimation to finally obtain the image target detection result. The CUI concatenated matching and posterior estimation process allows the target to be tracked between different frames. As new observation data is continuously acquired, the estimation of the target position or state can be continuously updated and improved, and the image target detection result is determined based on the CUI, thereby improving the quality and accuracy of the detection result.

[0097] In one embodiment, such as Figure 5 As shown, S242 includes:

[0098] S2422, input the two-dimensional scene image into the trained object detection network to obtain the feature map and the first class classification label corresponding to the two-dimensional scene image. The object detection network includes convolutional layers, pooling layers and classifiers.

[0099] Object detection networks are deep learning neural network architectures primarily used for image recognition and image processing. Convolutional neural networks (CNNs) are a type of object detection network. CNNs automatically extract features from images through convolution operations and represent and learn these high-level features through a layered neural network structure. This structure can include convolutional layers for convolution operations, pooling layers for pooling operations, and a classifier for mapping features to object categories. Before applying a CNN, it is usually trained based on the application task. In this embodiment, the initial CNN can be trained based on historical 2D scene images carrying object classification labels. The parameters of the CNN are continuously updated during training to improve model performance and reduce errors in practical applications, resulting in a well-trained object detection network. For example, a 2D scene image is input into the trained object detection network. The 2D scene image first undergoes a convolution operation to extract features from the input 2D scene image. For instance, a sliding convolutional kernel (a small window) extracts local features from the input 2D scene image and generates a feature map. Each convolutional kernel can learn different features, such as edges and textures. Pooling operations are typically performed on the feature maps obtained from convolution operations. Pooling reduces the spatial size of the feature maps, thereby reducing computational complexity. Common pooling operations include max pooling and average pooling. The classifier analyzes each input feature map and assigns a class label to each target, denoted as the first classification label. This label typically represents the category of multiple moving targets detected through image object detection, such as "car" or "pedestrian." The classifier can be trained using data with known target categories during the training phase. By comparing the input features with the features of known target categories, it identifies objects in the image.

[0100] Furthermore, non-linear activation functions can be used in convolutional neural networks to introduce non-linear properties, enabling the network to learn complex data relationships. The last layer of a convolutional neural network can be a fully connected layer, used for classification or regression tasks, transforming the previous feature maps into the final output.

[0101] S2424: Perform a feature pyramid operation on the feature map to obtain a multi-scale feature map corresponding to the two-dimensional scene image.

[0102] Feature pyramid operations are used to process multi-scale image information in order to detect and recognize objects or features at different scales. This enables algorithms to capture information about objects at different scales, thereby improving the performance of detection and recognition.

[0103] For example, a feature pyramid can include multiple scale layers starting from a pooled feature map, each layer being a different version of the pooled feature map, obtained through downsampling or upsampling operations. These layers have different resolutions to allow the algorithm to operate at different scales. At each pyramid level, the feature extraction algorithm can extract features from the image, such as edges, textures, corners, etc., and perform scale normalization on these features at different levels to ensure consistency of features across different scales, facilitating subsequent algorithms to perform object detection and recognition tasks at different scales.

[0104] S2426, based on multi-scale feature maps, determines the bounding box coordinates and bounding box size of multiple detected moving targets.

[0105] After obtaining the multi-scale features of a 2D scene image, bounding boxes corresponding to multiple moving targets in the image can be determined using sliding window detection or anchor box methods. Taking sliding window detection as an example, a sliding window, typically a rectangular region, needs to be defined first. This window can slide across the image to search for potential targets. Then, the sliding window is placed at a specific location in the 2D scene image, and features within the window are extracted, classified, and detected. Each time the window slides, extraction, classification, and detection operations can be performed at the new location to determine whether the window contains a target. After determining that the window contains a target, non-maximum suppression algorithms can be used to remove overlapping bounding boxes, thus obtaining the final bounding box information, including bounding box coordinates and dimensions.

[0106] In this embodiment, by performing convolution, pooling, classification, and feature pyramid operations on the two-dimensional scene image, an initial target detection result is obtained. This initial target detection result includes the bounding box coordinates, bounding box dimensions, and first classification labels of multiple moving targets. The above target detection process can automatically identify and locate objects in images or videos and runs in real time, enabling simultaneous processing of a large amount of image or video data, which helps to improve the efficiency of target detection and tracking.

[0107] In one embodiment, the pre-constructed Gaussian function includes a pre-constructed plane Gaussian function, such as... Figure 5 As shown, S400 includes:

[0108] S420: Determine the expression of the plane Gaussian function based on the bounding box coordinates, the projected two-dimensional coordinates, and the pre-constructed plane Gaussian function;

[0109] Determining the expression for the planar Gaussian function requires combining the bounding box coordinates, projected 2D coordinates, and a pre-built planar Gaussian function model. For example, the model of the planar Gaussian function needs to be defined first. For instance, the Gaussian function includes parameters such as mean, covariance matrix, and standard deviation. Since the data being processed is 2D, a planar Gaussian function model can be constructed, with the following expression:

[0110]

[0111] in, It is the normalization coefficient. and This represents the mean of the Gaussian function in the plane. and This represents the standard deviation of the Gaussian function in the plane.

[0112] Bounding box coordinates can provide an initial estimate of the mean, which can be an estimate of the center of the planar Gaussian function. Projected 2D coordinates can provide the positional information of the 3D point cloud coordinates on the 2D scene image. For each set of bounding box coordinates and projected 2D coordinates of moving targets, a set of Gaussian function mean and standard deviation can be estimated. Based on the estimates of the mean and standard deviation, the specific expression of the planar Gaussian function can be determined.

[0113] S440: Calculate the fitting error of the plane Gaussian function expression, adjust the parameters of the plane Gaussian function expression based on the fitting error, and determine the adjusted plane Gaussian function expression as the target fitting relationship when the fitting error is minimized.

[0114] After determining the planar Gaussian function, it can be optimized to better fit the bounding box coordinates and projected 2D coordinates of multiple moving targets. A suitable metric can be chosen first, and then the parameters of the planar Gaussian function expression can be adjusted under that metric to minimize the error between its fit and the target. Different metrics correspond to different approximation methods. For example, under the squared error metric, the least squares method can be used to minimize the squared error; under the mean square error metric, the least mean square error can be used to approximate and minimize the mean square error.

[0115] For example, the coordinates of the bounding box of the moving target can be calculated based on the projected 2D coordinates and the expression of the Gaussian function in the point cloud processing results. However, the calculated bounding box coordinates of the moving target have an error compared to the corresponding actual bounding box coordinates of the moving target in the 2D scene image processing results. Therefore, it is necessary to adjust the expression of the Gaussian function to minimize this error. Taking Euclidean distance as the selected metric, the Euclidean distance between the actual bounding box coordinates and the calculated bounding box coordinates of the moving target is calculated. Optimization algorithms, such as gradient descent, least squares, or other optimization algorithms, are used to minimize this Euclidean distance. The corresponding Gaussian function parameters are iteratively adjusted until the Gaussian function achieves a satisfactory fit for the bounding box coordinates and projected 2D coordinates of multiple sets of moving targets. The adjusted Gaussian function, due to its minimal error in fitting the target relationship under the Euclidean distance metric, can better fit the actual acquired data. Furthermore, the performance of the optimized Gaussian function can be validated and evaluated, including checking the fitting error to ensure that the optimized Gaussian function is applicable to multiple sets of data, not just the training data.

[0116] In this embodiment, the expression of the Gaussian function is determined based on the bounding box coordinates, the projected two-dimensional coordinates, and the pre-constructed Gaussian function. The expression of the Gaussian function is then optimized so that it can better fit the actual data under the selected metric, thereby improving the fitting accuracy and fitting efficiency.

[0117] In one embodiment, refer to Appendix Figure 5 The S1000 includes:

[0118] S1020: Perform posterior estimation on the updated two-dimensional scene image processing result to obtain the posterior estimation result of the updated two-dimensional scene image processing result.

[0119] Following the above embodiments, the updated 2D scene image processing result includes all targets detected through 2D image information and 3D point cloud information, and is presented in the 2D scene image, including bounding box coordinates, bounding box size, and classification labels. In the field of target tracking, the goal of posterior estimation is to estimate or update the state or attributes of the target based on existing target detection results and other data, such as updating the target's position in the next frame.

[0120] Posterior estimation methods can include Kalman filtering, particle filtering, etc., and the choice can be based on the actual target detection task; no limitation is made here. Taking the application of Kalman filtering to the 2D scene image processing results of the current frame and the next frame as an example, the first step of Kalman filtering is to establish a dynamic model of the system, which can be represented as a state transition matrix, used to describe how the system state evolves over time, and using the state vector to estimate the system state. At each time step, by multiplying the current system state and the state transition matrix, the system state of the next time step can be predicted. This prediction is usually accompanied by uncertainty, also known as state covariance. Simultaneously, an observation model is established, represented as an observation matrix, used to describe the relationship between sensor measurements and the system state. For example, after receiving the 2D scene image processing result of the current frame, the observation model can be applied to the prediction of the current state to generate a new state estimate, and this estimate will be used as the prediction processing result of the 2D scene image for the next time step. The Kalman filtering algorithm can handle noise well and has relatively low computational overhead, making it suitable for real-time target tracking scenarios.

[0121] Taking the two-dimensional scene image processing results of the current frame and the next frame as examples, we can obtain the two-dimensional scene image prediction processing results of the current frame and the next frame by performing posterior estimation on the two-dimensional scene image processing results of the current frame and the next frame respectively.

[0122] S1040: Based on the posterior estimation results of the updated two-dimensional scene image processing results, perform binary allocation association to obtain the motion trajectories of multiple moving targets.

[0123] Binary correlation refers to the process in multi-target tracking where targets measured by different sensors at different times are associated with their trajectories, forming a binary correlation matrix. This matrix is ​​used to determine the correlation between the detection results and the trajectory, thereby achieving multi-target tracking. Besides the binary correlation algorithm, various other tracking algorithms can be used, such as Kalman filtering, the Hungarian algorithm, or multiple hypothesis tracking; these are not limited here.

[0124] Taking the application of the binary association algorithm to the prediction processing results of the 2D scene images of the current frame and the next frame as an example, the binary association algorithm is used to associate moving targets in the 2D scene images of different frames, and to establish a trajectory data structure for each target, including information such as the target's position, velocity, appearance, and identifier, and to generate the final associated trajectory. Typically, this trajectory uses the target's unique identifier as an index and can be used for further target analysis, prediction, or control. This process includes creating a binary matrix where rows represent target trajectories and columns represent target detection results. The elements of this binary matrix indicate whether each target detection result is associated with each trajectory, which can be determined by calculating a correlation score, such as Euclidean distance. Further, it determines how to associate the target measurement results with the trajectory, for example, using one or more algorithms such as linear assignment, Hungarian algorithm, or maximum weight matching.

[0125] Furthermore, for newly emerging targets, a new trajectory needs to be initialized. This could involve creating a unique ID for the new target and initializing the trajectory information. Since multi-target tracking may require storing a large amount of trajectory data, a trajectory storage strategy needs to be considered, including trajectory retention, deletion, or saving. Appropriate operations can be performed based on the actual target tracking task.

[0126] In this embodiment, a posterior estimation result of the updated 2D scene image processing result is obtained by performing posterior estimation on the updated 2D scene image processing result. Then, a binary assignment association is performed based on the updated posterior estimation result to obtain the motion trajectories of multiple moving targets. Generating continuous trajectories based on target detection results helps in understanding the movement and behavior of moving targets. Furthermore, using a binary assignment association algorithm allows moving targets to be associated with existing trajectories, avoiding repeated processing of the same moving target and reducing processing workload.

[0127] To provide a clearer explanation of the target tracking method provided in this application, a specific embodiment and accompanying drawings are described below. Figure 5 The specific embodiment includes the following steps:

[0128] S220: Acquire a set of two-dimensional scene images and scene point cloud data. The two-dimensional scene image dataset includes multiple consecutive two-dimensional scene images.

[0129] S2422: Input the two-dimensional scene image into the convolutional neural network to obtain the feature map and the first class classification label corresponding to the two-dimensional scene image. The convolutional neural network includes convolutional layers, pooling layers and the first class classification label.

[0130] S2424: Based on multi-scale feature maps, determine the bounding box coordinates and bounding box size of multiple detected moving targets.

[0131] S2426: Input the multi-scale feature map into the classifier to obtain the first class label. The classifier is used to compare the target features with the known target categories and assign labels to the target.

[0132] S244: Perform cross-intersection-union-comparison (CUC) cascade matching and posterior estimation on the initial image target detection results to obtain the image target detection results.

[0133] S262: Perform target detection on scene point cloud data to obtain point cloud target detection results. The point cloud target detection results include the three-dimensional point cloud coordinates and second classification labels of multiple detected moving targets.

[0134] S264: Project the 3D point cloud coordinates of multiple moving targets onto the pixel coordinate system to obtain the projected 2D coordinates of the multiple moving targets.

[0135] S420: Determine the expression of the plane Gaussian function based on the bounding box coordinates, the projected two-dimensional coordinates, and the pre-constructed plane Gaussian function.

[0136] S440: Calculate the fitting error of the plane Gaussian function expression; adjust the parameters of the plane Gaussian function expression based on the fitting error, and determine the adjusted plane Gaussian function expression as the target fitting relationship when the fitting error is minimized.

[0137] S600: Locate undetected moving targets in the 2D scene image and determine the bounding box coordinates of the undetected moving targets in the 2D scene image based on the projected 2D coordinates of the undetected moving targets and the target fitting relationship.

[0138] S800: Locate undetected moving targets in the 2D scene image and determine the bounding box coordinates of the undetected moving targets in the 2D scene image based on the projected 2D coordinates of the undetected moving targets and the target fitting relationship.

[0139] S1020: Perform posterior estimation on the updated two-dimensional scene image processing results to obtain the posterior estimation results of the updated two-dimensional scene image processing results.

[0140] S1040: Based on the posterior estimation results of the updated two-dimensional scene image processing, perform binary allocation association to obtain the motion trajectories of multiple moving targets.

[0141] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0142] Based on the same inventive concept, this application also provides a target tracking device for implementing the target tracking method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more target tracking device embodiments provided below can be found in the limitations of the target tracking method described above, and will not be repeated here.

[0143] In one embodiment, such as Figure 6 As shown, a target tracking device 500 is provided, including: an information acquisition module 510, a fitting relationship confirmation module 520, a target fitting module 530, a result update module 540, and a trajectory generation module 550, wherein:

[0144] The information acquisition module 510 is used to acquire two-dimensional scene image processing results and scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets.

[0145] The fitting relationship confirmation module 520 is used to obtain the target fitting relationship based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, as well as a pre-constructed Gaussian function. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the two-dimensional scene image based on the projected two-dimensional coordinates of the moving target.

[0146] The target fitting module 530 is used to find undetected moving targets in the two-dimensional scene image and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship.

[0147] The result update module 540 is used to update the two-dimensional scene image processing results based on the bounding box coordinates of the missed moving targets in the two-dimensional scene image.

[0148] The trajectory generation module 550 is used to track the motion trajectories of multiple moving targets based on the updated two-dimensional scene image processing results.

[0149] In one embodiment, the information acquisition module 510 is specifically used for:

[0150] A two-dimensional scene image set and scene point cloud data are acquired. The two-dimensional scene image dataset includes multiple consecutive frames of two-dimensional scene images. These images are input into a convolutional neural network (CNN) to obtain feature maps corresponding to the two-dimensional scene images. The CNN includes convolutional layers and pooling layers. A feature pyramid operation is performed on the feature maps to obtain multi-scale feature maps corresponding to the two-dimensional scene images. The feature pyramid operation is used to extract features from images at different scales. The multi-scale feature maps are input into a classifier to obtain a first-class classification label. The classifier is used to compare the target features with known target categories and assign labels to the target. The initial image target detection results are then subjected to cross-intersection over union (CIU) cascaded matching and posterior estimation processing to obtain the final image target detection result.

[0151] In one embodiment, the fitting relationship confirmation module 520 is specifically used for:

[0152] Based on the bounding box coordinates, projected 2D coordinates, and a pre-constructed planar Gaussian function, the expression for the planar Gaussian function is determined. The expression for the planar Gaussian function is then optimized to obtain the target fitting relationship.

[0153] In one embodiment, the trajectory generation module 550 is specifically used for:

[0154] A posterior estimation is performed on the updated 2D scene image processing results to obtain the posterior estimate of the updated 2D scene image processing results. Based on the posterior estimate of the updated 2D scene image processing results, a binary assignment association is performed to obtain the motion trajectories of multiple moving targets.

[0155] Each module in the aforementioned target tracking device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a target tracking method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0157] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described target tracking method embodiments.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described target tracking method embodiments.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described target tracking method embodiments.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0162] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A target tracking method, characterized in that, The method includes: The process involves obtaining two-dimensional scene image processing results and scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets. Based on the bounding box coordinates and projected 2D coordinates of the same moving target, and a pre-constructed Gaussian function, a target fitting relationship is obtained. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the 2D scene image according to the projected 2D coordinates of the moving target. Find the undetected moving targets in the two-dimensional scene image, and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship; The two-dimensional scene image processing result is updated based on the bounding box coordinates of the missed moving target in the two-dimensional scene image; Based on the updated two-dimensional scene image processing results, the motion trajectories of the multiple moving targets are tracked; The pre-constructed Gaussian function includes a pre-constructed plane Gaussian function; The target fitting relationship is obtained based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, as well as a pre-constructed Gaussian function, including: The expression for the plane Gaussian function is determined based on the bounding box coordinates, the projected two-dimensional coordinates, and the pre-constructed plane Gaussian function. Calculate the fitting error of the expression for the plane Gaussian function; Based on the fitting error, the parameters of the plane Gaussian function expression are adjusted. When the fitting error is minimized, the adjusted plane Gaussian function expression is determined as the target fitting relationship.

2. The method according to claim 1, characterized in that, The acquisition of the two-dimensional scene image processing results and scene point cloud processing results includes: Acquire a two-dimensional scene image set and scene point cloud data, wherein the two-dimensional scene image dataset includes multiple consecutive two-dimensional scene images; Target detection is performed on the two-dimensional scene image and the scene point cloud data respectively to obtain image target detection results and point cloud target detection results. The image target detection results include the bounding box coordinates, bounding box size and first classification label of multiple detected moving targets. The point cloud target detection results include the three-dimensional point cloud coordinates and second classification label of multiple detected moving targets. The three-dimensional point cloud coordinates of multiple moving targets are projected onto the pixel coordinate system to obtain the projected two-dimensional coordinates of the multiple moving targets.

3. The method according to claim 2, characterized in that, Target detection is performed on the two-dimensional scene image to obtain image target detection results, including: Initial target detection is performed on the two-dimensional scene image to obtain the initial image target detection result; The initial image target detection results are sequentially processed by cross-intersection over union (CIU) cascade matching and posterior estimation to obtain the image target detection results.

4. The method according to claim 3, characterized in that, The image target detection results include the bounding box coordinates, bounding box size, and first classification label of multiple detected moving targets; The initial target detection of the two-dimensional scene image to obtain the initial image target detection result includes: The two-dimensional scene image is input into a trained target detection network to obtain a feature map corresponding to the two-dimensional scene image and the first classification label. The target detection network includes a convolutional layer, a pooling layer and a classifier. Perform a feature pyramid operation on the feature map to obtain a multi-scale feature map corresponding to the two-dimensional scene image; Based on the multi-scale feature map, the bounding box coordinates and bounding box size of the multiple detected moving targets are determined; The trained object detection network is trained based on historical two-dimensional scene images carrying object classification labels.

5. The method according to any one of claims 1 to 4, characterized in that, The step of tracking the motion trajectories of the multiple moving targets based on the updated two-dimensional scene image processing results includes: A posterior estimation is performed on the updated two-dimensional scene image processing result to obtain the posterior estimation result of the updated two-dimensional scene image processing result; Based on the posterior estimation results, a binary assignment association is performed to obtain the motion trajectories of multiple moving targets.

6. A target tracking device, used in the target tracking method as described in any one of claims 1 to 5, characterized in that, The device includes: The information acquisition module is used to acquire the two-dimensional scene image processing results and the scene point cloud processing results. The two-dimensional scene image processing results include the bounding box coordinates of multiple detected moving targets, and the scene point cloud processing results include the projected two-dimensional coordinates obtained from the three-dimensional point cloud coordinates of the multiple detected moving targets. The fitting relationship confirmation module is used to obtain the target fitting relationship based on the bounding box coordinates and projected two-dimensional coordinates of the same moving target, as well as a pre-constructed Gaussian function. The target fitting relationship is used to determine the bounding box coordinates of the moving target in the two-dimensional scene image based on the projected two-dimensional coordinates of the moving target. The target fitting module is used to find undetected moving targets in the two-dimensional scene image and determine the bounding box coordinates of the undetected moving targets in the two-dimensional scene image based on the projected two-dimensional coordinates of the undetected moving targets and the target fitting relationship. The result update module is used to update the two-dimensional scene image processing result based on the bounding box coordinates of the missed moving target in the two-dimensional scene image. The trajectory generation module is used to track the motion trajectories of the multiple moving targets based on the updated two-dimensional scene image processing results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images