Method, apparatus and device for target tracking by spatial positioning in images

Abstract

This application provides a method, apparatus, and device for target tracking using spatial positioning in images, which obtains first position coordinates of multiple target detection boxes at a target time point, predicts second position coordinates of candidate boxes at a next time point for each target detection box using the first position coordinates, calculates multidimensional similarities between each target detection box and the multiple candidate boxes using the second position coordinates and the third position coordinates of the target detection boxes at the next time point, including multidimensional similarities determined by a dynamic threshold of center-of-gravity distances, and uses a preset matching algorithm to output target tracking results at the next time point based on the multidimensional similarities. This realizes comprehensive consideration of the target's position in world space, effectively solving the problem of inaccurate positioning using single position information in two-dimensional images, and can improve the accuracy of positioning of overlapping targets by utilizing the difference in center-of-gravity distances between distant and nearby targets in successive imaging.

Description

[Technical Field]

[0001] The present application relates to the technical field of computer vision, and in particular to a target tracking method, apparatus, device and storage medium. [Background technology]

[0002] Target tracking algorithms identify abnormal vehicle behavior and potential hazards while driving, helping autonomous vehicles make more accurate decisions. The main steps of current target tracking algorithms involve taking the position of a target detection box in a 2D image as input, and using deep learning to detect targets, Kalman prediction, and Hungarian matching to output the predicted position of the target's detection box in the next frame. However, because the position change information in the 2D image is single, it cannot cover both long-range and short-range target tracking in the real world. Furthermore, in complex road test scenes such as parking lots and crosswalks, there are a large number of targets, and they often overlap in the 2D imaging plane of the image, which makes it easy for matching errors to occur during matching, and the accuracy of the target tracking results cannot be guaranteed. Summary of the Invention [Problem to be solved by the invention]

[0003] To solve the above technical problems, the present application provides a method, apparatus and device for target tracking by spatial positioning in an image. [Means for solving the problem]

[0004] The present application provides, as a first aspect, acquiring first position coordinates of a plurality of target detection boxes at target times, the first position coordinates including first pixel coordinates and first world coordinates; predicting second position coordinates of a candidate box at a next time point for each of the target detection boxes using the first position coordinates; calculating multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes according to the second position coordinates and third position coordinates of the target detection boxes at a next time point, wherein the multidimensional similarities include multidimensional similarities determined by a dynamic threshold of centroid distances, and the third position coordinates are actually measured position coordinates; and outputting the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm.

[0005] In some embodiments of the first aspect, the step of calculating multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes using the second position coordinates and third position coordinates of the target detection boxes at a next time point includes: For each of the target detection boxes, using a preset dynamic threshold of centroid distance, screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes; and calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes according to the second position coordinates and the third position coordinates.

[0006] In some embodiments of the first aspect, the step of screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes using a preset dynamic threshold of a centroid distance for each of the target detection boxes includes: Calculating a first centroid distance between each of the target detection boxes and a target tracking device according to the world coordinates of the first centroid of the target detection box and the pixel coordinates of the second centroid of the candidate box, and calculating second centroid distances between each of the target detection boxes and a plurality of the candidate boxes; and determining a dynamic threshold value corresponding to the first center-of-gravity distance according to a preset dynamic threshold value table of center-of-gravity distances.

[0007] The preset dynamic threshold value table of the center of gravity distance is Labeling information on past training data and calculating the centroid distance for each pair of labeled target detection boxes; Calculating the distribution status of the centroid distances and dividing the similarity threshold to generate a dynamic threshold table of centroid distances; and performing dynamic verification and optimization of the dynamic threshold table of the centroid distance based on each of the acquired target detection boxes.

[0008] For each of the target detection boxes, target candidate boxes whose distance to the second centroid of the target detection box is greater than the dynamic threshold are screened, and at least one of the target candidate boxes corresponds to each of the target detection boxes.

[0009] In some embodiments of the first aspect, The step of calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes using the second position coordinates and the third position coordinates includes: calculating a detection box centroid offset similarity between each of the target detection boxes and the target candidate boxes according to the pixel coordinate of the third centroid of the target detection box and the pixel coordinate of the second centroid of the candidate box; calculating a detection box shape change similarity and a detection box area similarity between each of the target detection boxes and the target candidate boxes according to pixel coordinates of a third contour of the target detection box and pixel coordinates of a second contour of the candidate box; and calculating a world coordinate offset similarity between each of the target detection boxes and the target candidate boxes using the third world coordinate of the target detection box and the second world coordinate of the candidate box. The step of calculating the multi-dimensional similarity between each of the target detection boxes and the target candidate boxes further comprises: The method includes calculating using a multidimensional similarity weighting model, wherein the multidimensional similarity weighting model includes at least a distance measurement algorithm, an edge feature matching algorithm, and a structural similarity index algorithm.

[0010] For each of the target detection boxes, the centroid offset similarity of the detection box, the shape change similarity of the detection box, the area similarity of the detection box, and the offset similarity of the world coordinates are weighted to obtain the multidimensional similarity.

[0011] In some embodiments of the first aspect, the step of acquiring first position coordinates of the plurality of target detection boxes at the target time point includes: obtaining first pixel coordinates at target points of a plurality of the target detection boxes; generating, for each of the target detection boxes, target ground point coordinates of the target detection boxes according to the first pixel coordinates; The method includes a step of calculating a spatial coordinate in world space of the target ground point coordinate according to the calibration internal parameters and the calibration external parameters of the target tracking device, wherein the spatial coordinate is a first world coordinate at the target time point of the target detection box.

[0012] The step of detecting the plurality of target detection boxes includes: performing distortion correction on the image collected by the target tracking device, performing target detection on the corrected image to generate a target detection box, and performing a correction process on the target detection box; When target detection is performed on the corrected image, a deep learning algorithm is used.

[0013] In some embodiments of the first aspect, the step of predicting second position coordinates of a candidate box at a next time point for each of the target detection boxes using the first position coordinates includes: For each of the target detection boxes, the method includes using a Kalman filter algorithm to predict second position coordinates of a candidate box of the target detection box at a next time point based on a first position coordinate of the target detection box at a target time point and a fourth position coordinate of the target detection box at a previous time point, wherein the Kalman filter algorithm further includes optimizing parameters for a state transition matrix and process noise based on motion data of the target detection box, and inputting the first position coordinates and the fourth position coordinates into a preset motion model based on the optimized parameters to predict a candidate box at a next time point, wherein the preset motion model is any one of a constant velocity model, a random acceleration model, a steering model, and a nonlinear module.

[0014] In some embodiments of the first aspect, the step of outputting the target tracking result at the next time point based on the multidimensional similarity using a preset matching algorithm includes: Using a Hungarian matching algorithm, matching each of the target detection boxes with a corresponding final target candidate box at a next time point based on multi-dimensional similarity between each of the target detection boxes and a plurality of the candidate boxes; and outputting the final target candidate box as the target tracking result.

[0015] The present application also provides, as a second aspect, an acquisition module used to acquire first position coordinates at target times of a plurality of target detection boxes, the first position coordinates including first pixel coordinates and first world coordinates; a prediction module used to predict second position coordinates of a candidate box at a next time point for each of the target detection boxes according to the first position coordinates; a calculation module used to calculate a multidimensional similarity between each of the target detection boxes and a plurality of candidate boxes according to the second position coordinate and a third position coordinate of the target detection box at a next time point, wherein the multidimensional similarity includes a multidimensional similarity determined by a dynamic threshold of a centroid distance, and the third position coordinate is an actually measured position coordinate; and an output module, which is used for outputting the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm.

[0016] The present application also provides, as a third aspect, a computer device including a processor and a memory, the memory being used to store a computer program, which, when executed by the processor, realizes the method for tracking an object by spatial positioning in an image according to the first aspect.

[0017] The present application also provides, as a fourth aspect, a computer-readable storage medium having a computer program stored thereon, the computer program implementing the method for tracking a target by spatial positioning in an image according to the first aspect when executed by a processor. [Effects of the Invention]

[0018] Compared with the prior art, the present invention has at least the following beneficial effects: The first position coordinates of a plurality of target detection boxes are obtained at a target time point, and the first position coordinates include a first pixel coordinate and a first world coordinate, thereby comprehensively considering the position of the target in world space and effectively solving the problem that the position change characteristics of the two-dimensional image are single, thereby effectively tracking distant targets and nearby targets in the real world, and effectively improving the accuracy of target tracking in the problem that multiple targets overlap in the two-dimensional image in a complex scene. The first position coordinates are used to predict the second position coordinates of the candidate box at the next time point for each of the target detection boxes, which, combined with the target position prediction, can be used to determine the position relative to the target. The present invention improves the accuracy of target positioning, and solves the problem of inaccurate positioning using single position information of a two-dimensional image. Multidimensional similarities between each of the target detection boxes and multiple candidate boxes are calculated using the second position coordinates and the third position coordinates of the target detection boxes at a next time point, and the multidimensional similarities include multidimensional similarities determined by a dynamic threshold of center-of-gravity distances. A preset matching algorithm is used to output the target tracking result at the next time point based on the multidimensional similarities. This improves the accuracy of target positioning when multiple targets overlap by utilizing the differences in center-of-gravity distances between distant targets and nearby targets and between small targets and large targets in the two-dimensional continuous imaging. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a flowchart of a method for tracking a target by spatial localization in an image, as shown in an embodiment of the present application. [Figure 2] FIG. 10 is a schematic diagram showing the relationship between the center of gravity distance and the dynamic threshold value shown in the examples of the present application. [Figure 3] 1 is a structural schematic diagram of a target tracking device based on spatial positioning in an image shown in an embodiment of the present application; [Figure 4] FIG. 1 is a structural schematic diagram of a computer device shown in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0020] The technical solutions of the embodiments of the present application will be described below clearly and completely in conjunction with the drawings of the embodiments of the present application. Needless to say, the described embodiments are only a part of the embodiments of the present application, and are not all of the embodiments. Based on the embodiments of the present application, other embodiments that a person skilled in the art can obtain without inventive work all fall within the scope of protection of the present application.

[0021]

[0023] Referring to Figure 1, Figure 1 is a flowchart of a method for tracking a target using spatial localization in an image provided by an embodiment of the present application. The method for tracking a target using spatial localization in an image of the embodiment of the present application can be used in a computing device, including but not limited to an in-vehicle terminal, a smartphone, a laptop, a tablet PC, a desktop PC, a physical server, a cloud server, etc., and a target tracking device (e.g., a camera) is provided or connected to the computing device. As shown in Figure 1, the method for tracking a target using spatial localization in an image of the embodiment includes steps S101 to S104, which are described in detail below. In step S101, first position coordinates of a plurality of target detection boxes at target times are obtained, and the first position coordinates include first pixel coordinates and first world coordinates.

[0022] In this step, the target detection box is a target bounding box obtained by performing bounding box detection on the two-dimensional image collected by the target tracking device. Optionally, the bounding box detection can be realized by an instance segmentation algorithm, which will not be described in detail here. Preferably, the target tracking device is a surround-view fisheye camera.

[0023] The first position coordinates are a set of position coordinates of the target detection box, which are actual measurements at a target time, including, but not limited to, centroid pixel coordinates describing the center of the target detection box, contour pixel coordinates describing the boundary of the target detection box, centroid world coordinates describing the center of the target detection box in world space, contour world coordinates describing the boundary of the target detection box, etc. Optionally, the pixel coordinates may be obtained during bounding box detection, and the world coordinates may be obtained by converting the pixel coordinates based on the coordinate relationship between the image and the world space (e.g., camera internal parameters, extrinsic parameters).

[0024] In some embodiments, step S101 includes: obtaining first pixel coordinates at target points of a plurality of the target detection boxes; generating, for each of the target detection boxes, target ground point coordinates of the target detection boxes according to the first pixel coordinates; The method includes a step of calculating a spatial coordinate in world space of the target ground point coordinate according to the calibration internal parameters and the calibration external parameters of the target tracking device, wherein the spatial coordinate is a first world coordinate at the target time point of the target detection box.

[0025] Here, the step of detecting the plurality of target detection boxes includes: The image collected by the target tracking device is subjected to distortion correction to eliminate possible distortion in the image and ensure the accuracy of subsequent target detection. The distortion correction may be performed by resampling pixels using an interpolation method such as bilinear interpolation or bicubic interpolation, and for radial distortion, a distortion correction formula may be used to adjust pixel coordinates, and for tangential distortion, additional parameters may be used to correct.

[0026] Target detection is performed on the corrected image to generate a target detection box, and the target detection box is subjected to a correction process, which may include, but is not limited to, the following: Multidimensional similarity matching utilizes the above-mentioned multidimensional similarity evaluation method to match target detection boxes between adjacent frames, thereby modifying the position and shape of the detection boxes.

[0027] Post-processing techniques can be used to further optimize the target detection boxes, such as non-maximum suppression (NMS) to resolve overlapping boxes, or bounding box regression to fine-tune the box positions.

[0028] Target tracking: A target tracking algorithm is used to utilize the motion information of the target between successive frames to optimize the position of the detection box, thereby improving the stability and continuity of detection.

[0029] Shape adjustment: based on the shape characteristics of the target, further shape adjustments are made to the detection box, thereby more accurately depicting the target's outline.

[0030] The corrected target detection results can be evaluated, and the accuracy and coverage of the detection boxes can be evaluated using metrics such as the intersection-over-union ratio (IoU). Based on the evaluation results, the entire detection system can be optimized, including adjusting the parameters of the target detection model, optimizing the post-processing algorithm, and adjusting the weights of the multidimensional similarity.

[0031] Here, when detecting targets in the corrected image, deep learning algorithms such as, but not limited to, Faster R-CNN, YOLO, and SSD are used for processing.

[0032] In this embodiment, the target ground point is used to represent the position of the target in world space. Illustratively, a deep learning algorithm is used to identify the name of the target detection box in the image of the target frame and the pixel coordinates of the target detection box (including the coordinates x and y of the upper left corner vertex, and the length w and width h of the target detection box), and the target ground point P (Px, Py) is equivalently generated by the pixel coordinates (x, y, w, h) of the target detection box. Optionally, Px = (x + w × 1 / 2), Py = y + h, that is, the midpoint of the bottom of the target detection box is the target ground point.

[0033] According to calibration parameters such as the internal parameters and external parameters of the target tracking device, a world coordinate value in world space of the point P is calculated, which is used to indicate the position of the target in world space. Optionally, the target tracking device is a surround-view fisheye camera, and the target ground point P is a fisheye image coordinate, thereby converting the fisheye image into a undistorted image and finding the pixel coordinate point corresponding to the target ground point P in the undistorted image, where the undistorted image is regarded as a pinhole camera model, and the world coordinate of the point P can be obtained according to the principle of pinhole imaging, and the calculation formula may be as follows: TIFF0007776670000001.tif38170Here, (u,v) are the pixel coordinates of the target ground point, and Z c is a constant, and the first and second matrices on the right side of the equation can be combined to obtain the camera intrinsic parameter matrix, and the third matrix is ​​the camera extrinsic parameter matrix, (X w ,Y w ,Z w ) are the world coordinates of the target ground point.

[0034] In step S102, second position coordinates of a candidate box at the next time point for each of the target detection boxes are predicted based on the first position coordinates.

[0035] In this step, the second position coordinate is an estimated value at the next time point. Because there may be an error between the position of the target detection box obtained by bounding box detection and the actual position, if bounding box detection is performed on every image frame, the accumulated error will continue to increase. In this embodiment, the motion model is used to predict the second position coordinate of the next candidate box of the target detection box, thereby correcting the detection error through position prediction and reducing the accumulated error.

[0036] In some embodiments, step S102 includes, for each of the target detection boxes, using a Kalman filter algorithm to predict second position coordinates of a candidate box of the target detection box at a next time point based on a first position coordinate of the target detection box at a target time point and a fourth position coordinate of the target detection box at a previous time point, wherein the Kalman filter algorithm further includes optimizing parameters for a state transition matrix and process noise based on motion data of the target detection box, and inputting the first position coordinates and the fourth position coordinates into a preset motion model based on the optimized parameters to predict a candidate box at a next time point, wherein the preset motion model is any one of a constant velocity model, a random acceleration model, a steering model, and a nonlinear module.

[0037] In this embodiment, the Kalman filter algorithm predicts based on the actual measurement value (i.e., the position of the target detection box obtained by the bounding box detection algorithm) and the estimated value (i.e., the position of the target detection box estimated by the calculation model), so as to effectively reduce the accumulated error in the target tracking process and the estimation error of the motion model. Note that the Kalman filter algorithm may be any known algorithm, and detailed description thereof will be omitted.

[0038] In step S103, multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes are calculated according to the second position coordinates and the third position coordinates of the target detection boxes at the next time point, and the multidimensional similarities include multidimensional similarities determined by a dynamic threshold of the center of gravity distance.

[0039] In this step, the third position coordinate is the actual measurement value at the next time. The multi-dimensional similarity is a weighted result of similarities calculated in multiple dimensions, and the multiple dimensional similarities include, but are not limited to, the detection box centroid offset similarity, the detection box shape change similarity, the detection box area similarity, and the world coordinate offset similarity. The detection box centroid offset similarity is the centroid offset similarity between the target detection box and the candidate box, the detection box shape change similarity is the shape change similarity between the target detection box and the candidate box, the detection box area similarity is the area similarity between the target detection box and the candidate box, and the world coordinate offset similarity is the world coordinate offset similarity between the target detection box and the candidate box. Optionally, the second position coordinate and the third position coordinate are both coordinate sets, and the corresponding similarities can be calculated using corresponding coordinate values ​​representing the centroid, shape, area, and world coordinates in the second position coordinate and the third position coordinate. The similarity calculation formula may be, but is not limited to, cosine distance similarity, Euclidean distance similarity, Manhattan distance similarity, or the like.

[0040] The determination of the dynamic threshold for centroid distance is a process of using the dynamic threshold for centroid distance to screen target candidate boxes and use the target candidate boxes for multidimensional similarity calculation. It should be noted that there may be significant differences between the centroid distance of a distant target and the centroid distance of a nearby target, or between the centroid distance of a large target and the centroid distance of a small target, between two consecutive frame images. For example, the centroid distance between a distant target a in image A of a target frame and a distant target a in image B of the next frame is D1, and the centroid distance between a nearby target b in image A of a target frame and a nearby target b in image B of the next frame is D2. Although the distant target a and the nearby target b are both in the same image, the centroid distance D1 of the distant target a is significantly smaller than the centroid distance D2 of the nearby target b. Therefore, in this application, a dynamic threshold is set for different target types to screen target candidate boxes, thereby reducing the computational complexity of candidate box matching and improving matching accuracy.

[0041] In some embodiments, step S103 includes: For each of the target detection boxes, using a preset dynamic threshold of centroid distance, screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes; and calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes according to the second position coordinates and the third position coordinates.

[0042] In this embodiment, optionally, the step of screening the target candidate boxes comprises: Calculating a first centroid distance between each of the target detection boxes and a target tracking device according to the world coordinates of the first centroid of the target detection box and the pixel coordinates of the second centroid of the candidate box, and calculating second centroid distances between each of the target detection boxes and a plurality of the candidate boxes; determining a dynamic threshold value corresponding to the first center-of-gravity distance according to a preset dynamic threshold value table of center-of-gravity distances; Here, the preset dynamic threshold table of center of gravity distance further includes: The past training data is labeled with information, including the position information of the target detection box, category information, etc. For each pair of labeled target detection boxes, a centroid distance is calculated, which may preferably be calculated as Euclidean distance or other distance measure.

[0043] The distribution of the centroid distances is statistically analyzed, and the similarity threshold is divided to generate a dynamic threshold table for the centroid distances. When dividing the similarity threshold, the threshold can be adjusted according to actual needs, thereby ensuring that different targets can be accurately distinguished during the target detection process.

[0044] The dynamic threshold table for centroid distance is dynamically verified and optimized based on each acquired target detection box. For example, during the dynamic verification process, cases of false positives, i.e., cases in which different targets are considered the same target or the same target is considered different targets, are recorded. Based on the false positives, the thresholds in the dynamic threshold table for centroid distance are dynamically adjusted, so that the system can more accurately distinguish targets in subsequent detections. The results of each dynamic optimization are fed back to the target detection system, thereby updating the system parameters or algorithms, thereby improving the robustness and accuracy of the system. By continuously monitoring the performance of the target detection system and periodically evaluating the system, the strategy for updating the dynamic threshold table for centroid distance is further optimized based on the evaluation results, so that the system can operate normally regardless of the scene or data. The dynamic threshold table for centroid distance is dynamically verified and optimized based on each acquired target detection box, thereby improving the performance and effectiveness of the target detection system.

[0045] For each of the target detection boxes, target candidate boxes whose distance to the second centroid of the target detection box is greater than the dynamic threshold are screened, and at least one of the target candidate boxes corresponds to each of the target detection boxes.

[0046] In this optional embodiment, a first centroid distance between the target detection box and the target tracking device is calculated according to the world coordinates of the first centroid and the world coordinates of the target tracking device, thereby determining whether the target detection box is a distant target or a nearby target; a corresponding dynamic threshold is determined according to the first centroid distance; the dynamic threshold is used to screen candidate boxes of the target detection box, thereby screening at least one target candidate box corresponding to the target detection box, and efficiently eliminating other candidate boxes that obviously do not meet the conditions of the candidate box of the target detection box; further reducing the calculation amount of the subsequent similarity calculation process and matching process; and reducing the interference of invalid candidate boxes in the matching process, thereby improving the accuracy of target tracking. For example, with respect to the dynamic threshold table of center of gravity distance, reference is made to the schematic diagram of the relationship between the dynamic threshold of center of gravity distance shown in FIG. 2, and the dynamic threshold table of center of gravity distance is represented in the schematic diagram. When the first center of gravity distance (target world distance) is 3 meters, the dynamic threshold is 3, and when the first center of gravity distance (target world distance) is 15 meters, the dynamic threshold is 3.

[0047] Optionally, the step of calculating the multi-dimensional similarity comprises: Calculating a detection box centroid offset (box_IOU) similarity between each of the target detection boxes and the target candidate boxes according to the pixel coordinates of the third centroid of the target detection box and the pixel coordinates of the second centroid of the candidate box; Calculating a detection box shape change (box_SHAPE) similarity and a detection box area (box_AREA) similarity between each of the target detection boxes and the target candidate boxes according to pixel coordinates of the third contour of the target detection box and pixel coordinates of the second contour of the candidate box; and calculating a world coordinate offset similarity between each of the target detection boxes and the target candidate boxes using the third world coordinate of the target detection box and the second world coordinate of the candidate box. Specifically, the step of calculating the multi-dimensional similarity between each of the target detection boxes and the target candidate boxes further includes: calculating using a multi-dimensional similarity weighting model, wherein said multi-dimensional similarity weighting model includes at least: A distance measurement algorithm such as Euclidean distance or Manhattan distance is used to calculate the spatial distance between the target detection box and the target candidate box.

[0048] An edge feature matching algorithm such as Canny edge detection is used to extract edge features of the target detection box and the target candidate box, and then an appropriate matching algorithm such as feature point matching, edge line matching, etc. is used to calculate the edge feature similarity between them.

[0049] A structural similarity index algorithm assesses the overall similarity of two images by comparing their brightness, contrast, and structural similarity.

[0050] For each of the target detection boxes, the centroid offset similarity of the detection box, the shape change similarity of the detection box, the area similarity of the detection box, and the offset similarity of the world coordinates are weighted to obtain the multidimensional similarity.

[0051] In this optional embodiment, the coordinate parameters of the candidate box are optionally divided by the coordinate parameters of the target detection box to obtain a similarity parameter.

[0052] In step S104, a preset matching algorithm is used to output the target tracking result at the next time point based on the multi-dimensional similarity.

[0053] In this step, since there may be multiple target detection boxes in two consecutive frame images, it becomes difficult to determine the target detection box in the image of the next frame that corresponds to the target target detection box. For example, if there are 20 target detection boxes in both consecutive images A and B, and even if the second position coordinates are obtained by predicting one of the target detection boxes a in image A, it is difficult to directly determine which target detection box in image B at the next point in time the second position coordinates correspond to. Therefore, in the present application, a preset matching algorithm is used to match the correspondence between the multiple target detection boxes in the image of the target frame and the multiple candidate boxes in the image of the next frame.

[0054] In some embodiments, step S104 includes using a Hungarian matching algorithm to match each of the target detection boxes with a corresponding final target candidate box at a next time point based on multi-dimensional similarities between each of the target detection boxes and a plurality of the candidate boxes, and outputting the final target candidate box as the target tracking result.

[0055] In this embodiment, all the multi-dimensional similarities obtained in step S103 are input into the Hungarian matching algorithm, which traverses and loops through each target detection box and each candidate box to obtain the candidate box corresponding to each target detection box.

[0056] In order to implement the method for tracking a target by spatial positioning in an image corresponding to the above method embodiment and achieve the corresponding functions and technical effects, please refer to Figure 3, which shows a structural block diagram of the target tracking device by spatial positioning in an image provided by the embodiment of the present application. For convenience of explanation, only the parts related to this embodiment are shown, and the target tracking device by spatial positioning in an image provided by the embodiment of the present application is as follows: An acquisition module 301 is used to acquire first position coordinates of a plurality of target detection boxes at target times, the first position coordinates including first pixel coordinates and first world coordinates; a prediction module 302 used to predict second position coordinates of a candidate box at a next time point for each of the target detection boxes according to the first position coordinates; a calculation module 303 used to calculate multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes according to the second position coordinates and third position coordinates of the target detection boxes at a next time point, wherein the multidimensional similarities include multidimensional similarities determined by a dynamic threshold of centroid distances, and the third position coordinates are actually measured position coordinates; and an output module 304 adapted to output the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm.

[0057] In some embodiments, the computing module 303: a screening module used for screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes using a preset dynamic threshold of centroid distance for each of the target detection boxes; a calculation module used to calculate a multi-dimensional similarity between each of the target detection boxes and the target candidate boxes according to the second position coordinates and the third position coordinates.

[0058] In some embodiments, the screening module specifically comprises: Calculating a first centroid distance between each of the target detection boxes and a target tracking device according to the world coordinates of the first centroid of the target detection box and the pixel coordinates of the second centroid of the candidate box, and calculating second centroid distances between each of the target detection boxes and a plurality of the candidate boxes; determining a dynamic threshold value corresponding to the first center-of-gravity distance according to a preset dynamic threshold value table of center-of-gravity distances; For each of the target detection boxes, the dynamic threshold is used to screen target candidate boxes whose distance from the second center of gravity of the target detection box is greater than the dynamic threshold, and each of the target detection boxes corresponds to at least one of the target candidate boxes.

[0059] In some embodiments, the calculation module specifically: Calculating a detection box centroid offset similarity between each of the target detection boxes and the target candidate boxes according to the pixel coordinate of the third centroid of the target detection box and the pixel coordinate of the second centroid of the candidate box; Calculating a detection box shape change similarity and a detection box area similarity between each of the target detection boxes and the target candidate boxes according to pixel coordinates of a third contour of the target detection box and pixel coordinates of a second contour of the candidate box; Calculating a world coordinate offset similarity between each of the target detection boxes and the target candidate boxes using the third world coordinate of the target detection box and the second world coordinate of the candidate box; For each target detection box, the center of gravity offset similarity of the detection box, the shape change similarity of the detection box, the area similarity of the detection box, and the world coordinate offset similarity are weighted to obtain the multidimensional similarity.

[0060] In some embodiments, the acquisition module 301 specifically includes: Obtaining first pixel coordinates at target points of the plurality of target detection boxes; For each of the target detection boxes, generating target ground point coordinates of the target detection box according to the first pixel coordinates; The calibration internal parameters and calibration external parameters of the target tracking device are used to calculate the spatial coordinates of the target ground point coordinates in world space, and the spatial coordinates are the first world coordinates of the target detection box at the target time point.

[0061] In some embodiments, the prediction module 302 specifically: For each of the target detection boxes, a Kalman filter algorithm is used to predict the second position coordinate of a candidate box at the next time point of the target detection box based on the first position coordinate of the target detection box at the target time point and the fourth position coordinate at the previous time point.

[0062] In some embodiments, the output module 304 specifically: Using a Hungarian matching algorithm, matching each of the target detection boxes with a corresponding final target candidate box at a next time point based on multi-dimensional similarity between each of the target detection boxes and a plurality of the candidate boxes; The final target candidate box is used to output the target tracking result.

[0063] The target tracking device using spatial positioning in an image can implement the target tracking method using spatial positioning in an image of the above method embodiment. The optional items in the above method embodiment also apply to this embodiment, so detailed descriptions will be omitted here. Other contents related to the embodiment of the present application may refer to the contents of the above method embodiment, and detailed descriptions will be omitted in this embodiment.

[0064] 4 is a structural schematic diagram of a computing device provided by one embodiment of the present application. As shown in FIG. 4, the computing device 4 of this embodiment includes at least one processor 40 (only one is shown in FIG. 4), a memory 41, and a computer program 42 stored in the memory 41 and executable by the at least one processor 40, and when the processor 40 executes the computer program 42, steps of any of the above method embodiments are realized.

[0065] The computer device 4 may be a computing device such as an in-vehicle terminal, a smartphone, a tablet PC, a desktop PC, a cloud server, etc. The computer device may include, but is not limited to, a processor 40 and a memory 41. It will be understood by those skilled in the art that FIG. 4 is merely an example of the computer device 4 and does not constitute a limitation on the computer device 4. The computer device 4 may include more or fewer components than those shown in the figure, or may combine several or different components, and may further include, for example, an input / output device, a network access device, etc.

[0066] The so-called processor 40 may be a central processing unit (CPU), or it may also be another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0067] In some embodiments, the memory 41 may be an internal storage unit of the computing device 4, such as a hard disk or memory of the computing device 4. In some embodiments, the memory 41 may also be an external storage unit of the computing device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, or a Flash card attached to the computing device 4. Furthermore, the memory 41 may include both an internal storage unit of the computing device 4 and an external storage unit. The memory 41 may be used to store, for example, program code of the computer program, such as an operating system, application programs, a boot loader program, data, other programs, etc. The memory 41 may also be used to temporarily store data that has been output or is to be output.

[0068] Furthermore, an embodiment of the present application provides a computer-readable storage medium having a computer program stored therein, the computer program implementing the steps of any of the above method embodiments when executed by a processor.

[0069] An embodiment of the present application provides a computer program product, which, when executed on a computer device, causes the computer device to implement the steps of each of the above method embodiments.

[0070] In some embodiments provided by the present application, each block in a flowchart or block diagram may represent a module, program segment, or part of code, and it can be understood that the module, program segment, or part of code includes one or more executable instructions for implementing a predetermined logical function. It should be noted that in some alternative embodiments, the functions shown in the blocks may occur in a different order than shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, or may even be executed in the reverse order, as determined by the target functionality.

[0071] The functions may be realized in the form of software functional modules and stored in a computer-readable storage medium when sold or used as an independent product. In this regard, the spirit of the technical solution of the present application or a portion of the technical solution that contributes to the prior art may appear in the form of a software product, which is stored in a storage medium and includes several instructions that enable a computer device to perform some or all of the steps of the methods described in each embodiment of the present application. The storage medium includes various media that can store program code, such as USB memory, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, and optical disk.

[0072] The above-mentioned specific examples further illustrate the objectives, technical solutions and beneficial effects of the present application. It should be noted that the above-mentioned examples are merely specific examples of the present application and are not intended to limit the scope of protection of the present application. However, any modifications, equivalent replacements, improvements, etc. that are within the scope of the spirit and principles of the present application that are apparent to those skilled in the art shall be included in the scope of protection of the present application.

Claims

1. acquiring first position coordinates of a plurality of target detection boxes at target times, the first position coordinates including first pixel coordinates and first world coordinates; predicting second position coordinates of a candidate box at a next time point for each of the target detection boxes using the first position coordinates; calculating multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes according to the second position coordinates and third position coordinates of the target detection boxes at a next time point, wherein the multidimensional similarities include multidimensional similarities determined by a dynamic threshold of centroid distances, and the third position coordinates are actually measured position coordinates; and outputting the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm.

2. the step of calculating multidimensional similarities between each of the target detection boxes and a plurality of candidate boxes using the second position coordinates and third position coordinates of the target detection boxes at a next time point, For each of the target detection boxes, using a preset dynamic threshold of centroid distance, screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes; and calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes using the second position coordinates and the third position coordinates.

3. the step of screening at least one target candidate box corresponding to the target detection box from the plurality of candidate boxes using a preset dynamic threshold of a centroid distance for each of the target detection boxes, Calculating a first centroid distance between each of the target detection boxes and a target tracking device according to the world coordinates of the first centroid of the target detection box and the pixel coordinates of the second centroid of the candidate box, and calculating second centroid distances between each of the target detection boxes and a plurality of the candidate boxes; determining a dynamic threshold value corresponding to the first centroid distance according to a preset centroid distance dynamic threshold value table; 3. The method for tracking a target by spatial localization in an image according to claim 2, further comprising: for each of the target detection boxes, screening target candidate boxes whose distance to the second centroid of the target detection box is greater than the dynamic threshold, wherein at least one of the target candidate boxes corresponds to each of the target detection boxes.

4. The preset dynamic threshold table of center of gravity distances is Labeling information on past training data and calculating the centroid distance for each pair of labeled target detection boxes; Calculating the distribution status of the centroid distances and dividing the similarity threshold to generate a dynamic threshold table of centroid distances; and performing dynamic verification and optimization of the dynamic threshold table of centroid distances based on each acquired target detection box.

5. the step of calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes using the second position coordinates and the third position coordinates, 3. The method for target tracking by spatial positioning in an image according to claim 2, further comprising the step of calculating a detection box centroid offset similarity between each of the target detection boxes and the target candidate boxes according to a pixel coordinate of a third centroid of the target detection box and a pixel coordinate of a second centroid of the candidate box.

6. The step of calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes using the second position coordinates and the third position coordinates further includes:

6. The method for target tracking by spatial positioning in an image according to claim 5, further comprising the step of calculating a detection box shape change similarity and a detection box area similarity between each of the target detection boxes and the target candidate boxes using pixel coordinates of a third contour of the target detection box and pixel coordinates of a second contour of the candidate box.

7. The step of calculating a multidimensional similarity between each of the target detection boxes and the target candidate boxes using the second position coordinates and the third position coordinates further includes:

7. The method for target tracking by spatial positioning in an image according to claim 6, further comprising the step of calculating a world coordinate offset similarity between each of the target detection boxes and the target candidate boxes using a third world coordinate of the target detection box and a second world coordinate of the candidate box.

8. the step of calculating a multi-dimensional similarity between each of the target detection boxes and the target candidate boxes comprises: calculating using a multidimensional similarity weighting model, wherein the multidimensional similarity weighting model includes at least a distance measurement algorithm, an edge feature matching algorithm, and a structural similarity index algorithm; and weighting, for each of the target detection boxes, the centroid offset similarity of the detection box, the shape change similarity of the detection box, the area similarity of the detection box, and the world coordinate offset similarity to obtain the multidimensional similarity.

9. The step of obtaining first position coordinates of a plurality of target detection boxes at a target time point includes: obtaining first pixel coordinates of a plurality of the target detection boxes at target time points; for each of the target detection boxes, generating target ground point coordinates of the target detection box using the first pixel coordinates; 2. The method for target tracking by spatial positioning in an image according to claim 1, further comprising: a step of calculating spatial coordinates in world space of the target ground point coordinates according to calibration internal parameters and calibration external parameters of a target tracking device, wherein the spatial coordinates are first world coordinates at a target time point of the target detection box.

10. Further, the step of detecting the plurality of target detection boxes includes: performing distortion correction on the image collected by the target tracking device, performing target detection on the corrected image to generate a target detection box, and performing a correction process on the target detection box; The method for tracking a target by spatial positioning in an image according to claim 9, characterized in that a deep learning algorithm is used to process the target detection in the corrected image.

11. the step of predicting second position coordinates of a candidate box at a next time point for each of the target detection boxes using the first position coordinates, 2. The method for target tracking by spatial localization in an image according to claim 1, further comprising the step of: for each of the target detection boxes, predicting a second position coordinate of a candidate box at a next time point of the target detection box based on a first position coordinate of the target detection box at a target time point and a fourth position coordinate of the target detection box at a previous time point using a Kalman filter algorithm.

12. The Kalman filter algorithm:

12. The method of claim 11, further comprising: optimizing parameters for a state transition matrix and a process noise based on the motion data of the target detection box; and inputting the first position coordinate and the fourth position coordinate into a preset motion model based on the optimized parameters to predict a candidate box at a next time point, wherein the preset motion model is one of a constant velocity model, a random acceleration model, a steering model, or a nonlinear module.

13. the step of outputting the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm, Using a Hungarian matching algorithm, matching each of the target detection boxes with a corresponding final target candidate box at a next time point based on multi-dimensional similarity between each of the target detection boxes and a plurality of the candidate boxes; and outputting the final target candidate box as the target tracking result.

14. an acquisition module used to acquire first position coordinates at target times of a plurality of target detection boxes, the first position coordinates including first pixel coordinates and first world coordinates; a prediction module used to predict second position coordinates of a candidate box at a next time point for each of the target detection boxes according to the first position coordinates; a calculation module used to calculate a multidimensional similarity between each of the target detection boxes and a plurality of candidate boxes according to the second position coordinate and a third position coordinate of the target detection box at a next time point, the multidimensional similarity including a multidimensional similarity determined by a dynamic threshold of a centroid distance, and the third position coordinate being an actually measured position coordinate; and an output module adapted to output the target tracking result at the next time point based on the multi-dimensional similarity using a preset matching algorithm.

15. A computer device comprising a processor and a memory, the memory being adapted to store a computer program, the computer program being executed by the processor to implement the method for tracking an object by spatial positioning in an image according to any one of claims 1 to 7.

Images