Target tracking method and device for multi-sensing equipment

Abstract

The invention provides a target tracking method for multi-sensing equipment. The target tracking method comprises the following steps of: performing near-end tracking on a target in a sensing range based on near-end data of an acquisition sensor; the data acquired by the acquisition sensor comprises near-end data and far-end data; establishing a track list of the targets, wherein tracks of the targets are stored in the track list; performing maintenance optimization on the trajectory based on the time and the pose of the target to obtain an optimized trajectory; and performing far-end tracking on the target by using the optimized trajectory and the far-end data, wherein the near-end data is closer to the data acquisition sensor than the far-end data. According to the method, a multi-target detection positioning algorithm framework based on graph optimization is established through establishing a front-end tracking and rear-end optimization thought for multi-target detection, and stable detection tracking of multi-target perception is realized. Information of multiple base stations is organically spliced, real-time positioning and tracking binding are carried out on a target moving in the multi-base-station sensing system, and the high detection rate and the low ID hopping rate are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and particularly relates to a target tracking method, device, terminal and computer-readable storage medium for multi-sensing devices. Background Art

[0002] According to the design principle of sensors, whether it is lidar or camera, there is the characteristic of "objects appear larger when they are closer and smaller when they are farther away". For cameras, this characteristic is manifested in that after the target moves far away, the pixel space it occupies shrinks significantly. For lidar, this characteristic is manifested in that the point cloud of distant targets becomes very sparse. This common characteristic results in the base station being unable to perform reliable target detection at a distance, reducing the detection rate and accuracy.

[0003] When the target moves to the intersection between base stations, the situation where both base stations lose the target simultaneously may occur. When the target continues to move and appears in the stable detection field of view of the next base station, it is already impossible to complete the binding with the target detected by the previous base station, and at this time, the detection consistency problem between base stations appears. Based on the perception information stitching software algorithm architecture of independent base stations, the sub-node information of the continuously arranged multi-base station detection system is not organically utilized as a whole.

[0004] Therefore, how to design an algorithm software architecture based on the overall system to organically stitch the information of multiple base stations, perform real-time positioning and tracking binding of the target moving in the multi-base station perception system, and achieve a high detection rate and a low ID jump rate. Summary of the Invention

[0005] The purpose of the present invention is to propose a target tracking method for multi-sensing devices, so as to solve the problem that the current cross-device target tracking is prone to loss.

[0006] The present invention proposes a target tracking method for multi-sensing devices, and the method includes:

[0007] Performing proximal tracking on the target within the sensing range based on the proximal data collected by the acquisition sensor; the data collected by the acquisition sensor includes proximal data and distal data;

[0008] Establishing a trajectory list of the target, where the trajectories of multiple targets are stored in the trajectory list;

[0009] Maintaining and optimizing the trajectory based on time and the pose of the target to obtain an optimized trajectory;

[0010] Performing distal tracking on the target using the optimized trajectory and distal data, where the proximal data is closer to the data acquisition sensor than the distal data.

[0011] In one alternative embodiment, a trajectory list of targets is established, and the trajectories of multiple targets are stored in the trajectory list, including: performing downsampling storage operations according to time and pose change thresholds.

[0012] In one alternative embodiment, the trajectory is maintained and optimized based on time and the pose of the target to obtain an optimized trajectory, including:

[0013] Obtaining the trajectory state of the target;

[0014] Selecting a distal trajectory list that meets the conditions according to the trajectory state for trajectory optimization of the target.

[0015] In one alternative embodiment, taking the trajectory state of the target includes: a leaving state and / or an occlusion state.

[0016] In one alternative embodiment, when the trajectory state is the leaving state, correspondingly, selecting a distal trajectory list that meets the conditions according to the trajectory state for trajectory optimization of the target, including:

[0017] Fitting the historical poses of the leaving target using a first-order Bezier curve, and judging whether there is a mis-match according to the fitting error; listing the targets with mis-matches in the distal trajectory list, reselecting the allocation result at the mis-match, and correcting the target ID.

[0018] In one alternative embodiment, when the trajectory state is the occlusion state, correspondingly, selecting a distal trajectory list that meets the conditions according to the trajectory state for trajectory optimization of the target, including:

[0019] Predicting all subsequent poses of the target according to the first-order Bezier function; achieving double re-matching of curves and IOU with the newly created trajectory in the trajectory list to complete broken-chain reconnection.

[0020] In one alternative embodiment, the method further includes: for the curves within the fitting error, establishing an optimization equation based on a graph to optimize the pose parameters and output the optimized pose of the target.

[0021] The present invention also proposes a target tracking device for multi-sensing devices, including:

[0022] A tracking module for performing proximal tracking on the targets within the sensing range based on proximal data of the acquisition sensor; the data collected by the acquisition sensor includes proximal data and distal data;

[0023] A storage module for storing the established trajectory list of targets, and the trajectories of multiple targets are stored in the trajectory list;

[0024] A trajectory optimization module, configured to maintain and optimize the trajectory based on the time and the pose of the target, so as to obtain an optimized trajectory;

[0025] The tracking module is further configured to perform remote tracking on the target by using the optimized trajectory and remote data, where the proximal data is closer to the data acquisition sensor than the remote data

[0026] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the method provided in any embodiment of the present invention.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] A target tracking method for multi-sensing devices provided by the present invention performs proximal tracking on a target within a sensing range based on proximal data of an acquisition sensor; the data acquired by the acquisition sensor includes proximal data and remote data; a trajectory list of the target is established, and the trajectory list stores the trajectories of multiple targets; the trajectory is maintained and optimized based on the time and the pose of the target to obtain an optimized trajectory; the target is remotely tracked by using the optimized trajectory and remote data, where the proximal data is closer to the data acquisition sensor than the remote data. This method establishes an idea of front-end tracking plus back-end optimization through multi-target detection, establishes a multi-target detection and positioning algorithm framework based on graph optimization, and realizes stable detection and tracking of multi-target perception. The information of multiple base stations is organically stitched together, and the real-time positioning and tracking binding of the target moving in the multi-base station sensing system are realized, so as to achieve a high detection rate and a low ID jump rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0030] Figure 1 is a flowchart of target tracking of multi-sensing devices in an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of a global tracking assignment strategy in an embodiment of the present invention;

[0032] Figure 3 is a schematic structural diagram of a target tracking device for multi-sensing devices in an embodiment of the present invention;

[0033] Figure 4 is a schematic structural diagram of a terminal in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0036] The present invention proposes a target tracking method for multi-sensing devices. This method is applicable to target tracking in a relatively large range. The sensing devices in the applicable sensing system can be lidar, millimeter-wave radar, cameras, or a combination of these sensors. The relatively large range described in this application generally refers to a range of 1 kilometer or more. See Figure 1 , this method includes:

[0037] Step S101: Perform proximal tracking on the targets within the sensing range based on the proximal data collected by the acquisition sensors; the data collected by the acquisition sensors includes proximal data and distal data.

[0038] Specifically: Any sensor can be used, such as lidar including radars with wavelengths of 905nm, 1550nm, etc.; millimeter-wave radar including radars in 77G, 70G, 24G frequency bands, etc.; images include RGB color cameras or near-infrared cameras. Use a simple deep learning-based detector to achieve target detection. For lidar, detectors such as pointpillars, second detectors, etc. are used; for millimeter-wave radar, cfar detectors, etc. are used; for images, yolov5 detectors, etc. are used. Output the target detection results at the front end n*[x,y,z,w,h,l,a]. Where n represents the number of targets in the current frame, and x, y, z, w, h, l, a respectively represent the three-dimensional coordinates, length, width, height, and heading angle of the target.

[0039] The tracking in this embodiment can adopt existing tracking algorithms. The Markov tracking algorithm can be selected. At this time, the Markov-based target tracking includes:

[0040] First, establish a state transition model for the input results:

[0041]

[0042] Among them, X is the state of the target, is the state probability distribution of the target; t is the current moment, and t-1 is the previous moment. For all newly born targets and existing targets Use the state at the previous moment to calculate the state at the current moment. The state transition function of each target satisfies the superposition distribution of the Poisson point process and the Bernoulli distribution .

[0043] Secondly, the observation model:

[0044]

[0045] Among them, X is the state of the target, is the state probability distribution of the target; t is the current moment, and t-1 is the previous moment. For all newly born targets and existing targets Use the state at the previous moment to calculate the state at the current moment. The state transition function of each target satisfies the superposition distribution of the Poisson point process and the Bernoulli distribution .

[0046] Construct a tracking assignment strategy based on global hypotheses. The assignment strategy is as Figure 2 shown. Save the top N local hypotheses and update the detected target state quantity through the Kalman filter.

[0047] Step S102; establish a trajectory list of the target. The trajectory list stores the trajectories of multiple targets. Establish and maintain a multi-target trajectory list at the back end. This trajectory can store the trajectories of M targets with the top N hypotheses and is a large storage database. Maintain and optimize the trajectory based on time and the pose of the target to obtain an optimized trajectory; optionally, set the frame number interval to k and the pose change threshold to e.

[0048] Step S103; maintain and optimize the trajectory based on time and the pose of the target to obtain an optimized trajectory; use the optimized trajectory and remote data to perform remote tracking on the target, where the proximal data is closer to the data acquisition sensor than the remote data. Repeat the above process of S101 - S103 to perform offline target trajectory tracking maintenance and optimization, and finally output the offline multi-target tracking result from the back end.

[0049] The target tracking method of the multi-sensing device in this application establishes an idea of front-end tracking plus back-end optimization through multi-target detection, establishes a multi-target detection and positioning algorithm framework based on graph optimization, and realizes stable detection and tracking of multi-target perception. Organically splice the information of multiple base stations, bind the real-time positioning and tracking of the targets moving in this multi-base station sensing system, and achieve a high detection rate and a low ID jump rate.

[0050] In one embodiment, the trajectory is maintained and optimized based on the pose of the target over time to obtain an optimized trajectory, including: obtaining the trajectory state of the target; selecting a list of distal trajectories that meet the conditions according to the trajectory state, and performing trajectory optimization of the target.

[0051] Optionally, the trajectory state of the target includes: a leaving state and / or an occlusion state.

[0052] When the trajectory state is the leaving state, correspondingly, selecting a list of distal trajectories that meet the conditions according to the trajectory state, and performing trajectory optimization of the target, including: fitting the historical pose of the leaving target using a first-order Bezier curve, and judging whether there is a mis-match according to the fitting error; listing the targets with mis-matches in the list of distal trajectories, re-selecting the allocation result at the mis-match, and correcting the target ID.

[0053] When the trajectory state is the occlusion state, correspondingly, selecting a list of distal trajectories that meet the conditions according to the trajectory state, and performing trajectory optimization of the target, including: predicting all subsequent poses of the target according to the first-order Bezier function; achieving double re-matching of the curve and IOU with the newly established trajectory in the trajectory list to complete broken-chain reconnection.

[0054] Optionally, for the curve within the fitting error, an optimization equation based on a graph is established to optimize the pose parameters, and the optimized pose is output as the optimized pose of the target.

[0055] Please refer to Figure 3 , Figure 3 FIG. is a schematic structural diagram of a target tracking device 10 of a multi-sensing device provided in an embodiment of the present invention. In this embodiment, each module included in the target tracking device of the multi-sensing device is used to execute each step in the corresponding embodiment of the present invention. For details, please refer to the relevant descriptions in the corresponding embodiments. For the sake of convenience of description, only the parts related to this embodiment are shown. Refer to Figure 3 FIG., the target tracking device 10 of the multi-sensing device may include: a tracking module 100, configured to perform proximal tracking on a target within a sensing range based on proximal data of an acquisition sensor; the data acquired by the acquisition sensor includes proximal data and distal data;

[0056] a storage module 200, configured to store a list of trajectories of the established target, and the list of trajectories stores the trajectories of multiple targets;

[0057] The tracking module 100 is further configured to maintain and optimize the trajectory based on the pose of the target over time to obtain an optimized trajectory; and perform distal tracking on the target using the optimized trajectory and distal data, where the proximal data is closer to the data acquisition sensor than the distal data.

[0058] As an alternative embodiment, the storage module 200 is configured to perform downsampling storage operations according to time and pose change thresholds.

[0059] As an alternative embodiment, the tracking module 100 is configured to obtain the trajectory state of the target; select a list of distal trajectories that meet the conditions according to the trajectory state, and perform trajectory optimization of the target.

[0060] As an alternative embodiment, when the trajectory state is the leaving state, the tracking module 100 is configured to fit the historical poses of the leaving target using a first-order Bezier curve, and determine whether a mis-match occurs according to the fitting error; list the targets with mis-matches in the distal trajectory list, re-select the allocation result at the mis-match, and correct the target ID.

[0061] As an alternative embodiment, when the trajectory state is the occlusion state, the tracking module 100 is configured to predict all subsequent poses of the target according to the first-order Bezier function; perform double re-matching of the curve and IOU with the newly created trajectory in the trajectory list to complete broken-chain reconnection.

[0062] As an alternative embodiment, the tracking module 100 is configured to establish a graph-based optimization equation for the curve within the fitting error, optimize the pose parameters, and output the optimized pose of the target.

[0063] Figure 4 FIG. is a schematic structural diagram of a terminal device provided by an embodiment of the present invention. As Figure 4 shown, the terminal device 700 of this embodiment includes: a processor 710, a memory 720, and a computer program 730 stored in the memory 720 and executable on the processor 710, such as a program for the target tracking method of a multi-sensing device. When the processor 710 executes the computer program 730, the steps in the respective embodiments of the above-mentioned target tracking method for multi-sensing devices are implemented, such as Figure 1 S101 to S103 shown. Alternatively, when the processor 710 executes the computer program 730, the functions of the respective modules in the corresponding embodiments are implemented. For example, Figure 3 the functions of the respective modules shown, for details please refer to Figure 4 the relevant descriptions in the corresponding embodiments.

[0064] Exemplarily, the computer program 730 may be divided into one or more modules, and one or more modules are stored in the memory 720 and executed by the processor 710 to implement the point cloud enhancement method provided by the embodiments of the present invention. One or more modules may be a series of computer program instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the computer program 730 in the terminal device 700. For example, the computer program 730 may implement the target tracking method of the multi-sensing device provided by the embodiments of the present invention.

[0065] The terminal device 700 may include, but is not limited to, the processor 710 and the memory 720. Those skilled in the art can understand that Figure 4 merely examples of the terminal device 700, and do not constitute a limitation on the terminal device 700. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0066] The so-called processor 710 may be a central processing unit, or may also be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0067] The memory 720 may be an internal storage unit of the terminal device 700, such as the hard disk or memory of the terminal device 700. The memory 720 may also be an external storage device of the terminal device 700, such as a plug-in hard disk, a smart memory card, a flash memory card, etc. equipped on the terminal device 700. Further, the memory 720 may also include both the internal storage unit and the external storage device of the terminal device 700.

[0068] The embodiments of the present invention provide a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program is executed by the processor to perform the target tracking method of the multi-sensing device in the above various embodiments.

[0069] The embodiments of the present invention provide a computer program product. When the computer program product runs on the terminal device, the terminal device is enabled to perform the target tracking method of the multi-sensing device in the above various embodiments.

[0070] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

[0071] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A target tracking method for multi-sensing devices, characterized in that, The method includes: Performing proximal tracking on a target within a sensing range based on proximal data of a collection sensor; the data collected by the collection sensor includes proximal data and distal data; Establishing a trajectory list of the target, where the trajectory list stores the trajectories of multiple targets; Maintaining and optimizing the trajectory based on time and the pose of the target to obtain an optimized trajectory; Performing distal tracking on the target using the optimized trajectory and distal data, where the proximal data is closer to the data collection sensor than the distal data.

2. The method according to claim 1, wherein Establishing a trajectory list of the target, where the trajectory list stores the trajectories of multiple targets, including: Performing downsampling storage operations according to time and pose change thresholds.

3. The method according to claim 1, wherein Maintaining and optimizing the trajectory based on time and the pose of the target to obtain an optimized trajectory, including: Obtaining the trajectory state of the target; Selecting a distal trajectory list that meets the conditions according to the trajectory state to perform trajectory optimization of the target.

4. The method according to claim 3, characterized in that, The trajectory state of the target includes: a leaving state and / or an occlusion state.

5. The method according to claim 4, wherein When the trajectory state is the leaving state, correspondingly, selecting a distal trajectory list that meets the conditions according to the trajectory state to perform trajectory optimization of the target, including: Fitting the historical pose of the leaving target using a first-order Bezier curve, and judging whether there is a mis-match according to the fitting error; listing the targets with mis-matches in the distal trajectory list, re-selecting the allocation result at the mis-match, and correcting the target ID.

6. The method according to claim 4, wherein When the trajectory state is the occlusion state, correspondingly, selecting a distal trajectory list that meets the conditions according to the trajectory state to perform trajectory optimization of the target, including: Predicting all subsequent poses of the target according to the first-order Bezier function; achieving double re-matching of the curve and IOU with the newly established trajectory in the trajectory list to complete broken-chain reconnection.

7. The method according to any one of claims 5 or 6, characterized in that The method further includes: For a curve within the fitting error, establishing a graph-based optimization equation to optimize the pose parameters and outputting the optimized pose of the target.

8. A target tracking device for a multi-sensing device, characterized in that, Including: A tracking module for performing proximal tracking on a target within a sensing range based on proximal data of a collection sensor; The data collected by the collection sensor includes proximal data and distal data; A storage module for storing the established trajectory list of the target, where the trajectory list stores the trajectories of multiple targets; The tracking module is further configured to maintain and optimize the trajectory based on time and the pose of the target to obtain an optimized trajectory; and perform distal tracking on the target using the optimized trajectory and distal data, where the proximal data is closer to the data collection sensor than the distal data.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

Images