Target tracking method and device and related product

Abstract

The invention provides a target tracking method and device and electronic equipment, and the method comprises the following steps: obtaining a first detection state quantity of a tracking target corresponding to a first video frame and a first millimeter wave radar frame, the first video frame and the first millimeter wave radar frame being obtained at a first moment; determining a second detection state quantity according to the first detection state quantity and a time difference between a first moment and a second moment, wherein the second moment is a moment after the first moment; and taking the second detection state quantity as the input of a particle filter algorithm, and obtaining the corresponding speed, position and acceleration of the tracking target at the second moment based on the particle filter algorithm. According to the method provided by the invention, the video frame and the millimeter wave radar frame can be comprehensively utilized to track the tracking target, and the finally obtained speed, position and acceleration accuracy of the tracking target is higher.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly to a target tracking method, apparatus and related products. Background Art

[0002] With the continuous development of technology, the number of motor vehicles is increasing rapidly. In order to meet people's increasingly high travel requirements, intelligent connected vehicles have emerged. Intelligent connected vehicles and ordinary vehicles drive on the road simultaneously.

[0003] For intelligent connected vehicles, it is very important to obtain information about the surrounding vehicles. The information about the surrounding vehicles generally refers to the information of the surrounding vehicles during the driving process of the vehicle, and the surrounding vehicles during the driving process of the vehicle can be called tracking targets. During the driving process of the vehicle, the vehicle will continuously obtain relevant information of the tracking target through sensors. The prior art generally obtains the information of the tracking target through sensors, but the information of the tracking target obtained through sensors is often of low accuracy.

[0004] Therefore, how to obtain more accurate information about the tracking target has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] Based on the above problems, the present application provides a target tracking method, apparatus and related products to solve the problem that the information of the tracking target obtained by the prior art is not comprehensive enough.

[0006] The present application provides a target tracking method, and the target tracking method includes the following steps:

[0007] Obtain a first detection state quantity corresponding to a tracking target in a first video frame and a first millimeter-wave radar frame, where the first detection state quantity is used to represent the speed, position and acceleration of the tracking target, and the first video frame and the first millimeter-wave radar frame are obtained at a first moment;

[0008] Determine a second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment, where the second detection state quantity is used to represent the speed, position and acceleration of the tracking target at the second moment, and the second moment is a moment after the first moment;

[0009] Use the second detection state quantity as the input of the particle filter algorithm, and obtain the speed, position and acceleration of the tracking target corresponding to the second moment based on the particle filter algorithm.

[0010] In a possible implementation manner, the method further includes:

[0011] Obtain the number identifier corresponding to the tracking target in the first video frame;

[0012] If the distance between the tracking target in the first video frame and the tracking target in the second video frame is less than a preset distance threshold, then use the number identifier corresponding to the tracking target in the first video frame as the number identifier of the tracking target in the second video frame;

[0013] Match the tracking target in the second video frame with the tracking target in the second millimeter-wave radar frame;

[0014] Add the corresponding number identifier to the tracking target in the second millimeter-wave radar frame that successfully matches the second video frame.

[0015] In a possible implementation manner, the method further includes:

[0016] Add a corresponding geometric figure to the tracking target in the first video frame, and the geometric figure can represent the position of the tracking target;

[0017] For the tracking targets with the same number identifier, if there are multiple radar points corresponding to the geometric figure in the first millimeter-wave radar frame, then cluster the multiple radar points into a radar point cluster, and the speeds of the radar points in the same radar point cluster are less than a preset speed difference threshold.

[0018] In a possible implementation manner, the obtaining the first detection state quantity corresponding to the tracking target in the first video frame and the first millimeter-wave radar frame includes:

[0019] Use the first video frame and the first millimeter-wave radar frame to obtain the position, speed, and acceleration corresponding to the tracking target with the same number identifier, and calculate according to a preset weight to obtain the first detection state quantity.

[0020] In a possible implementation manner, the method further includes:

[0021] Use the position, speed, and acceleration of the tracking target in the second video frame and the position, speed, and acceleration of the tracking target in the second millimeter-wave radar frame to determine a detection value;

[0022] Use the speed, position, and acceleration corresponding to the tracking target at the second moment obtained by the particle filter algorithm to determine a tracking value;

[0023] Calculate the Mahalanobis distance between the tracking value and the detection value of the tracking target, and the Mahalanobis distance is used to represent the gap between the tracking value and the detection value.

[0024] In a possible implementation, the tracked target has a corresponding particle swarm, and the method further includes:

[0025] Determining a second particle weight coefficient of the particle swarm at the second moment by using the first particle weight coefficient corresponding to the first moment of the particle swarm, the information in the second video frame, and the information in the second millimeter-wave radar frame;

[0026] Calculating a weight matrix of the weight of each particle at the second moment accounting for the total weight by using the second particle weight coefficient;

[0027] Taking the weight matrix as the input of a particle filter algorithm, and obtaining the speed, position, and acceleration corresponding to the tracked target at the second moment based on the particle filter algorithm.

[0028] In a possible implementation, the method further includes:

[0029] Judging the number of degenerated particles in the particle swarm by using the weight coefficient of each particle;

[0030] When the number of degenerated particles is greater than a preset value, resampling the particle swarm with a preset initial value.

[0031] In a possible implementation, the method further includes:

[0032] Taking the moment corresponding to the video frame acquired by the camera as a reference, and enabling the millimeter-wave radar to acquire a millimeter-wave radar frame at the moment when the camera acquires the video frame.

[0033] This application also provides a target tracking device, and the target tracking device includes the following modules:

[0034] A first detection state quantity acquisition module, configured to acquire a first detection state quantity corresponding to a tracked target in a first video frame and a first millimeter-wave radar frame, where the first detection state quantity is used to represent the speed, position, and acceleration of the tracked target, and the first video frame and the first millimeter-wave radar frame are acquired at a first moment;

[0035] A second detection state quantity acquisition module, configured to determine a second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment, where the second detection state quantity is used to represent the speed, position, and acceleration of the tracked target at the second moment, and the second moment is a moment after the first moment;

[0036] A particle filter module, configured to take the second detection state quantity as the input of a particle filter algorithm, and obtain the speed, position, and acceleration corresponding to the tracked target at the second moment based on the particle filter algorithm.

[0037] The present application also provides an electronic device, which includes a memory and a processor, wherein:

[0038] The memory is used to store a computer program;

[0039] The processor is used to execute the computer program to implement the above-mentioned target tracking method.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] The method provided by the present application comprehensively uses video frames and millimeter-wave radar frames. The first detection state quantity is obtained through the information in the video frames and the information in the millimeter-wave radar frames. The second detection state quantity is determined based on the particle filter algorithm by using the first detection state quantity and the time difference between the first moment and the second moment. The speed, position, and acceleration corresponding to the second detection state quantity are used as the speed, position, and acceleration corresponding to the tracking target at the second moment. In the method provided by the present application, the information of video frames and millimeter-wave radar frames is comprehensively considered. The speed, position, and acceleration of the tracking target can be obtained through video frames, and the speed, position, and acceleration of the tracking target can also be obtained through millimeter-wave radar frames. The present application comprehensively uses the information in video frames and millimeter-wave radar frames, and determines the speed, position, and acceleration of the tracking target based on the particle filter algorithm. Compared with the prior art, the method provided by the present application obtains the information at the video frame level and the information at the millimeter-wave radar frame level at the data acquisition level, and obtains more comprehensive information compared with the prior art. Based on the more comprehensive information obtained, the present application further uses the particle filter algorithm to obtain more accurate information such as the speed, position, and acceleration of the tracking target. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a flowchart of a target tracking method provided by an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of the synchronization in time of a camera and a millimeter-wave radar provided by an embodiment of the present application;

[0045] Figure 3 It is a detection schematic diagram when a millimeter-wave radar performs detection provided by an embodiment of the present application;

[0046] Figure 4 A schematic diagram of clustering by the DBSCAN clustering algorithm provided by an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the positional relationship between radar points and geometric figures provided by an embodiment of the present application;

[0048] Figure 6 An overall architecture diagram for tracking a tracking target provided by an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the preprocessing process of front radar information provided by an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the preprocessing process of front corner radar information provided by an embodiment of the present application;

[0051] Figure 9 A schematic diagram of the preprocessing process of rear corner radar information provided by an embodiment of the present application;

[0052] Figure 10 A schematic diagram of the preprocessing process of camera information provided by an embodiment of the present application;

[0053] Figure 11 A schematic diagram of the structure of a target tracking device provided by an embodiment of the present application. Detailed implementation manners

[0054] As described above, in order to meet people's increasingly high travel requirements, intelligent connected vehicles have emerged. Intelligent connected vehicles and ordinary vehicles drive on the road at the same time. This situation where intelligent connected vehicles and ordinary vehicles drive on the road at the same time is called heterogeneous traffic flow. It is very important for traffic safety and traffic operation efficiency that intelligent connected vehicles accurately obtain information about other vehicles around them in real time during driving.

[0055] After research, it is found that in related technologies, information about a tracking target is often obtained through a single type of sensor. However, a single type of sensor cannot obtain comprehensive enough information. For example, in a harsh weather environment, visual sensors such as cameras will inevitably be affected by the environment, thereby reducing the accuracy of the information obtained about the tracking target. In addition, if a large vehicle blocks a small vehicle, visual sensors such as cameras cannot obtain information about the small vehicle blocked by the large vehicle. Although using radar sensors such as millimeter-wave radars to obtain information about the tracking target has better accuracy in a harsh weather environment, radar sensors such as millimeter-wave radars are difficult to image, and the accuracy of the information about large vehicles obtained by radar sensors such as millimeter-wave radars is low. In order to obtain more accurate information about the tracking target, this application provides a target tracking method, device, and related products to improve the accuracy of obtaining information about the tracking target.

[0056] In order to enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0057] It can be understood that the method provided in this application can be applied to a processing device, which is a processing device that can obtain the first detection state quantity corresponding to the tracking target in the first video frame and the first millimeter-wave radar frame. For example, the second detection state quantity can be determined based on the particle filter algorithm according to the first detection state quantity and the time difference between the first moment and the second moment, such as a terminal device or a server. The method provided in this application can be independently executed by a terminal device or a server, or can be applied to a network scenario where a terminal device and a server communicate, and is executed in cooperation with the terminal device and the server. Among them, the terminal device can be a device such as a computer or a mobile phone. The server can be understood as an application server or a Web server. In actual deployment, the server can be an independent server or a cluster server.

[0058] Figure 1 The flowchart of a target tracking method provided in this application, the method includes the following steps:

[0059] S101: Obtain the first detection state quantity corresponding to the tracking target in the first video frame and the first millimeter-wave radar frame.

[0060] The first video frame can be collected by a visual sensor such as a camera, and the first millimeter-wave radar frame can be collected by a millimeter-wave radar. The processing device obtains the first detection state quantity of the tracking target corresponding to the first video frame and the first millimeter-wave radar frame. The first detection state quantity is used to represent the speed, position, and acceleration of the tracking target, and the first video frame and the first millimeter-wave radar frame are obtained at the first moment.

[0061] In a possible implementation, the visual sensor can be a camera. The processing device can use the moment corresponding to the video frame obtained by the camera as a reference, and let the millimeter-wave radar obtain the millimeter-wave radar frame at the moment when the camera obtains the video frame, so as to synchronize the camera and the millimeter-wave radar in time.

[0062] Since the data collected by the camera and the millimeter-wave radar have different focuses, in order to make full use of the information collected by the camera and the millimeter-wave radar, different processing can be performed on the data collected by different sensors.

[0063] In a possible implementation, the vehicle is equipped with a camera, a front millimeter-wave radar, and a corner millimeter-wave radar. For the information collected by the front millimeter-wave radar and the corner millimeter-wave radar, the information related to the road edge points collected by the front millimeter-wave radar and the corner millimeter-wave radar can be deleted. During the target tracking process, the main consideration is the state of the tracking target. The road edge point information is useless information for target tracking. The processing device can filter out the road edge point information from the information collected by the front millimeter-wave radar and the corner millimeter-wave radar and delete the relevant information.

[0064] For the information collected by the corner millimeter-wave radar, filtering and deletion processing can be performed according to conditions such as the number of consecutive frames in which the tracking target appears on multiple millimeter-wave radar frames collected by the corner millimeter-wave radar, the position range of the tracking target, the overlap situation between the tracking target and the vehicle itself, and the overlap situation between tracking targets. For example, on a millimeter-wave radar frame collected by a corner millimeter-wave radar, if the tracking target overlaps with the vehicle itself, at this time, the overlapping part of the tracking target and the vehicle itself can be deleted.

[0065] The front millimeter-wave radar and the corner millimeter-wave radar will collect static target objects, such as static target objects like trees and railings beside the road. The static target objects are also useless information for target tracking. At this time, the processing device can filter and delete the static target objects collected by the front millimeter-wave radar and the corner millimeter-wave radar in combination with the information collected by the camera.

[0066] For the front millimeter-wave radar, the processing device can record the front radar obstacles whose absolute speed exceeds the preset speed. The front radar obstacles have a high probability of being tracking targets, and the preset speed can be 6m / s.

[0067] For the information collected by the camera, the processing device can filter out the curb points and static objects in the information collected by the camera. The filtered curb points and static objects can help the processing device filter the information collected by the front millimeter-wave radar and the corner millimeter-wave radar. The camera can obtain the lane line information in the road, and the lane line information can help the processing device assign lane attributes to the tracking targets so that the processing device can track the tracking targets more accurately.

[0068] Figure 2 FIG. is a schematic diagram of synchronizing a camera and a millimeter-wave radar in time provided by this application. The camera collects image data at fixed time intervals, and the interval time is related to the frame rate of the camera. The frame rate of the camera refers to the number of images obtained by the camera per second. If the camera is 60 frames, then the camera will collect 60 video frames in 1 second, and the interval time between two video frames is 1 / 60 second. Figure 2 In the upper row, T0, T1, T2, T3... represent the moments when the camera collects each video frame. Figure 2 In the lower row, T0, T1, T2, T3... represent the moments when the millimeter-wave radar originally collects each millimeter-wave radar frame. Since the time intervals for the camera and the millimeter-wave radar to collect data are different, in order to make the video frames collected by the camera and the millimeter-wave radar frames collected by the millimeter-wave radar be at the same moment, the time for the millimeter-wave radar to collect millimeter-wave radar frames can be adjusted to achieve synchronization in time between the camera and the millimeter-wave radar. Figure 2 The time interval for the millimeter-wave radar to collect millimeter-wave radar frames in FIG. is greater than the time interval for the camera to collect video frames. At the moment T0, the millimeter-wave radar and the camera collect data simultaneously. Synchronize the moment T1 when the millimeter-wave radar in the lower row collects the millimeter-wave radar frame to the moment T1 in the upper row. The synchronization process is indicated by an arrow. Synchronize the moment T2 when the millimeter-wave radar in the lower row collects the millimeter-wave radar frame to the moment T2 in the upper row. The synchronization process is indicated by an arrow. Synchronize the moment T3 when the millimeter-wave radar in the lower row collects the millimeter-wave radar frame to the moment T3 in the upper row. The synchronization process is indicated by an arrow.

[0069] For comprehensive coverage of the tracking targets, multiple millimeter-wave radars are generally installed on the vehicle. According to the installation positions, the millimeter-wave radars can be divided into front corner millimeter-wave radars and front millimeter-wave radars. The front corner millimeter-wave radars and the front millimeter-wave radars use the same time interval to obtain adjacent millimeter-wave radar frames.

[0070] The tracking targets can be traffic participants such as motor vehicles, non-motor vehicles, and pedestrians around the vehicle. The processing device can use a particle filter to generate a particle swarm for each tracking target. Generally, the number of particle filters installed in the vehicle is not large, and the number of particle filters is determined according to the actual traffic conditions. The vehicle generally chooses to install 20 particle filters. Through the particle filter, corresponding particle swarms can be generated for up to 20 tracking targets such as motor vehicles, non-motor vehicles, and pedestrians. The number of particles in a particle swarm can be preset. In the method provided in this application, the number of particles in a particle swarm can be set to 100. A tracking target can correspond to one particle swarm or multiple particle swarms. In a possible implementation, when the number of tracking targets is less than 20, the processing device can allocate multiple particle swarms to specific tracking targets. For example, the vehicle is installed with 20 particle filters, and there are 10 tracking targets around the vehicle. The processing device first uses 10 particle filters to determine 10 corresponding particle swarms for the 10 tracking targets. Then, the processing device can use 4 particle filters to determine a second corresponding particle swarm for the four targets closest to the vehicle itself. The tracking target closest to the vehicle in the lateral direction can be determined by a camera, and the tracking target closest to the vehicle in the longitudinal direction can be determined by a millimeter-wave radar. Tracking targets with multiple corresponding particle swarms will obtain a faster convergence speed when performing the particle filter algorithm. After determining the particle swarm corresponding to the tracking target, the processing device can initialize the particle state quantity of the particle swarm corresponding to the tracking target.

[0071] The vehicle's driving on the road is a continuous process. During the vehicle's driving, the camera and millimeter-wave radar will continuously collect data. The processing device will use the earliest part of the data collected by the camera and millimeter-wave radar as the original data. For example, the processing device can use the first 10 frames of data collected by the camera and millimeter-wave radar as the original data. By presetting the original data, it is ensured that position, speed, acceleration, and other information can be obtained for any video frame and millimeter-wave radar frame except the original data.

[0072] If the first moment corresponding to the first video frame and the first millimeter-wave radar frame is the moment to start tracking the target, at this time, the processing device can determine the first detection state quantity by using the position, speed, and acceleration of the tracking target on the first video frame and the position, speed, and acceleration on the first millimeter-wave radar frame. When the processing device obtains the first detection state quantity of the tracking target, the first detection state quantity of the tracking target in the lateral direction is more inclined to the data of the first video frame, and the first detection state quantity of the tracking target in the longitudinal direction is more inclined to the data of the first millimeter-wave radar frame.

[0073] In a possible implementation, if the first moment is not the moment when tracking the target starts, and the third moment before the first moment is the moment when tracking the target starts, the third detection state quantity can be determined by the position, velocity, and acceleration of the tracking target on the third video frame and the position, velocity, and acceleration on the third millimeter-wave radar frame. The third video frame and the third millimeter-wave radar frame are obtained at the third moment. In this case, the terminal device can determine the first detection state quantity corresponding to the first video frame and the first millimeter-wave radar frame of the tracking target based on the third detection state quantity and the time difference between the first moment and the third moment using the particle filter algorithm.

[0074] S102: Determine the second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment.

[0075] The second moment is the moment after the first moment. The processing device can use the first detection state quantity and the time difference between the first moment and the second moment to determine the second detection state quantity.

[0076] In a possible implementation, the detection state quantity A corresponding to the k-th moment k = [x, y, v x , v y , a x , a y k , where x represents the lateral distance between the tracking target and the vehicle, y represents the longitudinal distance between the tracking target and the vehicle, v x represents the velocity of the tracking target in the lateral direction, v y represents the velocity of the tracking target in the longitudinal direction, a x represents the acceleration of the tracking target in the lateral direction, a y represents the acceleration of the tracking target in the longitudinal direction, and k represents the k-th video frame and the k-th millimeter-wave radar frame obtained at the k-th moment corresponding to the detection state quantity.

[0077] The detection state quantity at the k-th moment can be transferred to the particles at the k+1-th moment through the following formula:

[0078]

[0079] A k+1 is the detection state quantity at the k+1-th moment, and Δt is the time difference between the k-th moment and the k+1-th moment. When k = 1, A1 represents the first detection state quantity, and the second detection state quantity A2 can be obtained through the above formula.

[0080] S103: Use the second detection state quantity as the input of the particle filter algorithm, and obtain the velocity, position, and acceleration corresponding to the tracking target at the second moment based on the particle filter algorithm. ​

[0081] The processing device uses the second detected state quantity as a feature of the tracking target, and uses the particle swarm corresponding to the tracking target mentioned in S101 to find the second detected state quantity. Each particle has a similarity with the second detected state quantity, and the velocity, position, and acceleration corresponding to the particle state quantity of the particle with the highest similarity to the second detected state quantity are used as the velocity, position, and acceleration of the tracking target at the second moment.

[0082] The method provided in this application comprehensively considers the information collected by different types of sensors. The first detected state quantity is obtained through the information collected by the sensors, and the second detected state quantity is determined by using the first detected state quantity and the time difference between the first moment and the second moment. The second detected state quantity is used as the input of the particle filter algorithm, and the position, velocity, and acceleration of the tracking target at the second moment are determined based on the particle filter algorithm. In the method provided in this application, the information of the video frame and the millimeter-wave radar frame is comprehensively considered. The velocity, position, and acceleration of the tracking target can be obtained through the video frame, and the velocity, position, and acceleration of the tracking target can also be obtained through the millimeter-wave radar frame. This application comprehensively uses the information in the video frame and the millimeter-wave radar frame, and determines the velocity, position, and acceleration of the tracking target based on the particle filter algorithm. Compared with the prior art, the method provided in this application obtains the information at the video frame level and the millimeter-wave radar frame level at the data acquisition level, and obtains more comprehensive information compared with the prior art. Based on the more comprehensive information obtained, this application further uses the particle filter algorithm to obtain more accurate information such as the velocity, position, and acceleration of the tracking target.

[0083] In a possible implementation, the processing device can track multiple tracking targets. When the processing device tracks multiple tracking targets, the processing device can number a preset number of tracking targets. In a possible implementation, the first video frame includes ten tracking targets, which include eight small vehicles and two large vehicles. The processing device can obtain the corresponding number identifiers of these ten tracking targets. If the distance between the tracking targets in the first video frame and the tracking targets in the second video frame is less than a preset distance threshold, the number identifier corresponding to the tracking target in the first video frame is used as the number identifier of the tracking target in the second video frame. The preset distance threshold can include a preset longitudinal threshold and a preset lateral threshold. In a possible implementation, the longitudinal threshold can be 50 cm, and the lateral threshold can be 15 cm. In the second video frame, there is a vehicle. The lateral distance between this vehicle and the vehicle numbered 1 in the first video frame is 5 cm, and the longitudinal distance is 20 cm. Both the lateral distance and the longitudinal distance are less than the threshold. At this time, the processing device can use the number identifier 1 as the number identifier of this vehicle in the second video frame. After the vehicle in the second video frame obtains the corresponding number identifier, the processing device can match the tracking target in the second video frame with the tracking target in the second millimeter-wave radar frame. If the match is successful, add the corresponding number identifier to the tracking target that matches successfully in the second millimeter-wave radar frame.

[0084] The method provided in this application links the same tracking targets in different video frames through number identifiers, so that for the same tracking target, the processing device can accurately obtain the data corresponding to this tracking target in different video frames, and the data of different tracking targets will not be confused, making the video frame data and millimeter-wave radar frame data obtained by the sensor more credible for a specific tracking target.

[0085] Since the millimeter-wave radar will detect multiple radar points for a large vehicle, Figure 3 This is a detection schematic diagram when the millimeter-wave radar provided in this application performs detection. Figure 3 As can be seen, for the truck in the rightmost lane, the millimeter-wave radar will detect multiple radar points. It is very difficult to determine the specific information of the truck in the right lane only through the information obtained by the millimeter-wave radar.

[0086] In a possible implementation, the processing device may first add a corresponding geometric figure to the tracking target in the first video frame. For example, if there is a truck numbered 1 in the first video frame, the processing device may add a rectangular box to the truck numbered 1 in the first video frame to represent the position of the truck. In a possible implementation, there are multiple radar points in the first millimeter-wave radar frame that can be successfully matched with the truck numbered 1. At this time, the processing device may cluster the multiple radar points in the first millimeter-wave radar frame through the DBSCAN clustering algorithm to obtain a radar point cluster. The radar point cluster can be used as a matching object to match the tracking target in the video frame. The speed difference between the radar points in the same radar point cluster is less than a preset speed difference threshold.

[0087] Figure 4 FIG. is a schematic diagram of clustering by the DBSCAN clustering algorithm provided by the present application, and the points therein can be used as radar points. First, a core point E is selected among the multiple radar points. With the core point E as the center, radar points are searched in the first millimeter-wave radar frame through a preset search radius eps. With the core point E as the center, there are minPts radar points within a radius of eps. Calculate the speed difference between any two radar points among the minPts radar points. The processing device adds the same cluster label to all radar points whose speed difference is less than the preset speed difference threshold. If there is no speed difference between any two radar points within the circle with the core point E as the center and a radius of eps that is less than the preset speed difference threshold, it can be determined at this time that E is a boundary point. At this time, the processing device may reselect a center point in the first millimeter-wave radar frame for the DBSCAN clustering algorithm. The DBSCAN clustering algorithm stops until the processing device processes all the radar points in the first millimeter-wave radar frame.

[0088] In a possible implementation, the radar points corresponding to the radar point cluster in the millimeter-wave radar frame may not all fall within the geometric figure of the video frame, but this radar point cluster can represent the tracking target. At this time, the processing device may first dilate the geometric figure corresponding to the tracking target according to a preset dilation threshold. The processing device may dilate the geometric figure of the video frame by using a preset horizontal position dilation threshold, a vertical position dilation threshold, and a horizontal speed dilation threshold. The processing device may match the radar point cluster with the dilated geometric figure. Figure 5 FIG. is a schematic diagram of the positional relationship between a radar point and a geometric figure provided by the present application. Among them, P1, P2, P3, and P4 are the four endpoints corresponding to the geometric figure, and P is the radar point. The processing device can determine whether the radar point falls within the geometric figure through the cross product operation of points. The formula is as follows:

[0089]

[0090] If the radar point P satisfies the above formula, it is proved that the radar point is inside the geometric figure formed by P1, P2, P3, and P4. If the radar point P is not inside the inflated geometric figure, then the radar point is most likely unavailable and can be removed.

[0091] The method provided in this application matches and fuses the information of the video frame and the millimeter-wave radar frame obtained at the same moment corresponding to the same tracking target. Considering the particularity of detecting large vehicles in the millimeter-wave radar frame, this application uses the DBSCAN clustering algorithm to fuse the radar points of large vehicles in the millimeter-wave radar frame, reducing the difficulty of matching and fusing the information of the video frame and the millimeter-wave radar frame corresponding to the same tracking target, making it easier and faster for the processing device to match and fuse the information of the video frame and the millimeter-wave radar frame obtained at the same moment corresponding to the same tracking target.

[0092] In a possible implementation, the processing device can use the first video frame and the first millimeter-wave radar frame to obtain the position, speed, and acceleration corresponding to the tracking target with the same number identifier, and calculate the first detection state quantity according to the preset weight. As described above, if the first moment corresponding to the first video frame and the first millimeter-wave radar frame is the moment of starting to track the target, the processing device can use the position, speed, and acceleration of the tracking target on the first video frame and the position, speed, and acceleration on the first millimeter-wave radar frame to determine the first detection state quantity. When the processing device obtains the first detection state quantity of the tracking target, the weights of the position, speed, and acceleration of the first video frame in the horizontal direction in the first detection state quantity are greater, and the weights of the position, speed, and acceleration of the first millimeter-wave radar frame in the vertical direction in the first detection state quantity are greater.

[0093] The method provided in this application fully considers the different characteristics of different types of sensors. When obtaining the first detection state quantity, it mainly considers the position, speed, and acceleration of the video frame in the horizontal direction, and mainly considers the position, speed, and acceleration of the millimeter-wave radar frame in the vertical direction. The first detection state quantity fused in this way has higher accuracy.

[0094] In a possible implementation, the tracking target has a corresponding particle swarm. The processing device can obtain a representative particle that is closest to the state of the second particle through the particle filter algorithm. The processing device uses the velocity, position, and acceleration of this representative particle as the velocity, position, and acceleration of the tracking target at the second moment. The processing device can input the velocity, position, and acceleration corresponding to the second video frame and the second millimeter-wave radar frame into the particle swarm to obtain the velocity, position, and acceleration of each particle according to the sensor. The Mahalanobis distance D can be obtained by using the variance of each particle in the particle swarm corresponding to the sensor and the variance between each particle in the particle swarm after the particle filter algorithm. m The Mahalanobis distance D m can be calculated by the following formula:

[0095]

[0096] where x represents the variance between all particles in the particle swarm after inputting the velocity, position, and acceleration corresponding to the second video frame and the second millimeter-wave radar frame into the particle swarm. μ represents the variance between all particles in the particle swarm after the particle filter algorithm ends. The Mahalanobis distance D m can represent the distance between the velocity, position, and acceleration corresponding to the tracking target obtained through the particle wave algorithm and the velocity, position, and acceleration corresponding to the tracking target obtained through the sensor.

[0097] The Mahalanobis distance D m can be used as an indicator to determine whether the velocity, position, and acceleration corresponding to the tracking target at the second moment are accurate. In this regard, a Mahalanobis distance threshold can be preset in advance. If the Mahalanobis distance is less than the set Mahalanobis distance threshold, it proves that the velocity, position, and acceleration of the tracking target obtained through the particle filter algorithm at the second moment are accurate.

[0098] In a possible implementation, the Mahalanobis distance thresholds for different tracking targets can be different. For example, if a vehicle has the same identification number in the first video frame and the second video frame, then for this vehicle, the Mahalanobis distance threshold can be larger than the preset reference Mahalanobis distance threshold. If a vehicle does not appear in the first video frame and is a new vehicle that appears in the second video frame, then for this vehicle, the Mahalanobis distance threshold can be smaller than the preset reference Mahalanobis distance threshold.

[0099] In another possible implementation, the confidence threshold of the sensor can be determined in advance. The confidence threshold of the sensor includes the lateral position threshold dx, the longitudinal position threshold dy, the lateral velocity threshold dv x and the longitudinal velocity threshold dv yWhen the speed and position of a vehicle obtained by the sensor in the second video frame and the second millimeter-wave radar frame are less than the confidence threshold of the sensor, then for this vehicle, the Mahalanobis distance threshold can be greater than the preset reference Mahalanobis distance threshold.

[0100] This application proposes to use the Mahalanobis distance to represent the distance between the speed, position, and acceleration calculated by the particle filter algorithm and the speed, position, and acceleration obtained by the sensor for any tracking target. By comparing the Mahalanobis distance of any tracking target with the set Mahalanobis distance threshold, the accuracy of the speed, position, and acceleration calculated by the particle filter algorithm for each tracking target can be accurately understood, which is very instructive for the subsequent adjustment of algorithm parameters.

[0101] In a possible implementation, each tracking target in the second video frame and the second millimeter-wave radar frame has a corresponding Mahalanobis distance threshold. Using the Mahalanobis distance threshold corresponding to each tracking target, the detection value of each tracking target obtained from the second video frame and the second millimeter-wave radar frame, and the tracking value of each tracking target obtained by the particle filter algorithm as the input of the Hungarian algorithm, the Hungarian algorithm is used to match the detection value and the tracking value. As mentioned above, the tracking targets in the first video frame and the second video frame may have been assigned the same number identifier, that is, they have been matched. Before the processing device executes the Hungarian algorithm, the matching status of the tracking targets in the first video frame and the second video frame is initialized. For the tracking value of each tracking target, try to match it with the detection value of the tracking target. The processing device can start from the tracking value of a tracking target to find a detection value. If the tracking value finds a detection value, the above tracking value and the above detection value are successfully matched. If the tracking value does not find any detection value, it proves that this tracking target is unreachable. The processing device uses the Hungarian algorithm to match the tracking value and the detection value of each tracking target until all the tracking values and detection values are successfully matched. For each successfully matched tracking value and detection value, there will be a certain gap between the two values, which can be called the weight between the tracking value and the detection value, or the cost between the tracking value and the detection value. The gap between each successfully matched tracking value and detection value is related to the Mahalanobis distance threshold corresponding to the tracking target.

[0102] The method provided by this application can minimize the difference between the tracking value obtained by the sensor for each tracking target and the detection value obtained by the particle filter algorithm for the same tracking target. This application uses the Hungarian algorithm to match the tracking value and the detection value, and determines the same tracking target in different video frames and millimeter-wave radar frames through the matching result of the tracking value and the detection value. The matching result obtained by the Hungarian algorithm is often the optimal matching result.

[0103] At each moment, the particle wave algorithm can have a corresponding weight coefficient for each particle. In one possible implementation, at a certain moment, the processing device can first determine an initial particle weight. The processing device sets the initial particle weight coefficient to 1 / num, where num is the total number of particles in the particle swarm. If there are 100 particles in the particle swarm corresponding to a tracking target, then the initial particle weight coefficient is 1 / 100. For the particle weight coefficient W of the particle swarm at the k-th moment k can be obtained through the following formula:

[0104]

[0105] where W k represents the particle weight coefficient of the particle swarm at the k-th moment, and W k-1 represents the particle weight coefficient of the particle swarm at the (k - 1)-th moment. x is the state parameter, longitudinal distance, lateral distance, longitudinal velocity, lateral velocity, longitudinal acceleration, and lateral acceleration at the k-th moment. μ represents the average value of the longitudinal distance, lateral distance, longitudinal velocity, lateral velocity, longitudinal acceleration, and lateral acceleration obtained by all particles in the particle swarm through the sensor at the k-th moment. Among them, ∑ represents the variance between the longitudinal distance, lateral distance, longitudinal velocity, lateral velocity, longitudinal acceleration, and lateral acceleration of all particles obtained through the sensor at the k-th moment.

[0106] If the first moment is the moment when tracking the target starts, then W1 is 1 / 100, and the particle weight coefficient W2 corresponding to the second moment can be obtained through the above formula.

[0107] After obtaining the particle weight coefficient W of the particle swarm at the k-th moment k it is possible to obtain the weight coefficient w of the i-th particle in the particle swarm at the k-th moment k (i) , where i represents the i-th particle in the particle swarm, and i is a positive integer less than or equal to the total number of particles in the particle swarm. Through the normalization operation, the proportion of the weight of each particle in the total weight of the particle swarm at the k-th moment can be obtained The weight matrix of the proportion of the weight of each particle in the total weight can be calculated through the following formula:

[0108]

[0109] where N is the total number of particles in the particle swarm at the k-th moment, and w k (i) represents the weight coefficient w of the i-th particle in the particle swarm at the k-th moment k (i) . can be used as the input of the particle filter algorithm. and the second detected state quantity are used as the input of the particle filter algorithm, and based on the particle filter algorithm, the speed, position, and acceleration corresponding to the tracking target at the second moment are obtained.

[0110] The method provided by this application can, during the continuous driving of the vehicle, use the particle weight coefficients of the previous moment's particle swarm to determine the particle weight coefficients of the next moment's particle swarm, and based on the particle weight coefficients of the next moment's particle swarm, determine the weight matrix of the proportion of the weight of each particle used in the particle filter algorithm. By means of the weight matrix, the particle filter algorithm can obtain a more accurate output, enabling more precise tracking of the tracking target.

[0111] In a possible implementation manner, during the continuous tracking of the tracking target, the particles in the corresponding particle swarm of this tracking target may degenerate. The method provided by this application can judge whether the particles in the particle swarm have degenerated through the degeneration score The degeneration score can be calculated by the following formula:

[0112]

[0113] where N is the total number of particles in the particle swarm, i is the i-th particle in the particle swarm, i is a positive integer not less than N, is the proportion of the weight of each particle in the particle swarm at the k-th moment to the total weight. When the processing device finds that the number of degenerated particles is greater than the preset value, the processing device can resample the particle swarm using the preset initial value.

[0114] The method provided by this application can detect the degeneration situation of each particle in each particle swarm in real time. When the number of degenerated particles in the particle swarm is greater than the preset value, the particle swarm is resampled using the preset initial value. Degenerated particles will have an adverse impact on the result of the particle filter algorithm. The method provided by this application can eliminate the adverse impact of degenerated particles on the result of the particle filter algorithm through resampling, making the tracking of the tracking target more accurate.

[0115] Figure 6This is an overall architecture diagram for tracking a target provided by this application. Among them, the data preprocessing module is mainly responsible for obtaining information from different types of sensors and performing certain preprocessing. The data preprocessing module preprocesses camera information, front radar information, corner radar information, and panoramic information. The front radar information and corner radar information can be front millimeter-wave radar information and corner millimeter-wave radar information, and the panoramic information is the information obtained through vision sensors. Performing certain preprocessing on the information of different types of sensors can include determining the speed, position, and acceleration of any tracked target in each camera frame according to the camera information. The preprocessing performed by the data preprocessing module can also include determining the speed, position, and acceleration of any tracked target in each radar frame according to the front radar information and corner radar information. The preprocessing performed by the data preprocessing module can also include determining a numbered identifier for each tracked target according to the camera information. The preprocessing performed by the data preprocessing module can also include adding geometric figures to each tracked target according to the camera information.

[0116] The fusion module can fuse different types of information. The fusion module can perform associative fusion. For example, the fusion module can match the numbered identifiers of the same tracked target in the video frame and the radar frame at the same moment through the numbered identifier. The fusion module can also cluster multiple radar points corresponding to a larger tracked target in the radar frame into a radar point cluster based on the DBCAN clustering algorithm, and match the clustered radar point cluster with the tracked target in the video frame. The fusion module can also perform trajectory management. The fusion module can comprehensively utilize camera information, front radar information, corner radar information, and panoramic information to perform a particle filter algorithm, and calculate the position, speed, and acceleration of any tracked target at any moment through the particle filter algorithm. After calculating the position, speed, and acceleration of any tracked target at any moment through the particle filter algorithm, the fusion module can update the trajectory of any tracked target. For example, the fusion module can update the action trajectory of the tracked target during the tracking process, track new tracked targets that appear at any moment, and merge the tracking trajectories of the same tracked target, etc.

[0117] The output module is responsible for processing and outputting the information obtained by the fusion module. For example, the output module can output the tracking trajectory of any tracked target and manage the tracking trajectories of all tracked targets. In a possible implementation, if part of the tracking trajectory of a tracked target is missing, the output module can complete the missing tracking trajectory of this tracked target, that is, perform trajectory compensation, according to the position, speed, and acceleration of this tracked target in the missing part of the tracking trajectory obtained by the fusion module based on the particle filter algorithm. The output module can also screen the tracked targets and select the tracking trajectories of the tracked targets that the user wants to know.

[0118] Figure 7 A schematic diagram of the preprocessing process for the front radar information provided by this application. The preprocessing process for the front radar information may include the following steps:

[0119] S701: Filter curb points.

[0120] The terminal device can filter out the curb points in the front radar information.

[0121] S702: Delete curb points.

[0122] Since the curb points are worthless data for target tracking, the processing device can delete the curb point information filtered out in the front radar information.

[0123] S703: Record the number of frames of dynamic points.

[0124] In the front radar information, there may be one or more dynamic points, and these dynamic points may represent the tracking targets. The same dynamic points may exist in two adjacent millimeter-wave radar frames. The processing device can record the number of frames in which the positions of the dynamic points change.

[0125] S704: Perform a preliminary match on the obstacles filtered out by multiple sensors.

[0126] The processing device can comprehensively use the information of the front radar, front corner radar, and rear corner radar to perform a preliminary match on the filtered-out obstacles. The obstacles may be obstacles outside the road line or inside the road line. The obstacles inside the road line may be tracking targets such as vehicles or obstacles such as roadblocks on the road. The obstacles outside the road line may be obstacles such as trees or railings beside the road or pedestrians on the roadside. The processing device can match the obstacles filtered out by the front radar, front corner radar, and rear corner radar. For example, there is a static obstacle inside the road line in the front radar information, and this static obstacle is a roadblock. At the same moment, the same roadblock also exists in the front corner radar information. The processing device can match this roadblock in the information of the front radar, front corner radar, and rear corner radar.

[0127] S705: Determine the obstacle closest to the self-lane in the front radar information.

[0128] During the target tracking process, the obstacle closest to the self-lane is very likely to be the tracking target closest to the vehicle. The processing device determines the obstacle closest to the self-lane in the front radar information.

[0129] S706: Filter the obstacles in the front radar information according to the preliminary match result and the number of frames of dynamic points.

[0130] The processing device preliminarily determines whether the obstacle in the front radar information is a tracking target or a static obstacle according to the preliminary matching result and the dynamic point frame number. The dynamic point frame number is mainly used to determine the tracking target.

[0131] S707: Delete the static obstacles in the front radar information.

[0132] Some static obstacles in the front radar information are worthless data for target tracking, such as trees beside the road, railings beside the road, or roadblocks in the road. The processing device can delete the static obstacles in part of the front radar information.

[0133] S708: Assign lane attributes to the remaining obstacles.

[0134] After the processing device deletes some obstacles, the remaining obstacles can be preliminarily identified as tracking targets. The processing device can assign lane attributes to the tracking targets. The lane attributes can include which lane in the road the obstacle is located in.

[0135] Figure 8 The figure shows a schematic diagram of the preprocessing process for the front corner radar information provided by this application. The preprocessing process for the front corner radar information can include the following steps:

[0136] S801: Filter out curb points.

[0137] The terminal device can filter out the curb points in the front corner radar information.

[0138] S802: Delete the curb points.

[0139] Since the curb points are worthless data for target tracking, the processing device can delete the curb point information filtered out in the front corner radar information.

[0140] S803: Perform preliminary matching on the obstacles filtered out by multiple sensors.

[0141] The processing device can comprehensively use the information of the front radar, front corner radar, and rear corner radar to perform preliminary matching on the filtered obstacles. The obstacles may be obstacles outside the road line or inside the road line. The obstacles inside the road line may be tracking targets such as vehicles or static obstacles such as roadblocks in the middle of the road. The obstacles outside the road line may be obstacles such as trees or railings beside the road or pedestrians beside the road. The processing device can match the obstacles filtered out by the front radar, front corner radar, and rear corner radar. For example, there is a static obstacle in the front corner radar information, and this static obstacle is a roadblock. At the same time, this roadblock also exists in the front radar information. The processing device can match this roadblock in the front corner radar information and the front radar information.

[0142] S804: Determine the obstacle closest to the host lane in the front corner radar information.

[0143] During the target tracking process, the obstacle closest to the host lane is very likely to be the tracking target closest to the vehicle. The processing device determines the obstacle closest to the host lane in the front corner radar information.

[0144] S805: Determine whether a static obstacle can be deleted.

[0145] In S803, the processing device performs a preliminary match on the obstacles filtered by the multi-sensor, and can use the preliminary match result to determine whether a static obstacle can be deleted. For static obstacles, static obstacles such as trees beside the road or railings beside the road contribute less to target tracking, but static obstacles such as roadblocks in the middle of the road or vehicle wrecks caused by traffic accidents cannot be ignored for target tracking. The processing device can use the preliminary match result to determine whether a static obstacle can be deleted.

[0146] S806: Delete the deletable static obstacles.

[0147] The processing device can delete the static obstacles such as trees beside the road or railings beside the road in the front corner radar information, which are deletable static obstacles that contribute less to target tracking.

[0148] Figure 9 This is a schematic diagram of the preprocessing process for the rear corner radar information provided by this application. The preprocessing process for the rear corner radar information may include the following steps:

[0149] S901: Determine whether an obstacle can be deleted.

[0150] Generally, a camera is installed in front of the vehicle, and the front radar is also installed in front of the vehicle. The processing device can comprehensively use the road edge information collected by the camera to confirm the target objects outside the road edge of the radar. Since the vehicle is driving within the road edge, the obstacles outside the road edge contribute less to target tracking, and the obstacles outside the road edge can be determined as deletable obstacles.

[0151] S902: Delete the deletable obstacles.

[0152] The processing device can delete the deletable obstacles that contribute less to target tracking.

[0153] Figure 10 This is a schematic diagram of the preprocessing process for the camera information provided by this application. The preprocessing process for the camera information may include the following steps:

[0154] S1001: Filter out some obstacles in the camera information.

[0155] The processing device uses the camera information to confirm the obstacles appearing in the camera information. Since most of the camera information is presented in the form of video frames, the processing device can confirm the obstacles in the video frames. The processing device filters out some obstacles with a confidence level lower than the preset confidence level among the obstacles confirmed in the video frames. The filtered obstacles are generally obstacles far from the vehicle itself. The confidence level can evaluate the contribution of the obstacles to target tracking, and the preset confidence level can be 0.65.

[0156] S1002: Find the dynamic obstacle with the closest longitudinal distance among the obstacles.

[0157] The dynamic obstacle with the closest longitudinal distance has a high probability of being identified as the tracking target.

[0158] S1003: Assign lane attributes to the obstacles in the camera information and the obstacles in the front radar.

[0159] When the vehicle is driving on the road, the camera information includes the lane information of the road. The terminal device can assign lane attributes to the obstacles in the camera information and the obstacles in the front radar using the lane information. When assigning lane attributes, priority is given to assigning lane attributes to dynamic obstacles. The present application also provides a schematic structural diagram of a target tracking device as shown in Figure 11 The target tracking device 1100 includes the following modules:

[0160] The first detection state quantity acquisition module 1101 is used to acquire the first detection state quantity corresponding to the tracking target in the first video frame and the first millimeter-wave radar frame. The first detection state quantity is used to represent the speed, position, and acceleration of the tracking target. The first video frame and the first millimeter-wave radar frame are acquired at the first moment;

[0161] The second detection state quantity acquisition module 1102 is used to determine the second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment. The second detection state quantity is used to represent the speed, position, and acceleration of the tracking target at the second moment. The second moment is a moment after the first moment;

[0162] The particle filter module 1103 is used to use the second detection state quantity as the input of the particle filter algorithm and obtain the speed, position, and acceleration of the tracking target corresponding to the second moment based on the particle filter algorithm.

[0163] In a possible implementation manner, the device further includes a matching module, and the matching module is specifically used for:

[0164] Acquire the number identifier corresponding to the tracking target in the first video frame;

[0165] If the distance between the tracking target in the first video frame and the tracking target in the second video frame is less than a preset distance threshold, then use the number identifier corresponding to the tracking target in the first video frame as the number identifier of the tracking target in the second video frame;

[0166] Match the tracking target in the second video frame with the tracking target in the second millimeter-wave radar frame;

[0167] Add the corresponding number identifier to the tracking target in the second millimeter-wave radar frame that matches successfully with the second video frame.

[0168] In a possible implementation, the device further includes a clustering module, and the clustering module is specifically configured to:

[0169] Add a corresponding geometric figure to the tracking target in the first video frame, and the geometric figure can represent the position of the tracking target;

[0170] For the tracking targets with the same number identifier, if there are multiple radar points corresponding to the geometric figure in the first millimeter-wave radar frame, then cluster the multiple radar points into a radar point cluster, and the speeds of the radar points in the same radar point cluster are less than a preset speed difference threshold.

[0171] In a possible implementation, the first detection state quantity acquisition module is specifically configured to:

[0172] Use the first video frame and the first millimeter-wave radar frame to obtain the position, speed, and acceleration corresponding to the tracking target with the same number identifier, and calculate according to a preset weight to obtain the first detection state quantity.

[0173] In a possible implementation, the device further includes a Mahalanobis distance determination module, and the Mahalanobis distance determination module is specifically configured to:

[0174] Use the position, speed, and acceleration of the tracking target in the second video frame and the position, speed, and acceleration of the tracking target in the second millimeter-wave radar frame to determine a detection value;

[0175] Use the speed, position, and acceleration corresponding to the tracking target at the second moment obtained by the particle filter algorithm to determine a tracking value;

[0176] Calculate the Mahalanobis distance between the tracking value and the detection value of the tracking target, and the Mahalanobis distance is used to represent the gap between the tracking value and the detection value.

[0177] In a possible implementation, the tracking target has a corresponding particle swarm, and the device further includes a weight matrix determination module, which is specifically configured to:

[0178] Determine the second particle weight coefficient of the particle swarm at the second moment by using the first particle weight coefficient corresponding to the first moment of the particle swarm, the information in the second video frame, and the information in the second millimeter-wave radar frame;

[0179] Calculate the weight matrix of each particle weight accounting for the total weight at the second moment by using the second particle weight coefficient;

[0180] Use the weight matrix as the input of the particle filter algorithm, and obtain the speed, position, and acceleration corresponding to the tracking target at the second moment based on the particle filter algorithm.

[0181] In a possible implementation, the device further includes a resampling module, which is specifically configured to:

[0182] Judge the number of degenerated particles in the particle swarm by using the weight coefficient of each particle;

[0183] When the number of degenerated particles is greater than a preset value, resample the particle swarm by using a preset initial value.

[0184] In a possible implementation, the device further includes a timestamp synchronization module, which is specifically configured to:

[0185] Taking the moment corresponding to the video frame acquired by the camera as a reference, enable the millimeter-wave radar to acquire a millimeter-wave radar frame at the moment when the camera acquires the video frame.

[0186] An embodiment of the present application further provides a target tracking device, where the device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the steps of the target tracking method according to any embodiment of the present application.

[0187] In practical applications, the computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0188] A computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program which can be used by or in connection with an instruction execution system, apparatus, or device.

[0189] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0190] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the above.

[0191] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0192] It should be noted that the various embodiments in this specification are described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and reference can be made to the corresponding parts of the method embodiments for the relevant content. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components referred to as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0193] As described above, it is only a specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A target tracking method, characterized in that, Including: Obtain a first detection state quantity corresponding to a tracking target in a first video frame and a first millimeter-wave radar frame, where the first detection state quantity is used to represent the speed, position, and acceleration of the tracking target, and the first video frame and the first millimeter-wave radar frame are obtained at a first moment; Determine a second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment, where the second detection state quantity is used to represent the speed, position, and acceleration of the tracking target at the second moment, and the second moment is a moment after the first moment; Use the second detection state quantity as the input of a particle filter algorithm, and obtain the speed, position, and acceleration corresponding to the tracking target at the second moment based on the particle filter algorithm.

2. The method according to claim 1, wherein The method further includes: Obtain a number identifier corresponding to the tracking target in the first video frame; If the distance between the tracking target in the first video frame and the tracking target in the second video frame is less than a preset distance threshold, use the number identifier corresponding to the tracking target in the first video frame as the number identifier of the tracking target in the second video frame; Match the tracking target in the second video frame with the tracking target in the second millimeter-wave radar frame; Add a corresponding number identifier to the tracking target in the second millimeter-wave radar frame that matches successfully with the second video frame.

3. The method according to claim 2, wherein The method further includes: Add a corresponding geometric figure to the tracking target in the first video frame, and the geometric figure can represent the position of the tracking target; For the tracking target with the same number identifier, if there are multiple radar points corresponding to the geometric figure in the first millimeter-wave radar frame, cluster the multiple radar points into a radar point cluster, and the speeds of the radar points in the same radar point cluster are less than a preset speed difference threshold.

4. The method according to claim 3, characterized in that The obtaining of the first detection state quantity corresponding to the tracking target in the first video frame and the first millimeter-wave radar frame includes: Use the first video frame and the first millimeter-wave radar frame to obtain the position, speed, and acceleration corresponding to the tracking target with the same number identifier, and calculate according to a preset weight to obtain the first detection state quantity.

5. The method according to claim 2, wherein The method further includes: Determine a detection value using the position, speed, and acceleration of the tracking target in the second video frame and the position, speed, and acceleration of the tracking target in the second millimeter-wave radar frame; Determine a tracking value using the speed, position, and acceleration corresponding to the tracking target at the second moment obtained based on the particle filter algorithm; Calculate the Mahalanobis distance between the tracking value and the detection value of the tracking target, and the Mahalanobis distance is used to represent the gap between the tracking value and the detection value.

6. The method according to claim 2, wherein The tracking target has a corresponding particle swarm, and the method further includes: Determine a second particle weight coefficient of the particle swarm at the second moment using the first particle weight coefficient of the particle swarm corresponding to the first moment, the information in the second video frame, and the information in the second millimeter-wave radar frame; Calculate the weight matrix of each particle weight accounting for the total weight at the second moment using the second particle weight coefficient; Use the weight matrix as the input of the particle filter algorithm, and obtain the speed, position, and acceleration of the tracking target corresponding to the second moment based on the particle filter algorithm.

7. The method according to claim 6, characterized in that, The method further includes: Judge the number of degenerated particles in the particle swarm using the weight coefficient of each particle; When the number of degenerated particles is greater than a preset value, resample the particle swarm using a preset initial value.

8. The method according to claim 1, wherein The method further includes: Taking the moment corresponding to the video frame acquired by the camera as a reference, enable the millimeter-wave radar to acquire a millimeter-wave radar frame at the moment when the camera acquires the video frame.

9. A target tracking device, characterized in that, Including A first detection state quantity acquisition module, configured to acquire a first detection state quantity corresponding to a tracking target in a first video frame and a first millimeter-wave radar frame, where the first detection state quantity is used to represent the speed, position, and acceleration of the tracking target, and the first video frame and the first millimeter-wave radar frame are acquired at a first moment; A second detection state quantity acquisition module, configured to determine a second detection state quantity according to the first detection state quantity and the time difference between the first moment and the second moment, where the second detection state quantity is used to represent the speed, position, and acceleration of the tracking target at the second moment, and the second moment is a moment after the first moment; A particle filter module, configured to use the second detection state quantity as the input of the particle filter algorithm, and obtain the speed, position, and acceleration of the tracking target corresponding to the second moment based on the particle filter algorithm.

10. An electronic device, characterized in that, Including a memory and a processor, where: The memory is used to store a computer program; The processor is configured to execute the computer program to implement the target tracking method according to any one of claims 1-8.

Images