Millimeter wave radar and camera combined multi-target deblurring monitoring method and system

Abstract

The invention discloses a millimeter-wave radar and camera combined multi-target deblurring monitoring method and system, and relates to the technical field of target tracking, and the method comprises the steps: erecting a millimeter-wave radar and a camera on the same vertical plane for monitoring; a checkerboard is used for recording the corresponding focusing time of the camera under different distances, and the function relation between the target distance and the focusing time under different focal length segments is obtained; distance, speed and angle information of multiple targets is obtained according to millimeter wave radar detection, tracks of the multiple targets are obtained, a multi-target tracking algorithm is used for conducting unified prediction and management on the tracks, and a track list is obtained; circularly selecting targets in different periods, and selecting an unfocused fuzzy target in the current period for monitoring; determining a corresponding focal length section according to the distance of the fuzzy target, finally calculating the focusing time according to the function relation, and adjusting the camera to the corresponding focal length, so that the imaging of the current fuzzy target is clear. According to the invention, accurate focusing of a plurality of targets in different distance intervals is realized.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a multi-target deblurring monitoring method and system combining a millimeter-wave radar and a camera. Background Art

[0002] Traditional camera monitoring methods can obtain rich and intuitive image information, but they also have problems such as poor anti-interference performance, blurry shooting images, inaccurate target positioning, and high manual monitoring costs. Millimeter-wave radar can monitor targets all day and all weather, and has strong penetration and anti-interference capabilities. It can accurately measure key information such as the distance, speed, and angle of the target, but it is difficult to identify the type of target and obtain the target's image information. Therefore, the monitoring method using a combination of millimeter-wave radar and camera can combine the advantages of both and improve the performance of security monitoring.

[0003] However, there has been no good solution to the problem of focus blur in multi-target monitoring. When there are multiple targets at different distances, due to the limitation of the camera's depth of field, the camera cannot focus on multiple targets at the same time, so it is difficult to achieve multi-target monitoring, which often leads to the loss of information of key targets. There are two common methods for the problem of target tracking and monitoring in video surveillance. One is to use only camera monitoring, detect multiple targets through visual perception algorithms, and intelligently identify targets with larger weights for tracking and monitoring; the other is to use millimeter-wave radar to obtain parameter information such as distance and angle of multiple targets, and then use data association algorithms to track multiple targets, calculate the target with the largest weight, and control the camera to focus on this target to achieve tracking and monitoring of a single target. However, these methods can only track and monitor a single target continuously. In the case of multiple targets, there is always a problem of focus blur, so it is difficult to meet the needs of tracking and monitoring multiple targets in practical applications.

[0004] Therefore, there is an urgent need for a multi-target tracking and monitoring solution to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-target deblurring monitoring method and system combining millimeter-wave radar and camera, which uses millimeter-wave radar and camera to jointly perform multi-target monitoring and tracking, and can achieve precise focusing on multiple targets in different distance intervals, thereby obtaining clear imaging of multiple targets.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multi-target deblurring monitoring method combining millimeter wave radar and camera, comprising the following steps:

[0008] S1. System calibration: Set up the millimeter wave radar and camera on the same vertical plane for monitoring;

[0009] The focus time of the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain the functional relationship between the target distance and the focus time at different focal lengths;

[0010] S2. Start tracking: obtain the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtain the tracks of multiple targets, use the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, and obtain the track list tracks;

[0011] S3, continuous tracking: based on the track list tracks obtained in S2, cyclically select targets in different periods, and select an unfocused blurred target for monitoring in the current period;

[0012] The corresponding focal length segment is determined according to the distance of the blurred target, and finally the focus time of the blurred target is calculated according to the functional relationship in S1, and the camera is adjusted to the corresponding focal length to make the image of the current blurred target clear.

[0013] Furthermore, the S1, system calibration: the millimeter wave radar and the camera are mounted on the same vertical plane for monitoring; the focus time corresponding to the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain a functional relationship between the target distance and the focus time at different focal length segments, specifically including the following steps:

[0014] S101, setting up the millimeter wave radar and the camera on the same vertical plane for monitoring, ensuring the consistency of the two in solving the target distance information, and determining the appropriate range of the monitoring target according to the sensor;

[0015] S102, print a chessboard to calibrate the data, and divide it into D focal length segments, which are focal lengths f1, ..., f D , and initialize i=1;

[0016] S103, set the camera focal length to f i , the corresponding monitoring distance interval is [R i ,R i +N i d i ]; Set the step size to d i , move the chessboard from the distance R i Start with step length d i Increase the distance successively and record the distance as r j The time it takes to achieve clear focus j , denoted as (r j,t j ), a total of N i Group data, j = 1, ..., N i ;

[0017] S104, based on the collected N i Group data (r j ,t j ), using the numpy.linalg.lstsq function in Python to fit t i about The quadratic curve is: where t i Indicates that at the focal length f i The focus clear time under the condition of r is the monitoring distance, a is i , b i and c i are the parameters obtained after curve fitting;

[0018] S105, let i=i+1; if i>D, then proceed to the subsequent operation, otherwise continue to repeat steps S103 and S104, and finally obtain the functional relationship between the target distance and the focusing time under different focal length segments: Where i=1,…,D.

[0019] Further, the S2, starting tracking: obtaining the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtaining the tracks of the multiple targets, and using the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, specifically includes:

[0020] S201, initialize the relevant parameters of the millimeter wave radar: carrier frequency is f0, modulation slope is K, modulation bandwidth is B, sampling rate is f s , the number of sampling points is Nr, the number of chirps is Na, and the modulation period is T; the corresponding linear frequency modulation signal is sent using millimeter wave radar;

[0021] Initialize the track list tracks = []; initialize the monitoring period to T M The current processing cycle ordinal number k=0; the target traversal direction d is initialized to 1; the tracking target ordinal number i is initialized to 1;

[0022] S202, the radar receives the target echo signal, and uses the target detection and clutter suppression algorithm to process the received echo signal in real time, and uses the DBSCAN function in Python to condense the point traces of the same target obtained by real-time processing to obtain N moving target point traces and obtain their corresponding distances r j , radial velocity v j and angle θ j , where j = 1,…,N;

[0023] The moving target points are listed in [x j ,y j ], and store all points in the point list measurements, where x j =r j sinθ j ,y j =r j cosθ j ;

[0024] S203, traverse the track list tracks, for each track track, traverse the point track list measurements, use the minimum neighbor algorithm to associate the point track measurement with the track track, and finally obtain the associated point track list associations;

[0025] S204, traverse the point track list measurements, for each point track measurement, find whether it is in the associations, if not, add it to the list unmatched_measurements, and use the list unmatched_measurements to store all points that are not matched to tracks; initialize the instantiation object of the track class Track for all the points in this list, and add them to the track list tracks, where each Track class object contains an instantiation object k of the KalmanFilter class f , used for subsequent Kalman filter tracking;

[0026] S205. All tracks in the track list tracks are managed uniformly, and whether each track matches the target in the current cycle is recorded. Tracks that have not been matched to point tracks for more than M times are deleted, and the number of unmatched times for tracks that have been matched to point tracks is cleared to zero; the track list tracks is traversed, and the Kalman filter algorithm is used to predict all tracks, and the prediction results are still stored in the track list tracks.

[0027] Furthermore, S3, continuous tracking: based on the track list tracks obtained in S2, cyclically selecting targets in different periods, selecting an unfocused blurred target for monitoring in the current cycle; determining the corresponding focal length segment according to the distance of the blurred target, and finally calculating the focus time of the blurred target according to the functional relationship in S1, adjusting the camera to the corresponding focal length, so that the image of the current blurred target is clear, specifically includes the following steps:

[0028] S301. Denote the length of the track list "tracks" as l. If d = 1, then take i = min(i, l) as the currently selected target ordinal number, and then set i = i + 1. If i > l, then set d = -1. If d = -1, then take i = max(i, 1) as the currently selected target ordinal number, and then set i = i - 1. If i < 1, then set d = 1.

[0029] S302. Take y = tracks[i - 1].kf.y, that is, the ordinate of the current position of the target in the selected target's track, and compare it with R i respectively, where i = 1, …, D. If R i < y < R i+1 and i < D, or y > Ri and i = D, then set the camera focal length to f i ;

[0030] S303. In the expression , take r = y, so as to calculate the corresponding focusing time t i , and then call the camera interface function to focus on the current blurred target to obtain a focused and clear target image.

[0031] S304. Set k = k + 1. When k < T M , then jump to step S302 to continue monitoring, otherwise terminate the monitoring operation.

[0032] The present invention also provides a multi-target deblurring monitoring system combining a millimeter-wave radar and a camera for executing the multi-target deblurring monitoring method combining the millimeter-wave radar and the camera, including:

[0033] A calibration module for mounting the millimeter-wave radar and the camera on the same vertical plane for monitoring;

[0034] A function relationship determination module for using a checkerboard to record the focusing time corresponding to the camera at different distances, and fitting different distances and the corresponding focusing times into a focusing parameter curve to obtain the function relationship between the target distance and the focusing time under different focal length segments;

[0035] A starting tracking module for obtaining the tracks of multiple targets based on the distance, speed, and angle information of the multiple targets detected by the millimeter-wave radar, and using a multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets to obtain a track list "tracks";

[0036] The continuous tracking module is used to cyclically select targets in different periods based on the track list tracks, select an unfocused blurred target for monitoring in the current cycle; determine its corresponding focal length segment according to the distance of the blurred target, and finally calculate the focus time of the blurred target according to the functional relationship, adjust the camera to the corresponding focal length, and make the image of the current blurred target clear.

[0037] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the multi-target deblurring monitoring method and system provided by the present invention, which uses the millimeter-wave radar and the camera to jointly monitor and track multiple targets, detects target information through radar, and designs a segmented multi-target monitoring algorithm to monitor multiple targets, and can accurately focus on multiple targets in different distance intervals, thereby obtaining clear imaging of multiple targets. The present invention realizes simple, effective and low-cost multi-target deblurring monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 This is a flow chart of the multi-target deblurring monitoring method combined with a millimeter-wave radar and a camera according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of using a checkerboard to calibrate data;

[0041] Figure 3 This is a result diagram of monitoring multiple targets in the T1 period in Example 2 of the present invention;

[0042] Figure 4 This is a graph showing the monitoring results of multiple targets in the T2 period in Example 2;

[0043] Figure 5 This is a graph of the monitoring results of multiple targets in the T3 period in Example 2. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The purpose of the present invention is to provide a multi-target deblurring monitoring method and system combining millimeter-wave radar and camera, which uses millimeter-wave radar and camera to jointly perform multi-target monitoring and tracking, and can achieve precise focusing on multiple targets in different distance intervals, thereby obtaining clear imaging of multiple targets.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] like Figure 1 As shown, a multi-target deblurring monitoring method combining a millimeter-wave radar and a camera provided in an embodiment of the present invention includes the following steps:

[0048] S1. System calibration: Set up the millimeter wave radar and camera on the same vertical plane for monitoring;

[0049] The focus time of the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain the functional relationship between the target distance and the focus time at different focal lengths;

[0050] S2. Start tracking: obtain the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtain the tracks of multiple targets, use the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, and obtain the track list tracks;

[0051] S3, continuous tracking: based on the track list tracks obtained in S2, cyclically select targets in different periods, and select an unfocused blurred target for monitoring in the current period;

[0052] The corresponding focal length segment is determined according to the distance of the blurred target, and finally the focus time of the blurred target is calculated according to the functional relationship in S1, and the camera is adjusted to the corresponding focal length to make the image of the current blurred target clear.

[0053] In this embodiment, the S1, system calibration: the millimeter wave radar and the camera are mounted on the same vertical plane for monitoring; the focus time corresponding to the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain a functional relationship between the target distance and the focus time at different focal lengths, specifically including the following steps:

[0054] S101, setting up the millimeter wave radar and the camera on the same vertical plane for monitoring, ensuring the consistency of the two in solving the target distance information, and determining the appropriate range of the monitoring target according to the sensor;

[0055] S102, print a chessboard to calibrate the data, and divide it into D focal length segments, which are focal lengths f1, ..., f D , and initialize i=1;

[0056] S103, set the camera focal length to f i , the corresponding monitoring distance interval is [R i ,R i +N i d i ]; Set the step size to d i , move the chessboard from the distance R i Start with step length d i Increase the distance successively and record the distance as r j The time it takes to achieve clear focus j , denoted as (r j ,t j ), a total of N i Group data, j = 1, ..., N i ;

[0057] S104, based on the collected N i Group data (r j ,t j ), using the numpy.linalg.lstsq function in Python to fit t i about The quadratic curve is: where t i Indicates that at the focal length f i The focus clear time under the condition of r is the monitoring distance, a is i 、b i and c i are the parameters obtained after curve fitting;

[0058] S105, let i=i+1; if i>D, then proceed to the subsequent operation, otherwise continue to repeat steps S103 and S104, and finally obtain the functional relationship between the target distance and the focusing time under different focal length segments: Where i=1,…,D.

[0059] In this embodiment, S2, starting tracking: obtaining the distance, speed and angle information of multiple targets according to millimeter wave radar detection, obtaining the tracks of multiple targets, and using a multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, specifically includes:

[0060] S201, initialize the relevant parameters of the millimeter wave radar: carrier frequency is f0, modulation slope is K, modulation bandwidth is B, sampling rate is f s , the number of sampling points is Nr, the number of chirps is Na, and the modulation period is T; the corresponding linear frequency modulation signal is sent using millimeter wave radar;

[0061] Initialize the track list tracks = []; initialize the monitoring period to T M The current processing cycle ordinal number k=0; the target traversal direction d is initialized to 1; the tracking target ordinal number i is initialized to 1;

[0062] S202, the radar receives the target echo signal, and uses the target detection and clutter suppression algorithm to process the received echo signal in real time, and uses the DBSCAN function in Python to condense the point traces of the same target obtained by real-time processing to obtain N moving target point traces and obtain their corresponding distances r j , radial velocity v j and angle θ j , where j = 1,…,N;

[0063] The moving target points are listed in [x j ,y j ], and store all points in the point list measurements, where x j =r j sinθ j ,y j =r j cosθ j ;

[0064] S203, traverse the track list tracks, for each track track, traverse the point track list measurements, use the minimum neighbor algorithm to associate the point track measurement with the track track, and finally obtain the associated point track list associations;

[0065] S204, traverse the point track list measurements, for each point track measurement, find whether it is in the associations, if not, add it to the list unmatched_measurements, and use the list unmatched_measurements to store all points that are not matched to tracks; initialize the instantiation object of the track class Track for all the points in this list, and add them to the track list tracks, where each Track class object contains an instantiation object k of the KalmanFilter class f , used for subsequent Kalman filter tracking;

[0066] S205. Uniformly manage all the tracks in the track list "tracks", record whether each track matches the target in the current cycle, delete the tracks that have not matched the point cloud for more than M times, and clear the number of unmatched times of the tracks that have matched the point cloud; traverse the track list "tracks" and use the Kalman filter algorithm to predict all the tracks, and still store the prediction results in the track list "tracks".

[0067] In this embodiment, step S3. Continuous tracking: Based on the track list "tracks" obtained in step S2, select the target cyclically in different periods, and select an out-of-focus blurred target for monitoring in the current cycle; determine the corresponding focal length segment according to the distance of the blurred target, and finally calculate the focusing time of the blurred target according to the function relationship in step S1, and adjust the camera to the corresponding focal length to make the image of the current blurred target clear, which specifically includes the following steps:

[0068] S301. Denote the length of the track list "tracks" as l; if d = 1, then take i = min(i, l) as the current selected target ordinal number, and then let i = i + 1. If i > l, then let d = -1; if d = -1, then take i = max(i, 1) as the current selected target ordinal number, and then let i = i - 1. If i < 1, then let d = 1;

[0069] S302. Take y = tracks[i - 1].kf.y, that is, the ordinate of the current position of the target in the track of the selected target, and compare it with R i respectively, where i = 1,..., D; if R i < y < R i+1 and i < D, or y > Ri and i = D, then set the camera focal length to f i ;

[0070] S303. In the expression , take r = y, so as to calculate the corresponding focusing time t i , and then call the camera interface function to focus on the current blurred target to obtain a focused and clear target image;

[0071] S304. Let k = k + 1; when k < T M , then jump to step S302 to continue monitoring, otherwise terminate the monitoring operation.

[0072] Embodiment 2

[0073] This embodiment of the present invention is a further introduction to the multi-target deblurring monitoring method combining the millimeter-wave radar and the camera described in Embodiment 1.

[0074] Step 1. System calibration

[0075] (a) Sensor deployment. TI AWR1843 millimeter-wave radar and Hikvision 3746FWDA4 / F-IZS camera are used for monitoring. The millimeter-wave radar and camera are set up on the same vertical plane for monitoring to ensure the consistency of the target distance information calculated by the two. According to the characteristics of the sensor, the number of monitored multiple targets is set to no more than 32, and the monitoring distance range is 1 to 50 meters.

[0076] (b) Printing a checkerboard for calibration data such as Figure 2 As shown; divide the focal length into 3 segments, namely, focal length f1 = 7mm, f2 = 16mm, f3 = 25mm, and initialize i = 1;

[0077] (c) Set the camera focal length to f i , the corresponding monitoring distance interval is [R i ,R i +N i d i ]; Set the step size to d i , move the chessboard from the distance R i Start with step length d i Increase the distance successively and record the distance as r j The time it takes to achieve clear focus j , denoted as (r j ,t j ), a total of N i Group data, i.e. j = 1,…,N i ;

[0078] When i=1, the distance interval is [1,3], the step length is 0.5m, and 5 groups of data are collected, namely (1,0), (1.5,100), (2,200), (2.5,300), (3,300); when i=2, the distance interval is [3.5,6], the step length is 0.5m, and 6 groups of data are collected, namely (3.5,900), (4,1000), (4.5,1000), (5,1100), (5.5,1100), (6,1100); when i=3, the distance interval is [7,25], the step length is 1m, and 19 groups of data are collected. According to, they are (7,1500), (8,1600), (9,1700), (10,1800), (11,1900), (12,1900), (13,2000), (14,2000), (15,2000), (16,2000), (17,2000), (18,2100), (19,2100), (20,2100), (21,2100), (22,2100), (23,2100), (24,2100), (25,2100); the units of distance are m, t i The unit is ms;

[0079] (d) According to the collected N i Group data (r j ,t j ), using the numpy.linalg.lstsq function in Python to fit t i about The quadratic curve is: where t i Indicates that at the focal length f i The focus clear time under the condition of r is the monitoring distance, a is i 、b i and c i are the parameters obtained after curve fitting;

[0080] (e) Let i = i + 1; if i>3, then proceed to the subsequent operation, otherwise continue to repeat steps (c) and (d), and finally obtain the functional relationship between the target distance and the focusing time under the three focal length segments; the fitting curve when i = 1 is: The fitted curve when i=2 is: The fitted curve when i=3 is:

[0081]

[0082] Step 2: Start tracking

[0083] (a) Initialize the radar parameters: carrier frequency f0 is 77 GHz, modulation slope K is 30 MHz / us, modulation bandwidth B is 1500 MHz, sampling rate f s is 10MHz, the number of sampling points Nr is 128, the number of chirps Na is 128, and the modulation period T is 50us; use the millimeter wave radar to send the corresponding linear frequency modulation signal; initialize the track list tracks = []; initialize the monitoring cycle number T M The number of the current processing cycle is 2000, the ordinal number k = 0; the initialization target traversal direction d is 1; the initialization tracking target ordinal number i is 1;

[0084] (b) The radar receives the target echo signal and uses target detection and clutter suppression algorithms to process the received echo signal in real time. The DBSCAN function in Python is used to condense the points of the same target obtained by real-time processing to obtain N moving target points and their corresponding distances r. j , radial velocity v j and angle θ j , where j = 1, ..., N; these points are listed in [x j ,y j], and store all the traces in the list measurements, where x j =r j sinθ j ,y j =r j cosθ j ;

[0085] Taking the processing cycles T1, T2 and T3 as examples, when k=T1, four moving target traces are obtained, among which the distance of target 1 is 6.24m, the radial velocity is 1.21m / s, and the angle is 8.56°; the distance of target 2 is 16.52m, the radial velocity is 0.94m / s, and the angle is -1.95°; the distance of target 3 is 25.13m, the radial velocity is -0.83m / s, and the angle is -6.57°; the distance of target 4 is 26.84m, the radial velocity is 0.76m / s, and the angle is -3.61°. At this time, measurements=[[0.93,6.17],[-0.56,16.51],[-2.88,24.96],[-1.69,26.79]];

[0086] When k = T2, four moving target traces are obtained, among which the distance of target 1 is 6.81m, the radial velocity is 1.21m / s, and the angle is 8.31°; the distance of target 2 is 17.01m, the radial velocity is 0.92m / s, and the angle is -1.96°; the distance of target 3 is 24.78m, the radial velocity is -0.81m / s, and the angle is -6.57°; the distance of target 4 is 27.25m, the radial velocity is 0.79m / s, and the angle is -3.61°. At this time, measurements = [[0.98, 6.74], [-0.58, 17.00], [-2.83, 24.62], [-1.72, 27.20]];

[0087] When k = T3, four moving target traces are obtained, among which the distance of target 1 is 7.15m, the radial velocity is 1.22m / s, and the angle is 7.83°; the distance of target 2 is 17.22m, the radial velocity is 0.92m / s, and the angle is -1.97°; the distance of target 3 is 24.68m, the radial velocity is -0.81m / s, and the angle is -6.54°; the distance of target 4 is 27.41m, the radial velocity is 0.76m / s, and the angle is -3.59°. At this time, measurements = [[0.97, 7.08], [-0.59, 17.21], [-2.81, 24.52], [-1.72, 27.36]];

[0088] (c) Traverse the track list "tracks". For each track "track" in it, traverse the measurement list "measurements", and use the minimum proximity algorithm to associate the measurement "measurement" with the track "track", and finally obtain the associated measurement list "associations".

[0089] (d) Traverse "measurements". For each measurement "measurement" in it, check whether it is in the list "associations". If not, add it to the list "unmatched_measurements", that is, store all the measurements that do not match the track in this list; initialize an instantiated object of the Track class for all the measurements in this list, and add it to the list "tracks". Each object of the Track class contains an instantiated object "kf" of the KalmanFilter class, which is used for subsequent Kalman filter tracking.

[0090] (e) Uniformly manage all the tracks in the list "tracks", record whether each track matches the target in the current cycle, delete the tracks that have not matched the measurement for more than 5 times, and clear the unmatched times of the tracks that have matched the measurement; traverse the list "tracks", and use the Kalman filter algorithm to predict all the tracks, and the results are still stored in the list "tracks".

[0091] Step Three: Continuous Tracking

[0092] (a) Denote the length of the list "tracks" as l; if d = 1, then take i = min(i, l) as the currently selected target ordinal number, and then let i = i + 1. If i > l, then let d = -1; if d = -1, then take i = max(i, 1) as the currently selected target ordinal number, and then let i = i - 1. If i < 1, then let d = 1; where when k = T1, i = 1; when k = T2, i = 2; when k = T3, i = 3.

[0093] (b) Take y = tracks[i - 1].kf.y, that is, the ordinate of the current position of the target in the selected target track, and compare it with R i respectively, where i = 1,..., D; if R i < y < R i+1 and i < D, or y > R i and i = D, then set the camera focal length to f i; When k = T1, y = 6.17m, since 3m<6.17<7m, the camera focal length is set to f3 = 16mm; when k = T2, y = 17.00m, since 17.00m>7m, the camera focal length is set to f3 = 25mm; when k = T3, y = 24.52m, since 24.52m>7m, the camera focal length is set to f3 = 25mm;

[0094] (c) In the expression In the example, y=r is taken to calculate the corresponding focusing time t i , and then call the camera interface to zoom and get a clearly focused target image;

[0095] When k = T1, the focusing time is

[0096] The obtained monitoring image is as follows Figure 3 As shown;

[0097] When k = T2, the focusing time is

[0098] The obtained monitoring image is as follows Figure 4 As shown;

[0099] When k = T3, the focusing time is

[0100] The obtained monitoring image is as follows Figure 5 As shown;

[0101] (d) Let k = k + 1; when k <T M , jump to step (b) to continue monitoring, otherwise terminate the monitoring operation.

[0102] The modulation transfer function (MTF) is used to evaluate the clarity of the above surveillance images. The spectrum amplitude is usually used to approximate the MTF, that is:

[0103] Since the clarity of the focused target needs to be evaluated in the entire image, the local MTF value needs to be calculated. The image can be divided into M small blocks, and the target area can be divided into N small blocks. The normalized MTF mean is calculated as: The larger the value, the clearer the focused image.

[0104] After calculation, Figure 3 In , the normalized MTF average of target 1 is 1.10, the normalized MTF average of target 2 is 0.89, the normalized MTF average of target 3 is 1.03, and the normalized MTF average of target 4 is 1.04. Obviously, the camera focuses on target 1, making its image clear, while other targets are relatively blurred. Figure 4In , the normalized MTF average of target 1 is 0.82, the normalized MTF average of target 2 is 1.30, the normalized MTF average of target 3 is 1.15, and the normalized MTF average of target 4 is 1.20. Obviously, the camera focuses on target 2, making its image clear, while other targets are relatively blurred. Figure 5 In the figure, the normalized MTF average of target 1 is 0.83, the normalized MTF average of target 2 is 1.09, the normalized MTF average of target 3 is 1.32, and the normalized MTF average of target 4 is 1.26. At this time, the camera focuses on target 3, and the images of target 3 and target 4 close to it are clear, while other targets are relatively blurred.

[0105] Example 3

[0106] The embodiment of the present invention provides a multi-target deblurring monitoring system combined with a millimeter wave radar and a camera, which is used to execute the multi-target deblurring monitoring method combined with a millimeter wave radar and a camera described in Example 1, including:

[0107] The calibration module is used to set up the millimeter wave radar and the camera on the same vertical plane for monitoring;

[0108] A functional relationship determination module is used to use a checkerboard to record the focus time corresponding to the camera at different distances, and fit different distances and corresponding focus times into a focus parameter curve to obtain a functional relationship between the target distance and the focus time at different focal length segments;

[0109] The starting tracking module is used to obtain the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtain the tracks of multiple targets, use the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, and obtain the track list tracks;

[0110] The continuous tracking module is used to cyclically select targets in different periods based on the track list tracks, select an unfocused blurred target for monitoring in the current cycle; determine its corresponding focal length segment according to the distance of the blurred target, and finally calculate the focus time of the blurred target according to the functional relationship, adjust the camera to the corresponding focal length, and make the image of the current blurred target clear.

[0111] In summary, the multi-target deblurring monitoring method and system combining millimeter-wave radar and camera provided by the present invention can realize the monitoring and tracking of multiple targets, and can achieve precise focusing on multiple targets in different distance ranges; the present invention is improved on the basis of the previous millimeter-wave radar and camera combined monitoring method, and can solve the focus blur problem in multi-target monitoring while maintaining low cost, and has high practical application value.

[0112] The remaining technical features in this embodiment can be flexibly selected by those skilled in the art according to actual conditions to meet different specific practical needs. However, it is obvious to those skilled in the art that these specific details are not necessary to implement the present invention.

[0113] Modifications and changes made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the scope of protection of the claims attached to the present invention. In the above description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details are not necessary to practice the present invention. In other examples, in order to avoid confusing the present invention, well-known technologies, such as specific construction details, operating conditions and other technical conditions, are not specifically described.

[0114] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-target deblurring monitoring method combining millimeter wave radar and camera, characterized in that: The following steps are involved: S1. System calibration: Set up the millimeter wave radar and camera on the same vertical plane for monitoring; The focus time of the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain the functional relationship between the target distance and the focus time at different focal lengths; S2. Start tracking: obtain the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtain the tracks of multiple targets, use the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, and obtain the track list tracks; S3, continuous tracking: based on the track list tracks obtained in S2, cyclically select targets in different periods, and select an unfocused blurred target for monitoring in the current period; The corresponding focal length segment is determined according to the distance of the blurred target, and finally the focus time of the blurred target is calculated according to the functional relationship in S1, and the camera is adjusted to the corresponding focal length to make the image of the current blurred target clear.

2. The multi-target deblurring monitoring method of the millimeter wave radar and camera combined according to claim 1 is characterized in that: S1, system calibration: The millimeter wave radar and the camera are mounted on the same vertical plane for monitoring; the focus time of the camera at different distances is recorded using a checkerboard, and different distances and corresponding focus times are fitted into a focus parameter curve to obtain a functional relationship between the target distance and the focus time at different focal lengths, specifically including the following steps: S101, setting up the millimeter wave radar and the camera on the same vertical plane for monitoring, ensuring the consistency of the two in solving the target distance information, and determining the appropriate range of the monitoring target according to the sensor; S102, print a chessboard to calibrate the data, and divide it into D focal length segments, which are focal lengths f1, ..., f D , and initialize i=1; S103, set the camera focal length to f i , the corresponding monitoring distance interval is [R i ,R i +N i d i ]; Set the step size to d i , move the chessboard from the distance R i Start with step length d i Increase the distance successively and record the distance as r j The time it takes to achieve clear focus j , denoted as (r j ,t j ), a total of N i Group data, j = 1, ..., N i ; S104, based on the collected N i Group data (r j ,t j ), using the numpy.linalg.lstsq function in Python to fit t i about The quadratic curve is: where t i Indicates that at the focal length f i The focus clear time under the condition of r is the monitoring distance, a is i 、b i and c i are the parameters obtained after curve fitting; S105, let i=i+1; if i>D, then proceed to the subsequent operation, otherwise continue to repeat steps S103 and S104, and finally obtain the functional relationship between the target distance and the focusing time under different focal length segments: Where i=1,…,D.

3. The multi-target deblurring monitoring method of the millimeter wave radar and camera combined according to claim 1 is characterized in that: S2, starting tracking: obtaining the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtaining the tracks of the multiple targets, and using the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, specifically including: S201, initialize the relevant parameters of the millimeter wave radar: carrier frequency is f0, modulation slope is K, modulation bandwidth is B, sampling rate is f s , the number of sampling points is Nr, the number of chirps is Na, and the modulation period is T; the corresponding linear frequency modulation signal is sent using millimeter wave radar; Initialize the track list tracks = []; initialize the monitoring period to T M The current processing cycle ordinal number k=0; the target traversal direction d is initialized to 1; the tracking target ordinal number i is initialized to 1; S202, the radar receives the target echo signal, and uses the target detection and clutter suppression algorithm to process the received echo signal in real time, and uses the DBSCAN function in Python to condense the point traces of the same target obtained by real-time processing to obtain N moving target point traces and obtain their corresponding distances r j , radial velocity v j and angle θ j , where j = 1,…,N; The moving target points are listed in [x j ,y j ], and store all points in the point list measurements, where x j =r j sinθ j ,y j =r j cosθ j ; S203, traverse the track list tracks, for each track track, traverse the point track list measurements, use the minimum neighbor algorithm to associate the point track measurement with the track track, and finally obtain the associated point track list associations; S204, traverse the point track list measurements, for each point track measurement, find whether it is in the associations, if not, add it to the list unmatched_measurements, and use the list unmatched_measurements to store all points that are not matched to tracks; initialize the instantiation object of the track class Track for all the points in this list, and add them to the track list tracks, where each Track class object contains an instantiation object k of the KalmanFilter class f , used for subsequent Kalman filter tracking; S205. All tracks in the track list tracks are managed uniformly, and whether each track matches the target in the current cycle is recorded. Tracks that have not been matched to point tracks for more than M times are deleted, and the number of unmatched times for tracks that have been matched to point tracks is cleared to zero; the track list tracks is traversed, and the Kalman filter algorithm is used to predict all tracks, and the prediction results are still stored in the track list tracks.

4. The multi-target deblurring monitoring method of the millimeter wave radar and camera combined according to claim 1 is characterized in that: S3, continuous tracking: based on the track list tracks obtained in S2, select targets cyclically in different periods, select an unfocused blurred target for monitoring in the current cycle; determine the corresponding focal length segment according to the distance of the blurred target, and finally calculate the focus time of the blurred target according to the functional relationship in S1, adjust the camera to the corresponding focal length, so that the image of the current blurred target is clear, specifically including the following steps: S301, the length of the track list tracks is l; if d = 1, then i = min (i, l) is taken as the currently selected target ordinal number, and then i = i + 1, if i> l, then d = -1; if d = -1, then i = max (i, 1) is taken as the currently selected target ordinal number, and then i = i-1, if i<1, then d = 1; S302. Take y = tracks[i - 1].kf.y, that is, the ordinate of the current position of the target in the selected target track, and compare it with R i respectively, where i = 1,..., D; if R i < y < R i+1 and i < D, or y > Ri and i = D, then set the camera focal length to f i ; S303, in the expression In the example, we take r = y, and then calculate the corresponding focusing time t i , and then call the camera interface function to focus the current blurred target to obtain a clearly focused target image; S304, let k=k+1; when k <T M If yes, then jump to step S302 to continue monitoring, otherwise terminate the monitoring operation.

5. A multi-target deblurring monitoring system combined with a millimeter-wave radar and a camera, used to execute the multi-target deblurring monitoring method combined with a millimeter-wave radar and a camera according to any one of claims 1 to 4, characterized in that: include: The calibration module is used to set up the millimeter wave radar and the camera on the same vertical plane for monitoring; A functional relationship determination module is used to use a checkerboard to record the focus time corresponding to the camera at different distances, and fit different distances and corresponding focus times into a focus parameter curve to obtain a functional relationship between the target distance and the focus time at different focal length segments; The starting tracking module is used to obtain the distance, speed and angle information of multiple targets according to the millimeter wave radar detection, obtain the tracks of multiple targets, use the multi-target tracking algorithm to uniformly predict and manage the tracks of the multiple targets, and obtain the track list tracks; The continuous tracking module is used to cyclically select targets in different periods based on the track list tracks, select an unfocused blurred target for monitoring in the current cycle; determine its corresponding focal length segment according to the distance of the blurred target, and finally calculate the focus time of the blurred target according to the functional relationship, adjust the camera to the corresponding focal length, and make the image of the current blurred target clear.

Images