Radar and infrared thermal imaging composite target detection and tracking method and storage medium

Abstract

A target detection and tracking method and storage medium combining radar and infrared thermal imaging are disclosed. The method includes the following steps: Step S1: Acquiring millimeter-wave radar data and raw infrared thermal imaging data within a given space; Step S2: Performing imaging processing on the radar and infrared data respectively, removing background from the radar imaging results; Step S3: Performing time-dimensional registration on the infrared imaging results based on the radar refresh rate; performing CFAR detection processing on the matched results and clustering to output the target point cloud and centroid; Step S5: Using the transformation relationship between the spatial three-dimensional coordinate system and the infrared two-dimensional image coordinate system, obtaining the set of pixel positions of the target in the two-dimensional infrared thermal image; Step S6: Performing thermal feature calculation; Step S7: Outputting the target feature information. The storage medium is used to implement the above method. This invention has the advantages of simple principle, wide applicability, and high detection accuracy.

Description

Technical Field

[0001] This invention mainly relates to the field of rescue detection equipment technology, specifically a target detection and tracking method and storage medium that combines radar and infrared thermal imaging. Background Technology

[0002] Fire is one of the most common disasters threatening people's lives and property. How firefighters can quickly and effectively locate people hidden behind flames or smoke at a fire scene is undoubtedly a crucial issue. Although firefighting equipment is becoming increasingly advanced, there is still a lack of substantial solutions to the problem of how firefighters can quickly find people hidden by flames or smoke at a fire scene.

[0003] Infrared thermal imagers, as existing equipment, have been used by firefighters in fire scene rescues. However, when rescuers are surrounded and obscured by flames and thick smoke, these devices often struggle to detect those behind them, easily missing rescue opportunities. Millimeter-wave detection equipment, with its relatively longer wavelength, can easily penetrate obstacles such as thick smoke and thin wooden boards. Furthermore, it is unaffected by temperature and can detect life forms behind flames and high-temperature gases. However, it is prone to false alarms in complex environments.

[0004] Current fire rescue detection equipment mainly consists of infrared thermal imagers and millimeter-wave detectors. However, infrared thermal imagers are severely affected by factors such as dense smoke and flames when detecting living beings, making them prone to missing detections. Meanwhile, millimeter-wave detection equipment is prone to generating false alarms in complex fire environments, affecting the efficiency of rescue efforts. Summary of the Invention

[0005] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a target detection and tracking method and storage medium that combines radar and infrared thermal imaging, which is simple in principle, has a wide range of applications, and has high detection accuracy.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A target detection and tracking method combining radar and infrared thermal imaging, comprising the following steps:

[0008] Step S1: Collect millimeter-wave radar data and raw infrared thermal imaging data within a given space;

[0009] Step S2: Perform imaging processing on the radar and infrared data respectively, and remove the background from the radar imaging results;

[0010] Step S3: Based on the radar refresh rate, perform time dimension registration on the infrared imaging results using data interpolation methods; perform CFAR detection processing on the matched radar 3D imaging results, cluster the detection results, and output the target point cloud and centroid.

[0011] Step S4: Transform the output target point cloud into the infrared two-dimensional image coordinate system using the transformation relationship between the spatial three-dimensional coordinate system and the infrared two-dimensional image coordinate system to obtain the set of pixel positions of the target in the two-dimensional infrared thermal image.

[0012] Step S5: Calculate the thermal features of the projection set of the target on the two-dimensional infrared thermal image;

[0013] Step S6: Output the feature information of the target.

[0014] As a further improvement to the method of the present invention: in step S3, the imaging results between sensors are correlated and matched in the time dimension, the method being:

[0015] Radar sensor The thermal image corresponding to the frame imaging result is obtained through numerical interpolation:

[0016]

[0017] in The time interval between two adjacent radar frames. This represents the time interval between two adjacent frames of an infrared thermal imager. To obtain a heat map of the scene, where This indicates the frame number of the infrared thermal imager.

[0018] As a further improvement to the method of the present invention: the radar three-dimensional imaging results after background clutter filtering are subjected to constant false alarm rate (CFAR) detection, specifically as follows:

[0019]

[0020] Threshold value It can be calculated using the following formula:

[0021]

[0022] in, For testing points The signal mean within the reference cell, The number of reference units is set. The false alarm probability is initially set to a fixed value, denoted as . In subsequent detection processes, the false alarm probability parameters of the area will be adjusted based on the thermal characteristics of different locations. To obtain the scene's 3D imaging results after filtering out background clutter, where This indicates the frame number of the millimeter-wave radar.

[0023] As a further improvement to the method of the present invention: in step S3, the binarized result obtained from constant false alarm rate detection is... Perform clustering; the clustering results in the first... The set of 3D point clouds corresponding to the target class is denoted as , This represents the number of point clouds in this type of set.

[0024] As a further improvement to the method of the present invention: in step S3, the target point cloud set in three-dimensional space is mapped to a two-dimensional infrared thermal image through coordinate system transformation. The mapping result for the target class is:

[0025]

[0026] in The focal length of the thermal imager. Let be the coordinates of the optical center in the image. The first digit is obtained by calculating using the above formula. thermal image pixel set of the target class .

[0027] As a further improvement to the method of the present invention: in step S5, thermal features are calculated using the extracted thermal image pixel values.

[0028]

[0029] As a further improvement to the method of the present invention: the obtained target thermal features are fed back to CFAR detection, and the radar sensor in the first... In the frame, the first The false alarm probability parameter near the target class is calculated by the following formula:

[0030]

[0031] in for:

[0032]

[0033] in, For the reference values ​​of the thermal characteristics of the target of interest, where To set the maximum fluctuation value of thermal characteristics, This is the proportional coefficient corresponding to the maximum fluctuation parameter, and its value is often greater than 0. It is an adjustable parameter.

[0034] As a further improvement to the method of the present invention: in step S5, the calculation results are fed back to CFAR detection, tracking and track management in order to adaptively adjust the relevant parameters.

[0035] As a further improvement to the method of the present invention: in step S6, the feature information includes centroid position, velocity, millimeter-wave point cloud and infrared thermal image information.

[0036] The present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0037] Compared with the prior art, the advantages of the present invention are as follows:

[0038] 1. The target detection and tracking method and storage medium of the present invention, which combines radar and infrared thermal imaging, is simple in principle, has a wide range of applications, and high detection accuracy. It can integrate the feature information obtained by millimeter-wave radar and infrared thermal imager, and adaptively adjust the target detection coefficient and tracking coefficient to improve the detection performance of rescue equipment and reduce the probability of false alarms.

[0039] 2. The target detection and tracking method and storage medium of the present invention, which combines radar and infrared thermal imaging, enhances target detection and tracking performance by fusing feature information acquired from millimeter-wave radar and infrared thermal imagers, thereby solving the problems of poor target detection performance and high false alarm rate in complex environments such as fires and dense smoke. The present invention employs a multi-sensor composite detection method, which improves the device's perception of the environment and living organisms; by calculating the feature information of the infrared thermal imaging results, the present invention adaptively adjusts the detection and tracking parameters of the millimeter-wave radar, greatly reducing the probability of false alarms. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method of the present invention.

[0041] Figure 2 This is a schematic diagram of the starting trajectory in a specific application example of the present invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the present invention provides a target detection and tracking method combining radar and infrared thermal imaging, the steps of which include:

[0044] Step S1: Collect millimeter-wave radar data and raw infrared thermal imaging data within a given space;

[0045] Step S2: Perform imaging processing on radar and infrared data respectively, and then remove the background from the radar imaging results using a filtering algorithm;

[0046] Step S3: Based on the radar refresh rate, perform time dimension registration on the infrared imaging results using data interpolation methods; perform CFAR detection processing on the matched radar 3D imaging results, cluster the detection results, and output the target point cloud and centroid.

[0047] Step S4: Transform the output target point cloud into the infrared two-dimensional image coordinate system using the transformation relationship between the spatial three-dimensional coordinate system and the infrared two-dimensional image coordinate system to obtain the set of pixel positions of the target in the two-dimensional infrared thermal image.

[0048] Step S5: The projection set of the target on the two-dimensional infrared thermal image is used to calculate the thermal features according to the pre-set rules, and the calculation results are fed back to CFAR detection, tracking and track management to realize adaptive adjustment of relevant parameters in order to improve the detection performance of the equipment and reduce false alarms and missed alarms.

[0049] Step S6: Finally output the target's feature information; the feature information includes the centroid position, velocity, millimeter-wave point cloud and infrared thermal image information.

[0050] By using existing radar 3D imaging algorithms and background cancellation algorithms, the 3D imaging result of the scene after filtering out background clutter can be obtained, denoted as... ,in This indicates the frame number of the millimeter-wave radar.

[0051] Existing infrared thermal imaging technology can be used to obtain a thermal distribution map of a scene, denoted as... ,in This indicates the frame number of the infrared thermal imager.

[0052] Since radar and infrared thermal imagers often have different refresh rates, to facilitate feature fusion in the time dimension, in specific application examples, this invention further correlates and matches the imaging results between sensors in the time dimension. The specific method is as follows:

[0053] Radar sensor The thermal image corresponding to the frame imaging result can be obtained through numerical interpolation:

[0054] (1)

[0055] in The time interval between two adjacent radar frames. This represents the time interval between two adjacent frames of an infrared thermal imager.

[0056] Furthermore, this invention performs constant false alarm rate (CFAR) detection on the radar three-dimensional imaging results after filtering out background clutter, specifically as follows:

[0057] (2)

[0058] Threshold value It can be calculated using the following formula:

[0059] (3)

[0060] in For testing points The signal mean within the reference cell, The number of reference units is set. The false alarm probability is initially set to a fixed value, denoted as . In subsequent detection processes, the false alarm probability parameters of the area will be adjusted based on the thermal characteristics of different locations.

[0061] Subsequently, the present invention further binarizes the results obtained from constant false alarm rate (CFAR) detection. Perform clustering; the clustering results in the first... The set of 3D point clouds corresponding to the target class is denoted as , The number of point clouds in this type of set;

[0062] Furthermore, this invention maps a three-dimensional target point cloud set onto a two-dimensional infrared thermal image through coordinate system transformation. The mapping result for the target class is:

[0063] (4)

[0064] in The focal length of the thermal imager. Let be the coordinates of the optical center in the image. The first digit can be obtained by calculation using equation (4). thermal image pixel set of the target class .

[0065] Furthermore, in specific application examples, this invention performs thermal feature calculations using the extracted pixel values ​​of the thermal image:

[0066] (5)

[0067] Furthermore, this invention feeds back the obtained target thermal features to CFAR detection, specifically: the radar sensor in the... In the frame, the first The false alarm probability parameter near the target class can be calculated by the following formula:

[0068] (6)

[0069] in for:

[0070] (7)

[0071] in, The thermal characteristic reference value of the target of interest (which is a fixed value), where To set the maximum fluctuation value of thermal characteristics, This is the proportional coefficient corresponding to the maximum fluctuation parameter, and its value is often greater than 0. It is an adjustable parameter.

[0072] Furthermore, the present invention can also adjust the threshold of the initial trajectory in trajectory association by means of thermal characteristic parameters. The method for adjusting the initial trajectory is as follows:

[0073] As Figure 2 As shown, in the first At frame time, determine the first Whether a target class has established a track requires calculation of the first M frames detected. Total number of class targets Total number of times Threshold value of the starting track In comparison, when Time indicates the first Once the objective is achieved and the flight path is established, when The time indicates that the track establishment failed. The initial track can be adjusted by adjusting P. According to equation (3), the initial track can be adjusted by adjusting parameter P using the obtained thermal characteristic parameters:

[0074] (8)

[0075] in Indicates the first Did the first frame detect the first frame? A flag for the target class; its value is 1 if detected, and 0 otherwise. This indicates the total number of times a target is detected within the current window. , The initial value of the starting track threshold, and , This is an adjustable parameter. Once a target track is established, subsequent targets of this type do not need to undergo the same track establishment process again. Using the target detection and tracking methods described above, after stable detection and tracking, the relevant feature information of the target is finally output.

[0076] The present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0077] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A target detection and tracking method combining radar and infrared thermal imaging, characterized in that, The steps include: Step S1: Collect millimeter-wave radar data and raw infrared thermal imaging data within a given space; Step S2: Perform imaging processing on the radar and infrared data respectively, and remove the background from the radar imaging results; Step S3: Based on the radar refresh rate, perform time dimension registration on the infrared imaging results using data interpolation methods; perform CFAR detection processing on the matched radar 3D imaging results, cluster the detection results, and output the target point cloud and centroid. Step S4: Transform the output target point cloud into the infrared two-dimensional image coordinate system using the transformation relationship between the spatial three-dimensional coordinate system and the infrared two-dimensional image coordinate system to obtain the set of pixel positions of the target in the two-dimensional infrared thermal image. Step S5: Calculate the thermal features of the projection set of the target on the two-dimensional infrared thermal image; Step S6: Output the target's feature information; In step S3, the imaging results between sensors are correlated and matched in the time dimension, and the method is as follows: Radar sensor The thermal image corresponding to the frame imaging result is obtained through numerical interpolation: in The time interval between two adjacent radar frames. This represents the time interval between two adjacent frames of an infrared thermal imager. To obtain a heat map of the scene, where Indicates the frame number of the infrared thermal imager; In step S3, the target point cloud set in three-dimensional space is mapped to a two-dimensional infrared thermal image through coordinate system transformation. The mapping result for the target class is: in The focal length of the thermal imager. Let be the coordinates of the optical center in the image. The first digit is obtained by calculating using the above formula. thermal image pixel set of the target class .

2. The target detection and tracking method combining radar and infrared thermal imaging according to claim 1, characterized in that, The radar 3D imaging results, after filtering out background clutter, are subjected to constant false alarm rate (CFAR) detection, specifically as follows: Threshold value It can be calculated using the following formula: in, For testing points The signal mean within the reference cell, The number of reference units is set. The false alarm probability is initially set to a fixed value, denoted as . In subsequent detection processes, the false alarm probability parameters of the area will be adjusted based on the thermal characteristics of different locations. To obtain the scene's 3D imaging results after filtering out background clutter, where This indicates the frame number of the millimeter-wave radar.

3. The target detection and tracking method combining radar and infrared thermal imaging according to claim 1, characterized in that, In step S3, the binarized result obtained from constant false alarm rate detection is... Perform clustering; the clustering results in the first... The set of 3D point clouds corresponding to the target class is denoted as , This represents the number of point clouds in this type of set.

4. The target detection and tracking method combining radar and infrared thermal imaging according to any one of claims 1-3, characterized in that, In step S5, thermal features are calculated using the extracted pixel values ​​of the thermal image. 。 5. The target detection and tracking method combining radar and infrared thermal imaging according to claim 4, characterized in that, The obtained target thermal features are fed back to CFAR detection, and the radar sensor in the [missing information]... In the frame, the first The false alarm probability parameter near the target class is calculated by the following formula: in for: in, For the reference values ​​of the thermal characteristics of the target of interest, where To set the maximum fluctuation value of thermal characteristics, This is the proportional coefficient corresponding to the maximum fluctuation parameter, and its value is often greater than 0. It is an adjustable parameter.

6. The target detection and tracking method combining radar and infrared thermal imaging according to any one of claims 1-3, characterized in that, In step S5, the calculation results are fed back to CFAR detection, tracking and track management to adaptively adjust relevant parameters.

7. The target detection and tracking method combining radar and infrared thermal imaging according to any one of claims 1-3, characterized in that, In step S6, the feature information includes centroid position, velocity, millimeter-wave point cloud and infrared thermal image information.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method as described in any one of claims 1-7.

Images