Rescue site life detection method based on infrared imaging and laser radar

By combining infrared imaging and lidar detection methods, the data fusion of binocular infrared cameras and lidar is used to solve the accuracy of life recognition and in-depth information acquisition in emergency rescue sites, improving search and rescue efficiency and reducing safety risks.

CN119064948AInactive Publication Date: 2024-12-03SHAANXI QIANG POLARIZATION VISION TECH CO LTD

Patent Information

Application Number
CN202411564011.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional visualization and infrared imaging technologies are difficult to obtain accurate life recognition and in-depth information at emergency rescue sites, especially under complex environments and limited space, resulting in inefficient search and rescue and increased safety risks.

Method used

Using infrared imaging and lidar detection methods, images are captured through binocular infrared cameras, combined with target detection networks for identification, depth maps are generated using semi-global block matching algorithm, and fused with the three-dimensional point cloud data of lidar, and conditional filtering and iterate the nearest point registration algorithm to calculate accurate depth information.

Benefits of technology

Accurate life target detection and depth information acquisition in complex environments are achieved, error search caused by depth information errors are avoided, rescue efficiency is improved, resource waste and security risks are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119064948A_ABST
    Figure CN119064948A_ABST
Patent Text Reader

Abstract

The invention discloses a rescue site life detection method based on infrared imaging and a laser radar, and belongs to the technical field of imaging detection. The method comprises the following steps: constructing a detector, wherein the detector comprises a binocular infrared camera, an electric holder and a laser radar; shooting a binocular image, and inputting left view infrared data into the model for target detection; carrying out stereo matching on the life target in the binocular image, generating a disparity map, calculating a depth map according to the disparity map, and converting the depth map into three-dimensional point cloud data P; the electric holder rotates at a constant speed, the laser radar collects multiple groups of single-line data, each group of single-line data is projected into a three-dimensional space to obtain three-dimensional point cloud data Q, the three-dimensional point cloud data Q and the three-dimensional point cloud data P are registered, three-dimensional point cloud of the three-dimensional point cloud data Q is converted into a three-dimensional point cloud coordinate system of the three-dimensional point cloud data P, and the three-dimensional point cloud coordinate system of the three-dimensional point cloud data P is obtained. And the average depth information of the fused point clouds is solved, so that the depth information of the life target can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of imaging detection technology, and in particular to a rescue scene life detection method based on infrared imaging and laser radar. Background Art

[0002] Emergency rescue often requires searching and rescuing trapped people in complex environments, limited space, bad weather and other unfavorable factors. It is crucial to quickly identify trapped targets and obtain their accurate depth information, which helps rescue operations to be carried out quickly in the above environments, thereby saving more lives.

[0003] Traditional visual life detection equipment is mainly divided into visible light and infrared imaging technology. The imaging quality of visible light imaging technology is easily disturbed in emergency rescue sites such as dim environments and dusty environments. Even with active lighting, it is difficult to obtain good lighting conditions in narrow and foggy environments, so it is difficult to obtain reliable and accurate life recognition and depth measurement results; on the other hand, although life detectors based on infrared imaging technology can overcome the interference of adverse environments to a certain extent, due to the low spatial resolution of infrared imaging devices and other characteristics, this method cannot accurately obtain reliable depth information of the target. At the emergency rescue site, excessive errors in depth information often cause search and rescue personnel to search for a long time in the wrong area, resulting in reduced search and rescue efficiency, waste of search resources and increased safety risks for search and rescue personnel. Summary of the invention

[0004] In order to solve the problems of the prior art, the present invention provides a method for detecting life at a rescue site based on infrared imaging and laser radar, the method comprising:

[0005] S1: Building a detector, the detector comprising: a binocular infrared camera, an electric pan-tilt platform, and a laser radar arranged on the electric pan-tilt platform;

[0006] S2: calibrating and correcting the detector;

[0007] S3: taking a binocular image of the life target by using the binocular infrared camera;

[0008] S4: Inputting the left view infrared data in the binocular image into the trained target detection network model for target detection, outputting the recognition result, recognition accuracy and recognition frame of the life target, and extracting the position and size information of the recognition frame, and defining the area within the recognition frame as the ROI area of ​​interest;

[0009] S5: using the calibration and correction in S2 to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera, taking the pixel points in the ROI area as the object, stereo matching is performed on the life target in the binocular image through a semi-global block matching algorithm and a disparity map is generated, and a depth map of the life target is calculated according to the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera, and then the depth information in the depth map is converted into three-dimensional point cloud data P;

[0010] S6: The electric gimbal performs a uniform rotation motion, and the laser radar performs a rotation motion at a set time interval. Scan and collect multiple groups of single-line data, then combine the real-time quaternion posture read by the IMU in the electric gimbal, project each group of single-line data into three-dimensional space, fuse the three-dimensional space data of different frames, and use the conditional filtering algorithm to process the fused data to obtain three-dimensional point cloud data Q;

[0011] S7: Align the three-dimensional point cloud data Q and the three-dimensional point cloud data P, convert the three-dimensional point cloud of the three-dimensional point cloud data Q into the three-dimensional point cloud coordinate system of the three-dimensional point cloud data P, and fuse the two sets of point clouds to solve the average depth information of the fused point cloud, so as to obtain the depth information of the life target.

[0012] Furthermore, the S1 specifically includes:

[0013] The binocular infrared camera and the electric pan-tilt platform are installed on a guide rail through a card slot. The binocular infrared cameras are parallel to each other and are equipped with the same infrared lens. The laser radar and the electric pan-tilt platform are located between the binocular infrared cameras.

[0014] Furthermore, the guide rail includes: two vertical rods and a horizontal rod, the horizontal rod is fixed between the two vertical rods by screws, and the binocular infrared camera and the electric pan-tilt head are installed on the horizontal rod.

[0015] Furthermore, the detector also includes: a power supply module, a development board and a casing;

[0016] The power supply module is electrically connected to the development board, the binocular infrared camera, the electric pan-tilt platform and the laser radar, and the development board is also electrically connected to the binocular infrared camera, the electric pan-tilt platform and the laser radar;

[0017] The power supply module, the development board, the electric pan / tilt platform, the binocular infrared camera and the laser radar are located in the casing.

[0018] Furthermore, the S2 specifically includes:

[0019] The baseline distance of the binocular infrared camera is calculated according to the requirement of the maximum measurement distance, and the baseline distance is made to meet the requirement by adjusting the card slot on the horizontal rod. The infrared calibration plate is used as the target, and the binocular infrared camera is used to obtain sequential binocular infrared images, which are calibrated and corrected using the Zhang Zhengyou calibration method.

[0020] Further, the registering of the three-dimensional point cloud data Q and the three-dimensional point cloud data P in S7 specifically includes:

[0021] For each point of the three-dimensional point cloud data Q , find the point cloud data P with the same The closest matching point ;

[0022] Calculate the centroid of two sets of matching points and set the point and are the centroids of the matching points in the three-dimensional point cloud data Q and the three-dimensional point cloud data P, respectively, and the calculation method is as follows:

[0023] ;

[0024] in, Used to represent the number of points included in the three-dimensional point cloud data Q and the three-dimensional point cloud data P;

[0025] Decentralized processing to obtain a decentralized point set and :

[0026] ;

[0027] ;

[0028] in, Represents the point coordinates of the 3D point cloud data P before the i-th decentralized processing, Represents the coordinates of the point after the i-th decentralized processing of the 3D point cloud data P, Represents the coordinates of the point before the i-th decentralized processing of the 3D point cloud data Q, Represents the coordinates of the i-th point in the 3D point cloud data Q after decentralized processing;

[0029] Based on decentralized point set and , calculate the cross covariance matrix :

[0030] ;

[0031] in, express The transpose of

[0032] Cross-covariance matrix Perform singular value decomposition:

[0033] ;

[0034] in, represents an orthogonal matrix, represents a diagonal matrix, represents an orthogonal matrix, express The transpose of

[0035] Calculate the rotation matrix and translation vectors :

[0036] ;

[0037] ;

[0038] in, express The transpose of

[0039] Project the three-dimensional point cloud of the three-dimensional point cloud data Q to the three-dimensional point cloud coordinate system of the three-dimensional point cloud data P. , its corresponding point in the coordinate system of the three-dimensional point cloud data P It can be calculated by the following formula:

[0040] ;

[0041] Registration is complete.

[0042] The beneficial effects of the technical solution provided by the embodiment of the present invention are: the present invention uses infrared image data to realize life target detection under complex interference, and combines lidar data to provide reliable and accurate depth information, which can avoid the problem of search and rescue personnel conducting long-term searches in the wrong area due to depth information errors, and can effectively improve rescue efficiency, avoid waste of search resources, and reduce safety risks for search and rescue personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1It is a structural schematic diagram of a detector provided by the present invention;

[0045] Figure 2 The present invention provides a flowchart of a method for detecting life at a rescue scene based on infrared imaging and laser radar.

[0046] Figure numerals: 1-power supply module; 2-development board; 3-housing; 4-guide rail; 5-binocular infrared camera; 6-infrared lens; 7-electric pan / tilt; 8-laser radar. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] See also Figure 2 , a rescue scene life detection method based on infrared imaging and laser radar, comprising the following steps:

[0050] Step (1): Build the detector.

[0051] It should be noted that see Figure 1 The detector includes a power supply module 1, a development board 2, a housing 3, a guide rail 4, a binocular infrared camera 5, an infrared lens 6, an electric pan / tilt 7 and a laser radar 8.

[0052] The guide rail 4 includes two vertical rods and a horizontal rod. The horizontal rod is fixed between the two vertical rods by screws, and the position of the horizontal rod can be adjusted according to the screw holes of different lengths. When building, the binocular infrared camera 5 and the electric pan-tilt platform 7 are installed on the horizontal rod through the card slot. The binocular infrared camera 5 includes a left-eye infrared camera and a right-eye infrared camera. The left-eye infrared camera and the right-eye infrared camera are parallel to each other and are equipped with the same infrared lens 6. The electric pan-tilt platform 7 is located between the left-eye infrared camera and the right-eye infrared camera. The laser radar 8 is installed and fixed on the electric pan-tilt platform 7. The sensor plane of the laser radar 8 is on the same horizontal plane as the sensor plane of the binocular infrared camera 5. Before use, the distance of the baseline of the binocular infrared camera is calculated according to the requirements of the maximum measuring distance, and the baseline distance meets the requirements by adjusting the card slot on the horizontal rod, and fixed on the horizontal rod. The laser radar 8 is a single-line laser radar.

[0053] In addition, the power supply module 1 is electrically connected to the development board 2, the binocular infrared camera 5, the electric pan-tilt head 7, and the laser radar 8 to provide power for them; the development board 2 is also electrically connected to the binocular infrared camera 5, the electric pan-tilt head 7, and the laser radar 8; the power supply module 1, the development board 2, the binocular infrared camera 5, the electric pan-tilt head 7, and the laser radar 8 are located in the housing 3.

[0054] Step (2): Calibrate and calibrate the detector.

[0055] During calibration and correction, the baseline distance of the binocular infrared camera is first calculated according to the requirements of the maximum measurement distance. The baseline distance is adjusted to meet the requirements by adjusting the card slot on the horizontal rod. The infrared calibration plate is used as the target, and the binocular infrared camera is used to obtain sequential binocular infrared images. The Zhang Zhengyou calibration method is used for calibration and correction.

[0056] Step (3): Take a binocular image of the living target using a binocular infrared camera.

[0057] Step (4): Input the left view infrared data in the binocular image into the trained YOLOv7 model for target detection, output the recognition result, recognition accuracy and recognition frame of the life target, and extract the position and size information of the recognition frame. Define the area within the recognition frame as the ROI area of ​​interest.

[0058] Step (5): Using the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera obtained by calibration and correction in step (2), taking the pixel points in the ROI area as the object, the semi-global block matching algorithm (SGBM) is used to perform stereo matching on the life targets in the binocular image and generate a disparity map. The depth map of the life targets is calculated based on the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera, and then the depth information in the depth map is converted into three-dimensional point cloud data P.

[0059] Step (6): At the same time, after the detection starts, the electric gimbal rotates at a constant speed, and the laser radar moves at the set time interval. Scan and collect multiple groups of single-line data, each of which represents the point cloud features at the corresponding angle. Then, combined with the real-time quaternion posture read by the IMU in the electric gimbal, each group of single-line data is projected into three-dimensional space, and the three-dimensional space data of different frames are fused. After fusion, the conditional filtering algorithm is used for processing to obtain the three-dimensional point cloud data Q.

[0060] Each scan of the laser radar will produce a set of one-dimensional sequences, in which the data represents the arrive Angular range The distance r between the scanning point and the LiDAR. Assuming the angle The corresponding distance is , transform it into three-dimensional space coordinates The formula is as follows:

[0061] ;

[0062] Combined with the quaternion provided by the IMU, the 3D point is rotated to its corresponding spatial position. The rotated coordinates can be obtained by the following formula:

[0063] ;

[0064] Represents a quaternion ( , , , ) is the corresponding rotation matrix.

[0065] In order to ensure the reliability of the results, the obtained three-dimensional point cloud data Q is processed by designing conditional filtering of the minimum distance threshold and the Z-axis threshold to ensure that the obtained point cloud information is more reliable.

[0066] Step (7): After successfully acquiring the three-dimensional point cloud data Q of the laser radar and the three-dimensional point cloud data P of the binocular infrared camera, the two sets of point cloud data clouds need to be unified into the same coordinate system for registration.

[0067] Specifically include: using the iterative closest point (ICP) algorithm to solve the required rotation matrix and translation vectors .

[0068] Initialize the rotation matrix of the quantity to be solved and translation vectors ,in Initialized to the identity matrix, Initialized to the zero vector.

[0069] For each point of the 3D point cloud data Q , find the point cloud data P with the same The closest matching point .

[0070] Calculate the centroid of two sets of matching points and set the point and are the centroids of the matching points in the 3D point cloud data Q and 3D point cloud data P respectively, and the calculation method is as follows:

[0071] ;

[0072] in, Used to represent the number of points included in the three-dimensional point cloud data Q and the three-dimensional point cloud data P;

[0073] Decentralize each point to obtain a decentralized point set and :

[0074] ;

[0075] ;

[0076] in, Represents the point coordinates of the 3D point cloud data P before the i-th decentralized processing, Represents the coordinates of the point after the i-th decentralized processing of the 3D point cloud data P, Represents the coordinates of the point before the i-th decentralized processing of the 3D point cloud data Q, Represents the coordinates of the i-th point in the 3D point cloud data Q after decentralized processing;

[0077] Based on decentralized point set and , calculate the cross covariance matrix :

[0078] ;

[0079] in, express The transpose of

[0080] Cross-covariance matrix Perform singular value decomposition:

[0081] ;

[0082] in, represents an orthogonal matrix, represents a diagonal matrix, represents an orthogonal matrix, express The transpose of

[0083] Calculate the rotation matrix and translation vectors :

[0084] ;

[0085] ;

[0086] in, express The transpose of

[0087] Repeat the above operation to calculate the most suitable rotation matrix through continuous iteration. and translation vectors Project the 3D point cloud of the 3D point cloud data Q to the 3D point cloud coordinate system of the 3D point cloud data P. , its corresponding point in the coordinate system of the three-dimensional point cloud data P It can be calculated by the following formula:

[0088] ;

[0089] After the registration is completed, the lidar point cloud converted to the coordinate system of the binocular infrared camera is obtained. After the two sets of point clouds in the binocular infrared camera coordinate system are fused, the average depth information of the fused point cloud is solved to obtain the depth information of the life target.

[0090] It is worth noting that the present invention uses infrared image data to realize life target detection under complex interference, and combines lidar data to provide reliable and accurate depth information, which can avoid the problem of search and rescue personnel conducting long-term searches in the wrong area due to depth information errors. It can effectively improve rescue efficiency, avoid waste of search resources, and reduce safety risks for search and rescue personnel.

[0091] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting life at a rescue site based on infrared imaging and laser radar, characterized in that: The method comprises: S1: Building a detector, the detector comprising: a binocular infrared camera, an electric pan-tilt platform, and a laser radar arranged on the electric pan-tilt platform; S2: calibrating and correcting the detector; S3: taking a binocular image of the life target by using the binocular infrared camera; S4: Inputting the left view infrared data in the binocular image into the trained target detection network model for target detection, outputting the recognition result, recognition accuracy and recognition frame of the life target, and extracting the position and size information of the recognition frame, and defining the area within the recognition frame as the ROI area of ​​interest; S5: using the calibration and correction in S2 to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera, taking the pixel points in the ROI area as the object, stereo matching is performed on the life target in the binocular image through a semi-global block matching algorithm and a disparity map is generated, and a depth map of the life target is calculated according to the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared camera, and then the depth information in the depth map is converted into three-dimensional point cloud data P; S6: The electric gimbal performs a uniform rotation motion, and the laser radar performs a rotation motion at a set time interval. Scan and collect multiple groups of single-line data, then combine the real-time quaternion posture read by the IMU in the electric gimbal, project each group of single-line data into three-dimensional space, fuse the three-dimensional space data of different frames, and use the conditional filtering algorithm to process the fused data to obtain three-dimensional point cloud data Q; S7: Align the three-dimensional point cloud data Q and the three-dimensional point cloud data P, convert the three-dimensional point cloud of the three-dimensional point cloud data Q into the three-dimensional point cloud coordinate system of the three-dimensional point cloud data P, and fuse the two sets of point clouds to solve the average depth information of the fused point cloud, so as to obtain the depth information of the life target.

2. The rescue scene life detection method based on infrared imaging and laser radar according to claim 1 is characterized in that: The S1 specifically includes: The binocular infrared camera and the electric pan-tilt platform are installed on a guide rail through a card slot. The binocular infrared cameras are parallel to each other and are equipped with the same infrared lens. The laser radar and the electric pan-tilt platform are located between the binocular infrared cameras.

3. The rescue scene life detection method based on infrared imaging and laser radar according to claim 2 is characterized in that: The guide rail comprises: two vertical rods and a horizontal rod, the horizontal rod is fixed between the two vertical rods by screws, and the binocular infrared camera and the electric pan / tilt head are installed on the horizontal rod.

4. The method for detecting life at a rescue site based on infrared imaging and laser radar according to claim 3, characterized in that: The detector also includes: a power supply module, a development board and a casing; The power supply module is electrically connected to the development board, the binocular infrared camera, the electric pan-tilt platform and the laser radar, and the development board is also electrically connected to the binocular infrared camera, the electric pan-tilt platform and the laser radar; The power supply module, the development board, the electric pan / tilt platform, the binocular infrared camera and the laser radar are located in the casing.

5. The method for detecting life at a rescue site based on infrared imaging and laser radar according to claim 4, characterized in that: The S2 specifically includes: The baseline distance of the binocular infrared camera is calculated according to the requirement of the maximum measurement distance, and the baseline distance is made to meet the requirement by adjusting the card slot on the horizontal rod. The infrared calibration plate is used as the target, and the binocular infrared camera is used to obtain sequential binocular infrared images, which are calibrated and corrected using the Zhang Zhengyou calibration method.

6. The method for detecting life at a rescue site based on infrared imaging and laser radar according to claim 5, characterized in that: The registering of the three-dimensional point cloud data Q and the three-dimensional point cloud data P in S7 specifically includes: For each point of the three-dimensional point cloud data Q , find the point cloud data P with the same The closest matching point ; Calculate the centroid of two sets of matching points and set the point and are the centroids of the matching points in the three-dimensional point cloud data Q and the three-dimensional point cloud data P, respectively, and the calculation method is as follows: ; in, Used to represent the number of points included in the three-dimensional point cloud data Q and the three-dimensional point cloud data P; Decentralized processing to obtain a decentralized point set and : ; ; in, Represents the point coordinates of the 3D point cloud data P before the i-th decentralized processing, Represents the coordinates of the point after the i-th decentralized processing of the 3D point cloud data P, Represents the coordinates of the point before the i-th decentralized processing of the 3D point cloud data Q, Represents the coordinates of the i-th point in the 3D point cloud data Q after decentralized processing; Based on decentralized point set and , calculate the cross covariance matrix : ; in, express The transpose of Cross-covariance matrix Perform singular value decomposition: ; in, represents an orthogonal matrix, represents a diagonal matrix, represents an orthogonal matrix, express The transpose of Calculate the rotation matrix and translation vectors : ; ; in, express The transpose of Project the three-dimensional point cloud of the three-dimensional point cloud data Q to the three-dimensional point cloud coordinate system of the three-dimensional point cloud data P. , its corresponding point in the coordinate system of the three-dimensional point cloud data P It can be calculated by the following formula: ; Registration is complete.

Citation Information

Patent Citations

  • Survey, search and rescue robot with functions of simultaneous localization and map building

    CN105216905A

  • Vehicle safe driving assistance method, device, system and equipment and storage medium

    CN116778448A

  • Hyperbolic cooling tower inclination state identification method based on laser point cloud data

    CN118500353A

Cited By

  • Arm three-dimensional modeling method and system based on image processing and inertial sensing

    CN119850839A