A method for identifying trapped persons and detecting them in three dimensions in fire disaster environments

Abstract

The invention discloses a method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment, which belongs to the field of imaging detection technology and comprises: building a detector, wherein the detector comprises a binocular infrared polarization camera and a laser radar; calculating a left-eye infrared polarization image by taking a left-eye original image, then inputting the image into a model to obtain a target recognition result, and extracting the position and size information of a recognition frame; generating a disparity map by taking a pair of binocular images, calculating a depth map by taking the disparity map, converting the depth information in the depth map into dense three-dimensional point cloud data, acquiring sparse three-dimensional point cloud data of the target by means of a laser radar, performing coordinate conversion between the sparse three-dimensional point cloud data and the dense three-dimensional point cloud data, calculating the average depth of the sparse three-dimensional point cloud data in the recognition frame, setting a threshold, removing outliers greater than the threshold in the dense three-dimensional point cloud data, obtaining a three-dimensional reconstruction result, and improving the ability to identify and three-dimensionally detect trapped persons in a fire scene.

Description

Technical Field

[0001] The present invention relates to the field of imaging detection technology, and in particular to a method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment. Background Art

[0002] Fire disasters are extremely harmful to people's lives and property. When a disaster occurs, using a search and rescue device to search for trapped people can effectively reduce the casualties caused by the fire and ensure the personal safety of search and rescue personnel. At present, the field of indoor firefighting and disaster relief mainly uses infrared imaging search and rescue equipment to detect life, detect fire sources, monitor the spread of fire and carry out search and rescue. However, the high temperature and thick smoke at the fire scene will not only reduce the visibility of the human eye, but also affect the imaging quality of the infrared imaging equipment, reduce the imaging distance of the equipment, and increase the difficulty of search and rescue. Summary of the invention

[0003] In order to solve the problems of the prior art, the present invention provides a method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment, the method comprising:

[0004] S1: Building a detector, which includes a binocular infrared polarization camera and a laser radar;

[0005] S2: calibrating and correcting the detector;

[0006] S3: photographing a binocular image pair of the target by using the binocular infrared polarization camera;

[0007] S4: the left original image in the binocular image pair Perform a demosaicing operation to obtain the left-eye infrared image at four polarization angles. , and then the left eye infrared polarization image is obtained by calculation , the formula is as follows:

[0008] ;

[0009] ;

[0010] in, It represents the left-eye infrared image after fusion processing of the target photographed by the left-eye infrared polarization camera. , , and They represent the left-eye infrared images of the target photographed by the left-eye infrared polarization camera with polarization directions of 0°, 45°, 90° and 135° respectively;

[0011] The left-eye infrared polarization image Input the trained YOLOv6 model to obtain the target recognition result, and extract the position and size information of the recognition box;

[0012] S5: De-mosaicing the binocular image pair to obtain a binocular infrared image, obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera through S2, then correcting the binocular infrared image, and using a semi-global block matching algorithm to perform stereo matching and generate a disparity map, calculating a depth map based on the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera, and converting the depth information in the depth map into dense three-dimensional point cloud data , the sparse three-dimensional point cloud data of the target is obtained by the laser radar , the sparse three-dimensional point cloud data The 3D point cloud coordinate system is transformed into the dense 3D point cloud data In the three-dimensional point cloud coordinate system, the position and size information of the identification frame in S4 is obtained, and the sparse three-dimensional point cloud data in the identification frame is calculated. The average depth , set the threshold , remove the dense three-dimensional point cloud data Greater than threshold Outliers of the target are obtained by , and obtain the three-dimensional reconstruction result.

[0013] Furthermore, the S1 specifically includes:

[0014] The binocular infrared polarization camera and the laser radar are installed on a guide rail through a card slot. The binocular infrared polarization cameras are parallel to each other and are equipped with the same infrared lens. The laser radar is located between the binocular infrared polarization cameras.

[0015] Furthermore, the guide rail includes: two vertical rods and a horizontal rod, the horizontal rod is fixed between the two vertical rods by screws, and the binocular infrared polarization camera and the laser radar are installed on the horizontal rod through a slot.

[0016] Furthermore, the detector also includes: a power supply module, a development board and a casing;

[0017] The power supply module is electrically connected to the development board, the binocular infrared polarization camera and the laser radar, and the development board is also electrically connected to the binocular infrared polarization camera and the laser radar;

[0018] The power supply module, the development board, the binocular infrared polarization camera and the laser radar are located in the casing.

[0019] Furthermore, the S2 specifically includes:

[0020] S21: Calculate the baseline distance of the binocular infrared polarization camera according to the requirement of the maximum measurement distance, and adjust the card slot on the horizontal rod so that the baseline distance meets the requirement;

[0021] S22: photographing the infrared calibration plate with the binocular infrared polarization camera to obtain a binocular original infrared image of the calibration plate, and performing a demosaicing operation to obtain binocular infrared images of the calibration plate at four polarization angles;

[0022] S23: using the binocular infrared images of the calibration plate at four polarization angles to calculate and obtain a fused binocular infrared image of the calibration plate;

[0023] S24: Obtain the fused calibration plate sequence binocular infrared image through S22 and S23, and calibrate and correct the binocular infrared polarization camera using Zhang Zhengyou calibration method.

[0024] The beneficial effects of the technical solution provided by the embodiment of the present invention are as follows: the present invention uses a binocular infrared polarization camera combined with polarization imaging technology to obtain high-dimensional information on the surface of materials, the three-dimensional imaging capability of binocular imaging technology, and the detection capability of laser radar for indoor structural information, to construct a new type of fire detector based on infrared polarization binocular imaging and laser radar. The detection method of the detector is used to detect the fire environment, improve the imaging distance in the fire scene, the identification of trapped persons and the three-dimensional detection capability, and reduce the difficulty of search and rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 It is a structural schematic diagram of a detector provided by the present invention;

[0027] Figure 2 It is a flow chart of a method for identifying trapped persons and performing three-dimensional detection in a fire disaster environment provided by the present invention;

[0028] Figure 3 It is a binocular infrared image of a target provided by the present invention;

[0029] Figure 4 It is a calibration effect diagram provided by the present invention;

[0030] Figure 5 It is a correction effect diagram provided by the present invention;

[0031] Figure 6 It is a target recognition result graph provided by the present invention;

[0032] Figure 7 This is a three-dimensional reconstruction result diagram provided by the present invention.

[0033] Figure numerals: 1-power supply module; 2-development board; 3-housing; 4-guide rail; 5-binocular infrared polarization camera; 6-infrared lens; 7-laser radar. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] See also Figure 2 , a trapped person identification and three-dimensional detection method suitable for fire disaster environment, comprising the following steps:

[0037] Step (1): Build the detector.

[0038] It should be noted that see Figure 1 The detector includes: a power supply module 1, a development board 2, a housing 3, a guide rail 4, a binocular infrared polarization camera 5, an infrared lens 6 and a laser radar 7.

[0039] The guide rail 4 includes two vertical rods and a horizontal rod. The horizontal rod is fixed between the two vertical rods by screws, and the position of the horizontal rod can be adjusted according to the screw holes of different lengths. When building, the binocular infrared polarization camera 5 and the laser radar 7 are installed on the horizontal rod through the card slot. The binocular infrared polarization camera 5 includes a left-eye infrared polarization camera and a right-eye infrared polarization camera. The left-eye infrared polarization camera and the right-eye infrared polarization camera are parallel to each other and are equipped with the same infrared lens 6. The laser radar 7 is located between the left-eye infrared polarization camera and the right-eye infrared polarization camera. The sensor plane of the laser radar 7 is on the same horizontal plane as the sensor plane of the binocular infrared polarization camera 5. Before use, the distance of the baseline of the binocular infrared polarization camera is calculated according to the requirements of the maximum measurement distance, and the baseline distance meets the requirements by adjusting the card slot on the horizontal rod, and then fixed on the horizontal rod.

[0040] In addition, the power supply module 1 is electrically connected to the development board 2, the binocular infrared polarization camera 5 and the laser radar 7 to provide power for them; the development board 2 is also electrically connected to the binocular infrared polarization camera 5 and the laser radar 7; the power supply module 1, the development board 2, the binocular infrared polarization camera 5 and the laser radar 7 are located in the housing 3.

[0041] Step (2): Calibrate and correct the detector.

[0042] Specifically, the method includes: calculating the distance of the baseline of the binocular infrared polarization camera according to the requirement of the maximum measurement distance, adjusting the card slot on the horizontal rod so that the baseline distance meets the requirement, and fixing it on the horizontal rod.

[0043] The infrared calibration plate is photographed by a binocular infrared polarization camera to obtain the original binocular infrared image of the calibration plate , and perform a demosaicing operation to obtain the binocular infrared images of the calibration plate at four polarization angles and , the formula is as follows:

[0044] , (1);

[0045] , (2);

[0046] In formula (1) and formula (2), represents the demosaicing operation, Represents the original infrared image of the left eye of the calibration plate obtained by the left eye infrared polarization camera. Represents the original infrared image of the right eye of the calibration plate obtained by the right eye infrared polarization camera. and Respectively represent the height and width of the original infrared image of the right eye of the calibration plate and the original infrared image of the left eye of the calibration plate, represents real numbers; , , and They respectively represent the left-eye infrared images of the calibration plate taken by the left-eye infrared polarization camera with polarization directions of 0°, 45°, 90° and 135°; , , and They respectively represent the right-eye infrared images of the calibration plate taken by the right-eye infrared polarization camera with polarization directions of 0°, 45°, 90° and 135°.

[0047] Using the binocular infrared image of the calibration plate at four polarization angles and Calculate the binocular infrared image of the calibration plate after fusion processing , the formula is as follows:

[0048] (3);

[0049] (4);

[0050] In formula (3) and formula (4), It represents the infrared image of the left eye of the calibration plate after fusion processing. It represents the infrared image of the right eye of the calibration plate after fusion processing.

[0051] Taking the infrared calibration plate as the target, the binocular infrared image of the calibration plate sequence after fusion processing is obtained by shooting and calculating through equations (3) and (4): , the binocular infrared polarization camera is calibrated and corrected using Zhang Zhengyou calibration method.

[0052] Step (3): Use a binocular infrared polarization camera to capture a binocular image pair of the target.

[0053] Step (4): The left original image in the binocular image pair Perform a demosaicing operation to obtain the left-eye infrared image at four polarization angles. , and then calculate the left eye infrared polarization image , the formula is as follows:

[0054] (5);

[0055] (6);

[0056] In formula (5) and formula (6), It represents the left-eye infrared image after fusion processing of the target photographed by the left-eye infrared polarization camera. , , and They respectively represent the left-eye infrared images of the target photographed by the left-eye infrared polarization camera with polarization directions of 0°, 45°, 90° and 135°.

[0057] The left infrared polarization image Input the trained YOLOv6 model to obtain the target recognition result, and extract the position and size information of the recognition box.

[0058] Step (5): Obtain the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera through step (2), de-mosaic the binocular image pair through the method in step (2) to obtain the binocular infrared image of the target, then correct the binocular infrared image of the target, and use the semi-global block matching algorithm (SGBM) to perform stereo matching and generate a disparity map, calculate the depth map based on the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera, and convert the depth information in the depth map into dense 3D point cloud data , obtain the sparse 3D point cloud data of the target through LiDAR , the sparse 3D point cloud data The 3D point cloud coordinate system is transformed into dense 3D point cloud data In the three-dimensional point cloud coordinate system, obtain the position and size information of the recognition frame in step (4), and calculate the sparse three-dimensional point cloud data in the recognition frame The average depth , set the threshold ,Pick , remove dense 3D point cloud data Greater than threshold Outliers of the target are obtained by , and obtain the three-dimensional reconstruction result.

[0059] It is worth mentioning that in the present invention, a new type of fire detector based on infrared polarization binocular imaging and laser radar is constructed by combining a binocular infrared polarization camera with the advantages of polarization imaging technology in obtaining high-dimensional information on the surface of materials, the three-dimensional imaging capability of binocular imaging technology, and the detection capability of laser radar for indoor structural information. The detection method of the detector is used to detect the fire environment, improve the imaging distance in the fire scene, the identification of trapped persons and the three-dimensional detection capability, and reduce the difficulty of search and rescue.

[0060] It should be noted that Figure 2 is the overall flow chart of the present invention, showing the overall process from setting up the sensor to target recognition and 3D reconstruction results; Figure 3 It is a binocular infrared image of a target provided by the present invention, which is obtained by demosaicing and fusion. It can be observed that the intensity information of the image is clearly restored after the demosaicing operation, which verifies the effectiveness of the image fusion of the present invention; Figure 4It is a calibration effect diagram provided by the present invention, which is the calibration result of the binocular infrared polarization camera in the detector, wherein the first row is the corner point detection result of the checkerboard calibration plate, the coordinate (0, 0) is the origin of the detected corner point of the checkerboard calibration plate, and X and Y are the horizontal axis and vertical axis of the plane coordinate system where the corner point is located respectively; the second row is the recognition result of the binocular infrared polarization camera on the calibration plate, the left side is the recognized calibration plate position, and the right side is the binocular infrared polarization camera position. It can be observed that all corner points of the checkerboard calibration plate are accurately recognized and located, verifying the effectiveness of the present invention in the binocular infrared polarization camera calibration; Figure 5 This is a correction effect diagram provided by the present invention, which is the epipolar correction result of the binocular infrared polarization camera photographing the calibration plate after the detector is calibrated. It can be observed that the binocular infrared image of the calibration plate is accurately corrected, which verifies the effectiveness of the present invention in the calibration of the binocular infrared polarization camera; Figure 6 This is a target recognition result diagram of a left-eye image provided by the present invention. It can be observed that the category and depth information of the target are accurately recognized and located. The accuracy of the target recognition as a human is 0.94, and the depth of the target from the detector is 1513 mm, which verifies the effectiveness of the present invention in target recognition. Figure 7 This is a three-dimensional reconstruction result diagram provided by the present invention. It can be observed that the three-dimensional shape of the target surface is accurately reconstructed, verifying the effectiveness of the present invention in three-dimensional detection of the target.

[0061] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment, characterized in that: The method comprises: S1: Build a detector, which includes a binocular infrared polarization camera and a laser radar, specifically including: The binocular infrared polarization camera and the laser radar are installed on the guide rail through the card slot, the binocular infrared polarization cameras are parallel to each other and are equipped with the same infrared lens, and the laser radar is located between the binocular infrared polarization cameras; The guide rail comprises: two vertical rods and a horizontal rod, the horizontal rod is fixed between the two vertical rods by screws, and the binocular infrared polarization camera and the laser radar are installed on the horizontal rod through a card slot; S2: calibrating and correcting the detector; S3: photographing a binocular image pair of the target by using the binocular infrared polarization camera; S4: the left original image in the binocular image pair Perform a demosaicing operation to obtain the left-eye infrared image at four polarization angles. , and then the left eye infrared polarization image is obtained by calculation , the formula is as follows: ； ； in, It represents the left-eye infrared image after fusion processing of the target photographed by the left-eye infrared polarization camera. , , and They represent the left-eye infrared images of the target photographed by the left-eye infrared polarization camera with polarization directions of 0°, 45°, 90° and 135° respectively; The left-eye infrared polarization image Input the trained YOLOv6 model to obtain the target recognition result, and extract the position and size information of the recognition box; S5: De-mosaicing the binocular image pair to obtain a binocular infrared image, obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera through S2, then correcting the binocular infrared image, and using a semi-global block matching algorithm to perform stereo matching and generate a disparity map, calculating a depth map based on the disparity map and the intrinsic parameter matrix and extrinsic parameter matrix of the binocular infrared polarization camera, and converting the depth information in the depth map into dense three-dimensional point cloud data , the sparse three-dimensional point cloud data of the target is obtained by the laser radar , the sparse three-dimensional point cloud data The 3D point cloud coordinate system is transformed into the dense 3D point cloud data In the three-dimensional point cloud coordinate system, the position and size information of the identification frame in S4 is obtained, and the sparse three-dimensional point cloud data in the identification frame is calculated. The average depth , set the threshold , remove the dense three-dimensional point cloud data Greater than threshold Outliers of the target are obtained by , and obtain the three-dimensional reconstruction result.

2. The method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment according to claim 1, characterized in that: The detector also includes: a power supply module, a development board and a casing; The power supply module is electrically connected to the development board, the binocular infrared polarization camera and the laser radar, and the development board is also electrically connected to the binocular infrared polarization camera and the laser radar; The power supply module, the development board, the binocular infrared polarization camera and the laser radar are located in the casing.

3. The method for identifying and three-dimensionally detecting trapped persons in a fire disaster environment according to claim 2, characterized in that: The S2 specifically includes: S21: Calculate the baseline distance of the binocular infrared polarization camera according to the requirement of the maximum measurement distance, and adjust the card slot on the horizontal rod so that the baseline distance meets the requirement; S22: photographing the infrared calibration plate with the binocular infrared polarization camera to obtain a binocular original infrared image of the calibration plate, and performing a demosaicing operation to obtain binocular infrared images of the calibration plate at four polarization angles; S23: using the binocular infrared images of the calibration plate at four polarization angles to calculate and obtain a fused binocular infrared image of the calibration plate; S24: Obtain the fused calibration plate sequence binocular infrared image through S22 and S23, and calibrate and correct the binocular infrared polarization camera using Zhang Zhengyou calibration method.

Images