A method for human target detection in disaster environments based on multi-sensor fusion

Abstract

This invention discloses a method for human target detection in disaster environments based on multi-sensor fusion, fully utilizing the advantages of three sensor fusions to achieve the task of human target detection. First, visible light and infrared images are preprocessed and registered. Second, features from the visible light image are fused to the infrared image to obtain an infrared-enhanced image, which is further processed to obtain a salient target image. Next, the infrared camera and 3D LiDAR are jointly calibrated. Then, based on the correspondence established by the joint calibration, a 3D enhanced point cloud is obtained. Finally, a point cloud semantic segmentation neural network model is used to perform semantic segmentation on the 3D enhanced point cloud to achieve human target detection. This invention improves the accuracy of human target detection, demonstrates the practicality of a multi-sensor fusion-based method for human target detection in disaster environments, and can provide reference and assistance for human target detection under disaster conditions.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor environmental perception technology for unmanned systems and autonomous robots, and specifically relates to a method for detecting human targets in disaster environments based on multi-sensor fusion. Background Technology

[0002] With the rapid development of unmanned systems and autonomous robot technologies, relying solely on 3D LiDAR or visual sensors (such as visible light cameras and infrared cameras) is no longer sufficient for unmanned systems and autonomous robots to perform autonomous perception and scene understanding in complex environments. Therefore, the detection performance and robustness of various sensors differ in different search and rescue environments. Utilizing multiple sensors to complete the tasks of disaster environment perception and human target recognition can achieve higher accuracy and robustness.

[0003] In existing technologies, active imaging technology based on lidar has advantages such as high spatial resolution and the ability to acquire three-dimensional feature information of objects. However, due to different surface reflection effects of the target, reflection angle and occlusion can have a significant impact on the test data. Infrared imaging mainly depends on the radiation characteristics of the target being measured. It is less affected by light intensity, illumination angle, and occlusion, and has better robustness to the external environment. However, it has low spatial resolution and lacks spatial distance information. Visible light imaging has better visual perception effects such as resolution, contrast, and texture details than infrared imaging. However, it is greatly affected by light and, like infrared imaging, still has inherent defects such as low spatial resolution of two-dimensional images and lack of spatial distance information.

[0004] Due to the complementary nature of sensors, tasks that cannot be completed with a single sensor can be accomplished with a combination of multiple sensors. However, for commonly used sensors such as 3D LiDAR, infrared cameras, and visible light cameras, the detection accuracy of human target detection using these three types of sensors is relatively low. How to effectively utilize their advantages to achieve accurate and robust human target detection and semantic segmentation is an urgent problem to be solved. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a method for human target detection in disaster environments based on multi-sensor fusion. It fuses information from three sensors—infrared images, visible light images, and infrared images combined with 3D LiDAR calibration—to make human features more prominent in the point cloud. Furthermore, it employs a correlation representation group of 3D enhanced point cloud intensity features to better describe the point cloud. A fusion downsampling method is used to aggregate point cloud features, improving the network's accuracy in human target detection. Finally, a neural network for point cloud semantic segmentation is constructed, overcoming the low detection accuracy of existing methods using these three sensors for human target detection.

[0006] The technical solution of this invention is as follows:

[0007] A method for detecting human targets in disaster environments based on multi-sensor fusion includes the following steps:

[0008] Step 1: Input the image acquired by the sensor and preprocess the input image, specifically including: sharpening the input infrared image and downsampling and grayscale processing the input visible light image.

[0009] Step 2: Perform image registration on the preprocessed infrared image and visible light image;

[0010] Step 3: Based on the image registration result, the features of the visible light image are fused into the infrared image to obtain an infrared enhanced image, and then the infrared enhanced image is further processed to obtain a significant target image;

[0011] Step 4: Perform joint calibration of the infrared camera and the 3D LiDAR to obtain the mapping relationship between the pixels of the infrared image and the 3D Cartesian coordinates of the 3D LiDAR point cloud.

[0012] Step 5: Based on the mapping relationship between the pixels of the infrared image and the three-dimensional rectangular coordinates of the three-dimensional lidar point cloud, the intensity value of the point corresponding to the pixel in the point cloud is enhanced by the gray value of the pixel in the significant target image and the intensity value of the three-dimensional lidar point cloud to obtain the three-dimensional enhanced point cloud.

[0013] Step 6: Use the spatial rectangular coordinates and intensity feature related representation groups of the 3D enhanced point cloud to perform feature description of the 3D enhanced point cloud;

[0014] Step 7: Input the relevant representation group of the spatial rectangular coordinates and intensity features of the 3D enhanced point cloud into the point cloud semantic segmentation neural network model, perform point cloud semantic segmentation on the 3D enhanced point cloud, generate semantic labels for each point, and finally realize human target detection.

[0015] Furthermore, in step 1, the input infrared image is sharpened using the Laplace operator.

[0016] Furthermore, the image registration operation between the infrared image and the visible light image in step 2 is specifically performed as follows:

[0017] Step 2.1: Perform edge detection on the visible light image and the infrared image using the Canny operator to obtain the Canny edge maps of the visible light image and the infrared image, respectively;

[0018] Step 2.2: Perform ORB feature point detection on the Canny edge maps of the visible light image and the infrared image respectively, and perform preliminary feature point pairing. Then, perform RANSAC purification on the preliminary pairing results to obtain the accurate transformation relationship between the visible light and infrared images.

[0019] Step 2.3: Complete image registration between visible light and infrared images through the transformation relationship.

[0020] Furthermore, the specific process for obtaining the infrared enhanced image and the salient target image in step 3 is as follows:

[0021] Step 3.1: The features of the image-registered visible light image are fused to the image-registered infrared image using wavelet transform to obtain the infrared enhanced image I. irp (u,v), where u,v>0;

[0022] Step 3.2: Obtain the saliency map S(u,v) of the infrared enhanced image. Based on the grayscale contrast between pixels, the sum of the grayscale distances between the pixel and all pixels in the image is taken as the saliency value of the pixel. Define s k For each pixel in the saliency map S(u,v), s k The significance value is represented as SalS(S k As shown in formula (1):

[0023]

[0024] Among them, s j For any pixel on S(u,v), Gray(s) * ) represents pixel s * grayscale value, F m Represents pixel s m The frequency, N s This represents the total number of pixels at pixel point S(u,v);

[0025] right There is Gray(s) k ) = SalS(s k Let the mean gray level of S(u,v) be avg. S , greater than avg S Set to 1, less than avg S Setting it to 0 will yield the binarized mask image mask1(u,v) of S(u,v), which will then be used to transform the infrared enhanced image I. irp The Hadamard product of (u,v) and mask1(u,v) yields the sub-image Sub1(u,v), as shown in formula (2):

[0026] Sub1(u,v)=Iirp (u,v)⊙mask1(u,v) (2)

[0027] Step 3.3: Parametric gamma correction is used to separate the background from the foreground;

[0028] To further reduce the influence of the background and achieve separation between the background and foreground, parametric gamma correction is employed. The average gray level of Sub1(u,v) is denoted as... Then there is a grayscale threshold for a certain pixel sub1(k) on Sub1(u,v). Pixels smaller than the threshold will be brightened, and pixels larger than the threshold will be darkened, resulting in a sub-image Sub2(u,v). The gamma correction of a pixel sub2(k) on Sub2(u,v) is shown in Equation (3):

[0029]

[0030] Where γ1 and γ2 are gamma correction parameters and γ1≤1 and γ2>1, ε is a compensation coefficient to prevent the base from being 0;

[0031] Therefore, the above expression can be reformulated as formula (4):

[0032]

[0033] Where a, b, c, and d are undetermined coefficients;

[0034] Step 3.4: Perform a binarization masking on Sub2(u,v) to obtain the binarized mask image of Sub2(u,v), denoted as mask2(u,v). Perform the Hadamard product between mask2(u,v) and Sub1(u,v) to obtain the final salient target image I. irt (u,v), represented by formula (5):

[0035] I irt (u,v)=Sub1(u,v)⊙mask2(u,v) (5)

[0036] Furthermore, the specific process of obtaining the 3D enhanced point cloud in step 5 is as follows:

[0037] Step 5.1, for a point cloud P in a certain scene o Construct a KD tree with nodes (x,y,z) to obtain P. o any point in the middle p o The set of neighborhood points within the radius Δr is Among them, ||D km ||2 represents point p k Point P m The Euclidean distance;

[0038] Step 5.2, define P as the set of points mapped from the jointly calibrated salient target image onto the 3D LiDAR point cloud. uv For the existence of P uv A point p in k Its neighborhood point set is Define Γ k For p k For P uv The effective set is shown in formula (6):

[0039]

[0040] Among them, th p Here, Φ is the threshold for the number of neighborhood points, Φ is the empty set, crad(.) represents the number of elements in the set, and C... pi P uv For p k In set P uv The complement above;

[0041] Step 5.3, define the effective mapping point set of a point (u,v) on a salient target image to its corresponding point cloud as shown in formula (7):

[0042]

[0043] For salient target image I irt A pixel i on (u,v) k For its effective mapping point set A point that exists in Record its strength value as If so, then its enhanced strength value As shown in formula (8):

[0044]

[0045] The concrete representation of the adaptive function is shown in formula (9):

[0046]

[0047] Where α,β,μ,σ 2 The coefficients are adaptive and undetermined; sgn() is the unit step function.

[0048] Step 5.4, obtain the p o Enhanced 3D point cloud and At a certain point The strength value is shown in formula (10):

[0049]

[0050] Furthermore, the relevant representation group of the spatial rectangular coordinates and intensity features of the 3D enhanced point cloud in step 6 specifically includes:

[0051] For a point (x0, y0, z0) in a 3D point cloud, there is an intensity value in0. After fusing image information, the intensity value is IN0. The 3D augmented point cloud is represented as...

[0052] Among them, for a point p in the point cloud l , (x l ,y l ,z l ) is p l coordinates of the spatial rectangular coordinate system, IN l For p l The enhanced strength value is expressed by the relationship shown in formula (11):

[0053]

[0054] Where r l For p l The Euclidean distance to the origin O(0,0,0) of the 3D lidar; define angθ l For p l Spatial angular characteristics, let there be a reference vector on the xOy plane in space. It has the origin O(0,0,0) as its endpoint and passes through point A(x). A ,y A ,0), where x A <<y A In particular, because the laser used is offset by ±60° in the positive x-axis direction, therefore With 3D augmented point cloud The range of the angle θ between any two points is: That is, 3D augmented point cloud It is impossible for a point p to exist. e , making remember Let the origin O(0,0,0) be the endpoint, and the point passing through point p be the length of the path. l The vector, then and cosine similarity As shown in formula (12):

[0055]

[0056] Among them (OA) i and (p) l ) i Representing vectors sum vector The components; in particular, since the data of type float is stored in memory in a "sign bit-exponent-mantissa" distribution, x is used. A =1,y A =2 n >>1 is used to optimize computational efficiency, and for float type z l The non-signed position 0 has sign(z) l )=z l &0x8000, thus allowing z l Positive and negative properties ζ(z) l ) is defined as formula (13):

[0057]

[0058] An approximation of formula (12) is shown in formula (14):

[0059]

[0060] Therefore, angθ can be further defined. l As shown in formula (15):

[0061]

[0062] Where, Δ θ It is to make angθ l The error correction variable has a more continuous distribution;

[0063] The theoretical value of the point cloud intensity of three-dimensional lidar is shown in formula (16):

[0064]

[0065] Where A represents the beam aperture measured at a fixed angle, λ is the incident angle, re is the reflectivity of the object, σ is the absorptivity of the medium, in0 is the output intensity of the 3D lidar, and η or It is the efficiency of a 3D lidar system, η t This refers to the efficiency of the 3D lidar system. Therefore, it can be known that the intensity value of the 3D lidar is... l Strongly correlated with angθ in terms of variables l and r l .

[0066] Therefore, the correlation representation group of point cloud intensity features is adopted. To enhance the feature description of point clouds, so as to better train the point cloud semantic segmentation network in the next step;

[0067] Specifically, while traditional (x, y, z) methods are effective for detecting standing or walking human postures, in disaster environments, human postures can be complex, including standing, sitting, and even lying down. Traditional networks are not ideal for training such individuals. This paper proposes combining the spatial rectangular coordinates and intensity features of 3D augmented point clouds, using a related representation group of these coordinates and intensity features. By performing feature description on 3D enhanced point clouds, the feature description of point clouds becomes more effective, thereby making the network more accurate in detecting human targets.

[0068] Furthermore, the point cloud semantic segmentation neural network model in step 7 includes a downsampling module and the Local Feature Aggregation module in RandLA-Net. The specific process of constructing the point cloud semantic segmentation neural network model is as follows:

[0069] Step 7.1: Use a geometric random downsampling method that combines geometric sampling and random sampling to construct the downsampling module in the point cloud semantic segmentation neural network model;

[0070] Specifically, this involves calculating a K-neighborhood for each point, then calculating the angle between the normals of the point and its neighbors; the greater the curvature, the larger this angle value. Since calculating point cloud curvature is time-consuming, we use the above method to approximate the curvature effect; and setting an angle threshold thθ. l If there exists a point whose neighborhood angle with its nearest neighbor is greater than thθ l Then, it is placed into the geometric feature region G, thereby dividing the point cloud into the geometric feature region G and its region in the geometric feature region G. complement region Let the target number of samples be S, and the random downsampling rate be R. Then, for random samples G and The number of samples are S*(1-R) ​​and S*R, respectively;

[0071] Step 7.2: Combine the downsampling module with the Local Feature Aggregation module in RandLA-Net. The Local Feature Aggregation module includes Local Spatial Encoding, AttentivePooling, and Dilated Residual Block.

[0072] Step 7.3: For each point, the downsampling module uses the KNN algorithm to find its nearest neighbors. The Local FeatureAggregation module aggregates the features of the neighboring points. The features include the center point coordinates, the coordinates of the nearest neighbors, the coordinate difference, and the Euclidean distance from the center point to the nearest neighbor. The Nearest-Neighbor Interpolation method is used for Decoder upsampling, that is, for each point's nearest neighbor, the features of the nearest neighbor are used as the features of the point.

[0073] Step 7.4: Complete the construction of the point cloud semantic segmentation neural network model; the point cloud semantic segmentation neural network model is generally similar to an Encoder-Decoder structure.

[0074] The advantages of this invention are that, compared with the prior art, this invention fuses information from three sensors—infrared image sensor, visible light image sensor, and three-dimensional lidar sensor—to obtain a three-dimensional enhanced point cloud. Finally, it uses a point cloud semantic segmentation neural network model to detect human targets, demonstrating the practicality of a multi-sensor fusion-based human target detection method in disaster environments, thus enabling it to play a role in disaster area search and rescue scenarios. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of a method for detecting human targets in a disaster environment based on multi-sensor fusion, as described in an embodiment of the present invention.

[0076] Figure 2 This is a schematic diagram of infrared and visible light images in an embodiment of the present invention.

[0077] Figure 3 This is a schematic diagram of an infrared enhanced image and a prominent target image in an embodiment of the present invention.

[0078] Figure 4 This is a schematic diagram illustrating the joint calibration effect of an infrared camera and a three-dimensional lidar in an embodiment of the present invention.

[0079] Figure 5 This is a schematic diagram showing the comparison of point cloud enhancement before and after in an embodiment of the present invention.

[0080] Figure 6 This is a schematic diagram of the point cloud semantic segmentation network used in the embodiments of the present invention.

[0081] Figure 7 This is a schematic diagram illustrating the point cloud human target detection and semantic segmentation effects in an embodiment of the present invention.

[0082] Figure 8 This is a schematic diagram comparing the detection of seated human targets in point cloud before and after adding the intensity feature module in an embodiment of the present invention. Detailed Implementation

[0083] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0084] Example 1

[0085] In this embodiment, a disaster environment human target detection method based on multi-sensor fusion is used to detect human targets in actual human target detection tasks, such as at intersections and in dormitory scenarios.

[0086] The hardware selection is as follows: a TE-EV1 infrared camera manufactured by Beijing Huayuanhao Optoelectronic Technology Co., Ltd. with a resolution of 640*480 is used as the infrared image sensor; a Hikvision DS-E12USB camera with a resolution of 1920*1080 is used as the visible light image sensor; and a Robosense RS-LiDAR-M1 3D 125-line lidar with a ranging range of 200m is used as the 3D lidar sensor. The lidar has a 3D horizontal scanning angle range of 120°, with an average angular resolution of 0.2° in the horizontal direction; a vertical measurement angle range of 25°, with an average angular resolution of 0.2° in the vertical direction; each frame of laser data contains more than 70,000 laser points, and the laser data frequency is 10 Hz.

[0087] The 3D LiDAR, visible light camera, and infrared camera are all mounted on a simple platform to keep the relative positions of the three sensors fixed.

[0088] Applications such as Figure 1 As shown, a method for detecting human targets in disaster environments based on multi-sensor fusion includes the following steps:

[0089] S1: Downsample and grayscale the visible light image, and sharpen the infrared image. A schematic diagram of the infrared and visible light images is shown below. Figure 2 As shown;

[0090] S1.1: Downsample and grayscale the visible light image, and sharpen the infrared grayscale image using the Laplace operator.

[0091] S2: Image registration is performed between the preprocessed infrared image and the visible light image;

[0092] S2.1: Perform edge detection on the visible light image and the infrared image using the Canny operator to obtain the Canny edge maps of the visible light image and the infrared image respectively;

[0093] S2.2: Perform ORB feature point detection and preliminary feature point pairing on the Canny edge maps of the visible light image and the infrared image respectively, and perform RANSAC purification on the preliminary pairing results to obtain the accurate transformation relationship between the visible light and infrared images, and complete the registration between the visible light and infrared images through this transformation relationship.

[0094] S3: Fuse the features of the visible light image onto the infrared image to obtain an infrared-enhanced image, and further process the infrared-enhanced image to obtain a salient target image;

[0095] S3.1 The features of the visible light image are fused onto the infrared image to obtain the infrared enhanced image, that is, wavelet transform is used to fuse the registered images to obtain the infrared enhanced image I. irp (u,v), where u,v>0;

[0096] S3.2 Obtain the saliency map S(u,v) of the infrared enhanced image. The saliency map is based on the gray-level contrast between pixels; that is, the sum of the gray-level distances between a pixel and all pixels in the image is used as the saliency value of that pixel. Let s k Let s be a pixel in S(u,v), then s k The significance value is defined as shown in formula (1):

[0097]

[0098] Among them, s j For any pixel on S(u,v), Gray(s) * ) represents pixel s * grayscale value, F m Represents pixel s m The frequency, N s This represents the total number of pixels at pixel point S(u,v);

[0099] Suppose that... There is Gray(s) k ) = SalS(s k Let the mean gray level of S(u,v) be avg. S , greater than avg S Set to 1, less than avg S Setting it to 0 will yield the binarized mask image mask1(u,v) of S(u,v), which will then be used to transform the infrared enhanced image I. irp The Hadamard product of (u,v) and mask1(u,v) yields the sub-image Sub1(u,v), as shown in formula (2):

[0100] Sub1(u,v)=I irp (u,v)⊙mask1(u,v) (2)

[0101] S3.3 To further reduce the influence of the background and achieve separation between the background and foreground, parametric gamma correction is employed. Let the average gray level of Sub1(u,v) be... Then there is a grayscale threshold for a certain pixel sub1(k) on Sub1(u,v). Furthermore, pixels smaller than the threshold will be brightened, and pixels larger than the threshold will be darkened, thus obtaining the sub-image Sub2(u,v). That is, for a certain pixel sub2(k) on Sub2(u,v), it is as shown in formula (3):

[0102]

[0103] Where γ1 and γ2 are gamma correction parameters and γ1≤1 and γ2>1, ε is a compensation coefficient to prevent the base from being 0;

[0104] Therefore, the above expression can be reformulated as formula (4):

[0105]

[0106] Where a, b, c, and d are undetermined coefficients;

[0107] S3.4 Binarize Sub2(u,v) to obtain the binary mask image of Sub2(u,v), denoted as mask2(u,v). Perform Hadamard product between mask2(u,v) and Sub1(u,v) to obtain the final salient target image I. irt (u,v), as shown in formula (5):

[0108] I irt (u,v)=Sub1(u,v)⊙mask2(u,v) (5)

[0109] S4: Perform joint calibration of the infrared camera and the 3D LiDAR to obtain the mapping relationship between the infrared image pixels and the 3D Cartesian coordinates of the point cloud;

[0110] S5: After the joint calibration of the 3D laser camera and infrared camera is completed, the correspondence between the 3D coordinates of the starting point cloud and the image pixels has been established. Based on this correspondence, the gray value of a point in the salient target image and the intensity value of the 3D laser radar point cloud are used as independent variables to enhance the intensity value of the corresponding point in the point cloud. That is, for the obtained 3D enhanced point cloud, the intensity value of the point cloud has been fused with the gray value information of the salient target image and the intensity information of the original 3D laser radar point cloud.

[0111] S5.1: For a point cloud P in a certain scene o Construct a KD tree with nodes (x,y,z) to obtain P. oany point in the middle p o The set of neighborhood points within the radius Δr is Among them, ||D km ||2 represents point p k Point P m The Euclidean distance;

[0112] S5.2: Define P as the set of points mapped from the jointly calibrated salient target image onto the 3D LiDAR point cloud. uv For the existence of P uv A point p in k Its neighborhood point set is Define Γ k For p k For P uv If the set is valid, then we get formula (6):

[0113]

[0114] Among them, th p Here, Φ is the threshold for the number of neighborhood points, Φ is the empty set, and crad(.) represents the number of elements in the set. For p k In set P uv The complement above;

[0115] Therefore, the effective mapping point set of a point (u,v) on a salient target image to its corresponding point cloud can be defined as shown in Equation (7):

[0116]

[0117] S5.3: For salient target image I irt A pixel i on (u,v) k For its effective mapping point set A point that exists in Record its strength value as Then its enhanced strength value As shown in formula (8):

[0118]

[0119] In particular, in the experiment, we can visualize it as an adaptive function, as shown in formula (9):

[0120]

[0121] Where α,β,μ,σ 2 The coefficients are adaptive and undetermined; sgn() is the unit step function.

[0122] Thus, we can obtain the value of P. o Enhanced point cloud and At a certain point The strength value is shown in formula (10):

[0123]

[0124] S6: Input the relevant representation group of spatial rectangular coordinates and intensity features of the 3D enhanced point cloud into the neural network to perform point cloud semantic segmentation in order to achieve the detection of human targets.

[0125] S6.1: For a point (x0, y0, z0) in the original 3D point cloud, there is an intensity value in0. After fusing image information, the intensity value is IN0, thus representing the point cloud input features of the network as follows: Specifically, for a point p in the point cloud l , (x l ,y l ,z l ) is p l coordinates of the spatial rectangular coordinate system, IN l For p l The enhanced strength value is expressed by the relationship shown in formula (11):

[0126]

[0127] Where r l For p l The Euclidean distance to the origin O(0,0,0) of the 3D lidar; define angθ l For p l Spatial angular characteristics, let there be a reference vector on the xOy plane in space. It has the origin O(0,0,0) as its endpoint and passes through point A(x). A ,y A ,0), where x A <<y A In particular, because the laser used is offset by ±60° in the positive x-axis direction, therefore With 3D augmented point cloud The range of the angle θ between any two points is: That is, 3D augmented point cloud It is impossible for a point p to exist. e , making remember Let the origin O(0,0,0) be the endpoint, and the point passing through point p be the length of the path. l The vector, then and cosine similarity As shown in formula (12):

[0128]

[0129] Among them (OA) i and (p) l ) i Representing vectors sum vector The components; in particular, since the data of type float is stored in memory according to the distribution of "sign bit-exponent-mantissa", x can be chosen. A =1,y A =2 n >>1 is used to optimize computational efficiency, and for float type z l The non-signed position 0 has sign(z) l )=z l &0x8000, thus allowing z l Positive and negative properties ζ(z) l The definition is shown in formula (13):

[0130]

[0131] Approximating equation (12), we obtain equation (14):

[0132]

[0133] Therefore, angθ can be further defined. l The calculation formula is shown in formula (15):

[0134]

[0135] Where, Δ θ It is to make angθ l The error correction variable has a more continuous distribution;

[0136] Specifically, the theoretical formula for calculating the intensity value of the three-dimensional lidar point cloud is shown in formula (16):

[0137]

[0138] Where A represents the beam aperture measured at a fixed angle, λ is the incident angle, re is the reflectivity of the object, σ is the absorptivity of the medium, in0 is the output intensity of the 3D lidar, and η or It is the efficiency of a 3D lidar system, η t This refers to the efficiency of the 3D lidar system. Therefore, it can be known that the intensity value of the 3D lidar is... l Strongly correlated with angθ in terms of variables l and r l Therefore, we designed a related representation group for point cloud intensity features. To enhance the feature description of point clouds, so as to better train the point cloud semantic segmentation network in the next step;

[0139] Specifically, we found that the traditional (x,y,z) approach works well for detecting standing or walking postures, but in disaster environments, human postures can be complex and varied, including standing, sitting, and even lying down. Traditional networks are not ideal for training these postures. Therefore, we introduced... Feature descriptions are beneficial for improving the network's detection performance of human targets;

[0140] S6.2: A geometric random downsampling method combining geometric sampling and random sampling is used to construct the downsampling module in the point cloud semantic segmentation network, namely:

[0141] For each point, calculate its K-neighborhood, and then calculate the angle between the normals of the point and its neighbors. The greater the curvature, the larger this angle value. Since calculating point cloud curvature is time-consuming, we use the above method to approximate the curvature effect; we set an angle threshold thθ. l If there exists a point whose neighborhood angle with its nearest neighbor is greater than thθ l Then, it is placed into the geometric feature region G, thereby dividing the point cloud into the geometric feature region G and its region in the geometric feature region G. complement region Let the target number of samples be S, and the random downsampling rate be R. Then, for random samples G and The number of samples are S*(1-R) ​​and S*R, respectively;

[0142] S6.3: Utilizing the downsampling module from S6.2, plus the Local Feature Aggregation module from RandLA-Net, including Local Spatial Encoding, Attentive Pooling, and Dilated Residual Block, specifically, it uses the KNN algorithm to find the nearest neighbors for each point, and performs feature aggregation including the center point coordinates, nearest neighbor coordinates, coordinate differences, and Euclidean distance from the center point to the nearest neighbor. Then, it uses the Nearest-Neighbor Interpolation method for Decoder upsampling, i.e., for each point, it finds the nearest point and uses the features of that nearest point as the features of this point. The network is generally structured similarly to an Encoder-Decoder, taking the previously processed features and... After the 3D augmented point cloud is input into the network, a semantic label is generated for each point, which can detect human targets within it.

[0143] S6.4: Use the D-set dataset collected on campus to train and visualize the algorithm, and verify its effectiveness;

[0144] In this embodiment, the specific parameters are set as follows:

[0145] In step S3, the adaptive parameter is set to c = 2, The final salient target image is as follows Figure 3 The right figure shows the joint calibration effect of the 3D LiDAR and infrared camera in step S4. Figure 4 In step S5, the radius of the KD tree is Δr = 0.25, and the parameters of the empirical function are set to α = 1, β = 1, μ = 150, σ 2 =4, the effect of its point cloud enhancement before and after is as follows: Figure 5 In step S6, angθ l In the calculation formula, n = 8, Δ θ = -1.4601, the parameters of the sampling module are set to thθ = 2°, R = 25%, and the corresponding network framework is as follows. Figure 6 As shown;

[0146] In this embodiment, two scenarios are used to verify the effect, as follows:

[0147] Scenario 1: Crossroads scene, such as Figure 7 The left and right sub-images show the results before and after segmentation, respectively. In this scenario, the IOU of human segmentation is 73.81, indicating that the segmentation is basically accurate and meets the requirements for human target detection.

[0148] Scenario 2: Dormitory scene, such as Figure 8 The left and right sub-images show the effects before and after segmentation, respectively. In this scenario, the human body is in a sitting position. Before adding the intensity feature module, the IOU of the human body segmentation is 4.98. It is also clear from the image that the network's detection accuracy for the human body is poor. Only the upper left arm has some correctly detected points, indicating that the human body is misclassified by the network. After adding the intensity feature module, the IOU of the human body segmentation is 44.42, which is a significant improvement and meets the requirements for human target detection.

[0149] This invention enables human target detection in both intersection and dormitory scenarios, improves detection accuracy, and achieves excellent detection results. This demonstrates the practicality of the proposed multi-sensor fusion-based human target detection method in disaster environments, and provides reference and assistance for human target detection under disaster conditions.

[0150] This document uses specific examples to illustrate the principles and implementation schemes of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting human targets in disaster environments based on multi-sensor fusion, comprising the following steps: Step 1: Input the image acquired by the sensor and preprocess the input image, specifically including: The input infrared image is sharpened, while the input visible light image is downsampled and converted to grayscale. Step 2: Perform image registration on the preprocessed infrared image and visible light image; Step 3: Based on the image registration result, the features of the visible light image are fused into the infrared image to obtain an infrared enhanced image, and then the infrared enhanced image is further processed to obtain a significant target image; Step 4: Perform joint calibration of the infrared camera and the 3D LiDAR to obtain the mapping relationship between the pixels of the infrared image and the 3D Cartesian coordinates of the 3D LiDAR point cloud. Step 5: Based on the mapping relationship between the pixels of the infrared image and the three-dimensional rectangular coordinates of the three-dimensional lidar point cloud, the intensity value of the point corresponding to the pixel in the point cloud is enhanced by the gray value of the pixel in the significant target image and the intensity value of the three-dimensional lidar point cloud to obtain the three-dimensional enhanced point cloud. Step 6: Use the spatial rectangular coordinates and intensity feature related representation groups of the 3D enhanced point cloud to perform feature description of the 3D enhanced point cloud; Step 7: Input the spatial rectangular coordinates and intensity feature related representations of the 3D enhanced point cloud into the point cloud semantic segmentation neural network model, perform point cloud semantic segmentation on the 3D enhanced point cloud, generate semantic labels for each point, and finally realize human target detection.

2. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, In step 1, the input infrared image is sharpened using the Laplace operator.

3. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, The image registration operation between the infrared image and the visible light image in step 2 is as follows: Step 2.1: Perform edge detection on the visible light image and the infrared image using the Canny operator to obtain the Canny edge maps of the visible light image and the infrared image, respectively; Step 2.2: Perform ORB feature point detection on the Canny edge maps of the visible light image and the infrared image respectively, and perform preliminary feature point pairing. Then, perform RANSAC purification on the preliminary pairing results to obtain the accurate transformation relationship between the visible light and infrared images. Step 2.3: Complete image registration between visible light and infrared images through the transformation relationship.

4. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, The specific process for obtaining the infrared enhanced image and the salient target image in step 3 is as follows: Step 3.1: The features of the image-registered visible light image are fused to the image-registered infrared image using wavelet transform to obtain the infrared enhanced image I. irp (u,v), where u,v>0; Step 3.2: Obtain the saliency map S(u,v) of the infrared enhanced image. Based on the grayscale contrast between pixels, the sum of the grayscale distances between the pixel and all pixels in the image is taken as the saliency value of the pixel. Define s k For each pixel in the saliency map S(u,v), s k The significance value is represented as SalS(S k ): Among them, s j For any pixel on S(u,v), Gray(s) * ) represents pixel s * grayscale value, F m Represents pixel s m The frequency, N s This represents the total number of pixels at pixel point S(u,v); right There is Gray(s) k ) = SalS(s k Let the mean gray level of S(u,v) be avg. S , greater than avg S Set to 1, less than avg S Set to 0 to obtain the binarized mask image mask1(u,v) of S(u,v), and then use the infrared enhanced image I... irp The Hadamard product of (u,v) and mask1(u,v) yields the subimage Sub1(u,v), which is expressed by the following formula: Sub1(u,v)=I irp (u,v)⊙mask1(u,v) Step 3.3: Parametric gamma correction is used to separate the background from the foreground; Let the average gray level of Sub1(u,v) be... Then there is a grayscale threshold for a certain pixel sub1(k) on Sub1(u,v). Pixels smaller than the threshold will be brightened, and pixels larger than the threshold will be darkened, resulting in a sub-image Sub2(u,v). The gamma correction for a pixel sub2(k) on Sub2(u,v) is as follows: Where γ1 and γ2 are gamma correction parameters and γ1≤1 and γ2>1, ε is a compensation coefficient to prevent the base from being 0; Therefore, the above expression can be reformulated as: Where a, b, c, and d are undetermined coefficients; Step 3.4: Perform a binarization masking on Sub2(u,v) to obtain the binarized mask image of Sub2(u,v), denoted as mask2(u,v). Perform the Hadamard product between mask2(u,v) and Sub1(u,v) to obtain the final salient target image I. irt (u,v) is represented as: I irt (u,v)=Sub1(u,v)⊙mask2(u,v) 5. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, The specific process of obtaining the 3D enhanced point cloud in step 5 is as follows: Step 5.1, for a point cloud P in a certain scene o Construct a KD tree with nodes (x, y, z) to obtain P. o any point in the middle p o The set of neighborhood points within the radius Δr is Among them, ||D km ||2 represents point p k Point P m The Euclidean distance; Step 5.2, define P as the set of points mapped from the jointly calibrated salient target image onto the 3D LiDAR point cloud. uv For the existence of P uv A point p in k Its neighborhood point set is Define Γ k For p k For P uv The valid set of is represented as follows: Among them, th p Here, Φ is the threshold for the number of neighboring points, Φ is the empty set, and crad(.) represents the number of elements in the set. For p k In set P uv The complement of the above; Step 5.3, define the effective mapping point set of a point (u,v) on a salient target image to its corresponding point cloud as: For salient target image I irt A pixel i on (u,v) k For its effective mapping point set A point that exists in Record its strength value as Then its enhanced strength value for: The concrete representation of an adaptive function is as follows: Where α,β,μ,σ 2 The coefficients are adaptive and undetermined; sgn() is the unit step function. Step 5.4, obtain the p o Enhanced 3D point cloud and At a certain point The intensity value is:

6. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, The specific representation group of spatial rectangular coordinates and intensity features of the 3D enhanced point cloud in step 6 includes: For a point (x0, y0, z0) in a 3D point cloud, there is an intensity value in0. After fusing image information, the intensity value is IN0. The 3D augmented point cloud is represented as... Among them, for a point p in the point cloud l , (x l ,y l ,z l ) is p l coordinates of the spatial rectangular coordinate system, IN l For p l The enhanced strength values ​​are expressed by the following relationship: Where r l For p l The Euclidean distance to the origin O(0,0,0) of the 3D lidar; define angθ l For p l Spatial angular characteristics, let there be a reference vector on the xOy plane in space. It has the origin O(0,0,0) as its endpoint and passes through point A(x). A ,y A ,0), where x A <<y A In particular, because the laser used is offset by ±60° in the positive x-axis direction, therefore With 3D augmented point cloud The range of the angle θ between any two points is: That is, 3D augmented point cloud It is impossible for a point p to exist. e , making remember Let the origin O(0,0,0) be the endpoint, and the point passing through point p be the length of the path. l The vector, then and cosine similarity for: Among them (OA) i and (p) l ) i Representing vectors sum vector The components; specifically, using x A =1,y A =2 n >>1 is used to optimize computational efficiency, and for float type z l The non-signed position 0 has sign(z) l )=z l &0x8000, thus making z l Positive and negative properties ζ(z) l ) is defined as: right and cosine similarity Approximating, we get: This leads to the further definition of angθ l The calculation formula is: Where, Δ θ It is to make angθ l The error correction variable has a more continuous distribution; The theoretical formula for calculating the intensity value of a 3D lidar point cloud is: Where A represents the beam aperture measured at a fixed angle, λ is the incident angle, re is the reflectivity of the object, σ is the absorptivity of the medium, in0 is the output intensity of the 3D lidar, and η or It is the efficiency of a 3D lidar system, η t It refers to the efficiency of a 3D lidar system; 3D LiDAR intensity value in l Strongly correlated with angθ in terms of variables l and r l The correlation representation group of point cloud intensity features is adopted. Enhance the feature description of point clouds; By combining the spatial rectangular coordinates and intensity features of 3D enhanced point clouds, and using related representation groups of the spatial rectangular coordinates and intensity features of 3D enhanced point clouds... Feature description of 3D enhanced point cloud.

7. The method for detecting human targets in a disaster environment based on multi-sensor fusion according to claim 1, characterized in that, The point cloud semantic segmentation neural network model in step 7 includes a downsampling module and the LocalFeature Aggregation module in RandLA-Net. The specific process of constructing the point cloud semantic segmentation neural network model is as follows: Step 7.1: Use a geometric random downsampling method that combines geometric sampling and random sampling to construct the downsampling module in the point cloud semantic segmentation neural network model; Specifically, this involves calculating a K-neighborhood for each point, then calculating the angle between the normals from the point to its neighbors; the greater the curvature, the larger this angle value; and setting an angle threshold thθ. l If there exists a point whose neighborhood angle with its nearest neighbor is greater than thθ l Then, it is placed into the geometric feature region G, thereby dividing the point cloud into the geometric feature region G and its region in the geometric feature region G. complement region Let the target number of samples be S, and the random downsampling rate be R. Then, for random samples G and The number of samples are S*(1-R) ​​and S*R, respectively; Step 7.2: Combine the downsampling module with the Local Feature Aggregation module in RandLA-Net. The Local Feature Aggregation module includes Local Spatial Encoding, AttentivePooling, and Dilated Residual Block. Step 7.3: For each point, the downsampling module uses the KNN algorithm to find its nearest neighbors. The LocalFeature Aggregation module aggregates the features of the neighboring points. The features include the center point coordinates, the coordinates of the nearest neighbors, the coordinate difference, and the Euclidean distance from the center point to the nearest neighbor. The Nearest-Neighbor Interpolation method is used for Decoder upsampling, that is, for each point's nearest neighbor, the features of the nearest neighbor are used as the features of the point. Step 7.4: Complete the construction of the point cloud semantic segmentation neural network model.

Images