A method, device, and electronic equipment for identifying trapped people after a disaster based on region cue masking and confidence heatmap guidance.
By generating a complete 3D human body mesh and fusing dynamic features, the problems of occlusion and environmental interference in the identification of trapped people after disasters were solved, and high-precision 3D pose output and robust target separation were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-07-02
- Publication Date
- 2026-07-31
AI Technical Summary
Traditional methods for identifying trapped people after disasters based on visible light images suffer from performance degradation in environments with heavy occlusion, similar textures between rubble and human bodies, and dense smoke and dust. These methods also suffer from semantic repair ambiguity, rigid utilization of sensor information, and problems with the adhesion of multiple target instances, making it difficult to build a complete understanding of occluded targets in three-dimensional space.
A method based on region cue mask and confidence heatmap is adopted to generate a complete 3D human body mesh through a parametric human body model. Semantic repair and instance segmentation are performed by combining multimodal data. The feature fusion weights are dynamically adjusted by using confidence heatmap and instance segmentation is performed by combining 3D pose parameters.
It improves the accuracy and stability of personnel detection in extremely obstructed and harsh environments, and can output a complete three-dimensional posture model of each trapped person, providing more intuitive information support for rescue.
Smart Images
Figure CN122493498A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a method, device, and electronic device for identifying trapped people after a disaster based on region cue masking and confidence heatmap guidance. Background Technology
[0002] In earthquake emergency rescue, quickly and accurately locating trapped individuals severely obscured by collapsed buildings and rubble is a top priority. Traditional target detection and instance segmentation methods based on visible light (RGB) images suffer a sharp performance decline when faced with severe occlusion, similar textures between rubble and human bodies, and harsh environments such as dense smoke and dust, mainly due to three major bottlenecks:
[0003] (1) Semantic repair ambiguity: When the repair method based on two-dimensional images is highly similar to the visual features of the human body and the ruins such as bricks and steel bars, it is easy to generate ambiguity, mistakenly repairing the rubble as human body parts or repairing human body parts as wall extensions, resulting in false alarms.
[0004] (2) Rigid use of sensor information: Existing fusion methods lack dynamic adjustment capabilities when RGB / infrared sensors fail partially or completely due to environmental interference, resulting in overall system performance being compromised and poor robustness.
[0005] (3) Multi-target instance adhesion: When multiple trapped people are covered by the same large area of rubble, their features are completely adhered at the perception level. Existing instance segmentation methods have difficulty distinguishing them as independent individuals, leading to missed detection or confusion.
[0006] Although some studies have attempted to introduce three-dimensional sensors such as LiDAR to provide geometric information, most of them are only used as auxiliary features for two-dimensional perception and have failed to fundamentally build a complete understanding of occluded targets in three-dimensional space. Summary of the Invention
[0007] In view of this, this application proposes a method, device, and electronic device for identifying trapped personnel after a disaster based on region-based cue masks and confidence heatmaps. Specifically, this application is implemented through the following technical solution:
[0008] According to a first aspect of the embodiments of this specification, a method for identifying trapped persons after a disaster is provided based on region cue masks and confidence heatmaps, the method comprising the following steps:
[0009] Step S1: Obtain multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds;
[0010] Step S2: Based on the point cloud, the exposed human body parts are detected, and several candidate point clouds of human body parts are obtained. For each cluster of candidate point clouds of human body parts, the pose parameters and shape parameters of the parameterized human body model are predicted by the pose regression network to drive the parameterized human body model to generate the corresponding complete 3D human body mesh. All complete 3D human body meshes are projected onto the RGB image plane to generate a human body region cue mask and a confidence heatmap. Based on the human body region cue mask and the confidence heatmap, semantic repair is performed on the occluded human body regions in the RGB image to obtain an RGB repair map.
[0011] Step S3: Extract multi-scale features from the RGB restoration image, infrared image and depth map generated based on point cloud, and fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid.
[0012] Step S4: For each complete 3D human body mesh, construct an average pooling feature based on the human body region cue mask and the multi-scale fusion feature pyramid. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, construct an instance query vector. Input the instance query vector and the multi-scale fusion feature pyramid into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
[0013] According to a second aspect of the embodiments of this specification, a disaster-related trapped person identification device based on region cue mask and confidence heatmap guidance is provided, the device comprising:
[0014] The data acquisition unit is used to acquire multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds.
[0015] The image inpainting unit is used to detect exposed human body parts based on the point cloud, obtain several candidate point clouds of human body parts, predict the pose parameters and shape parameters of the parameterized human body model through a pose regression network for each cluster of candidate point clouds of human body parts, so as to drive the parameterized human body model to generate a corresponding complete 3D human body mesh; project all complete 3D human body meshes onto the RGB image plane to generate a human body region cue mask and a confidence heatmap, and perform semantic inpainting on the occluded human body regions in the RGB image based on the human body region cue mask and the confidence heatmap to obtain an RGB inpainted image;
[0016] The feature processing unit is used to extract multi-scale features from the RGB restoration image, the infrared image, and the depth image generated based on the point cloud, and to fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid.
[0017] The target segmentation unit is used to construct an average pooling feature based on the human region cue mask and the multi-scale fusion feature pyramid for each complete 3D human body mesh. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, an instance query vector is constructed. The instance query vector and the multi-scale fusion feature pyramid are input into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
[0018] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in the first aspect.
[0019] The embodiments of this application have at least the following technical effects:
[0020] (1) Breaking through the limitations of the pose library, achieving arbitrary pose completion: The embodiments of this application generate a complete three-dimensional human body mesh by driving the parameterized human body model, rather than based on the retrieval of the pose library. Therefore, the embodiments of this application can adapt to the ever-changing human body poses under the rubble, and improve the generalization ability and accuracy of three-dimensional reconstruction.
[0021] (2) The embodiments of this application utilize confidence heatmaps to dynamically adjust the fusion weights of different sensor features. For example, in areas where the confidence of the three-dimensional human body model is high, the features of RGB images and infrared images are used as the main focus; in areas where the confidence is low, the system automatically switches to using the depth geometric features provided by the lidar point cloud. This ensures that the reliability of personnel detection results will not be reduced due to the failure of a sensor (such as the camera being blocked by smoke and dust), and can improve robustness in harsh environments.
[0022] (3) Breakthrough in multi-target separation under sticky occlusion: The embodiments of this application inject the three-dimensional pose parameters generated in the generation stage as priors into the instance segmentation decoder, which provides a strong geometric basis for the instance segmentation decoder to distinguish different individuals with similar spatial positions, and can improve the target separation accuracy and recall rate in complex occlusion scenarios. Attached Figure Description
[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:
[0024] Figure 1 This is a schematic diagram illustrating a workflow for identifying people trapped after a disaster, as shown in an exemplary embodiment of this application.
[0025] Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a method for identifying trapped persons after a disaster based on a region cue mask and a confidence heatmap.
[0026] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment of this application;
[0027] Figure 4 This is a block diagram illustrating an exemplary embodiment of the present application of a disaster-relief trapped person identification device based on a region cue mask and confidence heatmap. Detailed Implementation
[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0029] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0030] The technical solution of this application constructs a collaborative processing architecture. Its core idea is to first use three-dimensional information to infer the complete posture of the occluded human body, then use this three-dimensional inference result to guide the restoration of the two-dimensional image, and finally use the geometric prior of the three-dimensional posture information to guide instance decoupling.
[0031] Specifically, the entire process is as follows Figure 1 As shown: First, multimodal environmental data of the post-disaster environment are collected; for example, point clouds of the post-disaster environment are collected using lidar sensors, RGB images are collected using cameras, and infrared images are collected using infrared sensors.
[0032] Then, in the 3D point cloud acquired by the LiDAR, the human body regions that only partially reveal the body are identified. These incomplete point clouds are then input into a pose regression network, which outputs the pose and body shape parameters required for a parameterized human body model (such as the SMPL model). Using these parameters, a complete 3D human body mesh can be generated, which can present any pose and conforms to the movement laws of human joints.
[0033] Next, the generated complete 3D human body mesh is projected onto the RGB image plane, thus obtaining two important pieces of information: one is a human body region cue mask, which reflects which pixels in the RGB image may belong to the human body, and the other is a confidence heatmap, which reflects the credibility of the generated complete 3D human body mesh. Using these two pieces of information as strong priors, the image inpainting network is guided to perform semantic inpainting on severely occluded human body regions in the RGB image, generating the repaired RGB image.
[0034] Finally, multimodal data fusion is performed using the restored RGB images, infrared images, and depth maps generated from point clouds. During the fusion process, the confidence heatmap acts as an arbitrator: in areas where the 3D generation results are reliable, image information is relied upon more; in areas where they are unreliable, depth geometry information is relied upon more. Simultaneously, the pose parameters corresponding to each complete 3D human mesh are combined with the average pooling features extracted from its projection area from the multi-scale fusion feature pyramid to construct an instance query vector. This instance query vector is input into the instance segmentation decoder, which can effectively distinguish between different trapped individuals who are obscured by rubble or whose spatial locations are very close.
[0035] Thus, this technical solution not only significantly improves the accuracy of personnel detection and system stability in extreme obstruction and harsh environments, but also outputs a complete three-dimensional posture model of each trapped person, providing more intuitive and valuable information support for rescue decisions.
[0036] The embodiments described in this specification will now be described in detail.
[0037] This application provides a method for identifying trapped people after a disaster based on region-based cue masks and confidence heatmaps. Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a method for identifying trapped persons after a disaster based on region-based cue masks and confidence heatmaps. Figure 2 As shown, the method for identifying people trapped after a disaster includes the following steps:
[0038] Step S1: Obtain multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds;
[0039] Step S2: Based on the point cloud, the exposed human body parts are detected, and several candidate point clouds of human body parts are obtained. For each cluster of candidate point clouds of human body parts, the pose parameters and shape parameters of the parameterized human body model are predicted by the pose regression network to drive the parameterized human body model to generate the corresponding complete 3D human body mesh. All complete 3D human body meshes are projected onto the RGB image plane to generate a human body region cue mask and a confidence heatmap. Based on the human body region cue mask and the confidence heatmap, semantic repair is performed on the occluded human body regions in the RGB image to obtain an RGB repair map.
[0040] Step S3: Extract multi-scale features from the RGB restoration image, infrared image and depth map generated based on point cloud, and fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid.
[0041] Step S4: For each complete 3D human body mesh, construct an average pooling feature based on the human body region cue mask and the multi-scale fusion feature pyramid. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, construct an instance query vector. Input the instance query vector and the multi-scale fusion feature pyramid into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
[0042] Next, a detailed explanation will follow. Figure 2 Each step in the process.
[0043] First, execute step S1 to collect multimodal environment data.
[0044] This embodiment allows a drone equipped with multimodal sensors to simultaneously acquire timestamp-aligned RGB images, infrared images, and point clouds. Before data acquisition, the multimodal sensors must be precisely jointly calibrated to obtain the mapping matrix between each sensor, ensuring that each image pixel has its corresponding three-dimensional point cloud coordinates in space.
[0045] In some embodiments, multimodal environmental data preprocessing is also performed, such as distortion correction, white balance and image enhancement for RGB images, distortion correction and image enhancement for infrared images; outlier filtering and ground point removal are performed on point clouds, and the data is converted into an ordered voxel mesh or the original point cloud structure is preserved for subsequent processing.
[0046] Next, step S2 is performed, which involves generating a complete 3D human body mesh and inpainting RGB images based on a parameterized human body model.
[0047] Step S2 includes sub-steps such as human body part detection, parametric pose prediction, human region cue mask and confidence heatmap generation, and semantic repair.
[0048] The human body part detection in sub-step S21 includes: voxelizing the point cloud to obtain a voxel mesh; using a 3D object detection network to detect the voxel mesh to obtain 3D candidate bounding boxes for the human body part; and clustering the point cloud within each 3D candidate bounding box to obtain candidate point cloud for the human body part.
[0049] For example, this embodiment employs a lightweight voxel-based 3D sparse convolutional network, such as Voxel-RCNN, to process the voxel mesh of point clouds. This 3D sparse convolutional network takes the voxel mesh generated from the point cloud as input and outputs a series of 3D candidate bounding boxes and their corresponding human body part category scores to detect any point cloud clusters that may belong to exposed parts of the human body, such as arms, torso, and head. Then, Euclidean distance clustering is performed on the point cloud clusters within each 3D candidate bounding box to obtain several independent candidate point clouds for human body parts, denoted as . N represents the total number of human body parts detected.
[0050] The parameterized pose prediction in sub-step S22 includes:
[0051] For each cluster of candidate point clouds of human body parts The process employs a deep learning-based pose regression network, which first extracts pose values using a PointNet++ module with shared weights. global geometric feature vectors ,Then The data is fed into a multilayer perceptron (MLP) regression head to predict pose parameters. and shape parameters For example, attitude parameters The 3-axis rotation parameters, including those of 24 joints, can be represented using a 6D rotation representation. This is achieved using predicted parameters. The system calls a standard parametric human body model, such as the differentiable forward function of the SMPL (Skinned Multi-Person Linear Model), to generate a corresponding complete 3D human body mesh. This complete 3D human body mesh includes vertices. and triangular dough pieces.
[0052] This step replaces the traditional pose library retrieval and can generalize to any pose.
[0053] Sub-step S23, generating the human region cue mask and confidence heatmap, includes: projecting the vertices of each complete 3D human mesh onto the RGB image plane to obtain a corresponding binary projection mask; merging the binary projection masks of all complete 3D human meshes to obtain the human region cue mask; for each complete 3D human mesh, calculating the fitting degree between the complete 3D human mesh and the corresponding human part candidate point cloud, and generating the confidence degree of the complete 3D human mesh based on the fitting degree; generating a confidence heatmap with the same size as the RGB image based on the human region cue mask and the confidence degree, wherein, for pixels within the human region cue mask, the confidence heatmap value of the pixel is the confidence degree corresponding to the binary projection mask to which the pixel belongs, and for pixels outside the human region cue mask, the confidence heatmap value is zero.
[0054] In this embodiment, after generating a complete 3D human body mesh, to evaluate the generation quality of the complete 3D human body mesh, the point cloud of the complete 3D human body mesh and its corresponding human body parts is calculated. The goodness of fit between them is used to generate the confidence score for each complete 3D human body mesh. .
[0055] Specifically, the fit is calculated by measuring the candidate point cloud of the human body parts. The fit is determined by the average Euclidean distance from each point to the surface of the complete 3D human body mesh. The fit is defined as... The candidate point cloud of the human body parts The total number of points in the middle is The expression for the fit is... as follows:
[0056] (1)
[0057] Where x is the candidate point cloud of the human body part. Three-dimensional spatial points, For the complete three-dimensional human body mesh The nearest neighbor of x on the surface.
[0058] After obtaining the aforementioned fit degree Then, it is converted into a confidence level constrained in the interval [0,1] using a negative exponential mapping function. The calculation formula is as follows:
[0059] (2)
[0060] in, This is a preset scaling factor that is greater than zero.
[0061] The degree of fit The smaller the value, the higher the geometric fit between the complete 3D human body mesh and the actual exposed human body parts, and the higher the corresponding confidence level. The larger the value, the better.
[0062] For each complete 3D human body mesh, its individual mesh vertices By projecting the camera's projection matrix onto the RGB image plane, a corresponding binary projection mask can be obtained. By merging all binary projection masks, the human body region cue mask can be obtained. .
[0063] Masking prompts in the human body area. Within the covered area, use the corresponding confidence level The area outside the mask is filled, and the area outside the mask is set to zero, thereby generating a confidence heatmap. .
[0064] The semantic insulation in sub-step S24 includes: concatenating the RGB image, the human body region cue mask, and the confidence heatmap along the channel dimension to obtain a combined tensor; inputting the combined tensor into an image insulation network, which is configured to perform semantic insulation on the occluded human body region in the RGB image within the region indicated by the human body region cue mask, based on the image context of the region and the guidance of the confidence value in the confidence heatmap, and outputting an RGB insulated image.
[0065] For example, this embodiment constructs an image inpainting network based on U-Net or Visual Transformer (ViT), with the input being a combined tensor: [RGB image, , Image inpainting networks in Within the indicated area, based on the surrounding image context and Guided by confidence level, it generates human body regions that are light-consistent and semantically reasonable.
[0066] Then, step S3 is executed to construct a dynamic feature fusion and multi-scale fusion feature pyramid based on regional credibility.
[0067] In some embodiments, step S3 specifically includes: extracting multi-scale features from the RGB restoration image, the infrared image, and the depth map generated from the point cloud, respectively, to obtain an RGB multi-scale feature pyramid, an infrared multi-scale feature pyramid, and a depth multi-scale feature pyramid; downsampling the confidence heatmap to the same resolution as each scale feature to obtain a multi-scale confidence feature pyramid; for each scale layer, inputting the corresponding scale's RGB features, infrared features, depth features, and confidence features to a dynamic routing unit, the dynamic routing unit adjusting the initial weight map according to the confidence features to obtain a modulated weight map corresponding to the RGB features, infrared features, and depth features, and fusing the RGB features, infrared features, and depth features based on the modulated weight map to obtain the fused features for that scale; and assembling the fused features of all scales into the multi-scale fused feature pyramid.
[0068] For example, in practical applications, lightweight convolutional neural networks can be used to extract multi-scale feature pyramids from RGB inpainted images, infrared images, and depth maps, respectively. , For RGB multi-scale feature pyramid, For infrared multi-scale feature pyramids, This is a deep multi-scale feature pyramid. Simultaneously, a confidence heatmap is used. Downsampling to the same resolution as each feature layer yields a multi-scale confidence feature pyramid. .
[0069] For the Layer features, employing lightweight dynamic routing units. These dynamic routing units use... and For input, the core of the dynamic routing unit is a miniature convolutional network, which outputs three initial weight maps. Using confidence features Modulation:
[0070] (3)
[0071] (4)
[0072] (5)
[0073] in, , , The initial RGB weight map, initial infrared weight map, and initial depth weight map are shown in the following order. , , The images, in order, are the modulated RGB weight map, the modulated infrared weight map, and the modulated depth weight map. This represents the confidence level feature.
[0074] This design enables high confidence regions ( In regions with high confidence (larger values), the weights of RGB and infrared features are enhanced; in low-confidence regions, the weights of depth (geometric) features dominate. Finally, the... The fusion characteristics of the layers are:
[0075] (6)
[0076] in, This represents the element-wise multiplication operator.
[0077] This embodiment involves all levels. Performing this operation yields a robust set of multi-scale fused feature pyramids. .
[0078] Finally, step S4 is executed to segment occlusion instances based on pose priors.
[0079] In this embodiment, the average pooling features are obtained through the following steps: For each complete 3D human body mesh, features within the area covered by the binary projection mask of the complete 3D human body mesh are extracted from the fusion feature map of any scale in the multi-scale fusion feature pyramid; the extracted features are then averaged to obtain the average pooling features of the complete 3D human body mesh.
[0080] The instance query vector in this embodiment is constructed through the following steps: fusing the average pooling feature with the three-dimensional pose parameters to generate the instance query vector of the complete three-dimensional human body mesh.
[0081] For example, in this embodiment, for each complete 3D human body mesh, its binary projection mask is used. Extract the binary projection mask of the complete 3D human body mesh from the fusion feature map at any scale in the multi-scale fusion feature pyramid. Features within the covered area By splicing or attention mechanisms and Merge to obtain the instance query vector .
[0082] This embodiment employs a query-based instance segmentation framework, such as the Mask2Former decoder. It integrates multi-scale feature pyramids. and instance query vector Input the Mask2Former decoder, and in the cross-attention layer of the decoder, query the vector for each instance. It interacts with multi-scale features, accurately aggregates relevant features belonging to the k-th target, effectively decouples instances, and outputs accurate segmentation masks, bounding boxes, category confidence scores, and 3D pose parameters for each detected trapped person.
[0083] Figure 3 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 3 At the hardware level, the device includes a processor 310, an internal bus 320, a network interface 330, memory 340, a hardware acceleration device 350, and non-volatile memory 360, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 310 reads the corresponding computer program from the non-volatile memory 360 into the memory 340 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0084] Figure 4 This is a structural block diagram illustrating an exemplary embodiment of a disaster-relief trapped personnel identification device guided by a region-hint mask and a confidence heatmap. The disaster-relief trapped personnel identification device can be applied to, for example... Figure 3 The electronic device shown implements the technical solution of this application. The disaster-stricken person identification device includes: a data acquisition unit 410, an image restoration unit 420, a feature processing unit 430, and a target segmentation unit 440, wherein:
[0085] The data acquisition unit 410 is used to acquire multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds.
[0086] Image inpainting unit 420 is used to detect exposed human body parts based on the point cloud, obtain several independent candidate point clouds of human body parts, predict the pose parameters and shape parameters of the parameterized human body model through a pose regression network for each cluster of candidate point clouds of human body parts, so as to drive the parameterized human body model to generate a corresponding complete three-dimensional human body mesh; project all complete three-dimensional human body meshes onto the RGB image plane to generate a human body region cue mask and a confidence heatmap, and perform semantic inpainting on the occluded human body regions in the RGB image based on the human body region cue mask and the confidence heatmap to obtain an RGB inpainted image;
[0087] The feature processing unit 430 is used to extract multi-scale features from the RGB restoration image, the infrared image and the depth image generated based on the point cloud, and to fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid.
[0088] The target segmentation unit 440 is used to construct an average pooling feature based on the human region cue mask and the multi-scale fusion feature pyramid for each complete 3D human body mesh. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, an instance query vector is constructed. The instance query vector and the multi-scale fusion feature pyramid are input into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
[0089] In some embodiments, the image restoration unit 420 is used to perform voxelization processing on the point cloud to obtain a voxel mesh; to use a three-dimensional object detection network to detect the voxel mesh to obtain 3D candidate bounding boxes of human body parts; and to cluster the point cloud within each 3D candidate bounding box to obtain candidate point cloud of the human body part.
[0090] In some embodiments, the feature processing unit 430 is configured to extract multi-scale features from the RGB restoration image, the infrared image, and the depth map generated from the point cloud, respectively, to obtain an RGB multi-scale feature pyramid, an infrared multi-scale feature pyramid, and a depth multi-scale feature pyramid; and to downsample the confidence heatmap to the same resolution as each scale feature to obtain a multi-scale confidence feature pyramid; for each scale layer, the RGB features, infrared features, depth features, and confidence features of the corresponding scale are input to the dynamic routing unit, which adjusts the initial weight map according to the confidence features to obtain a modulated weight map corresponding to the RGB features, infrared features, and depth features, and fuses the RGB features, infrared features, and depth features based on the modulated weight map to obtain the fused features of that scale; and to form the multi-scale fused feature pyramid by combining the fused features of all scales.
[0091] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0092] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.
[0093] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.
[0094] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0095] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0097] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0098] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.
[0099] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0100] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.
[0101] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0102] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for identifying trapped people after a disaster based on region-based cue masking and confidence heatmap guidance, characterized in that, The method includes the following steps: Step S1: Obtain multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds; Step S2: Based on the point cloud, the exposed human body parts are detected and a number of candidate point clouds of human body parts are obtained. For each cluster of candidate point clouds of human body parts, the pose parameters and shape parameters of the parameterized human body model are predicted by the pose regression network to drive the parameterized human body model to generate the corresponding complete three-dimensional human body mesh. All complete 3D human body meshes are projected onto the RGB image plane to generate human body region cue masks and confidence heatmaps. Based on the human body region cue masks and the confidence heatmaps, semantic repair is performed on the occluded human body regions in the RGB image to obtain an RGB repair map. Step S3: Extract multi-scale features from the RGB restoration image, infrared image and depth map generated based on point cloud, and fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid. Step S4: For each complete 3D human body mesh, construct an average pooling feature based on the human body region cue mask and the multi-scale fusion feature pyramid. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, construct an instance query vector. Input the instance query vector and the multi-scale fusion feature pyramid into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
2. The method according to claim 1, characterized in that, The candidate point cloud of human body parts in step S2 is obtained through the following steps: The point cloud is voxelized to obtain a voxel mesh; A three-dimensional object detection network is used to detect the voxel mesh to obtain 3D candidate bounding boxes for human body parts; Clustering is performed on the point cloud within each 3D candidate bounding box to obtain the candidate point cloud of the human body part.
3. The method according to claim 1, characterized in that, The human body region cue mask and confidence heatmap in step S2 are generated through the following steps: Projecting the vertices of each complete 3D human body mesh onto the RGB image plane yields the corresponding binary projection mask; The binary projection masks of all complete 3D human body meshes are merged to obtain the human body region cue mask; For each complete 3D human body mesh, calculate the fitting degree between the complete 3D human body mesh and the corresponding candidate point cloud of human body parts, and generate the confidence degree of the complete 3D human body mesh based on the fitting degree. Based on the human body region cue mask and the confidence level, a confidence heatmap with the same size as the RGB image is generated. For pixels within the human body region cue mask, the confidence heatmap value of the pixel is the confidence level corresponding to the binary projection mask to which the pixel belongs. For pixels outside the human body region cue mask, the confidence heatmap value is zero.
4. The method according to claim 1, characterized in that, The RGB restoration image in step S2 is generated through the following steps: The RGB image, the human body region cue mask, and the confidence heatmap are concatenated along the channel dimension to obtain a combined tensor; The combined tensor is input into an image inpainting network, which is configured to perform semantic inpainting on occluded human body regions in an RGB image within the region indicated by the human body region cue mask, based on the image context of the region and the guidance of confidence values in the confidence heatmap, and output an RGB inpainted image.
5. The method according to claim 1, characterized in that, Step S3 includes: Multi-scale features are extracted from the RGB restored image, the infrared image, and the depth map generated from the point cloud, respectively, to obtain RGB multi-scale feature pyramids, infrared multi-scale feature pyramids, and depth multi-scale feature pyramids; and the confidence heatmap is downsampled to the same resolution as each scale feature to obtain a multi-scale confidence feature pyramid. For each scale layer, the RGB features, infrared features, depth features, and confidence features of the corresponding scale are input to the dynamic routing unit. The dynamic routing unit adjusts the initial weight map according to the confidence features to obtain the modulated weight map corresponding to the RGB features, infrared features, and depth features. Based on the modulated weight map, the RGB features, infrared features, and depth features are fused to obtain the fused features of that scale. The fusion features of all scales are combined to form the multi-scale fusion feature pyramid.
6. The method according to claim 5, characterized in that, No. The expressions for each modulated weight map under the layer features are as follows: ， ， ， in, , , The initial RGB weight map, initial infrared weight map, and initial depth weight map are shown in the following order. , , The images, in order, are the modulated RGB weight map, the modulated infrared weight map, and the modulated depth weight map. This represents the confidence level feature.
7. The method according to claim 3, characterized in that, The average pooling feature in step S4 is obtained through the following steps: For each complete 3D human body mesh, features within the area covered by the binary projection mask of the complete 3D human body mesh are extracted from the fusion feature map at any scale in the multi-scale fusion feature pyramid. The extracted features are averaged pooled to obtain the averaged pooled features of the complete 3D human body mesh.
8. The method according to claim 1, characterized in that, The instance query vector in step S4 is constructed through the following steps: The average pooling feature is fused with the three-dimensional pose parameters to generate an instance query vector for the complete three-dimensional human body mesh.
9. A disaster-related personnel identification device based on region-cue masking and confidence heatmap guidance, characterized in that, The device includes: The data acquisition unit is used to acquire multimodal environmental data of the post-disaster environment, including time-aligned RGB images, infrared images, and point clouds. The image inpainting unit is used to detect exposed human body parts based on the point cloud, obtain several candidate point clouds of human body parts, predict the pose parameters and shape parameters of the parameterized human body model through a pose regression network for each cluster of candidate point clouds of human body parts, so as to drive the parameterized human body model to generate a corresponding complete 3D human body mesh; project all complete 3D human body meshes onto the RGB image plane to generate a human body region cue mask and a confidence heatmap, and perform semantic inpainting on the occluded human body regions in the RGB image based on the human body region cue mask and the confidence heatmap to obtain an RGB inpainted image; The feature processing unit is used to extract multi-scale features from the RGB restoration image, the infrared image, and the depth image generated based on the point cloud, and to fuse the multi-scale features using the confidence heatmap to obtain a multi-scale fused feature pyramid. The target segmentation unit is used to construct an average pooling feature based on the human region cue mask and the multi-scale fusion feature pyramid for each complete 3D human body mesh. Based on the average pooling feature and the pose parameters corresponding to the complete 3D human body mesh, an instance query vector is constructed. The instance query vector and the multi-scale fusion feature pyramid are input into the query-based instance segmentation decoder to output the segmentation mask, bounding box and 3D pose parameters of each trapped person.
10. An electronic device, characterized in that, include: processor; as well as A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 8.