Fire-fighting personnel identification and early warning system
By using human detection and posture analysis modules, the location and status of personnel at the fire scene can be monitored in real time, solving the problem of poor interaction among personnel at the fire scene and improving rescue efficiency and safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINESE PEOPLES LIBERATION ARMY NAVAL ACAD
- Filing Date
- 2025-09-28
- Publication Date
- 2026-06-19
Smart Images
Figure CN121330712B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of fire-fighting auxiliary analysis systems, and in particular relates to a personnel identification and early warning system for fire fighting. Background Technology
[0002] During natural disasters such as fires and typhoons, intense noise and limited visibility become significant obstacles to fire and rescue operations. In complex environments and with interference, the perception range and visibility of rescue personnel rapidly decline, often hindering effective interaction. This is a major factor limiting rescue efficiency and the safety of rescue workers. While technologies like radio voice communication can enhance interaction, they primarily improve on-site collaboration. In emergencies, timely and accurate message transmission may be impossible. Furthermore, in busy and complex situations, personnel may struggle to effectively observe the status of others. This problem not only affects the ability to respond to sudden disasters but also negatively impacts the safety of on-site personnel. Summary of the Invention
[0003] The purpose of this invention is to provide a fire-fighting personnel identification and early warning system applicable to scenarios such as fire fighting and disaster relief, which improves the perception ability of on-site personnel in complex environments, assists on-site personnel in continuously monitoring and analyzing the status of other personnel and objects on-site while carrying out intense operations, improves the ability to respond to emergencies, ensures personnel safety, and enhances on-site search and rescue capabilities.
[0004] To achieve the above objectives, the present invention adopts the following technical solution.
[0005] A fire-fighting personnel identification and early warning system includes a human detection network module and a human posture analysis module.
[0006] The human detection network module is used to extract human pixel features from video images. It mainly consists of an input layer, a backbone network layer, a neck network layer, and a prediction layer.
[0007] The backbone network consists of focus units, CBL convolutional units containing convolution, batch normalization and LeakyReLU activation functions, C3+ units, Swing Transformer units and pyramid pooling units;
[0008] The focus unit is used to perform video image segmentation operations, reducing the number of image channels and expanding the spatial size to facilitate further feature extraction and analysis. Specifically: for the input image... A new image is formed by dividing an image into blocks of fixed pixel size and then rearranging the pixels. ; can be represented as ;in Represents the pixel height of the original image, Represents the pixel width of the original image, Represents the number of channels in the original image. This represents the pixel height of the image obtained after segmentation. This represents the pixel width of the image obtained after segmentation. This indicates the arrangement operation performed on the focal cell;
[0009] The C3+ convolutional unit performs convolution operations on the image, using sliding convolution kernels to extract features contained in the image, including edges, textures, images, and objects. The C3+ convolutional unit controls the model's computational load and extracts feature gradient information, including local connection networks and local transfer networks. The local connection network is used to process the basic feature map... Segmentation is performed to obtain two parts of local features, namely The two local features are processed separately, one of which... Extracted from a local dense network The other part Directly passed to the local dense end; the output features of the locally connected network can be represented as , It will be transmitted through the transmission layer in the local transmission network, and its output will with After connection, it passes through another transmission layer before being output. ;
[0010] The Swin Transformer unit replaces the standard multi-head self-attention module with a shift-window-based multi-head self-attention module while keeping other parts unchanged. It consists of a shift-window-based multi-head self-attention module followed by a multilayer perceptron module containing GeLU nonlinearity, and uses residual connections between each module.
[0011] The pooling unit uses spatial pyramid pooling to control the feature dimension. For the input feature map, spatial pyramid pooling divides it into sub-regions of different scales, forming multiple grids of different sizes and constructing a multi-level spatial pyramid structure. Fast pooling is performed on each sub-region to obtain feature representations of different scales. The pooling results of different scales are concatenated to obtain a feature vector of fixed length.
[0012] The neck layer consists of feature pyramid network units and path aggregation network units. The feature pyramid network units fuse low-level feature maps with high-level feature maps through upsampling and convolution operations to generate new multi-scale feature maps. The aggregation network units add a bottom-up path enhancement network to pass low-level feature information back to high-level feature maps to enhance the model's utilization of feature information.
[0013] Based on the traditional feature prediction network, the prediction layer improves the feature capture ability of key nodes in the human prediction process by introducing a spatiotemporal channel attention structure, and fuses the obtained channel features and spatiotemporal features to improve the accuracy of the prediction results.
[0014] The human pose analysis module uses a graph convolutional network (GNN) to identify and extract key human nodes from the aforementioned human feature map. It determines the human pose by analyzing the pose of these key nodes. Specifically:
[0015] Based on the theory of Graph Convolutional Networks (GNNs), human feature maps can be represented as graphs consisting of nodes and the connections between them. ,in Represents a set of nodes. Represents an edge set. For human feature maps, the adjacency matrix is used. In Nodes ,gather The edge in the middle represents the edge connecting any two nodes. Information is stored in the adjacency matrix In the middle, if node and If there is a connection between them, then ,otherwise , with nodes The number of edges between vertices is equal to the degree of the node. The set they form is a degree matrix. ,in Changes in human posture can be represented as motion vectors. Where T refers to the number of frames the motion change lasts, and the vector of the human key nodes during the motion change process can be represented as: The key edge can be represented as ;
[0016] Based on the above, for the input human feature map ,use Spatial location can be obtained by convolution kernel convolution. The channel output model can be expressed as: ;
[0017] in It refers to the pixel coordinates of a spatial location. It refers to the sampling function. This refers to the weighting function;
[0018] Sampling function Based on the definition of a node's location in the node's neighborhood in the feature graph, when the distance of a node to a key node does not exceed the length of the edge associated with that key node... When sampling is performed, the weight function is used. Determined by the key nodes and edge features of the human body feature map, the nodes of the human body feature map can be divided into center nodes and edge nodes. Center nodes refer to nodes that are close to the center of gravity of the human body, while edge nodes refer to nodes that are far away from the center of gravity of the human body.
[0019] Based on the above principles, the human feature graph node convolutional model processed by a graph convolutional network (GNN) can be represented as:
[0020] ;
[0021] in It means Time Node The set of domain nodes; This indicates normalization processing; express Node of time Location;
[0022] Node feature analysis based on a human feature map node convolution model can generate spatiotemporal feature maps of human nodes at different action moments. ,in It refers to the set of all key nodes in an action sequence. It refers to the first In the action frame, the _ A key node; This refers to the node connection relationship. These refer to the connection relationships between an edge formed by any key node, an edge formed by adjacent key nodes, and an edge formed by the same node in adjacent frames and a certain key node.
[0023] Obtain the spatiotemporal feature map of human body nodes The model is trained and optimized by extracting spatiotemporal feature change data of human body nodes and combining it with spatiotemporal sequence features of key human body nodes under different postures obtained through simulation. The final model after training is used to extract data and classify postures and actions of real-time images to determine human posture information and design corresponding treatment plans based on different human posture information.
[0024] In a further improvement or specific implementation of the aforementioned fire-fighting personnel identification and early warning system, the human body detection network module also includes an input layer, which consists of a video or image preprocessing unit;
[0025] Used to periodically acquire image data from fire scenes via video or image acquisition equipment, and to perform preliminary processing to ensure the consistency of the acquired images and remove invalid data for further analysis.
[0026] A further improvement or specific implementation of the aforementioned fire-fighting personnel identification and early warning system involves improving the traditional convolution processing method within the neck network layer unit by applying a trainable refocusing transform to the pre-trained model to establish connections between parameters. Specifically, assuming the number of convolution input channels is... The number of output channels is The number of groups is The basic weight is The transformation weights are Establish identity mapping ;in This represents a convolution operation, which outputs each group by changing its weights. After transformation, the groups are regrouped and rearranged; the number of groups is... .
[0027] Further improvements or specific implementations of the aforementioned fire-fighting personnel identification and early warning system, in the prediction layer, for those divided into... Group of human feature diagrams The channel features and spatiotemporal features are extracted through two attention branches, respectively. The channel attention features are obtained using a single-layer transformation and can be represented as follows: ;in This refers to the pixel height of the input feature map. This refers to the pixel width of the input feature map, where , and The transformation parameters are determined through model training and optimization. This is the channel feature map obtained after transformation;
[0028] The spatiotemporal attention feature is obtained by processing the normalized input feature map. The result of the transformation can be expressed as follows: ;in and The transformation parameters are determined through model training and optimization. This is the channel feature map obtained after transformation. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the backbone network;
[0030] Figure 2 This is a schematic diagram of a spatial pyramid pooling system. Detailed Implementation
[0031] The present invention will be described in detail below with reference to specific embodiments.
[0032] The present invention discloses a personnel identification and early warning system for fire fighting, which is mainly used in fire fighting and other scenarios to locate and analyze personnel on site using real-time video image acquisition and analysis technology. The system determines the status of personnel on site through features such as posture, so as to assist fire fighting personnel in quickly locating personnel and determining their status in complex on-site environments, thereby improving the efficiency of fire fighting and rescue and helping them to complete fire fighting and rescue operations in low visibility and strong interference environments.
[0033] Due to the strong interference from smoke, bright light, and noise in special scenarios such as firefighting, as well as the obstruction caused by the heavy protective clothing and goggles worn by firefighters, the visual range and on-site identification and analysis capabilities of firefighters are greatly affected compared to normal environments. Information about other people scattered in the firefighting area is often difficult to observe and analyze effectively. Moreover, the fleeting nature of a fire can have a significant impact on people on-site in a short period of time. Improving the identification and analysis capabilities of on-site firefighters can effectively enhance the safety of firefighting operations and increase rescue efficiency.
[0034] To achieve on-site personnel location and identification, this invention employs a human detection network module based on the YOLO target recognition network model, which is used to extract personnel pixel features from video images. It mainly consists of an input layer, a backbone network layer, a neck network layer, and a prediction layer.
[0035] The input layer consists of video or image preprocessing units;
[0036] Used to periodically acquire image data from fire scenes via video or image acquisition equipment, and to perform preliminary processing to ensure the consistency of the acquired images and remove invalid data for further analysis.
[0037] like Figure 1 As shown, the backbone network consists of focus units, CBL convolutional units containing convolution, batch normalization and LeakyReLU activation functions, C3+ units, Swin Transformer units and pyramid pooling units;
[0038] The focus unit is used to perform video image segmentation operations, reducing the number of image channels and expanding the spatial size to facilitate further feature extraction and analysis. Specifically: for the input image... A new image is formed by dividing an image into blocks of fixed pixel size and then rearranging the pixels. ;
[0039] It can be represented as ;in Represents the pixel height of the original image, Represents the pixel width of the original image, Represents the number of channels in the original image. This represents the pixel height of the image obtained after segmentation. This represents the pixel width of the image obtained after segmentation. This indicates the arrangement operation performed on the focal cell;
[0040] The C3+ convolutional unit performs convolution operations on the image, using sliding convolution kernels to extract features contained in the image, including edges, textures, images, and objects. The C3+ convolutional unit controls the model's computational load and extracts feature gradient information, including local connection networks and local transfer networks. The local connection network is used to process the basic feature map... Segmentation is performed to obtain two parts of local features, namely The two local features are processed separately, one of which... Extracted from a local dense network The other part Directly passed to the local dense end; the output features of the locally connected network can be represented as , It will be transmitted through the transmission layer in the local transmission network, and its output will with After connection, it passes through another transmission layer before being output. ;
[0041] The Swin Transformer unit replaces the standard multi-head self-attention module with a shift-window-based multi-head self-attention module while keeping other parts unchanged. It consists of a shift-window-based multi-head self-attention module followed by a multilayer perceptron module containing GeLU nonlinearity, and uses residual connections between each module.
[0042] The pooling unit uses spatial pyramid pooling to control the feature dimension. For the input feature map, spatial pyramid pooling divides it into sub-regions of different scales, forming multiple grids of different sizes and constructing a multi-level spatial pyramid structure. Fast pooling is performed on each sub-region to obtain feature representations of different scales. The pooling results of different scales are concatenated to obtain a feature vector of fixed length.
[0043] Spatial pyramid pooling performs multiple max pooling operations on the input image, and then combines multiple pooling outputs with different output sizes; in each pooling operation, to ensure different input sizes Ultimately, they all have the same output size. The size and stride of the pooling sliding window are adaptively set, where the pooling sliding window size... Pooling sliding window step size ;
[0044] In this application, the neck network layer consists of a feature pyramid network unit and a path aggregation network unit. The feature pyramid network unit fuses the low-level feature map with the high-level feature map through upsampling and convolution operations to generate a new multi-scale feature map. The aggregation network unit adds a bottom-up path enhancement network to pass the low-level feature information back to the high-level feature map to enhance the model's utilization of feature information.
[0045] Simultaneously, the traditional convolution processing method is improved within the neck network layer unit by applying a trainable refocusing transformation to the pre-trained model to establish connections between parameters. Specifically, assuming the number of convolution input channels is... The number of output channels is The number of groups is The basic weight is The transformation weights are Establish identity mapping ;in This represents a convolution operation, which outputs each group by changing its weights. After transformation, the groups are regrouped and rearranged; the number of groups is... ;
[0046] Based on traditional feature prediction networks, the prediction layer introduces a spatiotemporal channel attention structure to improve the feature capture capability of key nodes during human prediction, and fuses the obtained channel features and spatiotemporal features to improve the accuracy of prediction results.
[0047] Specifically: for those divided into Group of human feature diagrams The channel features and spatiotemporal features are extracted through two attention branches, respectively. The channel attention features are obtained using a single-layer transformation and can be represented as follows: ;in This refers to the pixel height of the input feature map. This refers to the pixel width of the input feature map, where , and The transformation parameters are determined through model training and optimization. This is the channel feature map obtained after transformation;
[0048] The spatiotemporal attention feature is obtained by processing the normalized input feature map. The result of the transformation can be expressed as follows: ;in and The transformation parameters are determined through model training and optimization. This is the channel feature map obtained after transformation;
[0049] By identifying and extracting human features in the human detection network module, human data information in the image can be located, thereby obtaining personnel image data. Through continuous acquisition and analysis, personnel on site can be quickly located. However, in order to confirm the status of personnel, it is also necessary to judge their posture based on the personnel image. By classifying and matching their posture, the real-time status information of the personnel can be confirmed. When it is detected that the person in the picture is stationary for a long time, lying on the ground, or undergoing drastic changes in movement, it is necessary to intervene and check in time to confirm the abnormal situation, realize continuous mutual monitoring of personnel on site, and effectively improve the safety and collaborative rescue capabilities of firefighters on site.
[0050] The human pose analysis module uses a graph convolutional network (GNN) to identify and extract key human nodes from the aforementioned human feature map. It determines the human pose by analyzing the pose of these key nodes. Specifically:
[0051] Based on the theory of Graph Convolutional Networks (GNNs), human feature maps can be represented as graphs consisting of nodes and the connections between them. ,in Represents a set of nodes. Represents an edge set. For human feature maps, the adjacency matrix is used. In Nodes ,gather The edge in the middle represents the edge connecting any two nodes. Information is stored in the adjacency matrix In the middle, if node and If there is a connection between them, then ,otherwise , with nodes The number of edges between vertices is equal to the degree of the node. The set they form is a degree matrix. ,in Changes in human posture can be represented as motion vectors. Where T refers to the number of frames the motion change lasts, and the vector of the human key nodes during the motion change process can be represented as: The key edge can be represented as ;
[0052] Based on the above, for the input human feature map ,use Spatial location can be obtained by convolution kernel convolution. The channel output model can be expressed as: ;
[0053] in It refers to the pixel coordinates of a spatial location. It refers to the sampling function. This refers to the weighting function;
[0054] Sampling function Based on the definition of a node's location in the node's neighborhood in the feature graph, when the distance of a node to a key node does not exceed the length of the edge associated with that key node... When sampling is performed, the weight function is used. Determined by the key nodes and edge features of the human body feature map, the nodes of the human body feature map can be divided into center nodes and edge nodes. Center nodes refer to nodes that are close to the center of gravity of the human body, while edge nodes refer to nodes that are far away from the center of gravity of the human body.
[0055] Based on the above principles, the human feature graph node convolutional model processed by a graph convolutional network (GNN) can be represented as:
[0056] ;
[0057] in It means Time Node The set of domain nodes; This indicates normalization processing; express Node of time Location;
[0058] Node feature analysis based on a human feature map node convolution model can generate spatiotemporal feature maps of human nodes at different action moments. ,in It refers to the set of all key nodes in an action sequence. It refers to the first In the action frame, the _ A key node; This refers to the node connection relationship. These refer to the connection relationships between an edge formed by any key node, an edge formed by adjacent key nodes, and an edge formed by the same node in adjacent frames and a certain key node.
[0059] Obtain the spatiotemporal feature map of human body nodes The model is trained and optimized by extracting spatiotemporal feature change data of human body nodes and combining it with spatiotemporal sequence features of key human body nodes under different postures obtained through simulation. The final model after training is used to extract data and classify postures and actions of real-time images to determine human posture information and design corresponding treatment plans based on different human posture information.
[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. A fire-fighting personnel identification and early warning system, characterized in that Includes a human body detection network module and a human posture analysis module; The human detection network module is used to extract human pixel features from video images. It mainly consists of an input layer, a backbone network layer, a neck network layer, and a prediction layer. The backbone network consists of focus units, CBL convolutional units containing convolution, batch normalization and LeakyReLU activation functions, C3+ units, Swing Transformer units and pyramid pooling units; The focus unit is used to perform video image segmentation operations, reducing the number of image channels and expanding the spatial size to facilitate further feature extraction and analysis. Specifically: for the input image... A new image is formed by dividing an image into blocks of fixed pixel size and then rearranging the pixels. ; can be represented as ;in Represents the pixel height of the original image, Represents the pixel width of the original image, Represents the number of channels in the original image. This represents the pixel height of the image obtained after segmentation. This represents the pixel width of the image obtained after segmentation. This indicates the arrangement operation performed on the focal cell; The C3+ convolutional unit performs convolution operations on the image, using sliding convolution kernels to extract features contained in the image, including edges, textures, images, and objects. The C3+ convolutional unit controls the model's computational load and extracts feature gradient information, including local connection networks and local transfer networks. The local connection network is used to process the basic feature map... The segmentation yields two parts of local features, namely The two local features are processed separately, one of which... Extracted from a local dense network The other part Directly transmitted to the local dense end; The output features of a local connection network can be represented as , will be passed through a transfer layer in a local transfer network, whose output will be connected to and output after another transfer layer ; The Swin Transformer unit replaces the standard multi-head self-attention module with a shift-window-based multi-head self-attention module while keeping other parts unchanged. It consists of a shift-window-based multi-head self-attention module followed by a multilayer perceptron module containing GeLU nonlinearity, and uses residual connections between each module. The pooling unit uses Spatial Pyramid Pooling (SPP) to control the feature dimension. For the input feature map, Spatial Pyramid Pooling divides it into sub-regions of different scales, forming multiple grids of different sizes and constructing a multi-level spatial pyramid structure. Fast pooling operations are performed on each sub-region to obtain feature representations of different scales. The pooling results of different scales are concatenated to obtain a feature vector of fixed length. The neck layer consists of feature pyramid network units and path aggregation network units. The feature pyramid network units fuse low-level feature maps with high-level feature maps through upsampling and convolution operations to generate new multi-scale feature maps. The aggregation network units add a bottom-up path enhancement network to pass low-level feature information back to high-level feature maps to enhance the model's utilization of feature information. Based on the traditional feature prediction network, the prediction layer improves the feature capture ability of key nodes in the human prediction process by introducing a spatiotemporal channel attention structure, and fuses the obtained channel features and spatiotemporal features to improve the accuracy of the prediction results. The human pose analysis module uses a graph convolutional network (GNN) to identify and extract key human nodes from the aforementioned human feature map. It determines the human pose by analyzing the pose of these key nodes. Specifically: Based on the theory of Graph Convolutional Networks (GNNs), human feature maps can be represented as graphs consisting of nodes and the connections between them. ,in Represents a set of nodes. Represents an edge set. For human feature maps, the adjacency matrix is used. In Nodes ,gather The edge in the middle represents the edge connecting any two nodes. Information is stored in the adjacency matrix In the middle, if node and If there is a connection between them, then ,otherwise , with nodes The number of edges between vertices is equal to the degree of the node. The set they form is a degree matrix. ,in Changes in human posture can be represented as motion vectors. Where T refers to the number of frames the motion change lasts, and the vector of the human key nodes during the motion change process can be represented as: The key edge can be represented as ; Based on the above, for the input human feature map ,use Spatial location can be obtained by convolution kernel convolution. The channel output model can be expressed as: ; wherein denotes a pixel coordinate of the spatial position, denotes a sampling function, denotes a weight function; Sampling function According to the definition of the node position in the feature map, when the distance between a certain node and the key node is not more than the length of the edge associated with the key node , sampling is performed. Weight function According to the key nodes and edge features of the human feature map, the human feature map nodes can be divided into center nodes and edge nodes, the center nodes refer to the nodes close to the human body center of gravity, and the edges refer to the nodes away from the human body center of gravity. Based on the above theory, the convolutional model of human feature graph nodes after processing by a graph convolutional network (GNN) can be expressed as: ； wherein refers to time node field node set;, denotes normalization processing; denotes node of time position; Node feature analysis based on a human feature map node convolution model can generate spatiotemporal feature maps of human nodes at different action moments. ,in It refers to the set of all key nodes in an action sequence. It refers to the first In the action frame, the _ A key node; This refers to the node connection relationship. These refer to the connection relationships between an edge formed by any key node, an edge formed by adjacent key nodes, and an edge formed by the same node in adjacent frames and a certain key node. Obtain the spatiotemporal feature map of human body nodes The model is trained and optimized by extracting spatiotemporal feature change data of human body nodes and combining it with spatiotemporal sequence features of key human body nodes under different postures obtained through simulation. The final model after training is used to extract data and classify postures and actions of real-time images to determine human posture information and design corresponding treatment plans based on different human posture information.
2. The personnel identification pre-warning system for fire fighting according to claim 1, characterized in that, The human detection network module also includes an input layer, which consists of video or image preprocessing units; Used to periodically acquire image data from fire scenes via video or image acquisition equipment, and to perform preliminary processing to ensure the consistency of the acquired images and remove invalid data for further analysis.
3. The personnel identification pre-warning system for fire fighting according to claim 1, characterized in that, The traditional convolution processing method is improved within the neck network layer unit by applying a trainable refocusing transformation to the pre-trained model to establish connections between parameters. Specifically, assuming the number of convolution input channels is... The number of output channels is The number of groups is The basic weight is The transformation weights are Establish identity mapping ;in This represents a convolution operation, which outputs each group by changing its weights. After transformation, the groups are regrouped and rearranged; the number of groups is... .
4. The personnel identification pre-warning system for fire fighting according to claim 1, characterized in that, In the prediction layer, for those divided into Group of human feature diagrams The channel features and spatiotemporal features are extracted through two attention branches, respectively. The channel attention features are obtained using a single-layer transformation and can be represented as follows: ;in This refers to the pixel height of the input feature map. This refers to the pixel width of the input feature map, where , and The transformation parameters are determined through model training and optimization. This is the channel feature map obtained after transformation; Wherein the spatio-temporal attention feature is obtained by transforming the normalized input feature map , which can be expressed as ; wherein and is a transformation parameter, which is determined by model training optimization; is the channel feature map obtained after transformation.