An indoor fire pedestrian detection and re-identification system guided by a visual language model

Abstract

The application discloses a visual language model guided indoor fire pedestrian detection and re-identification system and belongs to the technical field of computer vision and artificial intelligence. In view of the problems of instability in complex scenes, discontinuity in identification, lack of semantic guidance mechanism and cross-frame semantic consistency of the existing indoor fire pedestrian detection system, the application encodes the pedestrian-related text description through a text feature generation module, extracts multi-scale visual features through a visual feature extraction module, generates a semantic guidance weight matrix through a text and image feature fusion module, enhances the visual features and outputs a detection result through a semantic guidance detection module, realizes cross-frame identity consistency maintenance through a semantic guidance pedestrian re-identification module, and dynamically updates a pedestrian feature library through an online learning module. The system improves the detection precision and robustness under the environment of smoke, uneven illumination and flame interference, and enhances the continuity and stability of identity re-identification.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence technology, and in particular relates to an indoor fire pedestrian detection and re-identification system guided by a visual language model. Background Technology

[0002] Indoor fires, as a typical public safety emergency, are characterized by complex environments such as enclosed spaces, high smoke concentrations, uneven lighting conditions, and significant interference from open flames, making them particularly common in typical indoor locations such as high-rise buildings and commercial complexes. In these extreme environments, rapid detection and reliable re-identification of pedestrians are crucial for assisting in assessing evacuation status, supporting emergency rescue decisions, and ensuring the effective operation of rescue systems. However, indoor fire environments are often accompanied by severe smoke obstruction, degradation of visible light information, and the coexistence of localized strong light and reflection interference. These problems are even more pronounced in indoor scenarios with complex spatial structures (such as high-rise buildings) or high population density, placing high demands on vision-based information perception systems. Currently, this technology mainly relies on computer vision-based detection systems, such as those using convolutional neural networks for target recognition and tracking. However, existing systems generally exhibit insufficient robustness and high false positive and false negative rates when facing the aforementioned complex interference, making it difficult to meet the stringent requirements for real-time performance and stability in emergency rescue operations.

[0003] In recent years, visual language modeling technology has demonstrated strong potential in the fusion and understanding of image and text semantics. These models can achieve deep semantic understanding of complex scenes, including semantic relationships between pedestrians, flames, smoke, and indoor spatial structures. However, due to their complex model structure, slow inference speed, and high computational resource requirements, they are difficult to directly embed into real-time detection systems that require low latency. Therefore, the current technological landscape lacks a system architecture that can effectively combine deep semantic understanding with efficient real-time detection.

[0004] Specifically, existing technologies suffer from several shortcomings that urgently need to be addressed. First, traditional visual detection systems lack the ability to model the unique semantic environment of fire scenes, leading to a significant decrease in discrimination capability when faced with smoke obscuring the view and intense flame light interference. Second, although visual language models possess powerful scene understanding capabilities, existing system architectures fail to integrate them with lightweight detection and recognition networks in real time. The lack of an effective semantic guidance mechanism at the system level prevents high-level semantic information from being applied to the extraction and optimization of low-level visual features. Finally, in pedestrian re-identification, traditional systems primarily rely on appearance features for cross-frame matching, failing to establish a semantic consistency-based association mechanism. This makes it impossible to guarantee the continuity of pedestrian identity features and the robustness of trajectories under conditions such as changes in person posture, brief occlusion, or drastic fluctuations in lighting. The main challenge in solving these problems lies in how to effectively transfer and integrate the knowledge of computationally intensive semantic understanding models into lightweight detection and tracking processes while ensuring real-time system responsiveness, and designing an efficient mechanism that can maintain cross-frame semantic consistency. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a visual language model-guided indoor fire pedestrian detection and re-identification system, which improves detection accuracy and robustness under conditions of smoke, uneven lighting, and flame interference, and enhances the continuity and stability of identity re-identification.

[0006] To achieve the above objectives, the present invention provides a visual language model-guided indoor fire pedestrian detection and re-identification system, comprising: The text feature generation module is used to generate text feature vectors related to pedestrians; The visual feature extraction module is used to extract multi-scale visual feature maps from the input image; The image-text feature fusion module is used to fuse the text feature vector with the multi-scale visual feature map to generate a semantic guidance weight matrix; The semantic guidance detection module is used to enhance the multi-scale visual feature map based on the semantic guidance weight matrix to obtain a semantically enhanced feature map. The detection module is used to perform pedestrian detection using the semantically enhanced feature map and output bounding boxes and confidence scores. The semantically guided pedestrian re-identification module is used to re-identify pedestrians detected in consecutive video frames across frames in order to maintain identity consistency. The online learning module is used to dynamically update the pedestrian feature database based on newly detected pedestrian features.

[0007] Optionally, the text feature generation module includes: The text prompt word definition unit is used to define a set of text prompt words related to pedestrians in indoor fire scenarios; A text encoding unit is used to encode the text prompt words using a text encoder based on a pre-trained visual language model to generate a text embedding vector; The projection unit is used to map the text embedding vector to the visual feature space through a learnable linear projection layer to obtain the text feature vector.

[0008] Optionally, the visual feature extraction module includes: The multi-scale feature extraction unit is used to extract multi-scale features from the input image using a convolutional neural network backbone network in conjunction with a feature pyramid network, thereby obtaining a set of feature maps with different spatial resolutions and semantic intensities.

[0009] Optionally, the image-text feature fusion module includes: The feature upsampling unit is used to upsample multi-scale visual feature maps to a uniform spatial size and stitch them together to form a fused visual feature map. The similarity calculation unit is used to calculate the similarity between each text feature vector and the feature vector of the fused visual feature map at each spatial location, and generate the initial response map; The weight generation unit is used to take the maximum value from the initial response values ​​corresponding to all text feature vectors for each spatial location to obtain the unnormalized guided weight matrix. The normalization unit is used to normalize and nonlinearly activate the unnormalized guiding weight matrix to generate a semantic guiding weight matrix.

[0010] Optionally, the semantic guidance detection module includes: The feature enhancement unit is used to multiply the semantic guidance weight matrix element-wise with the multi-scale visual feature map to generate a semantically enhanced feature map.

[0011] Optionally, the detection module includes: The detection head unit is used to process the semantically enhanced feature map and output the bounding box coordinates, confidence score, and class probability.

[0012] Optionally, the semantically guided pedestrian re-identification module includes: A multi-frame feature extraction unit is used to extract the appearance feature representation of the pedestrian detected in each frame. The appearance feature representation is obtained by extracting region features from the semantically enhanced feature map and concatenating them with CLIP image encoder features. The visual-text cross-frame association unit is used to construct the semantic description vector of pedestrian instances. It calculates the cosine similarity between appearance features and re-identified text prompts, and constructs a cross-frame association matrix to combine appearance similarity and semantic description vector similarity. The graph attention matching unit is used to optimize the cross-frame association matrix using a graph attention network and update pedestrian features; The temporal modeling unit is used to smooth pedestrian trajectories over time, maintaining identity continuity.

[0013] Optionally, the online learning module includes: The feature library management unit is used to maintain the pedestrian feature library and store the feature templates of pedestrians whose identities have been confirmed. The similarity calculation unit is used to calculate the similarity between newly detected pedestrian features and all template features in the feature library; The update unit is used to update the feature template of the corresponding ID using the momentum update strategy if the maximum similarity is higher than the high threshold; if the maximum similarity is lower than the low threshold, a new ID is assigned and the new feature is added to the feature library.

[0014] Technical Effects of this Invention: This invention discloses a visual language model-guided pedestrian detection and re-identification system for indoor fires. By introducing the semantic understanding capabilities of a visual language model, a semantically guided multi-module detection and re-identification architecture is constructed. The system utilizes a text feature generation module to encode pedestrian-related semantics and generates a semantically guided weight matrix through a graph-text feature fusion module, thereby enhancing the discriminative power of visual features. This significantly improves the accuracy and robustness of pedestrian detection in extreme fire environments with smoke, flames, and uneven lighting. Furthermore, the system employs a semantically guided pedestrian re-identification module, utilizing text embedding to construct semantic associations and employing a graph attention network to optimize cross-frame matching. This effectively suppresses feature drift caused by environmental disturbances, ensuring the continuity of pedestrian identity in rapidly changing scenarios. In addition, the online learning module dynamically updates the pedestrian feature library, enabling the system to adapt to dynamic changes in fire scenarios and maintain long-term stability without retraining the visual language model. This achieves a complete closed loop from semantic-level scene understanding to dynamic identity tracking. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the structure of an indoor fire pedestrian detection and re-identification system guided by a visual language model, according to an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0018] like Figure 1 As shown, this embodiment provides a visual language model-guided indoor fire pedestrian detection and re-identification system, including: The text feature generation module is used to generate text feature vectors related to pedestrians; The visual feature extraction module is used to extract multi-scale visual feature maps from the input image; The image-text feature fusion module is used to fuse the text feature vector with the multi-scale visual feature map to generate a semantic guidance weight matrix; The semantic guidance detection module is used to enhance the multi-scale visual feature map based on the semantic guidance weight matrix to obtain a semantically enhanced feature map. The detection module is used to perform pedestrian detection using the semantically enhanced feature map and output bounding boxes and confidence scores. The semantically guided pedestrian re-identification module is used to re-identify pedestrians detected in consecutive video frames across frames in order to maintain identity consistency. The online learning module is used to dynamically update the pedestrian feature database based on newly detected pedestrian features.

[0019] Furthermore, the text feature generation module includes: The text prompt word definition unit is used to define a set of text prompt words related to pedestrians in indoor fire scenarios; A text encoding unit is used to encode the text prompt words using a text encoder based on a pre-trained visual language model to generate a text embedding vector; The projection unit is used to map the text embedding vector to the visual feature space through a learnable linear projection layer to obtain the text feature vector.

[0020] Specifically, the implementation process of this embodiment includes: To incorporate prior textual semantic knowledge, this embodiment defines a set of textual prompt words related to pedestrians in indoor fire scenarios. ,like: These text prompts are fed into the text encoder of a pre-trained visual language model (such as CLIP). Obtain the corresponding text embedding vector. ,in, Indicates the first The text encoding vector of each text unit. This indicates that the output is a dimension of The text embedding is in a real vector space. To make the dimension of the text embedding compatible with the number of visual feature channels, a learnable linear projection layer is introduced. Mapping text embeddings to a visual feature space: ; in It is a weight matrix. It is a bias term. Indicates the first The text encoding vector of each text unit. This represents a mapping function. This step transforms abstract textual concepts into numerical representations that can be directly compared to pixel-level visual features, for use by subsequent image-text fusion modules.

[0021] Furthermore, the visual feature extraction module includes: The multi-scale feature extraction unit is used to extract multi-scale features from the input image using a convolutional neural network backbone network in conjunction with a feature pyramid network, thereby obtaining a set of feature maps with different spatial resolutions and semantic intensities.

[0022] Specifically, the implementation process of this embodiment includes: Given an input image This embodiment utilizes a CNN-based backbone network in conjunction with a Feature Pyramid Network (FPN) for multi-scale feature extraction, obtaining a set of feature maps with different spatial resolutions and semantic intensities, denoted as... ,in , It is the total number of layers in the feature map. For feature map of high, For feature map of high, This refers to the number of channels for visual features. These multi-scale feature maps can effectively capture visual information ranging from fine-grained details to high-level semantics, laying the foundation for detecting pedestrian targets at different scales.

[0023] Furthermore, the image-text feature fusion module includes: The feature upsampling unit is used to upsample multi-scale visual feature maps to a uniform spatial size and stitch them together to form a fused visual feature map. The similarity calculation unit is used to calculate the similarity between each text feature vector and the feature vector of the fused visual feature map at each spatial location, and generate the initial response map; The weight generation unit is used to take the maximum value from the initial response values ​​corresponding to all text feature vectors for each spatial location to obtain the unnormalized guided weight matrix. The normalization unit is used to normalize and nonlinearly activate the unnormalized guiding weight matrix to generate a semantic guiding weight matrix.

[0024] Specifically, the implementation process of this embodiment includes: To accurately inject textual semantic information into the corresponding spatial locations and semantic channels of visual features, fine feature alignment is required. First, this embodiment upsamples the multi-scale feature maps output by the FPN to a common, intermediate-scale spatial size using bilinear interpolation or deconvolution. Then, these feature maps are concatenated along the channel dimension, followed by channel integration and dimensionality reduction using a 1×1 convolutional layer. Finally, a unified-size visual feature map integrating multi-scale information is obtained. ,in For feature map of high, For feature map The height is high. Next, calculate the text embedding vector for each text. With fusion feature map Feature vector at each spatial location The similarity between the two text prompts is calculated, where i represents the position index in the height direction and j represents the position index in the width direction. The vector dot product is used as the similarity metric because it directly reflects the alignment of the two vectors in terms of direction. For each text prompt and each spatial location, an initial semantic response value is calculated. , in, Representing the features at spatial indices i and j, and the information in the c-th channel, K initial guided weight matrices are obtained by traversing all spatial locations and all text prompts. Each matrix represents the response strength of different locations in the image to specific text semantics.

[0025] After obtaining K initial response maps for different text prompts, they are merged into a comprehensive semantic guidance map. Direct averaging might dilute strong semantic signals; therefore, a strategy of taking the maximum value by category (i.e., by text prompt index) is adopted. Specifically, for spatial location... Compare the response values ​​generated by all K text prompts at that location. The maximum value among them is selected as the final unnormalized guiding weight for that position. , ; The essence of this operation is to ensure that each spatial location only generates a high response to the textual concept most relevant to it, thereby highlighting the image region that is highly relevant to any given textual description, thus forming the final guided weight matrix. .

[0026] Since the numerical range of dot product similarity may be unstable, directly using Using them as weights might interfere with the training process. Therefore, they need to be normalized and nonlinearly transformed to convert them into activation weight matrices with suitable value ranges and reasonable distributions. First, a batch normalization method based on global statistics is used to scale and translate them, so that their numerical distribution tends to be stable.

[0027] Specifically, calculation Mean and standard deviation over the entire matrix ; in The maximum value of the response generated by all K text prompts at that position is taken as the final, unnormalized guiding weight for that position. The height of the feature map, is the width of the feature map, and i and j are the indices (row and column) of the spatial location, respectively. for The global mean, for The global standard deviation is then calculated. Normalization is then performed. Next, the normalized matrix is ​​input into a Sigmoid activation function. The values ​​are mapped to the (0,1) interval to obtain the activation weight matrix. ; The S-curve characteristics of the Sigmoid function can suppress weak responses while enhancing moderate and strong responses, which is beneficial for highlighting important semantic regions.

[0028] Furthermore, the semantic guidance detection module includes: The feature enhancement unit is used to multiply the semantic guidance weight matrix element-wise with the multi-scale visual feature map to generate a semantically enhanced feature map.

[0029] Specifically, the implementation process of this embodiment includes: The obtained activation weight matrix A is used as the spatial attention weights and multiplied element-wise with the obtained fused visual feature map F to achieve semantically guided feature enhancement. For each channel c and each spatial location (i,j) on the feature map, the enhanced feature value is calculated as follows: The dot product operation means that in regions with strong semantic responses, the original visual features are preserved or even emphasized; while in regions with weak semantic responses, their feature values ​​are significantly suppressed. This mechanism can effectively improve the signal-to-noise ratio of regions in the feature map related to "pedestrians". The enhanced feature map is then fed into the detection head for bounding box coordinate offset. Regression, target confidence prediction, and category probability prediction.

[0030] Furthermore, the detection module includes: The detection head unit is used to process the semantically enhanced feature map and output the bounding box coordinates, confidence score, and class probability.

[0031] Specifically, the implementation process of this embodiment includes: Loss function of the detection module It is a weighted sum of multiple losses, designed to jointly optimize bounding box localization, target existence determination, and class differentiation. Its general form is: ; in, , , It is a hyperparameter that balances the weights of different loss terms.

[0032] 1) Bounding box regression loss The CIoU Loss (Complete Intersection over UnionLoss) method is adopted, taking into account the overlap area, center point distance, and aspect ratio. ; in, It is intersection, union, and comparison. It is Euclidean distance. It is the center point of the bounding box. It is the length of the diagonal of the smallest bounding rectangle. It is a parameter that measures the consistency of aspect ratio. .

[0033] 2) Target confidence loss Using labeled smoothed binary cross-entropy loss, we distinguish between prior boxes containing the target (positive samples) and prior boxes not containing the target (negative samples). ; in, It is the set of all sampled grid cells. It is a real label (0 or 1). It represents the confidence level of the prediction.

[0034] 3) Classification loss Using the same cross-entropy loss, we calculate the difference between the predicted class and the true class when the target exists. ; in, It is the total number of categories. It is an indicator function, when the first... The value is 1 if the predicted bounding box corresponds to the actual target, and 0 otherwise. It is a category One-hot encoded real labels, It is the predicted category probability distribution.

[0035] Furthermore, the semantically guided pedestrian re-identification module includes: A multi-frame feature extraction unit is used to extract the appearance feature representation of the pedestrian detected in each frame. The appearance feature representation is obtained by extracting region features from the semantically enhanced feature map and concatenating them with CLIP image encoder features. The visual-text cross-frame association unit is used to construct the semantic description vector of pedestrian instances. It calculates the cosine similarity between appearance features and re-identified text prompts, and constructs a cross-frame association matrix to combine appearance similarity and semantic description vector similarity. The graph attention matching unit is used to optimize the cross-frame association matrix using a graph attention network and update pedestrian features; The temporal modeling unit is used to smooth pedestrian trajectories over time, maintaining identity continuity.

[0036] Specifically, the implementation process of this embodiment includes: Building upon the detection module, this system also includes a semantically guided pedestrian re-identification module, used to process consecutive video frames and achieve cross-frame identity consistency. This module includes: Based on the detection module, the module units of this system process continuous video frame sequences. For each frame A series of pedestrian detection bounding boxes were obtained using the visual feature extraction module. ,in, Let the bounding box of the target be at the m-th time step. Let be the total number of times the m-th target appears in the sequence. For re-identification, a highly discriminative feature representation needs to be extracted for each detection box. A dual-path feature extraction strategy is employed to obtain rich information. The first path extracts the enhanced feature map from the detection module. Above, through the RoI Align (Region of Interest Align) operation, based on each detection box... The location is precisely extracted from the corresponding feature layer, and then compressed into a feature vector using global average pooling (GAP). ,in It is an eigenvector The vector contains visual information enhanced with textual semantics. The second approach, to introduce a more general and powerful semantic representation, involves cropping the detection box region from the original image. Image encoder for input CLIP model To obtain a deep semantic feature vector ,in This refers to the detection box area cropped from the original image. For feature vectors Finally, the two sets of features are concatenated and fused and dimensionality reduced through a fully connected layer to obtain the final pedestrian appearance feature representation used for re-identification. : ; in This represents vector concatenation. It involves fusing the weight matrix. Simultaneously, a more refined set of text prompts is defined, focusing on pedestrian appearance attributes. And its embedding is also obtained through the CLIP text encoder. These text embeddings will serve as semantic metrics for visual-text cross-frame association units.

[0037] The core of re-identification is calculating the similarity between pedestrian instances in different frames. This system's module units construct a cross-frame association matrix that simultaneously considers visual appearance and textual semantics. First, the intra-frame semantic description vector is calculated: for the m-th pedestrian instance in the n-th frame, its features... With all re-identified text prompts Calculate the cosine similarity to form a vector describing its semantic attributes. , ; Where, vector This quantitatively describes the degree to which the pedestrian instance conforms to various textual attributes. Then, a cross-frame correlation matrix is ​​constructed. Each element of the matrix This represents the overall correlation between the a-th pedestrian instance in frame i and the b-th pedestrian instance in frame j. It consists of two weighted components: the direct similarity of appearance features and the similarity of semantic description vectors. ; in, and These are hyperparameters used to balance the importance of appearance and semantic information. Semantic similarity decreases when appearance changes drastically due to occlusion or lighting conditions. It can provide more stable connections.

[0038] To fully utilize the output of the visual-text cross-frame association unit for global optimal matching, this system module modeles the re-identification problem as a graph optimization problem. First, an undirected graph is constructed. The node set V contains all pedestrian instances across all frames. The edge set E connects all pairs of nodes that are considered potentially related (e.g., nodes in frames that are temporally close), with each edge... The initial weights are calculated based on the correlation degree. Definition. Then, a Graph Attention Network (GAT) is applied to update node features. In each layer of the GAT, for node i, its neighboring nodes... Attention coefficient Calculated in the following way: ; ; in It is the learnable weight matrix of this layer. It is a learnable vector for the attention mechanism. This represents vector concatenation. It's the activation function. Then, the node... The new features are obtained by summing the attention-weighted features of its neighboring nodes and then performing a nonlinear transformation: .

[0039] After multiple GAT iterations, the features of each node All of these features incorporate contextual information from their local graph structure, making the features of different nodes belonging to the same person more similar, while increasing the differences in node features between different people. Based on the updated node features, the similarity matrix between all node pairs is recalculated. ,in Finally, using based on The bipartite graph matching algorithm assigns a unique and consistent identity ID to pedestrians in different frames.

[0040] For pedestrian feature sequences (i.e., a trajectory) that are initially linked together by the graph attention matching unit and belong to the same identity ID. To further leverage temporal continuity and smooth identity jumps caused by detection jitter or brief occlusions, a temporal model is used for post-processing. Here, a Transformer encoder is employed to model long temporal dependencies. First, a sinusoidal positional encoding is added to each feature in the sequence to inject temporal information, and then the input is fed into a network consisting of multiple Transformer Encoder blocks. Within each Encoder block, the core Multi-Head Self-Attention (MHA) mechanism is computed as follows: ; In this context, the query (Q), key (K), and value (V) matrices are all obtained by linear projection of the input sequence. This refers to the dimension of the key vector. It allows the model to focus on information from different positions in the sequence across different representation subspaces. After Transformer encoding, the output is a feature sequence enhanced with contextual information from the entire sequence. These features can be used for more accurate similarity calculations to correct errors in the initial matching.

[0041] Furthermore, the online learning module includes: The feature library management unit is used to maintain the pedestrian feature library and store the feature templates of pedestrians whose identities have been confirmed. The similarity calculation unit is used to calculate the similarity between newly detected pedestrian features and all template features in the feature library; The update unit is used to update the feature template of the corresponding ID using the momentum update strategy if the maximum similarity is higher than the high threshold; if the maximum similarity is lower than the low threshold, a new ID is assigned and the new feature is added to the feature library.

[0042] Specifically, the implementation process of this embodiment includes: To enable the system to adapt to dynamic changes in indoor fire scenarios (such as pedestrians' clothing becoming soiled or their posture changing), this system module includes an online learning and feature database update mechanism. The system maintains a pedestrian feature database. This contains high-quality pedestrian feature templates that have been verified. For each new frame, features extracted by the re-identification network... First, it is compared with the feature library. The similarity of all template features is compared. A high threshold is set. and a low threshold If the maximum similarity exceeds Then The ID is identified as the corresponding ID, and the momentum update strategy is used to update the template features of that ID in the feature library: ; in, This is the momentum coefficient; momentum updates can smooth feature changes and remain robust to noise. If the maximum similarity is lower than... Then it is believed Assign a new unique ID to a newly appearing pedestrian and initialize its characteristics as follows: The feature library is added. This mechanism ensures the system's adaptability to slow changes in pedestrian appearance while avoiding identity confusion caused by false detections or occlusions in a single frame.

[0043] The training of the re-identification module typically employs a joint loss function to simultaneously optimize the distinguishability and clustering properties of features.

[0044] 1) Identity Classification Loss (ID Loss): Treats re-identification as a multi-class classification problem, where each identity is a class. It uses cross-entropy loss to encourage the model to map features to the correct class centers.

[0045] ; in, This is the sample size in one batch. It is a sample The true identity label, It is the weight vector of the corresponding category. It represents the total number of identities in the training set.

[0046] 2) Triplet Loss: In a triplet (anchor sample) Positive samples Same ID), negative samples (For different IDs) the distance between the anchor point and the positive sample is reduced, and the distance between the anchor point and the negative sample is increased. ; in, It's a distance metric, and the margin is a preset interval. The total loss function of the re-identification module is a weighted sum of the above losses: ; in, , These are weighting coefficients used to balance the various losses.

[0047] This invention discloses a visual language model-guided pedestrian detection and re-identification system for indoor fires. By introducing the semantic understanding capabilities of a visual language model, a semantically guided multi-module detection and re-identification architecture is constructed. The system uses a text feature generation module to encode pedestrian-related semantics and a graph-text feature fusion module to generate a semantically guided weight matrix, thereby enhancing the discriminative power of visual features. This significantly improves the accuracy and robustness of pedestrian detection in extreme fire environments with smoke, flames, and uneven lighting. Furthermore, the system uses a semantically guided pedestrian re-identification module to construct semantic associations using text embedding and employs a graph attention network to optimize cross-frame matching, effectively suppressing feature drift caused by environmental disturbances and ensuring the continuity of pedestrian identity in rapidly changing scenes. In addition, an online learning module dynamically updates the pedestrian feature library, enabling the system to adapt to the dynamic changes in fire scenes and maintain long-term stability without retraining the visual language model, thus achieving a complete closed loop from semantic-level scene understanding to dynamic identity tracking.

[0048] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual language model-guided indoor fire pedestrian detection and re-identification system, characterized in that, include: The text feature generation module is used to generate text feature vectors related to pedestrians; The visual feature extraction module is used to extract multi-scale visual feature maps from the input image; The image-text feature fusion module is used to fuse the text feature vector with the multi-scale visual feature map to generate a semantic guidance weight matrix; The semantic guidance detection module is used to enhance the multi-scale visual feature map based on the semantic guidance weight matrix to obtain a semantically enhanced feature map. The detection module is used to perform pedestrian detection using the semantically enhanced feature map and output bounding boxes and confidence scores. The semantically guided pedestrian re-identification module is used to re-identify pedestrians detected in consecutive video frames across frames in order to maintain identity consistency. The online learning module is used to dynamically update the pedestrian feature database based on newly detected pedestrian features.

2. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The text feature generation module includes: The text prompt word definition unit is used to define a set of text prompt words related to pedestrians in indoor fire scenarios; A text encoding unit is used to encode the text prompt words using a text encoder based on a pre-trained visual language model to generate a text embedding vector; The projection unit is used to map the text embedding vector to the visual feature space through a learnable linear projection layer to obtain the text feature vector.

3. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The visual feature extraction module includes: The multi-scale feature extraction unit is used to extract multi-scale features from the input image using a convolutional neural network backbone network in conjunction with a feature pyramid network, thereby obtaining a set of feature maps with different spatial resolutions and semantic intensities.

4. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The image-text feature fusion module includes: The feature upsampling unit is used to upsample multi-scale visual feature maps to a uniform spatial size and stitch them together to form a fused visual feature map. The similarity calculation unit is used to calculate the similarity between each text feature vector and the feature vector of the fused visual feature map at each spatial location, and generate the initial response map; The weight generation unit is used to take the maximum value from the initial response values ​​corresponding to all text feature vectors for each spatial location to obtain the unnormalized guided weight matrix. The normalization unit is used to normalize and nonlinearly activate the unnormalized guiding weight matrix to generate a semantic guiding weight matrix.

5. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The semantic guidance detection module includes: The feature enhancement unit is used to multiply the semantic guidance weight matrix element-wise with the multi-scale visual feature map to generate a semantically enhanced feature map.

6. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The detection module includes: The detection head unit is used to process the semantically enhanced feature map and output the bounding box coordinates, confidence score, and class probability.

7. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The semantically guided pedestrian re-identification module includes: A multi-frame feature extraction unit is used to extract the appearance feature representation of the pedestrian detected in each frame. The appearance feature representation is obtained by extracting region features from the semantically enhanced feature map and concatenating them with CLIP image encoder features. The visual-text cross-frame association unit is used to construct the semantic description vector of pedestrian instances. It calculates the cosine similarity between appearance features and re-identified text prompts, and constructs a cross-frame association matrix to combine appearance similarity and semantic description vector similarity. The graph attention matching unit is used to optimize the cross-frame association matrix using a graph attention network and update pedestrian features; The temporal modeling unit is used to smooth pedestrian trajectories over time, maintaining identity continuity.

8. The visual language model-guided indoor fire pedestrian detection and re-identification system as described in claim 1, characterized in that, The online learning module includes: The feature library management unit is used to maintain the pedestrian feature library and store the feature templates of pedestrians whose identities have been confirmed. The similarity calculation unit is used to calculate the similarity between newly detected pedestrian features and all template features in the feature library; The update unit is used to update the feature template of the corresponding ID using the momentum update strategy if the maximum similarity is higher than the high threshold; if the maximum similarity is lower than the low threshold, a new ID is assigned and the new feature is added to the feature library.

Images