Fire fighting access occupation early warning grade analysis system and method based on large model

Abstract

The invention relates to the technical field of fire fighting access occupancy early warning, and particularly discloses a fire fighting access occupancy early warning grade analysis system and method based on a large model, and the method comprises the steps: monitoring and collecting the state of a fire fighting access in real time through a camera, so as to obtain a time sequence of a fire fighting access state monitoring key frame; and then detecting the fire fighting access state monitoring key frames by using a visual large model to identify the fire fighting access state, the object position and the change condition in each key frame so as to generate a corresponding detection result. And finally, carrying out fire fighting access occupation early warning grade analysis on detection results obtained through analysis by utilizing a large language model and prompt information to obtain analysis results. And compared with a traditional single-frame target detection analysis result, the method is beneficial to improving the speed and reliability of fire emergency response, so that potential safety hazards caused by blockage of a fire fighting access are reduced.

Description

Technical Field

[0001] The present application relates to the technical field of fire passage occupancy warning technology, and more specifically, to a large model-based fire passage occupancy warning level analysis system and method. Background Art

[0002] With the acceleration of urbanization and the increase in building density, fire safety has become increasingly important. Fire passages are important paths for firefighters to quickly reach the fire scene in an emergency, and their unobstructed access is the key to protecting life and property. However, in daily management, it is common for fire passages to be occupied, which not only increases the difficulty of rescue in the event of a fire, but also poses a potential threat to the lives and safety of residents.

[0003] Traditional fire escape occupancy detection usually relies on analyzing a single image captured by a camera to determine whether there is an object in the target area to determine whether it is occupied. This method is easily affected by factors such as lighting and angle, resulting in misjudgment. In addition, it cannot capture the position information of objects that change over time, which may lead to incorrect interpretation of dynamic scenes. In addition, past practices often require pre-setting multiple security levels, such as which type of target occupancy is considered high risk, etc. These standards are mostly based on human experience and lack flexibility and scientific basis. As environmental conditions change, the original standards may no longer apply, thus affecting the accuracy of judgment. Summary of the invention

[0004] The present application provides a large-model-based fire passage occupancy warning level analysis system and method, which can automatically identify and analyze the status of the fire passage, detect abnormalities in time and issue alarms, thereby ensuring that the fire passage remains unobstructed at all times.

[0005] In the first aspect, a fire passage occupancy warning level analysis method based on a large model is provided, comprising:

[0006] Obtain the time series of key frames of fire passage status monitoring collected by the camera;

[0007] Using the trained visual big model, each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frames is detected to obtain a time series of detection results, wherein the detection results include the fire passage state, object position and change;

[0008] After adding a prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, and the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether there is occupation; b. If there is occupation, how long is the duration; c. Give a reasonable warning level according to the occupancy duration and the type of occupied object".

[0009] In the second aspect, a fire passage occupancy warning level analysis system based on a large model is provided, including:

[0010] The fire passage status monitoring and acquisition module is used to obtain the time series of the fire passage status monitoring key frames collected by the camera;

[0011] A fire passage state detection module, used to detect each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frames using the trained visual large model to obtain a time series of detection results, wherein the detection results include the fire passage state, object position and change;

[0012] The fire passage occupancy warning level analysis module is used to add a prompt word at the end of the time series of the detection result, and then input it into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, wherein the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether there is occupation; b. If there is occupation, how long is the duration; c. Give a reasonable warning level according to the duration of occupation and the type of occupied object".

[0013] The present application provides a fire passage occupancy warning level analysis system and method based on a large model, which uses a camera to monitor and collect the status of the fire passage in real time to obtain a time series of key frames for monitoring the status of the fire passage, and then uses a large visual model to detect these key frames for monitoring the status of the fire passage to identify the status of the fire passage, the position of the object and the change in each key frame, thereby generating corresponding detection results. Finally, a large language model and prompt information are used to perform fire passage occupancy warning level analysis on the detection results obtained by these analyses to obtain analysis results. Compared with the traditional single-frame target detection analysis results, this helps to improve the speed and reliability of fire emergency response, thereby reducing the safety hazards caused by fire passage blockage. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0015] Figure 1 This is a schematic flowchart of a large model-based fire passage occupancy warning level analysis method according to an embodiment of the present application.

[0016] Figure 2 This is a data flow diagram of the fire passage occupancy warning level analysis method based on a large model in an embodiment of the present application.

[0017] Figure 3 This is a schematic flowchart of S3 in the fire passage occupancy warning level analysis method based on a large model in an embodiment of the present application.

[0018] Figure 4 This is a schematic flowchart of S32 in the fire passage occupancy warning level analysis method based on a large model in an embodiment of the present application.

[0019] Figure 5 This is a schematic flowchart of S321 in the fire passage occupancy warning level analysis method based on a large model in an embodiment of the present application.

[0020] Figure 6 This is a schematic flowchart of S322 in the fire passage occupancy warning level analysis method based on a large model in an embodiment of the present application.

[0021] Figure 7 This is a schematic block diagram of a large model-based fire passage occupancy warning level analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0023] In response to the above technical problems, in order to effectively monitor and warn of the occupancy of fire passages, intelligent monitoring systems based on artificial intelligence technology have come into being. Such systems can automatically identify and analyze the status of fire passages, detect anomalies in a timely manner and issue alarms, thereby ensuring that fire passages remain unobstructed at all times.

[0024] Based on this, in the technical solution of the present application, a method for analyzing the fire passage occupancy warning level based on a large model is proposed, which aims to use advanced machine learning technology, including a combination of a large visual model and a large language model, to provide a more reliable, flexible and intelligent solution. Specifically, the technical concept of the present application is to obtain a time series of key frames for monitoring the state of the fire passage by real-time monitoring and collecting the state of the fire passage through a camera, and then use a large visual model to detect these key frames for monitoring the state of the fire passage to identify the state of the fire passage, the position of the object and the change in each key frame, so as to generate corresponding detection results. Finally, the large language model and prompt information are used to analyze the fire passage occupancy warning level of these analyzed detection results to obtain the analysis results. Compared with the traditional single-frame target detection analysis results, this helps to improve the speed and reliability of fire emergency response, thereby reducing the safety hazards caused by fire passage blockage.

[0025] Specifically, Figure 1 and Figure 2 As shown, the steps of the fire passage occupancy warning level analysis method based on the large model include: S1, obtaining the time series of fire passage status monitoring key frames collected by the camera; S2, using the trained visual large model to detect each fire passage status monitoring key frame in the time series of the fire passage status monitoring key frame to obtain a time series of detection results, and the detection results include the fire passage status, object position and change; S3, after adding a prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, and the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether there is occupation; b. If there is occupation, how long is the duration; c. Give a reasonable warning level according to the occupancy duration and the type of occupied object".

[0026] Exemplarily, in step S1, a time series of key frames for monitoring the status of the fire passage collected by the camera is obtained. It should be understood that analyzing the situation of the fire passage through continuous key frames instead of a single picture can provide richer information and help the system understand the dynamic changes of objects in the fire passage, such as the movement, appearance or disappearance of objects, so as to more reliably determine whether there is occupancy. This method helps to reduce misjudgments caused by changes in lighting, angle problems or other instantaneous factors, improve the accuracy of detection results, and better adapt to complex situations in different scenarios.

[0027] In one embodiment, obtaining a time series of key frames for monitoring the status of a fire passage collected by a camera includes: It is necessary to install appropriate camera equipment around the fire passage, and these devices should have sufficient resolution and frame rate to ensure that clear and coherent images are captured. The location of the camera should be carefully selected to ensure that all important areas are covered, especially those that are easily occupied. Once the camera starts working, it will continuously record the video stream.

[0028] Exemplarily, in step S2, the trained visual large model is used to detect each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frame to obtain a time series of detection results, and the detection results include the fire passage state, object position and change. It should be understood that the traditional method relies on single-frame image analysis, which is easily affected by factors such as lighting and angle, and cannot capture the position information of the object changing over time, which may lead to misjudgment. However, by processing the key frames of the time series through the visual large model, these problems can be overcome and more reliable and detailed results can be provided.

[0029] In order to ensure that the large visual model can effectively handle fire escape scenarios, it is first necessary to build a high-quality dataset for model training. This dataset should contain a variety of images taken under different lighting conditions, angle changes, and weather conditions, and pay special attention to collecting various obstacles that may appear in fire escapes, such as debris, people, vehicles, etc. Each image needs to be manually or semi-automatically labeled to clearly mark the location of the fire escape, whether there are objects occupying the escape, and the category, size, and precise location of the occupied object.

[0030] Next, a suitable visual large model architecture is selected for training. In one embodiment, the trained visual large model used is an RT-DETR model based on the Transformer architecture. This architecture has been shown to have excellent target detection capabilities, especially when processing large-scale data, and it will not easily overfit. In addition, an important feature of the RT-DETR model is that it can realize open vocabulary target detection, that is, the model can also effectively identify new categories that are not covered in the training set. This is achieved by modifying the target classification layer weights at the end of the network: CLIP (Contrastive Language–Image Pre-training) is used to encode the categories, and the embedding results are used to replace the original target classification layer weights. In this way, when encountering new categories in the application stage, only the corresponding text description needs to be provided without retraining the entire model.

[0031] Once the model is trained, it can be applied to actual monitoring tasks. The system receives a real-time video stream from the camera and extracts a series of keyframes from it as input to the visual big model. These keyframes are selected based on specific criteria, such as when there is a significant change in the picture (such as a new object entering the field of view or the position of an existing object changes), or sampling at fixed time intervals. For each keyframe, the visual big model will output detailed detection results, including whether the fire passage is unobstructed and whether there are any obstacles; mark the specific coordinates of all objects that appear in the fire passage, which helps to understand the relative position of the objects relative to the passage boundary; analyze the movement trajectory of objects in the time series to determine whether they are temporarily stopped or occupy the passage for a long time. For example, if an object appears in a keyframe but disappears in the next few frames, it may be a temporary stop; conversely, if the object continues to exist in multiple consecutive keyframes, there may be a risk of long-term occupation.

[0032] In particular, considering the low reliability of single-frame results, this application proposes a method based on sequence analysis and result reorganization for a more in-depth analysis. Specifically, the number of key frames within a certain time period is designed, the time series data is evenly sampled, each frame of the video image is detected, and the detection results are matrix analyzed. First, it is calculated whether each target is in the area. For targets in the area, the proportion of the fire area occupied is calculated and summarized. Then, a time series position analysis is performed on a single target to determine whether there is any movement and to give a label. This method not only improves the accuracy of detection, but also enhances the understanding of dynamic changes.

[0033] By continuously monitoring and recording this information, the present application can construct a time series about the occupancy of the fire passage. This time series is not only a series of static results, but also a dynamic process record, reflecting the behavior patterns of objects in the fire passage and their development trends over time. For example, if an object is found to appear in the fire passage from a key frame and remains unchanged in the next multiple frames, then this may mean that the object has occupied the passage for a period of time and further attention needs to be paid to its potential safety risks. In order to make the detection results more intuitive and easy to understand, this information can also be converted into a concise text description. For example, "At the 5th second, the entrance to the fire passage was occupied by a pile of debris, the relative position of the object was (x1, y1, w, h), and the occupancy ratio was s". Here, the occupancy ratio s is obtained by calculating the ratio of the target occupied area to the entire fire passage area, which is different from the concept of intersection over union (IoU) in traditional target detection. In this application, for example, if the other category is s1 and the target area is s2, the intersection area s' is first calculated, and then the ratio of s' to s2 is calculated. After the entire sequence is run, a set T ={text1, text2, text3, ...} consisting of multiple such texts will be generated, and together with the specially designed prompt words (prompt), it will be used as input for the large language model to perform the final warning level analysis.

[0034] Exemplarily, in step S3, after adding the prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result. It should be understood that although the large visual model can provide accurate target detection information, it may be insufficient for decision support in complex scenarios (such as judging whether there is occupancy, duration, and reasonable warning level). The large language model has powerful natural language understanding and generation capabilities. Through the guidance of appropriate prompt words, it can help it better understand the meaning represented by the detection results, and then conduct a higher level of analysis. Specifically, the prompt word provides clear task guidance for the large language model, that is, to judge whether the fire passage is occupied based on the provided description, and output detailed warning information. This method not only improves the accuracy of the final analysis results, but also makes the whole process more automated and intelligent.

[0035] That is, after adding the prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result. By combining the time series of the detection result with the prompt word, the large language model can be helped to better understand the dynamic changes in the state of the fire passage in the time series data of the detection result and its meaning. For example, information such as the position movement, appearance or disappearance of objects in continuous frames can be conveyed to the model in this way, so that it can more accurately parse the meaning behind these changes, thereby obtaining more accurate fire passage occupancy warning level analysis results, which helps to provide more reasonable decisions.

[0036] In one embodiment, Figure 3 As shown, after the prompt word is added to the end of the time series of the detection results, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, including: S31, semantic embedding coding is performed on each detection result in the time series of the detection results to obtain a time series of detection result semantic embedding coding vectors; S32, context semantic association coding is performed on the time series of the detection result semantic embedding coding vectors to obtain a fire passage detection result time series context semantic association coding vector; S33, the prompt word is added to the end of the fire passage detection result time series context semantic association coding vector.

[0037] In the step S31, semantic embedding coding is performed on each detection result in the time series of the detection results to obtain a time series of semantic embedding coding vectors of the detection results. Considering that the original detection results (for example, object position, category and occupancy ratio, etc.) exist in the form of structured data or text description. In order for this information to be effectively processed by the large language model, it is necessary to convert them into a unified mathematical expression. Therefore, in the technical solution of the present application, semantic embedding coding is performed on each detection result in the time series of the detection results to obtain a time series of semantic embedding coding vectors of the detection results. Through the semantic embedding coding method, these different types of detection result data types can be converted into the same data expression form, and the implicit association relationship between different data types in each detection result can be captured, such as the relationship between the object position, category and occupancy ratio of the obstacle, which enhances the model's understanding ability, thereby facilitating hierarchical analysis and early warning of fire passage occupancy.

[0038] In one embodiment, a pre-trained language model, such as BERT or GPT series models, can be used to encode each descriptive text. This process involves segmenting the text into words or subword units, and then mapping each unit to a point in a multidimensional space based on the weight parameters learned inside the model. The advantage of this is that the semantic relationship between words can be captured, and even new concepts that are not covered in the training set can be understood to a certain extent.

[0039] In the step S32, context semantic association coding is performed on the time series of the detection result semantic embedding coding vector to obtain the fire passage detection result time series context semantic association coding vector. It should be understood that since each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector respectively contains the performance feature information about the fire passage state in each fire passage state monitoring key frame, including embedded semantic feature information such as object position, category and occupancy ratio, these detection result semantic embedding coding vectors can be regarded as a compact and meaningful representation of the fire passage state of the scene at that moment. However, when performing fire passage occupancy detection based on the fire passage state shown in a single frame image, it is necessary to manually set multiple security levels in advance, such as which type of target occupancy is considered high risk and other human experience, which can easily make the single frame analysis result unreliable and cause misjudgment problems, and there is no need to set the warning level scene based on human experience, such as vehicle occupancy as a high-risk situation. Furthermore, it is also considered that since the fire passage state features in each fire passage state monitoring key frame have temporal edge change features and correlation relationships in the time dimension, this correlation information reflects the temporal changes of information such as the location of fire passage obstacles, which helps to more accurately capture how objects in the fire passage move, or whether the occupancy situation is deteriorating, and provides a basis for the subsequent fire passage occupancy warning level analysis. Based on this, in order to better understand and predict the occupancy status and change trend of the fire passage, in the technical solution of the present application, the time series of the detection result semantic embedding coding vector is further contextualized semantically associated to obtain the fire passage detection result temporal contextual semantically associated coding vector. In particular, the contextual semantic association coding process is performed through a feature message passing network, which explicitly measures the propagation efficiency of the fire passage state features under each key frame in the network by introducing feature jump degree and message passing space span, thereby highlighting important time node information in the process of fire passage state information propagation and dynamically adjusting the information flow to obtain a fire passage detection result temporal contextual semantically associated feature message passing expression with good generalization ability and expressiveness.

[0040] In one embodiment, Figure 4As shown, in the step S32, contextual semantic association encoding is performed on the time series of the detection result semantic embedding coding vector to obtain the fire passage detection result temporal contextual semantic association coding vector, including: S321, calculating the feature jump degree and message transmission space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector to obtain the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transmission space span; S322, based on the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transmission space span, performing significant weighted aggregation on the time series of the detection result semantic embedding coding vector to obtain the fire passage detection result temporal contextual semantic association coding vector.

[0041] In one embodiment, Figure 5 As shown, in the step S321, the feature jump degree and message transfer space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector are calculated to obtain the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transfer space span, including: S3211, calculating the feature jump degree of each detection result semantic embedding coding vector to obtain the time series of the detection result semantic feature jump degree. Specifically, the process can be expressed as follows:

[0042]

[0043]

[0044]

[0045] in, is the time series of the semantic embedding encoding vector of the detection result, , , and are the first, second, and third time series of the semantic embedding encoding vector of the detection result. and The detection result semantic embedding encoding vector, yes Middle The eigenvalues ​​at the positions, yes The number of eigenvalues ​​in , and They are the time series of the semantic weighted average feature values ​​of the detection results. and The semantically weighted average feature value of the detection results, is the time series of the semantic feature jump degree of the detection result. The semantic feature jump degree of each detection result.

[0046] S3212, calculating the number of message transmissions between each detection result semantic embedding coding vector and the last detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector to obtain a sequence of detection result semantic transmission times; S3213, based on the sequence of detection result semantic transmission times, calculating the message transmission space span of each detection result semantic embedding coding vector to obtain the time series of the detection result semantic feature message transmission space span. Specifically, the process can be expressed as follows:

[0047]

[0048] in, express arrive The number of message delivery times, is the time series of the spatial span of the semantic feature message transmission of the detection result. The detection result semantic feature message conveys the spatial span.

[0049] Specifically, by calculating the feature jump degree of the semantic embedding coding vector of each detection result, the degree of scene change between different time points can be quantified. The so-called feature jump degree refers to the difference between the semantic embedding coding vectors at two or more adjacent time points. In a specific example, if an object suddenly appears in the fire escape and occupies a part of the space, the corresponding semantic embedding coding vector will change significantly, which will be reflected in the higher feature jump degree value. On the contrary, if the scene is relatively stable and there is no obvious movement or addition of objects, the feature jump degree will be lower. Therefore, the generated time series of the semantic feature jump degree of the detection results can help us identify which time periods have important events, such as the appearance of new obstacles or major changes in the positions of existing objects.

[0050] Next, the number of message passing between each encoding vector and the last semantic embedding encoding vector of the detection result in the time series is calculated, which is actually a measure of the frequency of information update at each time point. The number of message passing here refers to how many times new information is introduced from the current time point to the end of the sequence. In a practical application scenario, suppose that the camera in the underground garage takes a picture every few seconds and sends it to the large visual model for processing. When a vehicle enters the fire passage and stops there, the first few key frames may record this new situation, but as the vehicle continues to stop, subsequent key frames may no longer contain additional new information. At this point, the number of message passing can help distinguish those time points that truly bring new changes (such as the first entry of the vehicle) from other time points that just repeat the confirmation of known conditions.

[0051] Finally, based on the above-obtained message passing frequency sequence, the message passing space span of the semantic embedding encoding vector of each detection result is further calculated to evaluate the influence range of the information carried by each time point in the entire time series. The message passing space span reflects how the information of a certain period of time propagates and affects the state description of subsequent periods. In a typical scenario, if a private car is parked on a fire lane for a long time, the first few detection results will record this fact, and over time, even if the vehicle position does not change significantly, it still has a continuous impact on the entire time series. This influence can be reflected by the message passing space span: early key frames have a large spatial span because they define the basic conditions for all subsequent time points; while later key frames gradually reduce their influence unless new changes occur (such as vehicle departure). Therefore, the constructed time series of the message passing space span of the semantic feature of the detection result provides important clues about the information propagation path and intensity, allowing us to more accurately determine which factors play a key role in the overall trend.

[0052] In one embodiment, Figure 6 As shown, in the step S322, based on the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transmission space span, the time series of the detection result semantic embedding coding vector is significantly weighted and aggregated to obtain the fire channel detection result temporal context semantic association coding vector, including: S3221, based on the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transmission space span, the message transmission significant weights of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector are calculated to obtain the time series of the detection result semantic feature message transmission significant weights. Specifically, the process can be expressed by the formula:

[0053]

[0054]

[0055] in, and is the modulation parameter, is the time series of the significant factor of the semantic feature message transmission of the detection result The semantic feature message transmission significance factor of the detection results is is the normalization function, is a masking operation, is a predetermined threshold, is the time series of the significant weight of the semantic feature message of the detection result The semantic feature message of each detection result conveys significant weight.

[0056] S3222, using the time series of the semantic feature message of the detection result to transmit significant weights, calculate the weighted sum of the time series of the semantic embedding coding vector of the detection result to obtain the temporal context semantic association coding vector of the fire escape detection result. Specifically, the process can be expressed by the formula:

[0057]

[0058] in, is the number of vectors in the time series of the semantic embedding encoding vector of the detection result, It is the temporal context semantic association encoding vector of the fire channel detection result.

[0059] Specifically, by combining the time series of the jump degree of the semantic features of the detection results (i.e., the degree of scene change between adjacent time points) and the time series of the message transmission space span of the semantic features of the detection results (i.e., how the information of a certain period of time propagates and affects the state description of the subsequent period of time), the importance of each time point can be evaluated more comprehensively. A high jump degree means that the scene has changed significantly, which may indicate the occurrence of an important event; a large message transmission space span indicates that the information at this time point has a wide impact on the subsequent state. Taking these two factors into consideration, a more reasonable message transmission significance weight can be assigned to each time point, thereby ensuring that those time points that truly bring new changes or have long-term effects receive sufficient attention.

[0060] Finally, the calculated message transfer significance weight is used as the coefficient to perform weighted summation on the time series of the semantic embedding encoding vector of the entire detection result, and the temporal context semantic association encoding vector of the fire channel detection result can be obtained. This encoding vector is not just a simple numerical summary, but the result of integrating all time point information and their interrelationships. It retains the key features of the original data about the state changes of the fire channel and highlights those factors that play an important role in the overall trend. For example, in the above example, if the system detects that a private car occupies the fire channel for a long time, it can not only accurately report the specific time and location of the occupation, but also predict the impact of the situation on future traffic capacity based on the temporal context semantic association encoding vector. This prediction function enables the system to identify potential risks in advance and trigger high-priority alarms in time to notify relevant personnel to take action to remove the vehicle and avoid dangerous situations.

[0061] In one embodiment, based on the time series of the jump degree of the detection result semantic feature and the time series of the message transmission space span of the detection result semantic feature, the message transmission significance weights of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector are calculated to obtain the time series of the detection result semantic feature message transmission significance weights, including: based on the feature jump degree and the message transmission space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector, the message transmission significance factors of each detection result semantic embedding coding vector are calculated to obtain the time series of the detection result semantic feature message transmission significance factors; the time series of the detection result semantic feature message transmission significance factors are gated and screened to obtain the time series of the detection result semantic feature message transmission significance weights.

[0062] In one embodiment, based on the feature jump degree and message transmission space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector, the message transmission significance factor of each detection result semantic embedding coding vector is calculated to obtain the time series of the detection result semantic feature message transmission significance factor, including: extracting the detection result semantic feature jump degree and the detection result semantic feature message transmission space span corresponding to the first detection result semantic embedding coding vector from the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transmission space span; calculating the square of the detection result semantic feature jump degree corresponding to the first detection result semantic embedding coding vector to obtain the detection result semantic feature jump degree square modulation representation value; using the detection result semantic feature message transmission space span corresponding to the first detection result semantic embedding coding vector as the modulation coefficient to perform weighted modulation optimization representation on the detection result semantic feature jump degree square modulation representation value to obtain the detection result semantic feature message transmission significance factor.

[0063] In summary, the process of contextual semantic association encoding of the time series of the semantic embedding encoding vector of the detection result is to understand the broader temporal connectivity and long-distance temporal dependency of the fire passage state while maintaining the static feature representation of the fire passage state at each moment through the combination of feature jump degree and spatial span. This method integrates the dynamic change pattern of the fire passage state in the time dimension, so that the subsequent large language model or other decision modules can make accurate warning level judgments based on richer and more comprehensive data. In particular, the process of contextual semantic association encoding is to automatically identify the fire passage state of these key time nodes by calculating the message passing significance factor through the feature message passing network, and give a higher weight to ensure that the model can focus on the most relevant fire passage occupancy information for warning level analysis. This method is particularly suitable for application scenarios that require real-time response and accurate judgment, such as fire passage occupancy warning. It can effectively capture the development trend of occupancy behavior, timely discover potential risks, and provide scientific basis for managers so that they can take appropriate actions to ensure public safety.

[0064] In the step S33, the prompt word is added to the tail of the fire passage detection result temporal context semantic association coding vector. It should be understood that the fire passage detection result temporal context semantic association coding vector has been obtained through the above processing. Even with the fire passage detection result temporal context semantic association coding vector, the large language model still needs clear task guidance to effectively perform reasoning and decision-making. The role of the prompt word is to provide specific instructions to the large language model, telling it what to do next and how to interpret the input data. Specifically, in the prompt word, the task goal is first briefly explained, that is, to analyze whether the fire passage is occupied. At the same time, the model is required to judge the warning level based on the occupancy situation (such as occupancy time, object type). For example, if the occupancy lasts for a long time and is an object that affects traffic (such as personnel), it can be judged as a high warning. Furthermore, further restrictions can be added to the prompt words to generate more standardized analysis results. For example, the prompt words are "Please determine whether the fire escape is occupied based on the above description, and give the following information: a. Whether it is occupied (yes / no). b. If it is occupied, how long is the duration (in seconds). c. Give a reasonable warning level (low / medium / high) based on the duration of occupancy and the type of occupied object (for example, debris or personnel)".

[0065] Here, when each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector respectively represents the low-dimensional encoded semantic features of the corresponding detection result, when performing feature sequence message transmission based on feature jump degree, insufficient correspondence in the feature jump degree association of the global semantic space caused by the difference in the local semantic space of the generated encoded semantics will cause the sparse sequence message association transmission of the fire passage detection result temporal context semantic association coding vector, thereby reducing the accuracy of the fire passage occupancy warning level analysis result obtained by inputting the fire passage occupancy warning level analysis engine based on the large language model due to the lack of generated reasoning degree.

[0066] In a preferred example, after adding a prompt word to the tail of the fire passage detection result temporal context semantic association encoding vector, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, including:

[0067] Determine the number of super-distributed eigenvalues ​​whose difference from the feature mean is greater than a threshold in the fire channel detection result temporal context semantic association encoding vector, and calculate the reciprocal of the logarithm of the number of super-distributed eigenvalues ​​with base two and the exponential value of the reciprocal of the number of super-distributed eigenvalues ​​with a natural constant as the base to obtain the first fire channel detection result temporal context semantic association response field cell representation value The cell representation value of the response field associated with the second fire channel detection result temporal context semantics :

[0068]

[0069]

[0070]

[0071] in is the number of super-distributed eigenvalues ​​in the temporal context semantic association encoding vector of the fire channel detection result, and is the threshold hyperparameter, represents the temporal context semantic association encoding vector of the fire channel detection result, The first one represents the temporal context semantic association encoding vector of the fire channel detection result eigenvalues, Representation calculation The number of Represents the mean value of all eigenvalues ​​of the temporal context semantic association coding vector of the fire channel detection result;

[0072] Calculate the hyperbolic sine function value of the sum of squares of all eigenvalues ​​of the temporal context semantic association encoding vector of the fire channel detection result:

[0073]

[0074] in, represents the length of the fire channel detection result temporal context semantic association encoding vector, represents the hyperbolic sine function, Represents the value of the hyperbolic sine function;

[0075] And calculate the exponential value of the hyperbolic sine function value with the natural constant as the base and then divide it by the square of the length of the fire channel detection result temporal context semantic association encoding vector to obtain the fire channel detection result temporal context semantic association response multidimensional field manifold value ,in, represents a natural constant;

[0076] Calculate the multi-dimensional field manifold value of the fire channel detection result time series context semantic association response Relative to the first fire channel detection result, the temporal context semantic association response field cell representation value The second fire channel detection result temporal context semantic association response field cell representation value The fire channel detection result temporal context semantic association response covariance integration value:

[0077]

[0078] in, It represents the covariance integration value of the temporal context semantic association response of the fire channel detection result;

[0079] Calculate the power function feature vector of the fire channel detection result temporal context semantic association coding vector with the fire channel detection result temporal context semantic association response covariance integration value as the inverse, and calculate the autocorrelation matrix of the power function feature vector to obtain the fire channel detection result temporal context semantic association response adaptive representation matrix, that is:

[0080]

[0081] in, represents a power function feature vector whose inverse is the integrated value of the response covariance of the temporal context semantic association of the fire channel detection result for calculating the temporal context semantic association coding vector of the fire channel detection result, represents the transpose of a vector, represents matrix multiplication, Represents the fire channel detection result temporal context semantic association response adaptive representation matrix;

[0082] The fire channel detection result temporal context semantic association coding vector is matrix-multiplied with the fire channel detection result temporal context semantic association response adaptive representation matrix to obtain an optimized fire channel detection result temporal context semantic association coding vector, that is:

[0083]

[0084] Here, the fire channel detection result temporal context semantic association encoding vector is a row vector, Represents the optimized fire channel detection result temporal context semantic association encoding vector;

[0085] After the prompt word is added to the tail of the optimized fire passage detection result temporal context semantic association encoding vector, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result.

[0086] That is, in view of the lack of a global adaptive mapping mechanism caused by cross-distributed field associations in the attribute clusters of the fire passage detection result temporal context semantic association coding vector under the preset parameter discrete arrangement, a multi-dimensional tensor field cell representation framework is constructed by the mutual covariance coupling driving scheme of the fire passage detection result temporal context semantic association coding vector, and the deep interaction law of its parameter space is revealed by analyzing the fully connected topological association composite pattern. Therefore, by adopting the gradient-based multi-dimensional field implicit factor simulation method, a parameter space nonlinear association modeling system is established, thereby realizing the topological reconstruction of the probabilistic trajectory evolution pattern of the fire passage detection result temporal context semantic association coding vector. In this way, the accuracy of the fire passage occupancy warning level analysis result obtained by inputting the fire passage detection result temporal context semantic association coding vector into the fire passage occupancy warning level analysis engine based on the large language model is improved.

[0087] In summary, the fire passage occupancy warning level analysis method based on the large model according to the embodiment of the present application is explained, which uses the camera to monitor and collect the status of the fire passage in real time to obtain the time series of the fire passage status monitoring key frames, and then uses the visual large model to detect these fire passage status monitoring key frames to identify the fire passage status, object position and change in each key frame, so as to generate corresponding detection results. Finally, the fire passage occupancy warning level analysis is performed on these analyzed detection results using the large language model and prompt information to obtain the analysis results. Compared with the traditional single-frame target detection analysis results, this helps to improve the speed and reliability of fire emergency response, thereby reducing the safety hazards caused by fire passage blockage.

[0088] Figure 7 FIG. 1 is a schematic block diagram of a fire passage occupancy warning level analysis system based on a large model according to an embodiment of the present application. Figure 7As shown, the fire passage occupancy warning level analysis system 100 based on the large model includes: a fire passage state monitoring acquisition module 110, which is used to obtain the time series of fire passage state monitoring key frames collected by the camera; a fire passage state detection module 120, which is used to use the trained visual large model to detect each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frame to obtain a time series of detection results, and the detection results include fire passage state, object position and change; a fire passage occupancy warning level analysis module 130, which is used to add a prompt word at the end of the time series of the detection result, and then input it into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, and the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether there is occupation; b. If there is occupation, how long is the duration; c. Give a reasonable warning level according to the occupation duration and the type of occupied object".

[0089] In one embodiment, the trained large visual model is a RT-DETR model based on a transformer architecture.

[0090] Here, those skilled in the art can understand that the specific operations of the various modules and units in the fire passage occupancy warning level analysis system based on the large model have been described in the above reference. Figures 1 to 6 The description of the large model-based fire passage occupancy warning level analysis method has been introduced in detail, and therefore, its repeated description will be omitted.

[0091] An embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the methods in the above embodiments of the present application.

[0092] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer implements the methods in the above embodiments of the present application.

[0093] An embodiment of the present application also provides a chip, including a circuit, for executing the methods in the above embodiments of the present application.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] The prefixes such as "first" and "second" used in the embodiments of the present application are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers used to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0096] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] In the various embodiments of the present application, unless otherwise specified or logically conflicting, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fire passage occupancy warning level analysis method based on a large model, characterized in that: include: Obtain the time series of key frames of fire passage status monitoring collected by the camera; Using the trained visual big model, each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frames is detected to obtain a time series of detection results, wherein the detection results include the fire passage state, object position and change; After adding the prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, and the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether it is occupied; b. If occupancy exists, for what duration; c. Give a reasonable level of warning, depending on the duration of the occupancy and the type of object occupied".

2. The fire passage occupancy warning level analysis method based on a large model according to claim 1 is characterized in that: The trained visual model is a RT-DETR model based on the transformer architecture.

3. The fire passage occupancy warning level analysis method based on a large model according to claim 2 is characterized in that: After adding the prompt word at the end of the time series of the detection result, it is input into the fire passage occupancy warning level analysis engine based on the large language model to obtain the fire passage occupancy warning level analysis result, including: Performing semantic embedding coding on each detection result in the time series of the detection results to obtain a time series of semantic embedding coding vectors of the detection results; Performing context semantic association coding on the time series of the detection result semantic embedding coding vector to obtain a fire channel detection result time series context semantic association coding vector; The prompt word is added to the end of the temporal context semantic association encoding vector of the fire passage detection result.

4. The fire passage occupancy warning level analysis method based on a large model according to claim 3 is characterized in that: Performing context semantic association coding on the time series of the detection result semantic embedding coding vector to obtain the fire channel detection result time series context semantic association coding vector, including: Calculating the feature jump degree and message transfer space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector to obtain a time series of the detection result semantic feature jump degree and a time series of the detection result semantic feature message transfer space span; Based on the time series of the jump degree of the semantic features of the detection results and the time series of the spatial span of the semantic feature message transmission of the detection results, the time series of the semantic embedding coding vector of the detection results is significantly weighted and aggregated to obtain the temporal context semantic association coding vector of the fire escape detection result.

5. The fire passage occupancy warning level analysis method based on a large model according to claim 4 is characterized in that: Calculating the feature jump degree and message transfer space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector to obtain the time series of the detection result semantic feature jump degree and the time series of the detection result semantic feature message transfer space span, including: Calculating the feature jump degree of the semantic embedding coding vector of each detection result to obtain a time series of the semantic feature jump degree of the detection result; Calculating the number of message transmissions between each detection result semantic embedding coding vector and the last detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector to obtain a sequence of detection result semantic transmission times; Based on the sequence of the number of times the semantics of the detection results are transmitted, the message transmission space spans of the semantic embedding coding vectors of the respective detection results are calculated to obtain the time series of the message transmission space spans of the semantic features of the detection results.

6. The fire passage occupancy warning level analysis method based on a large model according to claim 5 is characterized in that: Based on the time series of the jump degree of the semantic feature of the detection result and the time series of the spatial span of the semantic feature message transmission of the detection result, the time series of the semantic embedding coding vector of the detection result is significantly weighted and aggregated to obtain the temporal context semantic association coding vector of the fire escape detection result, including: Based on the time series of the jump degree of the semantic feature of the detection result and the time series of the message transmission space span of the semantic feature of the detection result, calculating the message transmission significance weight of each detection result semantic embedding coding vector in the time series of the semantic embedding coding vector of the detection result to obtain the time series of the message transmission significance weight of the semantic feature of the detection result; The weighted sum of the time series of the semantic embedding coding vectors of the detection results is calculated based on the time series of the significant weights of the semantic feature messages of the detection results to obtain the temporal context semantic association coding vector of the fire escape detection result.

7. The fire passage occupancy warning level analysis method based on a large model according to claim 6 is characterized in that: Based on the time series of the jump degree of the semantic feature of the detection result and the time series of the message transmission space span of the semantic feature of the detection result, calculating the message transmission significance weight of each detection result semantic embedding coding vector in the time series of the semantic embedding coding vector of the detection result to obtain the time series of the message transmission significance weight of the semantic feature of the detection result, including: Based on the feature jump degree and message transfer space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector, calculating the message transfer significance factor of each detection result semantic embedding coding vector to obtain the time series of the detection result semantic feature message transfer significance factor; The time series of the semantic feature message transmission significance factor of the detection result is gated and screened to obtain the time series of the semantic feature message transmission significance weight of the detection result.

8. The fire passage occupancy warning level analysis method based on a large model according to claim 7 is characterized in that: Based on the feature jump degree and message transfer space span of each detection result semantic embedding coding vector in the time series of the detection result semantic embedding coding vector, the message transfer significance factor of each detection result semantic embedding coding vector is calculated to obtain the time series of the detection result semantic feature message transfer significance factor, including: Extracting the detection result semantic feature jump degree and the detection result semantic feature message transfer space span corresponding to the first detection result semantic embedding coding vector from the time series of the detection result semantic feature jump degree and the detection result semantic feature message transfer space span; Calculating the square of the jump degree of the semantic feature of the detection result corresponding to the first detection result semantic embedding coding vector to obtain a square modulation representation value of the jump degree of the semantic feature of the detection result; The detection result semantic feature message transmission space span corresponding to the first detection result semantic embedding coding vector is used as the modulation coefficient to perform weighted modulation optimization representation on the detection result semantic feature jump degree square modulation representation value to obtain the detection result semantic feature message transmission significance factor.

9. A fire passage occupancy warning level analysis system based on a large model, characterized in that: include: The fire passage status monitoring and acquisition module is used to obtain the time series of the fire passage status monitoring key frames collected by the camera; A fire passage state detection module, used to detect each fire passage state monitoring key frame in the time series of the fire passage state monitoring key frames using the trained visual large model to obtain a time series of detection results, wherein the detection results include the fire passage state, object position and change; The fire passage occupation warning level analysis module is used to add a prompt word at the end of the time series of the detection result, and then input it into the fire passage occupation warning level analysis engine based on the large language model to obtain the fire passage occupation warning level analysis result, wherein the prompt word is "Please judge whether the fire passage is occupied according to the above description, and give the following information: a. Whether it is occupied; b. If occupancy exists, for what duration; c. Give a reasonable level of warning, depending on the duration of the occupancy and the type of object occupied".

10. The fire passage occupancy warning level analysis system based on a large model according to claim 9 is characterized in that: The trained visual model is a RT-DETR model based on the transformer architecture.

Images