Fire fighting access occupation monitoring method and system based on large model

Abstract

The invention discloses a fire fighting access occupation monitoring method and system based on a large model, and the method comprises the steps: periodically obtaining at least one frame of monitoring image of a target fire fighting access, inputting the monitoring image into a multi-modal large model for preliminary target recognition, and guiding the multi-modal large model to output a first to-be-detected label set of the monitoring image through a preset prompt word template; updating a second to-be-detected label set with a fixed length by using the first to-be-detected label set, when the second to-be-detected label set changes, acquiring a video stream in the current period by the AI engine, decoding the video stream to obtain a plurality of frames of video stream images, and issuing the video stream images to the open target detection model; the open target detection model takes the capability parameter as an input parameter, completes target object detection of the video stream image, and outputs a target object information list; performing static tracking on a target object in the target fire fighting access region of interest, and outputting a tracked target object information list; and reporting when the tracked target object meets a preset alarm condition.

Description

Technical Field

[0001] The invention relates to an image processing technology for safety monitoring, and in particular to a fire passage occupancy monitoring method and system based on a large model. Background Art

[0002] As cities continue to expand and population density continues to increase, fire safety management faces increasingly severe challenges. Fire passages are important passages for evacuating people and for fire trucks to pass through when a fire occurs. Their smooth flow is directly related to the safety of people's lives and property. However, due to problems such as parking difficulties, it is not uncommon for fire passages to be occupied by motor vehicles, non-motor vehicles and other items, which seriously threatens public safety.

[0003] At present, the following methods are mainly used to monitor the occupancy of fire passages: First, monitoring is carried out through manual inspections, which has high labor costs and cannot guarantee 24-hour uninterrupted monitoring; second, monitoring is carried out using traditional video surveillance and computer vision technology. Although this method can realize automatic monitoring, it has the following shortcomings: 1. The detection target types are fixed and difficult to expand. Traditional video analysis methods usually use specific open target detection models, which can only identify pre-defined fixed types of targets, such as vehicles, non-motor vehicles, etc. When new types of occupied objects appear, the model needs to be retrained, which has poor adaptability.

[0004] 2. Insufficient detection accuracy and high false alarm rate. Traditional computer vision models are prone to missed detection and false detection in complex scenarios, especially in conditions of insufficient light and weather changes. In addition, due to the lack of continuous tracking and analysis capabilities of the target object, repeated alarms or false alarms often occur.

[0005] 3. Poor system scalability. Existing systems often tightly couple open target detection models with business logic. When new monitoring objects need to be added or monitoring strategies need to be adjusted, the entire system needs to be remodeled on a large scale, resulting in high system maintenance costs.

[0006] Therefore, there is an urgent need for a fire passage occupancy monitoring method and system with higher intelligence, wider detection range and higher accuracy. With the development of artificial intelligence technology, especially the breakthrough progress made by multimodal large models in the field of visual understanding, new technical means have been provided to solve the above technical problems. Summary of the invention

[0007] In order to address the deficiencies in the above-mentioned prior art, the present invention aims to utilize the advantages of multimodal large models and deep learning technology to achieve more intelligent and efficient fire passage occupancy monitoring.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes: The fire passage occupancy monitoring method based on a large model includes the following steps: S1. Periodically acquire at least one frame of monitoring image of the target fire passage, input it into the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected for the monitoring image; the first set of labels to be detected includes Chinese and English labels; S2. Use the first set of tags to be detected to update the second set of tags to be detected with a fixed length in chronological order. When the second set of tags to be detected changes, the AI ​​engine obtains the video stream in the current period and decodes it to obtain several frames of video stream images, and sends the video stream images and capability parameters to the open target detection model; S3. The open target detection model takes the capability parameter as an input parameter, completes the target object detection of the video stream image, and outputs a target object information list; S4. Static tracking of the target object in the target fire channel area of ​​interest is performed according to the target object information list, and the tracking target object information list is output; S5. When the tracking target object in the tracking target object information list meets the preset alarm condition, a fire passage occupation alarm event is reported.

[0009] Preferably, the capability parameters include: camera ID, video stream address, video stream format, area of ​​interest, AI capability name, tracking time threshold, and interval alarm period.

[0010] Preferably, the target object information list includes: target location, target category, target confidence, and image timestamp.

[0011] Preferably, the method for performing static tracking in step S4 includes: Filter the target objects in the target object information list that do not belong to the target fire channel interest area; Merge target objects whose intersection-and-union ratio is greater than a preset intersection-and-union threshold among the remaining target objects; If the remaining target objects do not appear at the target location within the preset time, it is determined that the target object is lost.

[0012] Preferably, the tracking target object information list includes: tracking ID, tracking status, target duration, target position, target category, target threshold, and image timestamp.

[0013] Preferably, the alarm condition includes: The duration of the tracking target object in the tracking target object information list at the target location is greater than the preset time; The number of reports of the tracking target objects in the tracking target object information list within the interval alarm period shall not exceed 1 time.

[0014] Preferably, if the interval alarm period is 0, the tracking target object in the tracking target object information list is reported only once in the whole monitoring period.

[0015] The present invention also provides a large model-based fire passage occupancy monitoring system, which is used to implement the above-mentioned large model-based fire passage occupancy monitoring method.

[0016] Beneficial Effects 1. The present invention adopts a technical solution that combines a multimodal large model with an open target detection model, which significantly improves the intelligence level and detection capability of the system. The multimodal large model can automatically understand and analyze the monitoring scene through a preset prompt word template, and dynamically output a set of labels to be detected. This method breaks through the limitation that traditional systems can only identify fixed types of targets, and greatly expands the detection range of the system. For example, when the fire escape is occupied by unconventional items such as temporarily stacked construction materials, express parcels, etc., the system can still accurately identify and issue timely alarms.

[0017] 2. The present invention innovatively introduces a static tracking mechanism, which effectively solves the problems of repeated alarms and false alarms in traditional systems. By continuously tracking and analyzing the target objects in the target fire passage area of ​​interest, the system can accurately determine the duration of the occupancy behavior and intelligently decide whether to trigger an alarm based on the preset alarm conditions. This method greatly improves the detection accuracy of the system and reduces the occurrence of false alarms.

[0018] 3. The present invention adopts a modular and service-oriented system architecture design, which significantly improves the scalability and maintainability of the system. By decoupling the multimodal large model, open target detection model and static tracking module, the system can flexibly expand functions and optimize upgrades. At the same time, the standardized parameter transmission interface design enables the system to easily access new AI capabilities, providing a good foundation for future functional expansion.

[0019] 4. The present invention has significant social and economic value in practical applications. The system can realize all-weather, automated monitoring of fire passages, greatly reducing the cost of manual inspections. At the same time, the system's high accuracy and real-time alarm capabilities effectively prevent illegal occupation of fire passages, improve the level of urban fire safety management, and make important contributions to protecting the lives and property of the people. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a flow chart of a fire passage occupancy monitoring method based on a large model provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0022] Embodiment 1 like Figure 1 As shown, this embodiment discloses a fire passage occupancy monitoring method based on a large model, comprising the steps of: S1. Periodically acquire at least one frame of monitoring image of the target fire passage, input it into the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected of the monitoring image; the first set of labels to be detected includes Chinese and English labels.

[0023] In this step, at least one frame of monitoring image of the target fire passage is first acquired through the camera at a preset time period (e.g., every 1 minute). This periodic sampling method can not only ensure the real-time monitoring of the fire passage status, but also avoid the waste of system resources caused by processing redundant image data.

[0024] After acquiring the monitoring image, it is input into the multimodal large model for preliminary target recognition. The multimodal large model can be any of the commonly used large models in the field. Preferably, the InternVL2 private deployment solution can be used, which adopts a dynamic resolution strategy to divide the image into blocks of different sizes, and adjusts the aspect ratio and resolution of the input image to better perceive the target object in the image. In order to guide the model to output standardized detection labels, the system uses a preset prompt word template.

[0025] Specifically, the structure of the prompt template can be divided into two roles: system / user. The specific examples are as follows: system: "You are a visual assistant that can identify the target types on and around the fire escape in the image based on the image information input by the user, and provide a set of Chinese and English labels.\nThe target type name should be as simple as possible; for example, 'car' should be represented as 'car', 'motorcycle' as 'motorcycle', 'bicycle' as 'bicycle', and so on.\nDo not include special characters in the returned results, such as: \\n,..., etc. Multiple targets are separated by commas.\nPlease refer to the returned answer examples: Car, Motorcycle, Bicycle, Bicycle".

[0026] user: "Please identify the objects on and around the fire escape in the image and return a set of Chinese and English labels. Multiple objects should be separated by commas and no special characters should appear."

[0027] The multimodal large model will analyze the input monitoring image based on this prompt word template and output the first set of labels to be detected. This Chinese-English label design has the following advantages: 1. Improve the versatility of the system and support applications in different language environments; 2. Facilitate standardized docking with downstream open target detection models; 3. It is conducive to system maintenance and data analysis; It should be noted that the multimodal large model mainly plays the role of scene understanding and label generation in this step, providing input parameters for subsequent precise target detection. This design fully utilizes the advantages of the multimodal large model in scene understanding, while avoiding the performance overhead that may be caused by directly using the multimodal large model for target detection.

[0028] S2. Use the first set of labels to be detected to update the second set of labels to be detected with a fixed length in chronological order. When the second set of labels to be detected changes, the AI ​​engine obtains the video stream in the current period and decodes it to obtain several frames of video stream images, and sends the video stream images and capability parameters to the open target detection model.

[0029] This step implements a dynamically updated tag management mechanism and triggers the target detection process based on tag changes. This step can be divided into three main parts for detailed description: The first is the dynamic update mechanism of the tag set. The system maintains a second tag set to be detected with a fixed length (for example, 20), which is updated in a first-in-first-out (FIFO) manner. When a new first tag set to be detected is obtained, the system will add the tags in it to the second tag set to be detected in chronological order. If the length of the second tag set to be detected exceeds 20, the earliest added tag will be automatically removed. This design ensures that the system always uses the latest detection targets, while optimizing the use of system resources by limiting the number of tags.

[0030] The second is change detection and video stream processing. The system monitors changes in the second set of labels to be detected. When a change is detected (such as adding a label, deleting a label, or modifying a label), the AI ​​engine immediately starts processing the video stream. Specifically, the AI ​​engine obtains the video stream data within the current time period and converts the video stream into a series of image frames through a decoder. This decoding process takes into account parameters such as video resolution and frame rate to ensure that the decoded image quality meets the requirements of target detection. It should be understood that the AI ​​engine is an independent deep learning inference service system that is responsible for performing actual visual recognition tasks. This service system has the following characteristics: 1. Service architecture design: The AI ​​engine adopts a service-oriented architecture, which provides external services through interfaces. This design allows the main system and the AI ​​inference engine to be decoupled, facilitating independent maintenance and upgrades.

[0031] 2. Parameter update mechanism: When the label set changes, the system knows which detection capabilities need to be updated because it implements a mapping relationship management mechanism: the system maintains a corresponding relationship table between labels and detection capabilities, and each label is associated with a specific detection capability.

[0032] 3. When the label changes, the system can determine which detection capabilities need to be updated by querying this mapping relationship.

[0033] It should be understood that the AI ​​engine can only obtain the video stream within the target period, or it can continuously obtain the video stream, and then pass the video stream within the specified period to the open target detection model according to the period range. The specific acquisition method can be designed by technicians in this field according to actual needs, and the present invention does not make further requirements.

[0034] In some preferred embodiments, an inference engine based on a deep learning framework, such as TensorRT and Pytorch, can be used to implement the AI ​​engine in the present invention. This part is not the focus of the present invention, so it will not be repeated here. Those skilled in the art can choose from the existing technology according to actual needs.

[0035] The last step is parameter delivery. The decoded video stream image is delivered to the open target detection model together with the complete capability parameters. The capability parameters are a set of configuration information, which defines the working mode and boundary conditions of the open target detection model when performing detection tasks. In the present invention, the capability parameters include the following core contents: basic configuration: including camera ID, video stream address and format (RTSP), etc. These parameters tell the model where to get input data; detection range: the model is limited to the specific area where the fire escape is located through the area of ​​interest parameter; target type: the object category to be detected is specified through the second set of labels to be detected; time control: including tracking duration threshold and interval alarm cycle, which are used to control the time dimension requirements of detection.

[0036] The relationship between these capability parameters and the open target detection model is a "configuration and execution" relationship. The open target detection model is a deep learning model that actually performs the object recognition task, and the capability parameters tell the model what to detect, where to detect, and how to detect. It should be understood that the open target detection model described in the present invention is a type of deep learning model specifically used for object positioning and recognition in images. It not only includes the basic functions of traditional target detection, but also breaks through the limitation that traditional models can only recognize fixed categories. It is represented by advanced models such as DINO (DETR with Improved deNoising anchOr boxes). The openness of this type of model makes it more adaptable and practical in practical applications within the scope of the present invention, and can better meet the target detection needs in various scenarios. Those skilled in the art can make reasonable choices in the prior art based on the hardware resource conditions on site and the requirements of real-time and detection accuracy, and the present invention does not make further restrictions.

[0037] In some preferred embodiments, specific examples of capability parameters are given. These capability parameters are a parameter set including multiple configuration items, specifically including: Camera ID: used to uniquely identify the video input source; Video stream address: specifies the data source location; Video stream format: specify the video format, such as RTSP format; Area of ​​interest: limit the detection range in the fire passage; AI Capability Name: Set to FireLaneObstruction; Tracking duration threshold: Set the time requirement for target tracking; Interval alarm period: controls the triggering frequency of the alarm.

[0038] S3. The open target detection model takes the capability parameter as an input parameter, completes the target object detection of the video stream image, and outputs a target object information list.

[0039] The open object detection model will parse the received capability parameters. These parameters affect many aspects of the detection process, such as the camera ID and video stream address determine the data source, and the region of interest parameter defines the image range that needs to be focused on. Together, these parameters build a complete context for a detection task. The target object information list contains detailed information for each detected target object, preferably including target location, target category, target confidence, and image timestamp.

[0040] S4. Static tracking is performed on the target object in the target fire passage area of ​​interest according to the target object information list, and the tracked target object information list is output.

[0041] The static tracking is a method for analyzing the persistence of an object based on the target detection results. Different from the traditional video target tracking algorithms (such as KCF, SORT, etc.), static tracking focuses on the persistence of the target object in a specific area without the need to accurately track the target's motion trajectory. In some preferred embodiments, specific methods for implementing static tracking are given, including: First, the target object information list is filtered by region. Specifically, it is checked whether the position coordinates of each target object are within the predefined fire channel interest region.

[0042] Next, target merging is performed. When multiple similar targets are detected (for example, the same vehicle is detected repeatedly), their intersection over union (IOU) ratios are calculated, and targets whose IOUs exceed a preset threshold (for example, 0.5) are merged into the same tracking target.

[0043] Then, a state record is maintained for each target that passes the screening, and a tracking target object information list is generated. This list contains richer information. In some preferred embodiments, the tracking target object information list includes: tracking ID, tracking state, target duration, target location, target category, target threshold, and image timestamp.

[0044] Finally, the target is lost. If there is no new detection result for a tracked target within a preset time (for example, 5 seconds), it will be marked as "lost".

[0045] The above static tracking mechanism has low computational overhead and does not require complex motion prediction; it has good fault tolerance for short-term occlusion of the target; it can accurately count the target's continuous stay time; and it avoids repeated counting caused by short-term movement of the target.

[0046] S5. When the tracking target object in the tracking target object information list meets the preset alarm condition, the fire passage occupation alarm event is reported. The alarm mechanism is the final decision-making link of the system. It determines whether to trigger an alarm by evaluating the state of the tracking target. Its design can be carried out by those skilled in the art according to the actual needs of the site. In some preferred embodiments, a method is provided to realize the reporting of the fire passage occupation alarm event by checking whether each tracking target meets two key alarm conditions, namely, duration and alarm frequency control, which specifically includes: Duration judgment: Check whether the target's stay time in the fire channel area exceeds the preset tracking time threshold.

[0047] Alarm frequency control: Avoid repeated alarms by setting the interval alarm cycle parameters to ensure that the same target only triggers an alarm once within a specified time period.

[0048] When a tracking target meets both conditions, the system will generate an alarm event. The data structure of the alarm event contains complete on-site information.

[0049] The design of this alarm mechanism takes into account the needs of multiple practical application scenarios: 1. By judging the duration, false alarms are avoided for objects that pass by temporarily. For example, a vehicle temporarily stops to load or unload goods, which will not trigger an alarm.

[0050] 2. Prevent the system from generating junk alarms by controlling the alarm frequency. For example, even if a car is parked in the fire passage continuously, the system will not frequently report the alarm repeatedly.

[0051] 3. Complete alarm information facilitates subsequent manual processing and data analysis. On-site managers can quickly locate problems based on alarm information, while data analysts can use these data for statistical analysis and trend research.

[0052] In the actual scenario of fire passage monitoring, some long-term occupancy situations may be encountered. For example, a car owner may keep the vehicle parked in the fire passage for several days or even weeks. In this case, if the alarm is continuously sent according to the conventional interval alarm cycle (for example, every 4 hours), a large number of repeated alarm messages will be generated. These repeated alarms will not only occupy system resources, but more importantly, they will cause information redundancy and reduced processing efficiency. Therefore, in some preferred embodiments, a "one-time alarm" mechanism is provided by setting the interval alarm cycle to a special value of 0. This means that for the same occupied object, the system will only send an alarm once when it first detects that it meets the alarm condition, and then no repeated alarms will be sent.

[0053] In some other preferred embodiments, the present invention also provides a large model-based fire passage occupancy monitoring system, which is used for the above-mentioned large model-based fire passage occupancy monitoring method.

[0054] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A fire passage occupancy monitoring method based on a large model, characterized in that: Includes steps: S1. Periodically acquire at least one frame of monitoring image of the target fire passage, input it into the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected for the monitoring image; the first set of labels to be detected includes Chinese and English labels; S2. Use the first set of tags to be detected to update the second set of tags to be detected with a fixed length in chronological order. When the second set of tags to be detected changes, the AI ​​engine obtains the video stream in the current period and decodes it to obtain several frames of video stream images, and sends the video stream images and capability parameters to the open target detection model; S3. The open target detection model takes the capability parameter as an input parameter, completes the target object detection of the video stream image, and outputs a target object information list; S4. Static tracking of the target object in the target fire channel area of ​​interest is performed according to the target object information list, and the tracking target object information list is output; S5. When the tracking target object in the tracking target object information list meets the preset alarm condition, a fire passage occupation alarm event is reported.

2. The fire passage occupancy monitoring method based on a large model as claimed in claim 1 is characterized in that: The capability parameters include: camera ID, video stream address, video stream format, area of ​​interest, AI capability name, tracking time threshold, and interval alarm period.

3. The fire passage occupancy monitoring method based on a large model as claimed in claim 1 is characterized in that: The target object information list includes: target location, target category, target confidence, and image timestamp.

4. The fire passage occupancy monitoring method based on a large model as claimed in claim 3 is characterized in that: The method for performing static tracking in step S4 includes: Filter the target objects in the target object information list that do not belong to the target fire channel interest area; Merge target objects whose intersection-and-union ratio is greater than a preset intersection-and-union threshold among the remaining target objects; If the remaining target objects do not appear at the target location within the preset time, it is determined that the target object is lost.

5. The fire passage occupancy monitoring method based on a large model as claimed in claim 1 is characterized in that: The tracking target object information list includes: tracking ID, tracking status, target duration, target position, target category, target threshold, and image timestamp.

6. The fire passage occupancy monitoring method based on a large model as claimed in claim 2 is characterized in that: The alarm conditions include: The duration of the tracking target object in the tracking target object information list at the target location is greater than the preset time; The number of reports of the tracking target objects in the tracking target object information list within the interval alarm period shall not exceed 1 time.

7. The fire passage occupancy monitoring method based on a large model as claimed in claim 6 is characterized in that: If the interval alarm period is 0, the tracking target object in the tracking target object information list is reported only once in the whole monitoring period.

8. The fire passage occupancy monitoring system based on the large model is characterized by: The system is used to implement the method according to any one of claims 1 to 7.

Images