Interaction system for monitoring occupation of fire fighting access

Abstract

The invention discloses an interactive system for monitoring occupation of a fire fighting access. The interactive system comprises an image acquisition device, a display device and a server, the server comprises a label acquisition module used for acquiring a monitoring image, inputting the monitoring image into a multi-mode large model to carry out preliminary target identification, outputting a first to-be-detected label set, updating a second to-be-detected label set with a fixed length, and notifying a built-in AI engine to start when the second to-be-detected label set changes; the built-in AI engine is used for acquiring a video stream in a current period and issuing a video stream image and a capability parameter to the open target detection model; the open target detection model is used for taking the capability parameters as input parameters, completing target object detection of the video stream image and outputting a target object information list; the static tracking module is used for statically tracking a target object according to a target object information list; and the alarm module is used for reporting a fire fighting access occupation alarm event to a user through a display device when a preset alarm condition is met.

Description

Technical Field

[0001] The present invention relates to the field of safety protection and detection technology, and in particular to an interactive system for monitoring fire passage occupancy. Background Art

[0002] Fire passages are important passages in buildings for emergency evacuation and fire rescue. Keeping fire passages unobstructed is of great significance to ensuring the safety of personnel. However, in actual applications, fire passages are often occupied at will, such as illegally parking vehicles and piling up debris, which seriously endangers the fire safety of buildings.

[0003] At present, the monitoring and interaction of fire passage occupancy mainly adopt the following methods: 1. Manual inspection method. This method requires security personnel to regularly inspect the fire passages and report the occupancy status to the management center through simple interactive methods such as intercoms or telephones. However, this method has problems such as high labor costs, low monitoring efficiency, untimely information transmission, and inability to guarantee all-weather monitoring.

[0004] 2. Traditional video surveillance. By installing ordinary surveillance cameras in the fire passage, the on-duty personnel can observe whether there is occupancy in real time through the display screen of the monitoring center, and exchange information with on-site personnel through the intercom system. Although this method realizes basic visual interaction, the interaction method is single and it is easy to miss the occupancy due to the distraction of personnel.

[0005] 3. Automatic monitoring based on simple image recognition. This method uses computer vision technology to analyze surveillance videos to detect whether the fire escape is occupied. However, the existing technology has the following shortcomings: The detection accuracy is low, and it is easy to have missed reports and false reports; it cannot accurately identify a variety of occupancy scenarios; the interactive interface is simple and lacks intuitive interactive functions such as real-time video stream display and target labeling; the alarm information display method is single and cannot meet the interactive needs in different scenarios; there is a lack of effective information feedback mechanism between the system and the user, which affects the disposal efficiency.

[0006] With the development of artificial intelligence technology, especially the progress of multimodal large models and target detection technology, new technical means have been provided to solve the above problems. However, there is no fire passage occupancy monitoring solution that combines multimodal large models with open target detection models and provides rich interactive functions. Summary of the invention

[0007] In order to solve the above technical problems, the present invention aims to provide an intelligent monitoring system with good human-computer interaction experience, which can realize real-time visual display, intelligent alarm and interactive disposal of occupancy behavior.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes: Interactive system for fire escape occupancy monitoring, including: An image acquisition device for acquiring a monitoring image of a target fire passage; A display device for interacting with a user; A server connected to the image acquisition device and the display device by signals respectively; the server is configured to: determine whether the fire passage is currently occupied based on the monitoring image of the target fire passage acquired by the image acquisition device, and if so, warn the user through the display device; The server includes a label acquisition module, a built-in AI engine, an open target detection model and an alarm module; The label acquisition module is configured to be signal-connected to the multimodal large model and the image acquisition device respectively, and is used to periodically acquire at least one frame of monitoring image of the target fire passage, input the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected of the monitoring image; use the first set of labels to be detected to update a second set of labels to be detected with a fixed length in chronological order, and when the second set of labels to be detected changes, notify the built-in AI engine to start; the first set of labels to be detected includes Chinese and English labels; The built-in AI engine is connected to the image acquisition device and the open target detection model signal, and is used to acquire the video stream in the current period and decode to obtain several frames of video stream images, and send the video stream images and capability parameters to the open target detection model; The open target detection model is connected to the alarm module signal, and is used to use the capability parameter as an input parameter to complete the target object detection of the video stream image and output a target object information list; The alarm module is connected to the display device signal, and is used to statically track the target object in the target fire passage interest area according to the target object information list, and output the tracking target object information list; when the tracking target object in the tracking target object information list meets the preset alarm condition, the fire passage occupancy alarm event is reported to the user through the display device.

[0009] Preferably, the alarm module includes a target tracking unit and an event reporting unit; The target tracking unit is used to statically track the target object in the target fire passage interest area according to the target object information list, and output the tracked target object information list; The event reporting unit is connected to the target tracking unit and the display device by signal respectively, and is used to determine whether the tracking target object in the tracking target object information list meets the preset alarm condition, and if so, reports the alarm information to the display device.

[0010] Preferably, the alarm module includes a video fusion unit and a video display unit; The video fusion unit is respectively connected to the built-in AI engine and the open target detection model signal, and is used to obtain the video stream image and the target object information list, and output the real-time video stream and the target object identification frame to the video display unit; The video display unit is connected to the display device signal, and is used to obtain the real-time video stream and the target object recognition frame, and send them to the display device after rendering for display to the user.

[0011] Preferably, the capability parameters include: camera ID, video stream address, video stream format, area of ​​interest, AI capability name, tracking time threshold, and interval alarm period.

[0012] Preferably, the target object information list includes: target position, target category, target confidence, image timestamp; the tracking target object information list includes: tracking ID, tracking status, target duration, target position, target category, target threshold, image timestamp.

[0013] Preferably, the method for the target tracking unit to perform static tracking includes: Filter the target objects in the target object information list that do not belong to the target fire channel interest area; Merge target objects whose intersection-and-union ratio is greater than a preset intersection-and-union threshold among the remaining target objects; If the remaining target objects do not appear at the target location within the preset time, it is determined that the target object is lost.

[0014] Preferably, the alarm condition includes: The duration of the tracking target object in the tracking target object information list at the target location is greater than the preset time; The number of reports of the tracking target objects in the tracking target object information list within the interval alarm period shall not exceed 1 time.

[0015] Preferably, if the interval alarm period is 0, the tracking target object in the tracking target object information list is reported only once in the whole monitoring period.

[0016] Preferably, the method for aligning the target object identification frame with the target object in the real-time video stream when the video display unit completes rendering includes: Obtain a target object information list in a real-time video stream, query the image timestamp, match and obtain the target object information list and the real-time video stream data packet with the same timestamp, render and send to a display device for display to the user.

[0017] Preferably, if the frame rate of the target object information list with the same timestamp is lower than the frame rate of the real-time video stream, the target information list data and the real-time video stream data packet within the preset time threshold are cached, and the cached target information list data is supplemented with the frame rate using linear interpolation.

[0018] Beneficial Effects 1. Improved the intelligence level and accuracy of fire passage occupancy monitoring: This application combines a multimodal large model with an open object detection model for all things to build a dual recognition mechanism: first, the multimodal large model is used to perform preliminary recognition of the monitoring image and output a set of labels to be detected; then, a professional object detection model is used for accurate recognition. This innovative technical solution significantly improves the detection accuracy of the system and effectively reduces the probability of false alarms and missed alarms.

[0019] 2. Achieved more flexible and efficient target tracking: By setting up the label acquisition module, the system can periodically acquire monitoring images and dynamically update the set of labels to be detected, so that the target detection model can adapt to scene changes in a timely manner. At the same time, through the static tracking mechanism and the preset intersection-over-union ratio threshold, the system can effectively identify and track continuously occupied target objects, avoid repeated alarms, and improve system operation efficiency.

[0020] 3. Provides rich interactive functions and intuitive visual interface: The system of this application integrates the real-time video stream with the target object recognition frame through the video fusion unit and the video display unit, so that users can intuitively understand the monitoring results. The system supports real-time display of detection and tracking status, and pushes alarm information to users in a timely manner through the display device, which greatly improves the interactive experience and use effect of the system.

[0021] 4. Avoid the floating frame phenomenon: The system of this application uses the alignment of the target object information list and the real-time video stream data packet timestamp, and fills the frame rate through linear interpolation when the frame rate does not match, so as to eliminate the phenomenon of the target frame and the picture being out of sync when displayed to the user, thereby improving the user interaction experience. 5. Reduce the cost of system maintenance and use: The system of this application uses an automated monitoring and alarm mechanism, which does not require continuous manual supervision, significantly reducing labor costs. The system's intelligent alarm mechanism can accurately locate occupied positions and improve the efficiency of problem handling. Through an intuitive interactive interface, user training costs are reduced and the system's ease of use is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a schematic diagram of the structure of an interactive system for fire passage occupancy monitoring provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0024] like Figure 1 As shown, this embodiment provides an interactive system for fire passage occupancy monitoring, including: An image acquisition device for acquiring monitoring images of a target fire passage.

[0025] A display device for interacting with a user. In this embodiment, the display device is an important component for realizing information interaction between the system and the user, and can be a device with display function such as a monitoring display screen in the fire control room, a mobile terminal device or a computer display. The display device provides users with comprehensive monitoring information and an interactive interface by being divided into functional areas such as real-time monitoring, target identification display, alarm information display, system status display and parameter configuration. Among them, the real-time monitoring area displays the processed video stream, including the identification frame and tracking trajectory of the target object; the alarm information area displays the detailed information of the alarm event in real time, and attracts the user's attention through sound and light prompts; the system status area feedbacks the working status of each functional module, such as the connection status of the image acquisition device, the running status of the model, etc. The display device also supports interactive functions such as adjusting monitoring parameters, viewing historical alarm records, confirming and processing alarm events through touch operations, thereby realizing efficient information interaction between the system and the user. This intuitive and convenient interactive method enables users to timely grasp the occupancy of the fire passage, significantly improving the availability and management efficiency of the system. It should be noted that those skilled in the art can appropriately adjust and expand the functions and interaction methods of the display device according to actual needs, and these changes should fall within the scope of protection of this application.

[0026] A server connected to the image acquisition device and the display device by signal respectively; the server is configured to: determine whether the fire passage is currently occupied according to the monitoring image of the target fire passage acquired by the image acquisition device, and if so, warn the user through the display device. In this embodiment, the server serves as the core processing unit of the system, and establishes a signal connection with the image acquisition device and the display device through a network or other communication methods. The server receives the monitoring images of the target fire passage from the image acquisition device, and these images can be single-frame images or continuous video streams. The server performs intelligent analysis and processing on the received monitoring images, and determines whether the fire passage is occupied through the built-in human algorithm, such as whether there are vehicles parked illegally, whether there are sundries piled up, etc. When the server detects that the fire passage is occupied, it will immediately trigger the alarm mechanism and send an alarm message to the display device through the signal connection channel with the display device. The alarm message may include key information such as the occupied location, the type of occupied object, and the time of occurrence. After receiving this information, the display device will promptly notify the user in a preset manner (such as a screen prompt, a sound reminder, etc.), so that the user can quickly understand the abnormal situation of the fire passage and take corresponding treatment measures.

[0027] The server includes a label acquisition module, a built-in AI engine, an open target detection model and an alarm module.

[0028] The label acquisition module is configured to be signal-connected to the multimodal large model and the image acquisition device respectively, and is used to periodically acquire at least one frame of monitoring image of the target fire passage, input the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected of the monitoring image; use the first set of labels to be detected to update a second set of labels to be detected with a fixed length in chronological order, and when the second set of labels to be detected changes, notify the built-in AI engine to start; the first set of labels to be detected includes Chinese and English labels.

[0029] The image acquisition device acquires at least one frame of monitoring image of the target fire passage at a preset time period (e.g., 1 minute interval). This periodic sampling method can ensure real-time monitoring of the fire passage status and avoid waste of system resources caused by processing redundant image data.

[0030] After acquiring the monitoring image, it is input into the multimodal large model for preliminary target recognition. The multimodal large model can be any of the commonly used large models in the field. Preferably, the InternVL2 private deployment solution can be adopted, which supports a dynamic resolution strategy, divides the image into blocks of different sizes, and adjusts based on the aspect ratio and resolution of the input image, so as to better perceive the target object in the image. In order to guide the model to output standardized detection labels, the system uses a preset prompt word template.

[0031] Specifically, the structure of the prompt template can be divided into two roles: system / user. The specific examples are as follows: system: "You are a visual assistant that can identify the target types on and around the fire escape in the image based on the image information input by the user, and provide a set of Chinese and English labels.\nThe target type name should be as simple as possible; for example, 'car' should be represented as 'car', 'motorcycle' as 'motorcycle', 'bicycle' as 'bicycle', and so on.\nDo not include special characters in the returned results, such as: \\n,..., etc. Multiple targets are separated by commas.\nPlease refer to the returned answer examples: Car, Motorcycle, Bicycle, Bicycle".

[0032] user: "Please identify the objects on and around the fire escape in the image and return a set of Chinese and English labels. Multiple objects should be separated by commas and no special characters should appear."

[0033] The multimodal large model will analyze the input monitoring image based on this prompt word template and output the first set of labels to be detected. This Chinese-English label design has the following advantages: 1. Improve the versatility of the system and support applications in different language environments; 2. Facilitate standardized docking with downstream open target detection models; 3. It is conducive to system maintenance and data analysis; It should be noted that the multimodal large model mainly plays the role of scene understanding and label generation in this step, providing input parameters for subsequent precise target detection. This design fully utilizes the advantages of the multimodal large model in scene understanding, while avoiding the performance overhead that may be caused by directly using the multimodal large model for target detection.

[0034] Furthermore, the tag acquisition module also implements a dynamic update mechanism for the tag set. Specifically, the system maintains a second tag set to be detected with a fixed length (for example, 20), which is updated in a first-in-first-out (FIFO) manner. When a new first tag set to be detected is obtained, the system will add the tags in it to the second tag set to be detected in chronological order. If the length of the second tag set to be detected exceeds 20, the earliest added tag will be automatically removed. This design ensures that the system always uses the latest detection targets, while optimizing the use of system resources by limiting the number of tags.

[0035] The second is change detection and video stream processing. The system monitors changes in the second set of labels to be detected. When a change is detected (such as adding a new label, deleting a label, or modifying a label), the built-in AI engine will be notified to start and start processing the video stream immediately. Specifically, the AI ​​engine obtains the video stream data within the current time period and converts the video stream into a series of image frames through a decoder. This decoding process takes into account parameters such as video resolution and frame rate to ensure that the decoded image quality meets the requirements of target detection. It should be understood that the AI ​​engine is an independent deep learning inference service system that is responsible for performing actual visual recognition tasks. This service system has the following characteristics: 1. Service architecture design: The AI ​​engine adopts a service-oriented architecture, which provides external services through interfaces. This design allows the main system and the AI ​​inference engine to be decoupled, facilitating independent maintenance and upgrades.

[0036] 2. Parameter update mechanism: When the label set changes, the system knows which detection capabilities need to be updated because it implements a mapping relationship management mechanism: the system maintains a corresponding relationship table between labels and detection capabilities, and each label is associated with a specific detection capability.

[0037] 3. When the label changes, the system can determine which detection capabilities need to be updated by querying this mapping relationship.

[0038] It should be understood that the AI ​​engine can only obtain the video stream within the target period, or it can continuously obtain the video stream, and then pass the video stream within the specified period to the open target detection model according to the period range. The specific acquisition method can be designed by technical personnel in this field according to actual needs, and the present invention does not make further requirements.

[0039] In some preferred embodiments, an inference engine based on a deep learning framework, such as TensorRT and Pytorch, can be used to implement the AI ​​engine in the present invention. This part is not the focus of the present invention, so it will not be repeated here. Those skilled in the art can choose from the existing technology according to actual needs.

[0040] The built-in AI engine is connected to the image acquisition device and the open target detection model signal, and is used to acquire the video stream in the current cycle and decode to obtain several frames of video stream images, and send the video stream images and capability parameters to the open target detection model. Specifically, the built-in AI engine sends the decoded video stream image together with the complete capability parameters to the open target detection model. The capability parameters are a set of configuration information, which defines the working mode and boundary conditions of the open target detection model when performing the detection task. In the present invention, the capability parameters include the following core contents: basic configuration: including camera ID, video stream address and format (RTSP), etc. These parameters tell the model where to get input data; detection range: the model only needs to pay attention to the specific area where the fire escape is located through the area of ​​interest parameter; target type: the object category to be detected is specified through the second set of labels to be detected; time control: including tracking duration threshold and interval alarm cycle, which are used to control the time dimension requirements of detection.

[0041] The relationship between these capability parameters and the open target detection model is a "configuration and execution" relationship. The open target detection model is a deep learning model that actually performs object recognition tasks, and the capability parameters tell the model "what to detect", "where to detect" and "how to detect". It should be understood that the open target detection model described in the present invention is a type of deep learning model specifically used for object positioning and recognition in images. It not only includes the basic functions of traditional target detection, but also breaks through the limitation that traditional models can only recognize fixed categories. It is represented by advanced models such as DINO (DETR with Improved deNoising Object Detector). The openness of this type of model makes it more adaptable and practical in practical applications within the scope of the present invention, and can better meet the target detection needs in various scenarios. Those skilled in the art can make reasonable choices in the existing technology based on the hardware resource conditions on site and the requirements of real-time and detection accuracy, and the present invention does not make further restrictions.

[0042] In some preferred embodiments, specific examples of capability parameters are given. These capability parameters are a parameter set including multiple configuration items, specifically including: Camera ID: used to uniquely identify the video input source; Video stream address: specifies the data source location; Video stream format: specify the video format, such as RTSP format; Area of ​​interest: limit the detection range in the fire passage; AI Capability Name: Set to FireLaneObstruction; Tracking duration threshold: Set the time requirement for target tracking; Interval alarm period: controls the triggering frequency of the alarm.

[0043] The open target detection model is connected to the alarm module signal, and is used to use the capability parameters as input parameters to complete the target object detection of the video stream image and output the target object information list. The open target detection model will parse the received capability parameters. These parameters will affect many aspects of the detection process, such as the camera ID and video stream address determine the data source, and the area of ​​interest parameter defines the image range that needs to be paid attention to. These parameters together construct a complete context for a detection task. The target object information list contains detailed information of each detected target object, preferably including target location, target category, target confidence, and image timestamp.

[0044] The alarm module is connected to the display device signal, and is used to statically track the target object in the target fire passage interest area according to the target object information list, and output the tracking target object information list; when the tracking target object in the tracking target object information list meets the preset alarm condition, the fire passage occupancy alarm event is reported to the user through the display device.

[0045] The alarm mechanism is the final decision-making link of the system. It determines whether an alarm needs to be triggered by evaluating the state of the tracked target. Its design can be carried out by technicians in this field according to the actual needs of the site. In some preferred embodiments, the alarm module includes a target tracking unit and an event reporting unit; The target tracking unit is used to statically track the target object in the target fire channel interest area according to the target object information list, and output the tracking target object information list; the static tracking is an object persistence analysis method based on the target detection result. Different from the traditional video target tracking algorithm (such as KCF, SORT, etc.), static tracking mainly focuses on the continuous existence state of the target object in a specific area, without the need to accurately track the movement trajectory of the target. In some preferred embodiments, specific methods for implementing static tracking are given, including: First, the target object information list is filtered by region. Specifically, it is checked whether the position coordinates of each target object are within the predefined fire channel interest area.

[0046] Next, target merging is performed. When multiple similar targets are detected (for example, the same vehicle is detected repeatedly), their intersection over union (IOU) ratios are calculated, and targets whose IOUs exceed a preset threshold (for example, 0.5) are merged into the same tracking target.

[0047] Then, a state record is maintained for each target that passes the screening, and a tracking target object information list is generated. This list contains richer information. In some preferred embodiments, the tracking target object information list includes: tracking ID, tracking state, target duration, target location, target category, target threshold, and image timestamp.

[0048] Finally, the target is lost. If there is no new detection result for a tracked target within a preset time (for example, 5 seconds), it will be marked as "lost".

[0049] The above static tracking mechanism has low computational overhead and does not require complex motion prediction; it has good fault tolerance for short-term occlusion of the target; it can accurately count the target's continuous stay time; and it avoids repeated counting caused by short-term movement of the target.

[0050] The event reporting unit is connected to the target tracking unit and the display device by signal respectively, and is used to determine whether the tracking target object in the tracking target object information list meets the preset alarm condition, and if so, reports the alarm information to the display device.

[0051] In some preferred embodiments, a method is provided to report the fire passage occupation alarm event by checking whether each tracking target meets two key alarm conditions, namely, duration and alarm frequency control, which specifically includes: Duration judgment: Check whether the target's stay time in the fire channel area exceeds the preset tracking time threshold.

[0052] Alarm frequency control: Avoid repeated alarms by setting the interval alarm cycle parameters to ensure that the same target only triggers an alarm once within a specified time period.

[0053] When a tracking target meets both conditions, the system will generate an alarm event. The data structure of the alarm event contains complete on-site information.

[0054] The design of this alarm mechanism takes into account the needs of multiple practical application scenarios: 1. By judging the duration, false alarms are avoided for objects that pass by temporarily. For example, a vehicle temporarily stops to load or unload goods, which will not trigger an alarm.

[0055] 2. Prevent the system from generating junk alarms by controlling the alarm frequency. For example, even if a car is parked in the fire passage continuously, the system will not frequently report the alarm repeatedly.

[0056] 3. Complete alarm information facilitates subsequent manual processing and data analysis. On-site managers can quickly locate problems based on alarm information, while data analysts can use these data for statistical analysis and trend research.

[0057] In the actual scenario of fire passage monitoring, some long-term occupancy situations may be encountered. For example, a car owner may keep the vehicle parked in the fire passage for several days or even weeks. In this case, if the alarm is continuously sent according to the conventional interval alarm cycle (for example, every 4 hours), a large number of repeated alarm messages will be generated. These repeated alarms will not only occupy system resources, but more importantly, they will cause information redundancy and reduced processing efficiency. Therefore, in some preferred embodiments, a "one-time alarm" mechanism is provided by setting the interval alarm cycle to a special value of 0. This means that for the same occupied object, the system will only send an alarm once when it first detects that it meets the alarm condition, and then no repeated alarms will be sent.

[0058] In order to realize the real-time visual display of the fire passage occupancy detection results and improve the human-computer interaction effect of the system, in some preferred embodiments, the alarm module includes a video fusion unit and a video display unit. The video fusion unit is respectively connected to the built-in AI engine and the open target detection model signal to obtain the video stream image and the target object information list, and output the real-time video stream and the target object identification frame to the video display unit; the video display unit is connected to the display device signal to obtain the real-time video stream and the target object identification frame, and send it to the display device after rendering for display to the user.

[0059] The design of the alarm module enables the system to fuse the video stream image from the built-in AI engine with the target object information output by the open target detection model in real time, accurately superimpose the target recognition box in the original video screen, and ensure the display effect through special rendering processing. This modular design not only solves the problem of separation between detection results and video images in traditional systems, but also ensures the smoothness and real-time nature of video display through an independent video display unit, allowing users to intuitively observe the real-time status of the fire escape and the detection results of the system. At the same time, this structural design also provides good scalability for the system, which is convenient for the subsequent addition of new visual effects or interactive functions, reflecting the innovation and practical value of this application in the design of intelligent monitoring systems.

[0060] In actual application scenarios, the synchronization problem between the front-end video stream and the target rendering display has always been a technical difficulty that needs to be solved urgently. In traditional solutions, due to the differences in the processing mechanisms of target detection reasoning and video stream transmission, the "floating frame" phenomenon often occurs, in which the target frame is not aligned with the target object in the video stream. In particular, when the inference frame rate of target detection cannot reach the frame rate of the video stream, the target frame position remains stationary while the target object in the video screen continues to move, which seriously affects the availability and user experience of the monitoring system. To address this problem, the existing technology usually adopts a processing method of decoding the video stream on the end side, directly rendering the target frame on the decoded image, and then re-encoding it into a video stream. Although this solution solves the synchronization problem in principle, it has many limitations in actual applications: First, the end-side devices generally have the problem of limited CPU performance, which makes it difficult to support the decoding, rendering and encoding of multiple videos; second, the re-encoding process significantly increases the performance overhead of the system, affecting the real-time performance of the system; finally, in the scenario of multiple users accessing at the same time, since the video stream and the target frame have been encoded together, it is impossible to dynamically adjust the rendering method of the target frame according to the personalized needs of different users, which greatly reduces the flexibility of the system.

[0061] In order to fundamentally solve these problems, the present application innovatively proposes a video stream and target frame synchronous rendering scheme based on timestamp matching. In this scheme, the video display unit first extracts the target object information list from the real-time video stream, and through the timestamp query and matching mechanism, ensures that each frame of the video screen can find the target object information that accurately corresponds to it. This timestamp-based matching mechanism avoids the forced synchronization processing in the traditional scheme, which not only ensures the rendering accuracy, but also reduces the system overhead. More importantly, in order to solve the problem that the target detection frame rate is lower than the video stream frame rate, the system innovatively introduces a data cache and frame rate compensation mechanism: by caching the target information list data and the real-time video stream data packet within the preset time threshold range, and combining the linear interpolation algorithm to supplement the frame rate of the target information list data, a smooth transition of the target frame position is achieved. This scheme not only effectively solves the synchronization problem between the target frame and the video screen, but also reduces the requirements for the performance of the end-side device through algorithm optimization. At the same time, since the rendering process of the target frame is relatively independent of the transmission process of the video stream, the system can flexibly support personalized display requirements in multi-user scenarios, such as displaying target frames of different styles according to the permission levels or preferences of different users, which greatly improves the practicality and interactive experience of the system. The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An interactive system for fire passage occupancy monitoring, characterized in that: include: An image acquisition device for acquiring a monitoring image of a target fire passage; A display device for interacting with a user; A server connected to the image acquisition device and the display device by signals respectively; the server is configured to: determine whether the fire passage is currently occupied based on the monitoring image of the target fire passage acquired by the image acquisition device, and if so, warn the user through the display device; The server includes a label acquisition module, a built-in AI engine, an open target detection model and an alarm module; The label acquisition module is configured to be signal-connected to the multimodal large model and the image acquisition device respectively, and is used to periodically acquire at least one frame of monitoring image of the target fire passage, input the multimodal large model for preliminary target recognition, and use a preset prompt word template to guide the multimodal large model to output a first set of labels to be detected of the monitoring image; use the first set of labels to be detected to update a second set of labels to be detected with a fixed length in chronological order, and when the second set of labels to be detected changes, notify the built-in AI engine to start; the first set of labels to be detected includes Chinese and English labels; The built-in AI engine is connected to the image acquisition device and the open target detection model signal, and is used to acquire the video stream in the current period and decode to obtain several frames of video stream images, and send the video stream images and capability parameters to the open target detection model; The open target detection model is connected to the alarm module signal, and is used to use the capability parameter as an input parameter to complete the target object detection of the video stream image and output a target object information list; The alarm module is connected to the display device signal, and is used to statically track the target object in the target fire passage interest area according to the target object information list, and output the tracked target object information list; When the tracking target object in the tracking target object information list meets the preset alarm condition, the fire passage occupation alarm event is reported to the user through the display device.

2. The interactive system for fire passage occupancy monitoring according to claim 1, characterized in that: The alarm module includes a target tracking unit and an event reporting unit; The target tracking unit is used to statically track the target object in the target fire passage interest area according to the target object information list, and output the tracked target object information list; The event reporting unit is connected to the target tracking unit and the display device by signal respectively, and is used to determine whether the tracking target object in the tracking target object information list meets the preset alarm condition, and if so, reports the alarm information to the display device.

3. The interactive system for fire passage occupancy monitoring according to claim 2, characterized in that: The alarm module includes a video fusion unit and a video display unit; The video fusion unit is respectively connected to the built-in AI engine and the open target detection model signal, and is used to obtain the video stream image and the target object information list, and output the real-time video stream and the target object identification frame to the video display unit; The video display unit is connected to the display device signal, and is used to obtain the real-time video stream and the target object recognition frame, and send them to the display device after rendering for display to the user.

4. The interactive system for fire passage occupancy monitoring according to claim 1, characterized in that: The capability parameters include: camera ID, video stream address, video stream format, area of ​​interest, AI capability name, tracking time threshold, and interval alarm period.

5. The interactive system for fire passage occupancy monitoring according to claim 1, characterized in that: The target object information list includes: target position, target category, target confidence, and image timestamp; the tracking target object information list includes: tracking ID, tracking status, target duration, target position, target category, target threshold, and image timestamp.

6. The interactive system for fire passage occupancy monitoring according to claim 2, characterized in that: The method for the target tracking unit to perform static tracking includes: Filter the target objects in the target object information list that do not belong to the target fire channel interest area; Merge target objects whose intersection-and-union ratio is greater than a preset intersection-and-union threshold among the remaining target objects; If the remaining target objects do not appear at the target location within the preset time, it is determined that the target object is lost.

7. The interactive system for fire passage occupancy monitoring according to claim 5, characterized in that: The alarm conditions include: The duration of the tracking target object in the tracking target object information list at the target location is greater than the preset time; The number of reports of the tracking target objects in the tracking target object information list within the interval alarm period shall not exceed 1 time.

8. The interactive system for fire passage occupancy monitoring according to claim 7, characterized in that: If the interval alarm period is 0, the tracking target object in the tracking target object information list is reported only once in the entire monitoring period.

9. The interactive system for fire passage occupancy monitoring according to claim 3, characterized in that: The method for aligning the target object recognition frame with the target object in the real-time video stream when the video display unit completes rendering includes: Obtain a target object information list in a real-time video stream, query the image timestamp, match and obtain the target object information list and the real-time video stream data packet with the same timestamp, render and send to a display device for display to the user.

10. The interactive system for fire passage occupancy monitoring according to claim 9, characterized in that: The rendering method includes: if the frame rate of the target object information list with the same timestamp is lower than the frame rate of the real-time video stream, then caching the target information list data and the real-time video stream data packet within a preset time threshold range, and using a linear interpolation method to supplement the frame rate of the cached target information list data.

Images