An escalator safety monitoring method and system, processing device and storage medium

Abstract

The present application relates to a kind of escalator safety monitoring method, system, processing device and storage medium in station, the method includes: obtaining the monitoring video of automatic escalator in station, and using SAM image segmentation model, based on the input prompt word set and mask feature map, generate segmentation mask picture, and obtain the position information of each escalator related area relative to corresponding monitoring scene picture;Adaptive matching of prompt word is carried out to monitoring scene picture using multi-modal large model based on thought chain, and it is determined whether high-risk abnormal behavior event occurs in the region of automatic escalator in station;When high-risk abnormal behavior event occurs in the region of automatic escalator in station, according to the alarm result of multi-modal large model based on thought chain and the position information of each escalator related area relative to corresponding monitoring scene picture, the alarm position and alarm level of automatic escalator in station are determined, and the present application can be widely applied in safety monitoring field.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring, and in particular to a method, system, processing equipment, and storage medium for safety monitoring of escalators within stations. Background Technology

[0002] Escalators, as an important component of modern urban public transportation infrastructure, are widely used in public places such as railway stations, shopping malls, subway stations, and airports, playing a crucial role in transporting large volumes of passengers. However, due to the high density of people in escalator areas, the variety of luggage carried by passengers, and the unpredictable behavior of passengers, escalator falls occur frequently, posing serious safety hazards to passengers. Therefore, efficient and accurate monitoring of escalator safety is urgently needed.

[0003] With the development of artificial intelligence, some companies are using video surveillance and deep semantic understanding analysis based on multimodal models to monitor and alert on abnormal behavior on escalators in real time. However, due to the different positions, angles, and models of escalator surveillance cameras, the escalator scenes displayed by different cameras often vary greatly, making it difficult for the same monitoring method to be effectively generalized to different scenarios. Currently, multimodal models have significant limitations in abnormal event detection tasks. Their ability to perceive spatial location information is weak, and they cannot effectively distinguish events occurring in different areas. In escalator safety monitoring scenarios, the monitoring screen usually includes both the escalator and stair areas. When abnormal situations such as falls occur in the stair area, the multimodal model may misjudge them as events in the escalator area, causing frequent false triggering of the alarm system and seriously affecting the accuracy and reliability of detection.

[0004] Furthermore, in escalator monitoring scenarios, camera models vary, resulting in diverse image resolutions. High-resolution images convey rich details, while low-resolution images suffer from missing feature information. Simultaneously, the different configurations of escalator areas, such as single-escalator and multi-escalator operation, make traditional single-prompt methods (commands, large-scale models) ill-suited to complex and varied detection needs, severely limiting the generalization ability and accuracy of anomaly detection. Therefore, designing and constructing a universally applicable prompt system has become a pressing issue for the industry. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide a method, system, processing equipment, and storage medium for escalator safety monitoring within stations that is accurate, highly reliable, and applicable to various scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, it provides a method for safety monitoring of escalators within stations, comprising:

[0007] The monitoring video of the escalator in the station is acquired, and the SAM image segmentation model is used to generate a segmentation mask image based on the input prompt word set and mask feature map to determine the escalator-related area corresponding to each monitoring scene image in the monitoring video, and to obtain the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0008] A multimodal large model based on the thought chain is used to adaptively match prompt words on the monitored scene images. Core detection is performed based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0009] When a high-risk abnormal behavior event occurs in the escalator area of ​​the station, the alarm location and alarm level of the escalator in the station are determined based on the alarm results of the multimodal big model based on the thinking chain and the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0010] Further, the process involves acquiring surveillance video of the escalators within the station and employing the SAM image segmentation model. Based on the input set of prompt words and the mask feature map, a segmentation mask image is generated to determine the escalator-related regions corresponding to each surveillance scene image in the surveillance video. The positional information of each escalator-related region relative to its corresponding surveillance scene image is obtained, including:

[0011] Obtain surveillance video of the escalators within the station;

[0012] Frame extraction is performed on the acquired surveillance video to obtain the corresponding surveillance scene images;

[0013] Each monitoring scene image is input into the image encoder of the SAM image segmentation model for processing, and the monitoring scene images are converted into image embeddings.

[0014] The text prompts are input into the Prompt encoder of the SAM image segmentation model for processing, and the text prompts are converted into prompt embeddings.

[0015] The mask feature map is input into the downsampling module of the SAM image segmentation model for processing to obtain the downsampled mask feature map;

[0016] The transformed image embedding, cue embedding, and downsampled mask feature map are input into the lightweight mask decoder of the SAM image segmentation model for fusion to generate a segmentation mask image with confidence scores.

[0017] The SAM image segmentation model outputs the position information of each escalator-related area relative to the corresponding monitoring scene image, based on segmentation mask images with confidence scores.

[0018] Furthermore, the process of generating the segmentation mask image is as follows:

[0019]

[0020] in, This is the final segmentation mask image used to determine the escalator area in the surveillance scene image; The input image is a picture of the monitored scene; A set of prompt words; These are the parameters of the model.

[0021] Furthermore, the method employs a multimodal large model based on thought chains to adaptively match cue words to the monitored scene images, and performs core detection based on the generated segmentation mask image and the matched cue words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station, including:

[0022] A multimodal large model based on the thought chain is used to analyze the scene and resolution features of the image, and obtain the resolution classification result and the escalator number classification result of the image.

[0023] A dynamic adaptive prompt selection model is constructed, and the corresponding prompt words are adaptively matched based on the resolution classification results and escalator quantity classification results of the monitored scene images.

[0024] The generated segmentation mask image and the matched prompt words are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0025] Furthermore, the method employs a multimodal large model based on thought chain to analyze the scene and resolution features of the image, obtaining resolution classification results and escalator quantity classification results, including:

[0026] A multimodal large model based on the thought chain is used to classify the monitoring scene images according to their resolution, dividing the monitoring scene images into high-definition images and standard-definition images, and obtaining the resolution classification results of the monitoring scene images. Among them, high-definition images include high-definition single escalator images and high-definition double escalator images, and standard-definition images include standard single escalator images and standard-definition double escalator images.

[0027] The number of escalators in the surveillance scene images is analyzed, and the surveillance scene images are divided into multiple escalator images and single escalator images to obtain the classification results of the number of escalators in the surveillance scene images.

[0028] The resolution classification results and escalator quantity classification results of the monitored scene images are organically integrated to form a composite classification selection criterion, which serves as a reference dimension in large-scale model classification tasks.

[0029] Furthermore, the dynamic adaptive Prompt selection model is as follows:

[0030]

[0031] in, The prompt word with the highest confidence level in the prompt word set. and These represent scene features and resolution features, respectively. An embedded representation of the cue word set; This is a similarity calculation function; and Weighting parameters to balance the influence of scene features and resolution features.

[0032] Furthermore, the generated segmentation mask image and the matched prompt words are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station, including:

[0033] The generated segmentation mask image is input into an image encoder based on a multimodal large model of thought chain to obtain the encoded image vector;

[0034] The matched prompt words are input into a text encoder based on a multimodal large model of thought chain to obtain the encoded text vector;

[0035] Cross-modal feature alignment is performed on the obtained image vectors and text vectors to map the visual features to the language model space, resulting in a fused feature vector.

[0036] The fused feature vectors are input into the large language model decoder for inference output to determine whether a high-risk abnormal behavior event has occurred in the escalator area of ​​the station. When a high-risk abnormal behavior event occurs in the escalator area of ​​the station, an alarm is triggered.

[0037] Secondly, a safety monitoring system for escalators within a station is provided, including:

[0038] The segmentation mask generation module is used to acquire the monitoring video of the escalators in the station, and uses the SAM image segmentation model to generate segmentation mask images based on the input prompt word set and mask feature map, so as to determine the escalator-related areas corresponding to each monitoring scene image in the monitoring video, and obtain the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0039] The core detection module uses a multimodal large model based on the thought chain to adaptively match prompt words on the monitored scene images, and performs core detection based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0040] The alarm location determination module is used to determine the alarm location and alarm level of the escalator in the station when a high-risk abnormal behavior event occurs in the escalator area. This is based on the alarm results of the multimodal big model based on the thinking chain and the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0041] Thirdly, a processing device is provided, including computer program instructions, wherein when the computer program instructions are executed by the processing device, they are used to implement the steps corresponding to the above-mentioned escalator safety monitoring method.

[0042] Fourthly, a computer-readable storage medium is provided, wherein computer program instructions are stored on the computer-readable storage medium, wherein the computer program instructions, when executed by a processor, are used to implement the steps corresponding to the above-described escalator safety monitoring method.

[0043] The present invention has the following advantages due to the adoption of the above technical solutions:

[0044] 1. This invention adopts a pre-processing mechanism, embedding an image segmentation algorithm before the multimodal large model. This algorithm accurately divides the monitoring screen into regions and pre-identifies the spatial boundaries of escalators and stairs. Thus, only the visual information of the escalator area is input into the large model for analysis. This hierarchical processing architecture effectively makes up for the lack of sensitivity of the large model to spatial location information, and significantly improves the regional positioning accuracy and overall reliability of abnormal event detection.

[0045] 2. This invention introduces a chain-of-thought approach to construct a dynamic interactive detection framework. The large model first performs in-depth analysis of scene features and resolution in the video stream, accurately identifying scene parameters such as the number of escalators and their layout structure, while simultaneously quantifying image clarity and information density. Based on the above analysis results, this invention dynamically matches the optimal Prompt template through an inference decision-making mechanism, achieving precise adaptation of the detection strategy to scene features and image quality, effectively improving the model's robustness and task adaptability under multi-source heterogeneous data.

[0046] 3. This invention can adaptively select appropriate prompt words for different escalator scenarios, solving the problem that a single prompt word cannot adapt to multiple scenarios, and has strong generalization.

[0047] 4. This invention is not only applicable to escalator scenarios, but also to other monitoring fields, demonstrating high versatility.

[0048] 5. This invention can realize real-time monitoring and analysis of escalator areas, effectively alleviate manual labor pressure, and has a high level of intelligence.

[0049] In summary, this invention can be widely applied in public places such as railway stations, subway stations, shopping malls, and airports to optimize escalator safety monitoring. Attached Figure Description

[0050] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0051] Figure 1 This is a schematic diagram of a method flow provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of an image segmentation algorithm provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of scene and resolution feature analysis provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of Prompt adaptive matching provided in an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of large model detection provided in an embodiment of the present invention. Detailed Implementation

[0056] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0057] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0058] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0059] Currently, multimodal large models have significant limitations in anomaly detection tasks. Their ability to perceive spatial location information is weak, making it difficult to effectively distinguish events occurring in different areas. In escalator safety monitoring scenarios, the monitoring footage typically includes both escalator and stair areas. When anomalies such as falls occur in the stair area, the multimodal large model may misclassify them as escalator events, leading to frequent false alarms and severely impacting the accuracy and reliability of detection. Furthermore, in escalator monitoring scenarios, camera models vary, resulting in diverse resolutions in the captured footage. Additionally, different escalator configurations, such as single-escalator and multi-escalator operation, make traditional single Prompt statements (commands from the large model) unsuitable for complex and varied detection needs, severely restricting the generalization ability and accuracy of anomaly detection. This invention provides a method for safety monitoring of escalators within a station, comprising: acquiring surveillance video of the escalator within the station; using a SAM image segmentation model, generating a segmentation mask image based on an input set of prompt words and a mask feature map to determine the escalator-related areas corresponding to each surveillance scene image in the video, and obtaining the positional information of each escalator-related area relative to the corresponding surveillance scene image; employing a multimodal large model based on thought chain to adaptively match prompt words to the surveillance scene images, and performing core detection based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area within the station; when a high-risk abnormal behavior event occurs in the escalator area within the station, determining the alarm location and alarm level of the escalator within the station based on the alarm results of the multimodal large model based on thought chain and the positional information of each escalator-related area relative to the corresponding surveillance scene image. This invention can significantly improve the regional positioning accuracy and overall reliability of abnormal event detection.

[0060] Example 1

[0061] like Figure 1 As shown, this embodiment provides a method for safety monitoring of escalators within a station, including the following steps:

[0062] 1) Acquire surveillance video of the escalators within the station, and use the SAM image segmentation model. Based on the input cue word set and mask feature map, generate segmentation mask images to determine the escalator-related areas corresponding to each surveillance scene image in the video, and obtain the positional information of each escalator-related area relative to the corresponding surveillance scene image, specifically:

[0063] 1.1) Obtain surveillance video of the escalators in the station.

[0064] 1.2) Perform frame extraction on the acquired surveillance video to obtain the corresponding surveillance scene images.

[0065] 1.3) Input all the images of each monitoring scene into the image encoder of the SAM (Segment Anything Model) image segmentation model for processing, and convert each monitoring scene image into an image embedding.

[0066] Specifically, the SAM image segmentation model is based on the Transformer architecture, such as... Figure 2 As shown, the surveillance scene image is input into the image encoder of the SAM image segmentation model and transformed into an image embedding, achieving accurate understanding and efficient processing of the image content. The process of transforming the surveillance scene image into an image embedding can be represented as:

[0067]

[0068] in, Embed the obtained image; The input image is a picture of the monitored scene. These are the model parameters.

[0069] 1.4) Input the text prompt (e.g., escalator in this invention, used to guide the SAM image segmentation model to segment the desired image region) into the Prompt encoder of the SAM image segmentation model for processing, and convert the text prompt into a prompt embedding.

[0070] 1.5) Input the mask feature map into the downsampling module of the SAM image segmentation model for processing to obtain the downsampled mask feature map.

[0071] Specifically, the downsampling module performs progressive dimensionality reduction using two 2×2 convolutional layers with a stride of 2, and performs channel mapping using a 1×1 convolution. The purpose of downsampling the mask feature map is that the feature map output by the image encoder from the surveillance scene image is only 1 / 16 the size of the original surveillance scene image, so the mask also needs to be reduced accordingly to correspond with the image embedding.

[0072] 1.6) The transformed image embedding, cue embedding, and downsampled mask feature map are input into the lightweight mask decoder of the SAM image segmentation model for efficient fusion, generating a segmentation mask image with confidence scores, thereby achieving accurate identification of escalator-related areas in various monitoring scene images.

[0073] Specifically, the downsampled mask feature map, along with the transformed image embedding and cue embedding, is input into the lightweight mask decoder. The lightweight mask decoder integrates the outputs of the image encoder and cue encoder through the Transformer's self-attention and cross-attention mechanisms, and deeply integrates the downsampled mask feature map into the inference process. The lightweight mask decoder fuses these features, enabling the SAM image segmentation model to comprehensively consider image content, spatial location information, and the position and shape of the mask cue, thereby generating a more accurate segmented mask image of the target object. In this process, the downsampled mask feature map provides prior information about the mask and a coarse segmentation region. The lightweight mask decoder generates a more accurate segmented mask image through progressive upsampling and refinement.

[0074] Specifically, the process of generating the segmentation mask image is as follows:

[0075]

[0076] in, The final segmentation mask image is used to determine the precise area of ​​the escalator in the surveillance scene image; This is a set of prompt words (including multiple pre-set prompt words, such as the boundary points of the escalator, etc.). These are the parameters of the model.

[0077] 1.7) The SAM image segmentation model is based on a segmentation mask image with confidence scores. The area of ​​the region is appropriately expanded to ensure complete coverage of the escalator scene. The model outputs the position information of each escalator-related area relative to the corresponding monitoring scene image, providing key regional basic information for subsequent tasks such as passenger behavior detection or abnormal situation identification.

[0078] 2) A multimodal large model based on the thought chain is adopted to adaptively match the prompt words in the monitoring scene images, and core detection is performed based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0079] Specifically, the core principle of the thought chain introduced in this invention is to allow the model to mimic the human problem-solving process, breaking down complex problems into a series of intermediate reasoning steps. In terms of implementation, this typically employs cue engineering, that is, providing examples or guiding statements to the model in the input. Therefore, the specific process of this step is as follows:

[0080] 2.1) As Figure 3 As shown, a multimodal large model based on the thought chain is used to analyze the scene and resolution features of the image, resulting in resolution classification and escalator quantity classification results:

[0081] 2.1.1) A multimodal large model based on the thinking chain is adopted to classify the monitoring scene images according to their resolution, dividing the monitoring scene images into high-definition images and standard-definition images, and obtaining the resolution classification results of the monitoring scene images. Among them, high-definition images include high-definition single escalator images and high-definition double escalator images, and standard-definition images include standard single escalator images and standard-definition double escalator images.

[0082] Specifically, resolution, as a key indicator of image quality, reflects the level of detail and information contained in an image. High-definition images often provide richer textures and more delicate image features, while standard-definition images are relatively lower in terms of data volume and clarity.

[0083] Specifically, when classifying surveillance scene images by resolution, the classification is based on the image size of the surveillance scene images. and resolution threshold Classify the relationships:

[0084]

[0085] in, The classification results are based on resolution (1 for high definition, 0 for standard definition).

[0086] 2.1.2) Analyze the number of escalators in the monitoring scene images, divide the monitoring scene images into multiple escalator images and single escalator images, and obtain the classification results of the number of escalators in the monitoring scene images.

[0087] Specifically, the number of escalators means that the complexity of the scene varies. A single escalator scene is relatively simple, while a multi-escalator scene involves more complex factors such as the spatial relationship and layout between escalators.

[0088] Specifically, when classifying surveillance scene images according to the number of escalators, the classification is based on the number of escalators detected. With quantity threshold Categorize:

[0089]

[0090] in, The classification results are based on the number of escalators.

[0091] 2.1.3) The resolution classification results and escalator quantity classification results of the monitored scene images are organically integrated to form a composite classification selection criterion, which serves as a reference dimension in the large model classification task.

[0092] 2.2) As Figure 4As shown, a dynamic adaptive prompt selection model is constructed, and based on the resolution classification results and escalator quantity classification results of the monitored scene images, the corresponding prompt words are adaptively matched:

[0093] 2.2.1) Constructing a dynamic adaptive Prompt selection model:

[0094]

[0095] in, The prompt word with the highest confidence level in the prompt word set. and These represent scene features and resolution features, respectively. For the set of prompt words Embedded representation; For similarity calculation functions (such as cosine similarity); and The weighting parameters are used to balance the influence of scene features and resolution features. This formula is used to select the prompt that best matches the current scene features and resolution features from the prompt word set, achieving dynamic adaptive prompt selection.

[0096] 2.2.2) The constructed dynamic adaptive prompt selection model is adopted, which adaptively matches the prompt words corresponding to the monitoring scene images based on the resolution classification results and escalator number classification results of the monitoring scene images.

[0097] Specifically, after the multimodal big model based on the thought chain performs scene and resolution feature analysis on the monitoring scene image, it can accurately determine its category and then retrieve the matching prompt words. The core logic of adaptive Prompt is that the lower the image information density and the more complex the scene, the more the prompt words need to make up for the missing information through structured constraints.

[0098] Specifically, the dynamic adaptive Prompt selection model includes four levels: Level 1, Level 2, Level 3, and Level 4. Level 1 is the most lenient, Level 2 is relatively lenient, Level 3 is relatively strict, and Level 4 is the most strict.

[0099] For images of tall escalators, due to their high image quality and relatively simple elements, the prompts are designed at the first level, aiming to fully explore the potential information in the images: there are no format restrictions on the prompts, allowing the model to play its role independently, and only the most basic task description needs to be provided, such as "analyze passenger behavior in the escalator area and determine whether there is an abnormal state".

[0100] For high-definition images of two escalators, although the image quality is high, the presence of multiple escalators necessitates a balance between information capture and accuracy. Therefore, the designed prompts are at the second level: in addition to the most basic task description, the prompts also need to include some scene information descriptions to guide the model to focus on scene information, such as "There are multiple escalator areas in the image. Please analyze whether any abnormal passenger behavior has occurred in these escalator areas."

[0101] For standard definition escalator images, due to their lower image resolution, more precise positioning and identification are required. Therefore, the designed prompts are at the third level: in addition to the task description and scene information mentioned above, the prompts add some constraints to avoid false alarms caused by the lack of key information in standard definition images, such as "The image is a standard definition image with low resolution. Please analyze whether there is any abnormal passenger behavior in the escalator area of ​​the image. If you are unsure, please do not answer."

[0102] For standard definition (SD) images of two escalators, the detection is most difficult due to their low resolution and numerous elements. Therefore, the prompts are designed at level four: the prompts have the most restrictions and require a structured descriptive framework, such as: "1. Prompt: The image is SD with low resolution and contains multiple escalator areas; 2. Task: For each escalator area, determine whether there is any abnormal behavior by personnel; 3. Judgment criteria: Judgment is based solely on clearly identifiable visual features, without inferring the behavior of blurred areas; 4. Output: Output "Yes" if there is abnormal behavior, and "No" if there is no abnormal behavior; 5. Prohibition: Do not make unfounded speculations, do not describe non-abnormal behaviors (such as normal standing or walking), and do not output conclusive statements for uncertain content."

[0103] 2.3) As Figure 5 As shown, the generated segmentation mask image and the matched prompt are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0104] 2.3.1) Input the generated segmentation mask image into the image encoder based on the multimodal large model of the thought chain to obtain the encoded image vector.

[0105] 2.3.2) Input the matched prompt words into the text encoder of the multimodal large model based on the thought chain to obtain the encoded text vector.

[0106] 2.3.3) Perform cross-modal feature alignment on the obtained image vectors and text vectors, mapping the visual features to the language model space to obtain the fused feature vectors.

[0107] 2.3.4) Input the obtained fused feature vector into the large language model decoder for inference output to determine the detection result, that is, to determine whether a high-risk abnormal behavior event such as a person falling has occurred in the escalator area of ​​the station. When a high-risk abnormal behavior event occurs in the escalator of the station, an alarm is pushed.

[0108] Specifically, the core task of the thought chain-based multimodal big data model at this stage is to perform intelligent monitoring and analysis of the operational status of escalators within the station, focusing on detecting and identifying high-risk abnormal behavioral events such as falls. Based on the input images and refined guiding prompts, the thought chain-based multimodal big data model will output analysis results, providing crucial evidence for safety monitoring decisions.

[0109] 3) When a high-risk abnormal behavior event occurs in the escalator area of ​​the station, the alarm location and alarm level of the escalator in the station are determined based on the alarm results of the multimodal big model based on the thinking chain and the position information of each escalator area relative to the corresponding monitoring scene image, so that the on-site personnel can respond quickly and avoid further expansion of the danger.

[0110] Specifically, the alarm levels are handled differently depending on the type of high-risk abnormal behavior event, including Level 1 and Level 2 alarms. Level 1 alarms are triggered by high-risk abnormal behavior events such as carrying large luggage or strollers on escalators; Level 2 alarms are triggered by falls in the relevant areas of the escalator. Level 1 alarms only require notifying monitoring personnel to focus on high-risk events occurring in the relevant areas of the escalator. Level 2 alarms, in addition to notifying monitoring personnel, also involve remotely controlling the escalator's start and stop via a button, as the event has already occurred, to prevent further danger.

[0111] Example 2

[0112] This embodiment provides an escalator safety monitoring system within a station, including:

[0113] The segmentation mask generation module is used to acquire the monitoring video of the escalators in the station, and uses the SAM image segmentation model to generate segmentation mask images based on the input prompt word set and mask feature map, so as to determine the escalator-related areas corresponding to each monitoring scene image in the monitoring video, and obtain the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0114] The core detection module uses a multimodal large model based on thought chain to adaptively match prompt words in the monitored scene images, and performs core detection based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events such as people falling have occurred in the escalator area of ​​the station.

[0115] The alarm location determination module is used to determine the alarm location and alarm level of the escalator in the station when a high-risk abnormal behavior event occurs in the escalator area. This is based on the alarm results of the multimodal big model based on the thinking chain and the position information of each escalator-related area relative to the corresponding monitoring scene image.

[0116] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0117] Example 3

[0118] This embodiment provides a processing device corresponding to the escalator safety monitoring method provided in Embodiment 1. The processing device can be applied to client processing devices, such as mobile phones, laptops, tablets, desktop computers, etc., to execute the method of Embodiment 1.

[0119] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to communicate with each other. The memory stores a computer program that can run on the processing device. When the processing device runs the computer program, it executes the escalator safety monitoring method provided in Embodiment 1.

[0120] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0121] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0122] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] Those skilled in the art will understand that the structure of the above-described computing device is only a partial structure related to the present invention and does not constitute a limitation on the computing device to which the present invention is applied. A specific computing device may include more or fewer components, or combine certain components, or have different component arrangements.

[0124] Example 4

[0125] This embodiment provides a computer program product corresponding to the escalator safety monitoring method in station provided in Embodiment 1. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the escalator safety monitoring method in station described in Embodiment 1 are loaded.

[0126] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0127] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] The above embodiments are only used to illustrate the present invention. The structure, connection method and manufacturing process of each component can be varied. All equivalent transformations and improvements made on the basis of the technical solution of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A method for safety monitoring of escalators within a station, characterized in that, include: The monitoring video of the escalator in the station is acquired, and the SAM image segmentation model is used to generate a segmentation mask image based on the input prompt word set and mask feature map to determine the escalator-related area corresponding to each monitoring scene image in the monitoring video, and to obtain the position information of each escalator-related area relative to the corresponding monitoring scene image. A multimodal large model based on the thought chain is used to adaptively match prompt words on the monitored scene images, and core detection is performed based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events have occurred in the escalator area of ​​the station. When a high-risk abnormal behavior event occurs in the escalator area of ​​the station, the alarm location and alarm level of the escalator in the station are determined based on the alarm results of the multimodal big model based on the thinking chain and the position information of each escalator-related area relative to the corresponding monitoring scene image. The process involves acquiring surveillance video of the escalators within the station and using the SAM image segmentation model. Based on the input set of prompt words and the mask feature map, a segmentation mask image is generated to determine the escalator-related areas corresponding to each surveillance scene image in the video. The positional information of each escalator-related area relative to its corresponding surveillance scene image is obtained, including: Obtain surveillance video of the escalators within the station; Frame extraction is performed on the acquired surveillance video to obtain the corresponding surveillance scene images; Each monitoring scene image is input into the image encoder of the SAM image segmentation model for processing, and the monitoring scene images are converted into image embeddings. The text prompts are input into the Prompt encoder of the SAM image segmentation model for processing, and the text prompts are converted into prompt embeddings. The mask feature map is input into the downsampling module of the SAM image segmentation model for processing to obtain the downsampled mask feature map; The transformed image embedding, cue embedding, and downsampled mask feature map are input into the lightweight mask decoder of the SAM image segmentation model for fusion to generate a segmentation mask image with confidence scores. The SAM image segmentation model is based on segmentation mask images with confidence scores, and outputs the position information of each escalator-related area relative to the corresponding monitoring scene image. The method employs a multimodal large model based on thought chain to adaptively match cue words in the monitored scene images. Based on the generated segmentation mask image and the matched cue words, core detection is performed to determine whether high-risk abnormal behavior events have occurred in the escalator area within the station, including: A multimodal large model based on the thought chain is used to analyze the scene and resolution features of the image, and obtain the resolution classification result and the escalator number classification result of the image. A dynamic adaptive prompt selection model is constructed, and the corresponding prompt words are adaptively matched based on the resolution classification results and escalator quantity classification results of the monitored scene images. The generated segmentation mask image and the matched prompt words are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events have occurred in the escalator area of ​​the station.

2. The method for safety monitoring of escalators within a station as described in claim 1, characterized in that, The process of generating the segmentation mask image is as follows: in, This is the final segmentation mask image used to determine the escalator area in the surveillance scene image; Images of the monitored scene; A set of prompt words; These are the parameters of the model.

3. The method for safety monitoring of escalators within a station as described in claim 1, characterized in that, The method employs a multimodal large model based on thought chain to analyze the scene and resolution features of the image, obtaining resolution classification results and escalator quantity classification results, including: A multimodal large model based on the thought chain is used to classify the monitoring scene images according to their resolution, dividing the monitoring scene images into high-definition images and standard-definition images, and obtaining the resolution classification results of the monitoring scene images. Among them, high-definition images include high-definition single escalator images and high-definition double escalator images, and standard-definition images include standard single escalator images and standard-definition double escalator images. The number of escalators in the surveillance scene images is analyzed, and the surveillance scene images are divided into multiple escalator images and single escalator images to obtain the classification results of the number of escalators in the surveillance scene images. The resolution classification results and escalator quantity classification results of the monitored scene images are organically integrated to form a composite classification selection criterion, which serves as a reference dimension in large-scale model classification tasks.

4. The method for safety monitoring of escalators within a station as described in claim 1, characterized in that, The dynamic adaptive Prompt selection model is as follows: in, The prompt word with the highest confidence level in the prompt word set. ; and These represent scene features and resolution features, respectively. An embedded representation of the cue word set; This is a similarity calculation function; and Weighting parameters to balance the influence of scene features and resolution features.

5. The method for safety monitoring of escalators within a station as described in claim 1, characterized in that, The generated segmentation mask image and matched prompt words are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events have occurred in the escalator area of ​​the station, including: The generated segmentation mask image is input into an image encoder based on a multimodal large model of thought chain to obtain the encoded image vector; The matched prompt words are input into a text encoder based on a multimodal large model of thought chain to obtain the encoded text vector; Cross-modal feature alignment is performed on the obtained image vectors and text vectors to map the visual features to the language model space, resulting in a fused feature vector. The fused feature vectors are input into the large language model decoder for inference output to determine whether a high-risk abnormal behavior event has occurred in the escalator area of ​​the station. When a high-risk abnormal behavior event occurs in the escalator area of ​​the station, an alarm is triggered.

6. A safety monitoring system for escalators within a station, characterized in that, include: The segmentation mask generation module is used to acquire the monitoring video of the escalators in the station, and uses the SAM image segmentation model to generate segmentation mask images based on the input prompt word set and mask feature map, so as to determine the escalator-related areas corresponding to each monitoring scene image in the monitoring video, and obtain the position information of each escalator-related area relative to the corresponding monitoring scene image. The core detection module uses a multimodal large model based on the thought chain to adaptively match prompt words on the monitored scene images, and performs core detection based on the generated segmentation mask image and the matched prompt words to determine whether high-risk abnormal behavior events have occurred in the escalator area of ​​the station. The alarm location determination module is used to determine the alarm location and alarm level of the escalator in the station when a high-risk abnormal behavior event occurs in the escalator area. This is based on the alarm results of the multimodal big model based on the thinking chain and the location information of each escalator-related area relative to the corresponding monitoring scene image. The process involves acquiring surveillance video of the escalators within the station and using the SAM image segmentation model. Based on the input set of prompt words and the mask feature map, a segmentation mask image is generated to determine the escalator-related areas corresponding to each surveillance scene image in the video. The positional information of each escalator-related area relative to its corresponding surveillance scene image is obtained, including: Obtain surveillance video of the escalators within the station; Frame extraction is performed on the acquired surveillance video to obtain the corresponding surveillance scene images; Each monitoring scene image is input into the image encoder of the SAM image segmentation model for processing, and the monitoring scene images are converted into image embeddings. The text prompts are input into the Prompt encoder of the SAM image segmentation model for processing, and the text prompts are converted into prompt embeddings. The mask feature map is input into the downsampling module of the SAM image segmentation model for processing to obtain the downsampled mask feature map; The transformed image embedding, cue embedding, and downsampled mask feature map are input into the lightweight mask decoder of the SAM image segmentation model for fusion to generate a segmentation mask image with confidence scores. The SAM image segmentation model is based on segmentation mask images with confidence scores, and outputs the position information of each escalator-related area relative to the corresponding monitoring scene image. The method employs a multimodal large model based on thought chain to adaptively match cue words in the monitored scene images. Based on the generated segmentation mask image and the matched cue words, core detection is performed to determine whether high-risk abnormal behavior events have occurred in the escalator area within the station, including: A multimodal large model based on the thought chain is used to analyze the scene and resolution features of the image, and obtain the resolution classification result and the escalator number classification result of the image. A dynamic adaptive prompt selection model is constructed, and the corresponding prompt words are adaptively matched based on the resolution classification results and escalator quantity classification results of the monitored scene images. The generated segmentation mask image and the matched prompt words are input into a multimodal large model based on thought chain for core detection to determine whether high-risk abnormal behavior events have occurred in the escalator area of ​​the station.

7. A processing device, characterized in that, It includes computer program instructions, wherein when the computer program instructions are executed by the processing device, they are used to implement the steps corresponding to the escalator safety monitoring method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, wherein when the computer program instructions are executed by a processor, they are used to implement the steps corresponding to the escalator safety monitoring method in any one of claims 1-5.

Images