Elevator security behavior detection method, device and equipment and computer storage medium

Abstract

The embodiment of the invention provides an elevator security behavior detection method, device and equipment and a computer storage medium, and belongs to the technical field of Internet of Things. Monitoring images of the elevator and sensing data collected by an elevator sensor are obtained, and the actual operation scene where the elevator is located is determined; target visual features of the elevator real-time monitoring image are extracted, and map semantic features related to an actual operation scene are extracted from the elevator scene behavior knowledge map; encoding the sensing data, and determining sensing data features; according to an actual operation scene, scene weights of the visual features, the map semantic features and the sensing data features are determined; and attention fusion is conducted on the visual features, the map semantic features and the sensing data features on the basis of the scene weight, a fusion feature vector is generated, whether an abnormal behavior occurs in the elevator or not is detected, and under the condition that the detection result indicates that the abnormal behavior occurs in the elevator, a first interface comprising the detection result is displayed. And therefore, the accuracy of elevator security behavior detection is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Things, and particularly relates to an elevator security behavior detection method and device, equipment and a computer storage medium. BACKGROUND

[0002] With the popularity of the concepts of intelligent buildings and smart cities, elevators, as indispensable vertical transportation tools in modern buildings, have always been a focus of attention for safe operation. Intelligent monitoring of illegal or abnormal behaviors such as “electric vehicles entering the elevator”, “smoking”, “person falling” in the elevator is a core requirement for ensuring building safety. In response to this requirement, a series of solutions have been created, such as traditional elevator visual recognition solutions, general knowledge graph assisted recognition solutions, and existing visual-graph simple fusion solutions.

[0003] Most traditional elevator visual recognition solutions are based on convolutional neural networks (YOLOv8, FasterR-CNN) for end-to-end training of images in the elevator. However, this process requires a large amount of labeled data, mainly due to the following reasons: the small space in the elevator, the variable lighting conditions caused by the elevator top light and door gap light, which make it difficult to adapt to the elevator scene; cross-brand elevators require a long time period (2-3 weeks) to re-label a large number of pictures and have poor model generalization ability; in addition, the visual features of luggage carts, baby strollers, etc. in the elevator are similar to those of electric vehicles, which can cause high multi-interference false alarm rates (for example, the probability of misjudging a luggage cart as an electric vehicle is as high as 18%).

[0004] The general knowledge graph assisted recognition solution is another existing technical approach, which extracts visual features and compares them with pre-set matching rules to obtain the output result. However, elevator behavior knowledge is scattered in “Elevator Safety Operation Regulations” and “Violation Case Reports”, and it takes a lot of time (2 months) to manually organize complete rules and build a graph covering 5 types of elevator behaviors; in addition, since this solution only relies on visual features without fusing elevator load and door state sensor data, it causes multi-modal data fragmentation and high recognition misjudgment rate (for example, the probability of misjudging “person carrying a long object” as “smoking” is as high as 22%).

[0005] The existing visual-graph simple fusion solution is a solution that uses the general knowledge graph as a visual feature screening tool without a deep fusion mechanism. Since only features are screened by the graph, the fusion degree is shallow, and cross-modal interaction between “vision-graph-sensor” cannot be achieved, resulting in low detection accuracy (only 75%).

[0006] Application Content The embodiments of the present application provide an elevator security behavior detection method, device, equipment and computer storage medium to improve the accuracy of elevator abnormal behavior detection.

[0007] In a first aspect, the embodiments of the present application provide an elevator security behavior detection method, which comprises: obtaining a monitoring image of an elevator and sensing data collected by an elevator sensor; determining an actual running scene of the elevator according to the monitoring image and the sensing data; extracting a target visual feature of the real-time monitoring image of the elevator to determine a visual feature, extracting a graph semantic feature related to the actual running scene from an elevator scene behavior knowledge graph, and encoding the sensing data to determine a sensing data feature; the elevator scene behavior knowledge graph comprises a plurality of entity nodes and relationships between different entity nodes, the plurality of entity nodes comprise elevator component nodes, behavior subject nodes, sensing data nodes and rule event nodes, and the relationships comprise structural relationships, behavior rule relationships and sensing linkage relationships connecting the plurality of entity nodes; determining scene weights of the visual feature, the graph semantic feature and the sensing data feature according to the actual running scene; performing attention fusion on the visual feature, the graph semantic feature and the sensing data feature based on the scene weights to generate a fusion feature vector; detecting whether an abnormal behavior occurs in the elevator according to the fusion feature vector to obtain a detection result; in a case where the detection result indicates that the abnormal behavior occurs in the elevator, outputting a first interface comprising the detection result.

[0008] In an implementable embodiment, the method further comprises: extracting the graph semantic feature related to the actual running scene from the elevator scene behavior knowledge graph, which comprises: for each entity node in the elevator scene behavior knowledge graph, aggregating features of K-order neighbors to obtain an embedding representation of the entity node; combining the embedding representations related to the actual running scene to form the graph semantic feature.

[0009] In an implementable embodiment, the method further comprises: performing attention fusion on the visual feature, the graph semantic feature and the sensor feature based on the scene weights to generate a fusion feature vector, which comprises: constructing visual-graph feature pairs based on different visual features and graph semantic features, respectively; calculating feature similarities between each visual-graph feature pair to obtain an initial attention matrix; increasing an attention weight in a visual-graph feature pair in which the feature similarity in the initial attention matrix is greater than a preset similarity to obtain an adjusted attention matrix; The visual features, the atlas semantic features and the sensor features are fused based on the scene weight and the adjusted attention matrix to generate a fusion feature vector.

[0010] In an implementable embodiment, the method further comprises: obtaining multi-source heterogeneous data of the elevator, the multi-source heterogeneous data comprising elevator domain document data and historical elevator abnormal behavior visual data; parsing the multi-source heterogeneous data to obtain multi-source parsed information, the multi-source parsed information comprising key-value pairs, text description information and associated visual attribute information; recognizing and extracting elevator scene-specific entities and their attributes from the multi-source parsed information by using a pre-trained language model enhanced by an elevator domain dictionary; establishing an association relationship between the entities based on preset elevator scene relationships and the extracted entities and their attributes, and forming an initial atlas structure; resolving conflicting knowledge from different data sources based on preset priority rules, and verifying doubtful knowledge based on multi-modal verification rules, and outputting an elevator scene behavior knowledge graph.

[0011] In an implementable embodiment, the method further comprises: parsing the multi-source heterogeneous data to obtain multi-source parsed information, comprising: extracting structured data in the elevator domain document data to obtain key-value pairs; extracting information features from the historical elevator abnormal behavior visual data to obtain visual attribute features.

[0012] In an implementable embodiment, the method further comprises: The historical elevator abnormal behavior visual data comprises historical elevator abnormal behavior image data and historical elevator abnormal behavior video data; and the information features of the historical elevator abnormal behavior visual data comprise: performing optical character recognition on the historical elevator abnormal behavior image data to obtain visual attribute features of a target in the historical elevator abnormal behavior image data; extracting time sequence information of the historical elevator abnormal behavior video data to obtain time sequence features of elevator abnormal behavior events.

[0013] In an implementable embodiment, the method further comprises: After outputting the first interface comprising the detection result, the method further comprises: receiving a first input to the first interface; in response to the first input, displaying a second interface, the second interface comprising a visual evidence area associated with the abnormal behavior, an atlas rule area and a sensor data area; The visual evidence area includes at least one of the following: an image frame and / or a video clip corresponding to the abnormal behavior in the monitoring image, a marking box in the image frame for marking an entity corresponding to the abnormal behavior, and entity feature information; The atlas rule area includes an entity node related to the abnormal behavior and a relationship between the nodes; The sensor data area includes at least one of the following: a change curve of sensor data, a relationship diagram of elevator door status and time, and smoke concentration data.

[0014] In an implementable embodiment, the method further includes: In response to a second input to the marking box, displaying an attention weight of the entity feature information in the attention matrix; In response to a third input to a target node in the node, searching for an associated relationship corresponding to the target node from the elevator scene behavior knowledge graph; and displaying the associated relationship; In response to a fourth input to a target time point in the change curve of the sensor data, displaying an attention weight of the sensor data at the target time in the attention matrix.

[0015] In an implementable embodiment, the method further includes: The second interface further includes an export control, and after the second interface is displayed, the method includes: In response to a fifth input to the export control, exporting a report of the abnormal behavior.

[0016] In an implementable embodiment, the method further includes: The second interface further includes a feedback control, and the feedback control is used to indicate a detection error; the abnormal behavior detection is obtained by using the elevator cross-modal attention fusion network, After the second interface is displayed, the method includes: In response to a sixth input to the feedback control, recording the detection result as a detection error case; Based on the detection error case, updating the elevator cross-modal attention fusion network.

[0017] In a second aspect, an embodiment of the present application provides an elevator security behavior detection device, and the device includes: A data acquisition module is configured to acquire monitoring images of an elevator and sensor data collected by sensors of the elevator; A scene determination module is configured to determine an actual running scene of the elevator based on the monitoring images and the sensor data; The feature extraction module is used to extract target visual features from real-time elevator monitoring images and determine the visual features; extract graph semantic features related to the actual operation scenario from the elevator scene behavior knowledge graph; encode sensor data and determine sensor data features; the elevator scene behavior knowledge graph includes multiple entity nodes and the relationships between different entity nodes. The multiple entity nodes include elevator component nodes, behavior subject nodes, sensor data nodes, and rule event nodes; the relationships include structural relationships connecting multiple entity nodes, behavior rule relationships, and sensor linkage relationships; The weight determination module is used to determine the scene weights of visual features, graph semantic features, and sensor data features based on the actual operating scenario. The feature vector fusion module is used to perform attention fusion on visual features, map semantic features and sensor data features based on scene weights to generate a fused feature vector. The detection module is used to detect whether abnormal behavior occurs inside the elevator based on the fused feature vector and obtain the detection results; The display module is used to display a first interface including the detection results when the detection results indicate that abnormal behavior has occurred inside the elevator.

[0018] Thirdly, embodiments of this application provide an elevator security behavior detection device, the device including: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement any one of the elevator security behavior detection methods.

[0019] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement any one of the elevator security behavior detection methods.

[0020] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the elevator security behavior detection methods described in the above embodiments.

[0021] The elevator security behavior detection method, apparatus, device, and computer storage medium of this application embodiment can simultaneously acquire and fuse elevator monitoring images and sensor data to determine the actual operating scenario, then extract visual features, graph semantic features, and sensor data features, and dynamically calculate the weights of each modality according to the scenario before performing attention fusion, finally generating a fused feature vector for abnormal behavior detection. Thus, this application can perform deep fusion of multimodal data features based on scene weights and attention mechanisms, and detect elevator security behavior based on these deeply fused features, thereby improving the accuracy of elevator security behavior detection. Attached Figure Description

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced. Those skilled in the art can further obtain other drawings according to these drawings without any creative effort.

[0023] Figure 1 is a flowchart of a process for constructing an elevator scene behavior knowledge graph provided by the embodiments of the present application; Figure 2 is a flowchart of an elevator security behavior detection method provided by the embodiments of the present application; Figure 3 is a flowchart of specific steps of S250 provided by the embodiments of the present application; Figure 4 is an interactive flowchart of reasoning logic visualization provided by the embodiments of the present application; Figure 5 is a structural diagram of an elevator security behavior detection device provided by the embodiments of the present application; Figure 6 is a structural diagram of an elevator security behavior detection device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0024] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of these specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0025] It should be noted that, in this paper, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0026] As we know from the background technology, most traditional elevator visual recognition solutions are based on end-to-end training of elevator images using convolutional neural networks (YOLOv8, Faster R-CNN). However, this process relies on a large amount of labeled data. The main reasons are: the small space inside the elevator and the variable lighting conditions caused by the elevator ceiling lights and door gaps make it difficult to adapt to the elevator scene; when adapting to elevators from different brands, a large number of images need to be re-labeled and a long time period (2-3 weeks) is required, which will result in poor model generalization ability; in addition, the visual features of luggage carts, strollers and other items inside the elevator are similar to those of electric vehicles, which will lead to a high false alarm rate due to multiple interferences (for example, the probability of misidentifying a luggage cart as an electric vehicle is as high as 18%).

[0027] A general knowledge graph-assisted recognition scheme is another existing technical approach. It extracts visual features and compares them with preset matching rules to obtain the output result. However, elevator behavior knowledge is scattered in the "Elevator Safety Operation Procedures" and "Violation Case Reports". Manually compiling complete rules and building a graph covering five types of elevator behavior would take a lot of time (2 months). In addition, since this scheme only relies on visual features and does not integrate elevator load and door status sensor data, it results in fragmented multimodal data and a high misjudgment rate (for example, the probability of misjudging "person carrying a thin object" as "smoking" is as high as 22%).

[0028] Existing simple vision-graph fusion schemes use general knowledge graphs as visual feature selection tools without a deep fusion mechanism. Because they only select features through the graph, the fusion level is shallow, and cross-modal interaction between vision, graph, and sensing cannot be achieved, resulting in low detection accuracy (only 75%).

[0029] To address the problems in the prior art, this application provides an elevator security behavior detection method, apparatus, device, and computer storage medium.

[0030] It should be noted that the elevator security behavior detection method provided in this application is based on a newly constructed elevator scene behavior knowledge graph. To better understand the embodiments of the elevator security behavior detection method provided in this application, the embodiment of the elevator scene behavior knowledge graph construction method is first introduced.

[0031] Figure 1 This illustration shows a flowchart of constructing an elevator scene behavior knowledge graph according to an embodiment of this application. Figure 1 As shown, the steps S110-S150 may be included: S110: Acquire multi-source heterogeneous data of elevators, including elevator-related document data and historical visual data of abnormal elevator behavior.

[0032] Documents and data related to elevators can include: "Elevator Safety Operation Procedures", "Elevator Violation Case Reports", and elevator parameter tables.

[0033] Historical elevator violation video clips: Videos labeled with behaviors such as "electric vehicle in elevator" and "smoking" are used for knowledge verification.

[0034] In this way, by integrating such structured and unstructured multi-source data, the system provides rich and realistic input sources for subsequent automated parsing and knowledge extraction, effectively supporting the construction of elevator scene behavior knowledge graph.

[0035] S120, parse multi-source heterogeneous data to obtain multi-source parsing information, which includes key-value pairs, text description information, and associated visual attribute information.

[0036] Key-value pairs are a common data structure that associates a unique key with a corresponding value. In this structure, the key is an identifier used to look up or reference the value, while the value is the data to be stored or retrieved. For example, a certain type of elevator: load capacity threshold = 1000kg.

[0037] Text description information refers to natural language text content parsed from elevator-related documents that describes rules, events, or scenarios, such as "electric vehicles are prohibited from entering the elevator car".

[0038] The associated visual attribute information refers to the objective features that are extracted from historical elevator abnormal behavior images or videos and are associated with a specific target or scene through visual analysis technology, such as the 48V logo identified by OCR or the number of wheels = 2 obtained by target detection.

[0039] Parsing multi-source heterogeneous data typically uses proprietary parsing algorithms, as shown in the example below: Document parsing and table parsing are typically used for text description information.

[0040] The document parsing process involves using tools to extract text, table, and image information, tailored to the characteristics of elevator documents. The table parsing process involves automatically recognizing structured tables such as "elevator model - load threshold" and "behavior type - penalty standard" and converting them into key-value pairs. Image parsing and video parsing are commonly used for processing visual attribute information.

[0041] The image parsing process involves extracting visual attributes from elevator images in violation cases using OCR; the video parsing process involves sampling historical violation videos at 10 frames per second and extracting the "subject-subject-action" time sequence information using YOLOv8 to generate dynamic knowledge such as "electric vehicle. wheel. cross threshold = yes (time: 00:03:15)".

[0042] In order to achieve the effect that text and visual data can be read by machines, S120 may include steps A1 and A2: A1: Extract structured data from elevator-related document data to obtain key-value pairs.

[0043] The structured data includes elevator model - load threshold and / or behavior type - penalty standard.

[0044] A2: Extract information features from the visual data of historical elevator abnormal behavior to obtain visual attribute features.

[0045] As an example, the system automatically identifies and extracts structured data from elevator-related document data, such as elevator parameter tables in Excel format, using a table parsing algorithm. For instance, it associates elevator models with rated loads to form key-value pairs of the type "model-load threshold". Simultaneously, for historical elevator abnormal behavior visual data, such as an MP4 surveillance video labeled "electric vehicle entering the elevator", the system samples frame by frame and analyzes it using an object detection model to extract temporal visual attribute features such as the main body (electric vehicle), the sub-body (wheel), the action (crossing the threshold), and the time of occurrence. This transforms the raw multimodal data into structured information that can be directly used by the knowledge building module.

[0046] In this way, through automated table parsing and visual feature extraction techniques, structured data in elevator domain documents and unstructured information in historical visual data are transformed into machine-readable key-value pairs and visual attribute features, respectively. This process achieves efficient and accurate structuring of multi-source heterogeneous data, providing standardized and high-quality input for the subsequent automated construction of knowledge graphs. It effectively solves the problems of low efficiency, error-proneness, and difficulty in aligning multimodal data caused by reliance on manual processing in traditional methods, significantly improving the automation level and reliability of the entire system's knowledge acquisition and processing stages.

[0047] To achieve the desired effect, S120 may also include steps B1-B2: B1: Perform optical character recognition on historical elevator abnormal behavior image data to obtain the visual attribute features corresponding to the target in the historical elevator abnormal behavior image data; B2: Extract the temporal information from historical elevator abnormal behavior video data to obtain the temporal characteristics of elevator abnormal behavior events.

[0048] As an example, the system performs optical character recognition on historical elevator abnormal behavior image data to extract the markings affixed to the elevator body, thereby obtaining visual attribute features. Simultaneously, for a segment of historical abnormal behavior video data, the system extracts key action sequences through a temporal analysis model, obtaining dynamic temporal features such as "person, posture, from standing to lying down," as well as the precise start and end time frames of the event in the video.

[0049] A specific example is that the system performs optical character recognition on historical elevator abnormal behavior image data to extract the "48V" label affixed to the vehicle body, thereby obtaining the visual attribute features of "electric vehicle. battery. voltage = 48V". Simultaneously, for a 10-second monitoring video recording the process of a "person falling", the system extracts the key action sequence through a time-series analysis model, obtaining the dynamic time-series features of the person's posture from standing to lying down, as well as the precise start and end time frames of the event in the video.

[0050] In this way, by combining OCR text recognition with video temporal analysis, a comprehensive extraction of embedded text attributes and dynamic behavioral process features from static images is achieved. This technology can transform unstructured visual data into structured knowledge such as attribute-value pairs and time-stamped behavioral sequences. This not only enriches the descriptive dimensions of entities in the knowledge graph but, more importantly, captures the dynamic evolution of violations. This provides an indispensable, temporally contextualized structured input for subsequently constructing an elevator behavior knowledge graph capable of accurately describing and reasoning about complex spatiotemporal events, significantly improving the refinement of behavior modeling and the accuracy of recognition and reasoning.

[0051] As an example of S120, the system automatically parses and processes the acquired multi-source heterogeneous data. For text-based document data, the system extracts normative clauses and descriptive text related to elevator safety through content analysis. For structured table or list data, the system converts it into machine-readable key-value pair format. Simultaneously, for historical data containing visual information, the system uses image recognition and video analysis technologies to extract visual features and attributes related to the subject of abnormal behavior, key components, and behavioral processes. Finally, the system outputs a comprehensive analytical result integrating text rules, structured parameters, and visual attribute descriptions, providing information input for subsequent knowledge extraction and graph construction.

[0052] A specific implementation involves the system receiving the aforementioned multi-source heterogeneous data and initiating a parsing process to extract structured information. For the acquired PDF-format "Elevator Safety Operation Procedures," the system automatically parses the text content, extracting key rules such as "Electric vehicles are prohibited from entering the elevator car," forming textual description information. For the Excel-format elevator parameter table, the system identifies the structured tables within, converting their content into a series of key-value pairs such as "Model: Brand X Model X, Rated Load: 1000kg, Car Dimensions: 1.6m × 1.8m." For historical violation video clips, the system samples keyframes at a fixed frequency and calls a target detection model to analyze the image content, extracting visual attribute information such as "Subject: Electric Vehicle," "Subject: Wheel," and "Action: Crossing the Threshold," while recording the time of occurrence, forming a visual attribute description containing temporal relationships. Through this process, the original heterogeneous data is transformed into a unified multi-source parsing information set containing key-value pairs, text rules, and visual attributes, laying the data foundation for the subsequent automated construction of the knowledge graph.

[0053] In this way, the system automatically parses and processes the acquired multi-source heterogeneous data. For text-based document data, the system extracts normative clauses and descriptive texts related to elevator safety through content analysis. For structured table or list data, the system converts it into machine-readable key-value pair format. Simultaneously, for historical data containing visual information, the system uses image recognition and video analysis technologies to extract visual features and attributes related to the subject of abnormal behavior, key components, and behavioral processes. Finally, the system outputs a comprehensive analytical result integrating text rules, structured parameters, and visual attribute descriptions, providing information input for subsequent knowledge extraction and graph construction.

[0054] S130 utilizes a pre-trained language model enhanced with an elevator domain dictionary to identify and extract elevator-specific entities and their attributes from multi-source parsing information.

[0055] The pre-trained language model structure includes a BERT pre-trained model, a newly added elevator scene entity dictionary, and a fine-tuned entity recognition head; the elevator scene entity dictionary contains more than 200 exclusive entities such as "car", "threshold", "load sensor" and "folding joint".

[0056] The definition of an entity includes: Core entities include elevator components, actors, and sensor data; Entity attributes such as the number of wheels on an electric vehicle, the load threshold of an elevator, and the type of human action.

[0057] As an example, the system inputs the multi-source parsed information obtained in the preceding steps into a pre-trained language model enhanced with elevator domain knowledge for processing. This model utilizes a built-in elevator-specific dictionary to automatically identify and extract entities representing core elevator components, actors, and sensors from text descriptions and key-value pairs, and further identifies and reduces the key attributes of these entities. Through this process, unstructured parsed information is transformed into structured "entity-attribute" knowledge units, providing standardized element inputs for subsequent relation modeling and graph construction, effectively improving the accuracy and coverage of elevator domain knowledge extraction.

[0058] A specific implementation could be as follows: identifying electric vehicles and wheels from text (quantity = 2); and identifying elevators and load capacity from key-value pairs (threshold = 1000 kg). This model achieves a recognition accuracy of 96.8% for elevator-related entities, an improvement of approximately 18.5% compared to the unenhanced general BERT model, thus providing high-quality structured elements for the subsequent construction of an accurate elevator behavior knowledge graph.

[0059] In this way, by introducing an elevator-specific dictionary to enhance and fine-tune the general pre-trained language model, high-precision and automated extraction of elevator-specific entities and attributes from multi-source heterogeneous information is achieved. This method significantly improves the accuracy and domain adaptability of knowledge extraction, effectively addresses the shortcomings of the general model in recognizing elevator terminology, and transforms unstructured knowledge that originally relied on manual interpretation and organization into standardized structured data that can be directly processed by machines. This provides core elements for the subsequent construction of an accurate and usable elevator behavior knowledge graph, and is a key step towards fully automated knowledge construction.

[0060] S140: Based on the preset elevator scene relationships and the extracted entities and their attributes, establish the association relationships between entities to form an initial graph structure.

[0061] The preset elevator scene relationships include: component association, behavior constraints, and sensor linkage.

[0062] Among them, component relationships include, for example, an elevator includes a car, and a car includes a door sill; Behavioral restrictions, such as prohibiting electric vehicles from entering the elevator and triggering smoke alarms when smoking; Sensor-based linkages, such as an electric vehicle entering an elevator triggering an increase in load and the door closing, are linked to a specific timing sequence.

[0063] Methods for establishing relationships include rule extraction and model extraction.

[0064] Among them, rule extraction, such as for "electric vehicles are prohibited from entering the elevator, and an alarm must be triggered if the load exceeds 800kg", automatically extracts the relationship of "electric vehicle - trigger - load alarm"; For complex relationships such as "wheel crossing the threshold → electric vehicle entering the elevator", the model uses an attention-based relationship extraction model, combined with video temporal information, achieving an extraction accuracy of 92.3%.

[0065] As an example, the system connects the "elevator" entity with the "car" entity based on structural relationships, associates the "electric vehicle" entity with the "no entry" event based on behavioral rules, and links the "increased load" attribute with the "electric vehicle entering the elevator" behavior based on sensor linkage relationships. By comprehensively applying methods such as rule matching and model prediction, the system weaves discrete entities and attributes into an initial graph network structure with rich semantic connections, laying the foundation for subsequent knowledge conflict resolution and graph improvement.

[0066] One specific implementation involves the system automatically establishing a "electric vehicle - prohibition - entry" behavioral constraint relationship from the text "electric vehicles prohibited from entering the elevator" through rule matching. By analyzing video temporal information and utilizing an attention-based relationship extraction model, the system identifies the spatiotemporal behavioral relationship of "wheels" "crossing" the "threshold" at a specific time point, achieving an accuracy rate of 92.3% for such complex relationship extraction. Simultaneously, based on sensor data logic, a sensor linkage relationship of "electric vehicle entering the elevator - trigger - load increase" is established. Ultimately, the system connects these discrete entities and attributes through various semantic relationships, forming an initial graph structure containing specific nodes and connections, providing a core framework for constructing a complete elevator behavior knowledge graph.

[0067] In this way, by combining a pre-defined elevator scene relationship framework with automated relationship extraction technology, discrete entities and attributes are systematically associated into a network structure with rich semantics. This process not only achieves the automated construction of relationships from simple component associations to complex spatiotemporal behaviors and sensor linkages, significantly improving the efficiency and structuring level of knowledge graph construction, but also directly encodes the domain logic and constraints of the elevator scene in the initial graph, providing a core knowledge foundation for subsequent high-precision, interpretable cross-modal behavior recognition.

[0068] S150, based on preset priority rules, adjudicates conflicting knowledge from different data sources, and verifies questionable knowledge based on multimodal verification rules, displaying a knowledge graph of elevator scene behavior.

[0069] Conflicting knowledge from different data sources can include document conflicts and visual-sensory conflicts.

[0070] Document conflicts, such as "Elevator Operation Procedure A" specifying a load threshold of 1000kg, while "Procedure B" specifies 800kg; A visual-sensor conflict could occur if the system detects an electric vehicle but the load remains unchanged, indicating that the image is actually a sticker.

[0071] Corresponding priority rules are preset for conflicts, such as: if the elevator model matches the load threshold conflict, then the elevator parameter table > operating procedures > case report.

[0072] Verification of questionable knowledge based on multimodal verification rules includes: if the load increases by less than 50kg within 5 seconds after visual detection of "electric vehicle", it is judged as a false target and is not added to the elevator scene behavior knowledge graph.

[0073] This can automatically resolve more than 90% of conflicts, with the remaining 10% confirmed through the operations and maintenance personnel interface, improving efficiency by 80% compared to manual conflict resolution.

[0074] In addition, the Elevator Behavior Knowledge Graph (EBKG) can be stored as follows: using Neo4j graph database, with ≥500 nodes (including 20 types of core entities) and ≥1200 relations (including 8 types of core relations). The hierarchical structure, taking the "electric vehicle entering the elevator" behavior as an example, is as follows: elevator ├─ Includes → Car (Dimensions: 1.6m × 1.8m) │ ├─ Includes → Threshold (Height: 5cm) │ └─ Associate → Load sensor (threshold: 1000kg) ├─ Prohibited → Electric vehicles (main body) │ ├─ Includes → Wheels (sub-body, quantity: 2, diameter: 50cm) │ ├─ Includes → Frame (Sub-body, Folding Joints: Existing) │ └─ Action → Enter (Associated: Wheel crosses threshold = Yes, Time: 00:03:15) └─ Trigger → Alarm (Conditions: Electric vehicle enters + Load > 1000kg + Door closed) As an example of the S150, when integrating multi-source knowledge to form the initial knowledge graph structure, the system automatically adjudicates knowledge conflicts arising from different data sources according to preset priority rules. Simultaneously, the system invokes multimodal verification rules to cross-validate certain questionable or single-source knowledge entries. After an automated conflict resolution and credibility verification process, the system ultimately outputs an internally consistent, multi-source calibrated elevator scene behavior knowledge graph, serving as the core knowledge base for subsequent behavior recognition and analysis.

[0075] In a specific implementation, when integrating conflicting knowledge from "Elevator Operation Procedure A" (which specifies a load threshold of 1000kg) and "Case Report B" (which records a threshold of 800kg for a certain elevator model), the system makes a decision based on a preset priority rule of "Elevator Parameter Table > Operation Procedure > Case Report," automatically adopting the load threshold (e.g., 1000kg) for the corresponding model in the parameter table. Simultaneously, for questionable knowledge such as the vision module reporting "electric vehicle detected" but the load sensor showing a change of less than 50kg within 5 seconds, the system triggers a multimodal verification rule, classifying it as a "false target" and excluding it. Through this series of automated conflict resolution and verification, the system ultimately outputs an elevator scene behavior knowledge graph stored in the Neo4j graph database. This graph contains over 500 nodes (covering 20 core entities such as elevator components and behavioral subjects) and over 1200 relationships, possessing internal consistency and high reliability, and can be directly used for subsequent cross-modal behavior recognition and reasoning.

[0076] In this way, through preset priority rules and multimodal verification mechanisms, automated adjudication of knowledge conflicts from different data sources and cross-validation of questionable knowledge are achieved, thereby outputting an internally consistent elevator scene behavior knowledge graph that has undergone multi-source calibration. This process significantly improves the credibility and reliability of the elevator scene behavior knowledge graph, effectively solves the ambiguity and errors that are difficult to avoid during manual integration, and provides a high-quality, reliable, and structured knowledge foundation for subsequent graph-based cross-modal behavior recognition, enhancing the accuracy and robustness of the entire system.

[0077] In summary, this embodiment achieves efficient and accurate transformation from multi-source heterogeneous data to a high-quality elevator scene behavior knowledge graph by constructing a fully automated elevator scene behavior knowledge graph generation pipeline. This method significantly improves the accuracy and coverage of knowledge extraction through domain-enhanced models and multimodal parsing techniques; and ensures the internal consistency and reliability of the graph through structured relationship modeling and conflict resolution mechanisms. This process completely changes the traditional manual, time-consuming, and error-prone graph construction mode, providing a dynamically updated, authoritative, and reliable structured knowledge core for subsequent cross-modal behavior recognition and reasoning, and is the cornerstone for improving the overall system's intelligence level.

[0078] In addition, embodiments of this application also include a method for updating the elevator scene behavior knowledge graph, as shown below: The system automatically triggers the knowledge graph update process in the following two situations: Proactive feedback trigger: Elevator maintenance personnel can mark a test result as "error" through the feedback control on the EIVI interface, including false alarms or missed alarms.

[0079] Passive discovery trigger: The system detects a certain behavior, but its confidence level is lower than a preset threshold, indicating that it has encountered a new situation that is not fully covered or is ambiguous in the training data.

[0080] Once triggered, the system executes the following automated steps: High-value sample selection: Using active learning algorithms such as BALD (Bayesian Active Learning by Disagreement), 10-15 sample images with the most information content and learning value are selected from real-time elevator data related to the triggering event.

[0081] Graph Entity and Attribute Expansion: Using this new batch of samples, the entity extraction model in the EBKG-ACM module is fine-tuned. For example, when "electric vehicle with child seat" is found, the model learns to recognize the new sub-entity "child seat" and adds the attribute "electric vehicle.child seat = exists" and association rules to the "electric vehicle" entity in the graph.

[0082] Fusion network parameter tuning: Synchronously update the scene weight factors in the ECMAFN network. For newly emerging features, adjust their base weights or reliability coefficients in specific scenes to enable the fusion network to more accurately assess the importance of new evidence.

[0083] As an example, the system triggered the knowledge graph update process in two separate scenarios: First, maintenance personnel marked a "electric vehicle entering the elevator" alarm as a "false alarm" on the EIVI interface, when the actual object was a bicycle with a large pet carrier. Then, the system detected an "unknown object" in another elevator with a confidence level of only 0.70, below the 0.85 threshold. The system then performed an automated update: 1) Using the BALD algorithm, it intelligently selected 15 high-value image samples from the associated videos of these two events, clearly showing the "pet carrier" and the "special structure of the bicycle"; 2) Using these samples, it fine-tuned the entity extraction model of the EBKG-ACM module to recognize the new entity "bicycle" and its attribute "carrying.pet carrier = yes", and established a new rule of "bicycle - prohibited - entering the elevator"; 3) Simultaneously updated the scene weight factors of the ECMAFN network, optimizing the reliability coefficients of the "pet carrier" visual features and the "bicycle frame" features under specific lighting conditions. The entire targeted update process was completed within 2.5 hours. While accurately identifying the new violation of "bicycles carrying pet carriers into elevators," the system maintained the same detection accuracy as all previous violations.

[0084] In this way, by integrating a trigger-based incremental learning mechanism, the elevator behavior knowledge graph achieves low-cost, high-efficiency, and automated continuous evolution. This mechanism can accurately and purposefully absorb new knowledge, correct erroneous perceptions, and optimize fusion weights without manual annotation or full model retraining. This not only gives the system a self-evolving capability of "use-feedback-optimization" to effectively respond to new violations and scenario changes, but more importantly, this process can be completed within hours, completely avoiding catastrophic forgetting and ensuring long-term stability of business continuity and recognition performance, thus building a dynamic technological barrier for the system's sustainable leadership.

[0085] Based on the elevator scene behavior knowledge graph constructed above, this application embodiment also provides an elevator security behavior detection method.

[0086] The steps for identification in the elevator security behavior detection method provided in this application embodiment are described below.

[0087] Figure 2 A flowchart illustrating an embodiment of the elevator security behavior detection method of this application is shown. Figure 2 As shown, the method may include the following steps S210-S270: S210, acquire the monitoring images of the elevator and the sensor data collected by the elevator sensors; As an example, image acquisition devices deployed inside the elevator continuously acquire monitoring images of the elevator car's interior. Simultaneously, various sensors connected to the elevator control system collect real-time sensor data related to the elevator's operating status.

[0088] One specific embodiment could be a camera deployed on the top of an elevator car of brand X, model X, in a residential community, continuously capturing real-time monitoring images—an RGB video stream with a resolution of 1920×1080. Simultaneously, multiple sensors integrated into the elevator system collect sensor data: a load sensor returns a real-time reading of the current car load (650kg), a door status sensor returns an "open" or "closed" switch signal, and a smoke concentration sensor returns a real-time monitoring value of 15μg / m³. This multimodal data is aggregated through an IoT gateway and aligned based on a unified timestamp, forming time-synchronized monitoring image frames and sensor data streams. This provides real-time, synchronous raw data input for subsequent cross-modal behavior recognition.

[0089] In this way, data from different physical sources are aggregated through communication links and organized and cached under a unified temporal framework to form a synchronous multimodal data stream, providing real-time input that integrates visual and physical state information for subsequent behavior recognition and analysis modules.

[0090] S220 determines the actual operating scenario of the elevator based on monitoring images and sensor data.

[0091] As an example, the system comprehensively utilizes real-time monitoring images and synchronized sensor data streams. By analyzing visual cues in the images (such as object distribution and lighting conditions) and sensor readings (such as load changes and door opening / closing status), it makes a comprehensive judgment and classification of the elevator's current actual operating scenario. For example, it identifies different scenario states such as "normal operation," "peak hours," or "abnormal event occurrence." This judgment result serves as crucial contextual information, guiding the fusion and weight allocation of multimodal features in subsequent steps.

[0092] In one specific embodiment, the system analyzes the current monitoring image, detects that the elevator ceiling light is on and the light is uniform, with an average brightness of 220 lux, and simultaneously reads from sensor data that the door status signal is "closed" and has been maintained for 3 seconds, and the load sensor reading is stable at 780 kg. Based on this information and a preset scene determination logic, the system classifies the current elevator's actual operating scenario as a "normal operation with the door closed" scenario, and calculates a door closure coefficient of 0.6 accordingly.

[0093] In this way, by integrating real-time visual information with multi-dimensional sensor data, automated and accurate perception of the elevator's current operating scenario is achieved. It can dynamically identify the elevator's specific state, such as door opening / closing, lighting conditions, and load, providing crucial contextual information for subsequent multimodal feature fusion. This allows the behavior recognition system to adaptively adjust its analysis strategy based on the actual operating scenario, significantly improving the system's understanding and adaptability to complex and changing elevator environments.

[0094] S230: Extract target visual features from real-time elevator monitoring images and determine visual features; extract graph semantic features related to the actual operation scenario from the elevator scene behavior knowledge graph; encode sensor data and determine sensor data features; the elevator scene behavior knowledge graph includes multiple entity nodes and relationships between different entity nodes. The multiple types of entity nodes include elevator component nodes, behavior subject nodes, sensor data nodes, and rule event nodes; the relationships include structural relationships, behavior rule relationships, and sensor linkage relationships connecting multiple types of entity nodes.

[0095] To achieve strong semantic features, S230 may include: For each entity node in the elevator scene behavior knowledge graph, the features of the K-order neighbors are aggregated to obtain the embedded representation of the entity node; The embedded representations related to the actual operating scenario are combined to form graph semantic features.

[0096] As an example, the system extracts key entity nodes from a pre-built knowledge graph for the current focused real-world operational scenario of electric vehicles entering elevators. For the "electric vehicle" entity node, the GraphSAGE algorithm is used to aggregate features from its K=3 order neighbors, including directly connected sub-entity nodes such as wheels and frames, as well as the "elevator car" node connected through "prohibition" relationships. Through layer-by-layer iteration using an encoding formula, a 512-dimensional "electric vehicle" node embedding representation is ultimately generated. Similarly, the system obtains the embeddings of other related nodes such as wheels and thresholds. Subsequently, the system concatenates (CONCAT) or weights these node embedding representations strongly related to the actual operational scenario to form a comprehensive graph semantic feature vector. This vector contains key scenario semantic information such as "electric vehicle includes wheels," "electric vehicle prohibited from entering elevator," and "wheels crossing thresholds," which is then used by the cross-modal fusion network.

[0097] In this way, by using GraphSAGE to aggregate multi-level neighbor features of entity nodes related to the current scene in the knowledge graph, and combining scene focus filtering and combined node embedding, deep encoding and scene-based representation of structured knowledge in the elevator domain are achieved. This technology can automatically capture and fuse complex semantic relationships between entities, transforming static graph data into dynamic feature vectors rich in contextual semantics. This not only provides high-quality, interpretable knowledge input for cross-modal fusion, significantly improving the semantic understanding and reasoning accuracy of behavior recognition systems in complex scenes, but also fundamentally solves the problem that traditional methods struggle to effectively utilize structured domain knowledge for dynamic scene adaptation.

[0098] As an example of the S230, the system processes real-time monitoring images using a deployed object detection model to locate and extract visual feature representations of key targets (such as people, vehicles, and objects) in the images. Simultaneously, based on the currently determined actual operating scenario, the system retrieves and extracts entity nodes and their relationships highly relevant to the scenario from the constructed elevator scene behavior knowledge graph, forming graph semantic features. Furthermore, raw data from various sensors is transformed into unified numerical vector features through a feature encoder. Through the above parallel processing flow, the system generates three modal features corresponding to visual content, graph knowledge, and physical state, respectively, providing input for subsequent cross-modal deep fusion and behavior recognition.

[0099] A specific embodiment of S230 can be as follows: The system inputs real-time acquired elevator monitoring images into a vision processing module composed of YOLOv8 and ResNet-50, extracting visual feature vectors of bounding box coordinates and wheel positions from detected electric vehicle targets. Simultaneously, upon detecting an electric vehicle approaching the elevator car, the system encodes entity nodes strongly related to "electric vehicle entering the elevator," such as "electric vehicle," "wheels," "threshold," and "load threshold," along with their relationships ("containment," "prohibition," and "trigger"), into graph semantic features using GraphSAGE (K=3) from a pre-built knowledge graph. At the same time, the current load data of 780kg and the door status signal (closed) are encoded into sensor data features of a unified dimension through a multilayer perceptron.

[0100] It should be noted that the elevator cross-modal attention fusion network includes a multimodal extraction layer, a scene weight calculation layer, a cross-modal attention fusion layer, and a behavior classification output layer.

[0101] The multimodal extraction layer is mainly used to extract visual, graphical, and sensory features.

[0102] Visual feature extraction can use ResNet-50+YOLOv8 neck features to focus on small targets inside the elevator, such as wheels; GraphSAGE, for example, integrates the semantic associations of "elevator-car-threshold" through K=3 order neighbor aggregation; The sensing is like an MLP mapping, which maps the load or gate state to a 512-dimensional vector with timestamp alignment.

[0103] The encoding of semantic features in a graph can include: The neighbor aggregation strategy involves aggregating the sub-sub-features of "wheels," "frame," and "folding joints," as well as the scene features of "elevator" and "cabin," for the "electric vehicle" node. The encoding formula is shown in formula (1): in Let v be a third-order neighbor of the elevator scene node; The output associated features are 512-dimensional graph semantic vectors, such as scene semantics including "electric vehicle.prohibited.entering the elevator" and "wheel.crossing the threshold".

[0104] In this way, by processing real-time visual content, structured domain knowledge, and dynamic sensor information in parallel and transforming them into a unified feature representation, deep fusion of multimodal data at the feature level is achieved. This effectively breaks down the barriers between heterogeneous data modalities, providing aligned and semantically rich inputs for subsequent attention-based cross-modal interactions. This enables the behavior recognition system to simultaneously utilize intuitive visual evidence, structured knowledge rules, and precise changes in physical state, thereby supporting more accurate, robust, and interpretable recognition results.

[0105] S240 utilizes the scene weight calculation layer in the elevator cross-modal attention fusion network to determine the scene weights of visual features, graph semantic features, and sensor data features based on the actual operating scenario.

[0106] The Scene Weight Calculation Layer (ESW-FCM) is used to dynamically adjust the weights of each modality.

[0107] The weighting formula is shown in formula (2): in, For modality The final weight, Basic weights; For scene coefficients; For modal reliability.

[0108] As an example, the multimodal features extracted in step S230 are input into the scene weight calculation layer (ESW-FCM) in the elevator cross-modal attention fusion network. Based on the actual elevator operating scene (such as door status and lighting conditions) determined in step S220, and combined with the reliability assessment of each modality, this layer dynamically calculates the weight values ​​that visual features, map semantic features, and sensor data features should be assigned to in the current scene. This achieves the goal of adaptively adjusting the importance of multimodal information according to different elevator operating conditions, laying a weighted foundation for subsequent feature fusion.

[0109] One specific implementation could be to define the actual operating scenario as "elevator doors closed and lighting normal". The Scene Weight Calculation Layer (ESW-FCM) sets scene coefficients based on this scenario. The weight is 0.6, and combined with the reliability of each modality: the sensor data is stable when the door is closed, with a reliability of 0.9; the vision is clear under normal illumination, with a reliability of 0.8; and the reliability of the map rule is 0.7. Substitute into formula (2) for calculation, where the basic weights are set as: visual 0.3, map 0.2, and sensor 0.5. The calculated weights are: visual feature weight = 0.3×(1+0.6×0.8)=0.444, map semantic feature weight = 0.2×(1+0.6×0.7)=0.284, and sensor data feature weight = 0.5×(1+0.6×0.9)=0.77. These dynamically calculated weights will be directly used for subsequent cross-modal attention fusion.

[0110] In this way, by dynamically adjusting the weights of each modality based on the real-time operation of the elevator through the scene weight calculation layer, adaptive allocation of the importance of multimodal information is achieved. This technology enables the system to flexibly respond to different elevator operating conditions, strengthen more reliable modalities in specific scenarios, and suppress modalities with higher uncertainty, thereby optimizing the strategy basis for multimodal fusion and effectively improving the overall accuracy and scene robustness of subsequent behavior recognition.

[0111] S250 utilizes the cross-modal attention fusion layer in the elevator cross-modal attention fusion network to perform attention fusion on visual features, graph semantic features, and sensor data features based on scene weights, generating a fused feature vector.

[0112] A cross-modal attention fusion layer is used to achieve feature interaction and fusion.

[0113] The attention matrix is ​​used to calculate the similarity between visual features and graph semantics, thereby strengthening the "wheel-threshold" association features. Fusion method: weighted summation, the specific calculation method is shown in formula (3): in, To fuse feature vectors.

[0114] Cross-modal attention fusion can include: Computational visual features With graph semantic features The cosine similarity is shown in formula (4): To achieve the goal of generating more discriminative unified feature vectors, the flowchart of the specific steps in S250 is as follows: Figure 3 As shown, steps S310-S340 may be included: S310, based on different visual features and graph semantic features, construct visual-graph feature pairs respectively; S320, calculate the feature similarity between each visual-spectral feature pair to obtain the initial attention matrix; S330, Increase the attention weights in visual-map feature pairs in the initial attention matrix whose feature similarity is greater than the preset similarity, to obtain the adjusted attention matrix; S340 performs attention fusion on visual features, map semantic features, and sensor features based on scene weights and the adjusted attention matrix to generate a fused feature vector.

[0115] As an example, the system first combines multiple features extracted by the visual module with the graph semantic features encoded by GraphSAGE to construct multiple visual-graph feature pairs. The system calculates the cosine similarity of each feature pair according to formula (4) to form an initial attention matrix. The system increases the attention weights corresponding to similarities higher than a preset threshold to obtain an adjusted attention matrix, thereby strengthening the contribution of highly correlated feature pairs such as "wheel outline-number of wheels". The system combines the calculated scene weights and the adjusted attention matrix to perform weighted aggregation of visual, graph, and sensor features to generate the final fused feature vector.

[0116] Thus, by decomposing cross-modal attention fusion into four sub-steps—feature pair construction, similarity calculation, weight adjustment, and weighted aggregation—the synergistic mechanism of "scene weights" and "attention mechanisms" is clarified. This technique first establishes a direct channel for feature interaction between heterogeneous modalities by constructing visual-graph feature pairs; second, it quantifies cross-modal semantic consistency through cosine similarity and selectively strengthens key associations based on a threshold strategy, enabling the fusion process to focus on the most discriminative evidence combinations; finally, it combines the adjusted attention with dynamic scene weights for final fusion. This process not only makes the fusion decision-making process transparent and interpretable, but more importantly, through hierarchical and refined processing, it achieves optimal complementarity and integration of multimodal information at the feature level, thereby maximizing the accuracy, robustness, and traceability of the behavior recognition model in complex elevator scenarios.

[0117] As an example of S250, the multimodal features extracted in step S230 and the scene weights calculated in step S240 are input together into the cross-modal attention fusion layer. This layer first calculates the interrelationships and influence between different modal features through an attention mechanism, especially emphasizing the parts with high semantic consistency between visual features and graph semantic features. Subsequently, based on the scene weights of each modality, this layer performs weighted aggregation of the interacting features, ultimately generating a unified fusion feature vector that comprehensively reflects visual evidence, knowledge rules, and physical state information for subsequent behavior classification.

[0118] A specific implementation can be as follows: the multimodal features after dynamic weighting by S240 are input into the cross-modal attention fusion layer. This layer first calculates the cosine similarity between the wheel outline and the electric vehicle.wheel.number=2 using formula (4), for example, obtaining a similarity value of 0.95. Since this value is greater than the preset threshold of 0.8, the system increases the attention weight of this feature pair to 0.8 to strengthen the key semantic association between "wheel outline" and "wheel number=2". Next, this layer combines all modal features (including sensor data features) and their corresponding scene weights with the adjusted attention matrix, and performs final fusion using the weighted summation formula (3) to generate a unified fusion feature vector rich in cross-modal semantic information for subsequent classification. Compared with "simple visual + map splicing", the behavior recognition accuracy of the fusion feature is improved by 19.2%.

[0119] In this way, by combining a cosine similarity-based attention mechanism with dynamic scene weights through a cross-modal attention fusion layer, deep interaction and adaptive fusion of visual features, graph semantics, and sensor data are achieved. This technology can accurately identify and strengthen highly consistent semantic associations across modalities, effectively suppress irrelevant or conflicting information, and thus generate a unified fusion feature vector with stronger discriminative power and richer contextual semantics. This not only significantly improves the overall accuracy and robustness of behavior recognition, but more importantly, the fusion process itself is interpretable, laying a direct foundation for subsequent generation of visualized reasoning paths and solving the tracing problems of traditional "black box" models.

[0120] S260 utilizes the behavior classification output layer in the elevator cross-modal attention fusion network to detect whether abnormal behavior occurs inside the elevator based on the fused feature vector, and obtains the detection result.

[0121] The behavior classification output layer is used to determine the type of behavior inside the elevator.

[0122] Specific types of behavior inside elevators can include: "cars entering the elevator", "smoking", "people falling" and other behaviors.

[0123] As an example, the fused feature vector generated in step S250 is input into the behavior classification output layer of the elevator cross-modal attention fusion network. This layer typically consists of a fully connected layer and a classification function. By calculating and analyzing the feature vector that integrates multimodal information, it determines whether a preset category of abnormal behavior has occurred in the elevator and outputs the corresponding behavior category and confidence level as the detection result, thus completing the entire process from multimodal data input to final behavior discrimination.

[0124] In one specific embodiment, the system inputs the 512-dimensional fused feature vector generated in step S250 into the behavior classification output layer. This layer consists of a fully connected network and a Softmax classifier. After the fully connected network performs the final transformation on the features, the Softmax function outputs a probability distribution corresponding to a preset abnormal behavior. For example, the system calculates the probability of the "electric vehicle entering the elevator" category to be 0.982, or 98.2%, and determines that it exceeds the preset threshold of 0.85. Therefore, it generates the detection result: behavior type "electric vehicle entering the elevator", confidence level 98.2%, and attaches the current timestamp "2024-05-20 14:35:22". This result is then passed to the visualization interface module for display.

[0125] In this way, the behavior classification output layer transforms the feature vectors, which are deeply integrated with multimodal information, into specific and quantifiable behavior detection results. Based on the highly discriminative features provided by the preceding steps, this technology achieves high-precision, high-confidence classification and judgment of various abnormal behaviors within elevators. Its structured output not only directly meets the real-time alarm and decision-making needs of security systems but also provides clear input for the visualization and tracing of subsequent reasoning processes, marking the achievement of a complete technical loop from raw multimodal data to final security insights.

[0126] S270, if the detection result indicates that abnormal behavior has occurred inside the elevator, outputs a first interface including the detection result.

[0127] The first interface is the initial display interface that pushes and displays core alarm information to the user and provides further interaction entry when the system detects abnormal behavior in the elevator.

[0128] As an example, when the system's behavior classification output layer determines that abnormal behavior has occurred inside the elevator, the system will generate and output a primary interface. This interface is usually displayed on the elevator maintenance terminal or the monitoring center's screen, clearly presenting the core results of the detection, such as the type of abnormal behavior, the confidence level of the determination, and the precise timestamp, providing maintenance personnel with initial abnormal alarms and status awareness, and serving as the interactive entry point to trigger subsequent detailed reasoning and visualization processes.

[0129] In one specific implementation, when the behavior classification output layer determines that the confidence level of the "electric vehicle entering the elevator" abnormal behavior is 98.2%, the system immediately pushes and displays a first interface in full screen on the 7-inch maintenance touchscreen deployed next to the elevator car. The top of this interface prominently displays "Electric Vehicle Entering Elevator Alarm" in bold red font, and below it displays key information: the behavior confidence level is presented as a progress bar marked "98.2%", and the event timestamp is precisely displayed as "2024-05-20 14:35:22". The interface also provides a "View Details" touch button, serving as a direct interactive entry point for the user to trigger the subsequent second interface displaying the complete inference path.

[0130] In this way, by generating and pushing a first interface that integrates key alarm information and interactive entry points in real time, the anomaly detection results are efficiently and intuitively communicated from the backend algorithm to the front-end maintenance personnel. This technology transforms abstract algorithm output into a user interface with clear visual guidance and operability. It not only meets the basic requirements of security systems for real-time performance and clarity, but more importantly, it builds a bridge connecting behavior detection and deep reasoning visualization, providing maintenance personnel with the ability to "trace the source with one click." This effectively solves the problems of isolated alarm information and poor traceability in traditional systems, improving the response efficiency and user experience of the entire security process.

[0131] In some embodiments, after outputting a first interface including the detection results, the elevator security behavior detection method may further include: Receive the first input to the first interface; In response to the first input, a second interface is displayed, which includes a visual evidence area associated with the abnormal behavior, a map rule area, and a sensor data area. The visual evidence area includes at least one of the following: image frames and / or video clips in the surveillance images corresponding to the abnormal behavior, bounding boxes in the image frames used to mark entities corresponding to the abnormal behavior, and entity feature information; The graph rule area includes entity nodes related to abnormal behavior and the relationships between nodes; The sensor data area includes at least one of the following: sensor data change curves, elevator door status versus time graphs, and smoke concentration data.

[0132] The first input is an interactive operation triggered by the user on the first interface to view the detailed reasoning path, such as clicking the "View Details" button.

[0133] As an example, when maintenance personnel click the "View Details" button on the first interface displaying "Electric Vehicle Entering Elevator Alarm," the system immediately responds to the interaction and displays a second interface in full screen. This interface is divided into three core areas: the left visual evidence area, which displays a key image frame of the event, in which the main body of the electric vehicle is marked with a red box, its wheels are marked with yellow boxes, and it is labeled "Number of Wheels = 2 (Match Degree 0.95)", while also providing a 3-second looping video clip of the event; the middle knowledge graph rules area, which graphically highlights relevant nodes such as "electric vehicle", "wheel", and "threshold" in the knowledge graph, and uses arrows to clearly show the core relationships on which judgments such as "electric vehicle.contains.wheel" and "wheel.crosses.threshold" are based; and the right sensor data area, which synchronously displays the data from 5 seconds before the event to the current time, showing the weight increase from 650kg to 780kg and the door status, showing the time the door closed.

[0134] In this way, through responsive interactive design, a second interface that displays a fused multimodal evidence chain is triggered with a single click upon detecting an anomaly. This transforms the "black box" decision-making process within the algorithm into completely transparent and verifiable 3D visual evidence. This technology not only presents visual positioning, graph rules, and sensor changes side-by-side, allowing maintenance personnel to intuitively understand the judgment criteria without requiring specialized technical background, greatly improving anomaly detection efficiency and system credibility; more importantly, the fixed elements of this interface, such as "red box marking - node highlighting - curve association," directly correspond to the core technical features of this patented method, forming tamper-proof and easily obtainable infringement evidence. This enhances product usability while building a robust technical protection barrier.

[0135] Based on the above, this application also implements a visualization interface for elevator behavior recognition and reasoning. The interface design principles may include: Strongly bound to the algorithm: The inference path displayed on the interface corresponds exactly to the fusion process of the ECMAFN network and the rule matching logic of the EBKG graph, and the core technology cannot be bypassed by modifying the interface; Observable features: It includes three layers of visual elements: "visual evidence - graph rules - sensor data", all of which are direct evidence for infringement. Elevator scenario adaptation: The interface layout is adapted to the elevator operation and maintenance terminal, and the operation steps are ≤3.

[0136] As an optional embodiment of this application, in order to achieve the purpose of reasoning visualization and thus facilitate infringement evidence collection and safety tracing in elevator scenarios, such as Figure 4 As shown, steps S410-S430 may be included: The core areas and interaction logic of the visual interface may include: Top: Behavior results display area.

[0137] Includes: Behavior type: Electric vehicle entering the elevator; Confidence level: 98.2%; Timestamp: 2024-05-20 14:35:22; The interaction logic is that clicking the result expands the reasoning path area below.

[0138] It should be noted that behavior types can be displayed in bold red font, confidence levels can be represented by progress bars and numerical values, and timestamps need to be accurate to the second.

[0139] This achieves the effect of highlighting the test results.

[0140] S410, in response to the second input to the bounding box, displays the attention weights of entity feature information in the attention matrix.

[0141] The second input is the user's interactive operation on the visual marker box that marks a specific entity in the visual evidence area of ​​the second interface, such as clicking the box.

[0142] As an example, the left side of the visualization interface is the visual evidence area, which includes: elevator images of the main and sub-subjects, key features and matching degree of the markers, and time-series animation.

[0143] Images inside elevators with electric vehicles marked in red and wheels and / or frames marked in yellow. The key features and matching degrees are marked as follows: number of wheels = 2 (match degree 0.95), folding joint = present (match degree 0.92), etc.

[0144] Play a 3-second clip (10 frames per second) showing the wheels crossing the threshold and the car body entering the ladder.

[0145] When users need to query the reasoning logic for tracing or evidence collection, they can click the red box to interact and display the ECMAFN weight of the feature, such as "Wheel feature weight 0.444".

[0146] In this way, by enabling deep interaction with the visual marker boxes, the decision weight allocation mechanism within the cross-modal attention fusion network is made transparent in a way that can be explored in real time at the front end. This design allows users to directly verify the influence of specific visual features in the final behavior judgment, thereby transforming the algorithm from a "black box" decision-making process to a "white box" verification process. This not only greatly enhances the trust and acceptability of the system's decision-making process among operations and maintenance personnel, but more importantly, this interactive logic is deeply bound to the core backend algorithm, making it difficult for any product that provides similar weight tracing functions to bypass the reliance on this patented technology solution. Thus, while improving product explainability, it also constructs a dual barrier that combines user experience value and technical protection value.

[0147] S420, responding to the third input of the target node in the node, searches for the association relationship corresponding to the target node from the elevator scene behavior knowledge graph; displays the association relationship.

[0148] The third input is the user's interactive operation on a certain graph entity node in the graph rules area of ​​the second interface, such as long-pressing or dragging the node.

[0149] The center of the visualization interface is the graph rules area, which includes graph nodes, relationship lines, and rule text.

[0150] Graph nodes can highlight “electric vehicle”, “wheel”, “elevator car”, and “load threshold” nodes and fill them with green; The relationship lines are labeled "electric vehicle.includes.wheel", "wheel.crosses.threshold", and "electric vehicle.triggers.load alarm", including the arrow direction and the corresponding weight of the line width; The rule text displays the matching EBKG rule, such as electric vehicle entering the elevator = wheel crossing the threshold + load increase > 50kg.

[0151] When users need to query reasoning logic for tracing or evidence collection, they can drag nodes to display all the relationships associated with that node, such as "electric vehicle" being associated with "no entry to elevator" and "triggered alarm".

[0152] This allows us to display the path of the graph matching.

[0153] S430, in response to a fourth input to the target time point in the change curve of the sensor data, displays the attention weight of the sensor data at the target time in the attention matrix.

[0154] Based on the above solution, the core area and interaction logic of the visual interface can also include: The right side of the visualization interface is the sensor data area, which includes: load curve, door status timing, and smoke concentration.

[0155] The load curve is shown in blue, marked "Initial 650kg → 780kg after entering the elevator", and the red dashed line marks "Threshold 1000kg". The door status sequence is represented by a bar chart, labeled "Open (0-3 seconds) → Closed (3-8 seconds)"; Smoke concentration is expressed as a numerical value plus color, such as: normal: green < 50 μg / m³, abnormal: red > 50 μg / m³.

[0156] When users need to query the reasoning logic for tracing or evidence collection, they can click on the curve to display the ECMAFN sensor weight at that point in time, such as "weight 0.77 at 780kg".

[0157] As an example, when a user engages in deep interaction on the second interface displaying the evidence chain of "electric vehicle entering the elevator": the user clicks on the yellow box marked "wheel" in the visual evidence area, and a floating window pops up, displaying "The weight of the 'wheel outline' visual feature in the cross-modal attention fusion matrix is: 0.80"; the user long-presses and drags the "electric vehicle" node in the graph rules area, and the system queries the background knowledge graph in real time and pops up a list, displaying all the relationships of the node, such as "electric vehicle - prohibited - entering the elevator", "electric vehicle - contains - wheel", "electric vehicle - trigger - load alarm"; the user clicks on the load change curve in the sensor data area corresponding to the time point when the load jumps from 650kg to 780kg, and the interface marks above that time point as "The weight of the load sensor data at this moment in the fusion attention matrix is: 0.77".

[0158] In this way, by designing multi-layered interactive visualization elements, the weight allocation within the cross-modal attention fusion network and the structured association with the knowledge graph are transparently displayed in a way that can be explored in real time on the front end. This design not only gives users the ability to trace the details of algorithmic decisions layer by layer—from the attention weights of visual features to the complete relationship network of graph nodes, and then to the influence of sensor data at specific moments, greatly enhancing the interpretability and user trust of the system—but more importantly, it strongly binds these interactive processes to the core backend algorithm, making it difficult for any product implementing similar interactive logic to avoid relying on this patented technology solution. Thus, while improving the product experience, it builds a visualization technology barrier that combines deep traceability capabilities and strong evidence collection effectiveness.

[0159] In some embodiments, the second interface also includes a feedback control, which is used to indicate that the detection result is incorrect; the abnormal behavior detection is obtained through an elevator cross-modal attention fusion network. After displaying the second interface, the method may also include: In response to the sixth input to the feedback control, the detection result is recorded as a detection error case; Based on the detection of error cases, the elevator cross-modal attention fusion network is updated.

[0160] The sixth input is the user's interactive operation on the second interface with the feedback control used to indicate errors in the detection result, such as clicking the "Error" option.

[0161] Based on the above solution, the core area and interaction logic of the visual interface can also include: The bottom of the visual interface is the operation area, which includes an export button and a feedback button.

[0162] The export button is used to export a PDF inference report, which includes all visualization elements and is watermarked with "China Mobile IoT". The feedback buttons include a correct button and an incorrect button. When a user clicks an incorrect button, it is automatically marked as an active learning sample.

[0163] When a user needs to generate a report, clicking the export button will generate the report within 10 seconds. The report contains a timestamp and device number and cannot be tampered with.

[0164] This allows for result export and feedback, and triggers incremental learning of the scenario to update the elevator cross-modal attention fusion network.

[0165] As an example, a feedback button is located at the bottom of the second interface, with two options: correct and incorrect. After the maintenance personnel review the complete reasoning path of the "electric vehicle entering the elevator" alarm, they determine that it is a false alarm based on the actual situation, such as the actual object in the image being a luggage cart, and then click the incorrect option. The system responds to this input by packaging the current detection results, including image frames, extracted features, matching graph rules, sensor data stream, and the final judgment result, marking it as a detection error case and storing it in the background database. Subsequently, the system activates the incremental learning process: based on this case, the BALD algorithm automatically filters out high-value samples containing the "luggage cart" feature from the video stream of that time period, and fine-tunes the scene weight calculation model of the elevator cross-modal attention fusion network, such as reducing the initial weight of the "metal frame" feature in a specific scene, completing the targeted optimization of network parameters within 2 hours without retraining the entire model.

[0166] Thus, the bottom of the second interface features a feedback control with "Correct" and "Incorrect" options. When maintenance personnel determine that a "electric vehicle entering the elevator" alarm is a false alarm, they click the "Incorrect" option to complete the sixth input. The system then records all relevant data, including image frames, features, map rule matching results, and sensor time-series data, as a detection error case. Subsequently, the system initiates an incremental learning process: based on this case, the BALD algorithm is used to automatically select 10-15 high-value samples containing identified luggage carts from the relevant time period videos; then, the scene weight calculation model of the elevator cross-modal attention fusion network is fine-tuned. For example, the basic weight Wm0 for the "metal frame" visual feature in the door-opening scene is adjusted from 0.3 to 0.15; this optimization process is completed within 2 hours without retraining the entire network model and without losing the ability to detect the original violations.

[0167] In addition, to adapt to various elevator scenarios, incremental learning is required to improve the accuracy of recognition within elevator scenarios. Furthermore, the elevator cross-modal attention fusion network can be updated, including steps C1-C4: C1: The default trigger condition is to automatically trigger incremental learning when the EIVI interface reports an "error" error, including false alarms or false negatives, or when the detection confidence level is <0.85. C2: The BALD (Bayesian Active Learning by Disagreement) algorithm is used to select 10-15 high-value samples from real-time elevator data; C3: Fine-tune the EBKG-ACM entity extraction model with new samples and add new attributes; C4: Update the scene weight factor of ECMAFN.

[0168] As an example, when the system's confidence level in recognizing an electric vehicle with a child seat is only 0.68, the system automatically initiates an incremental learning process: First, the system uses the BALD algorithm to proactively select 12 high-value image samples from the elevator monitoring video stream associated with the event. These samples best represent the "child seat" feature and are currently the most uncertain for the model. Then, the system uses these samples to fine-tune the entity extraction model of the EBKG-ACM module, enabling it to accurately identify and extract the new sub-entity "electric vehicle.child seat" and its attributes. Furthermore, the system updates the scene weight factor of the ECMAFN network, for example, increasing the base weight of the "child seat" visual feature in the relevant scene from 0.2 to 0.4. The entire targeted optimization process is completed within 2 hours without retraining the entire model, and the system retains its detection capability for all existing behaviors while accurately identifying this new type of violation.

[0169] In this way, through an automated closed loop of "confidence threshold / user feedback trigger → proactive learning and sample selection → targeted updating of knowledge model and fusion weights," the system achieves rapid adaptation and accurate learning of unknown or ambiguous violations. This mechanism can absorb and internalize new knowledge within hours without retraining the overall model or interfering with existing business operations. It also ensures that while enhancing the ability to identify new scenarios, it fully maintains the detection accuracy for all existing violations. This endows the system with the core capability of continuous evolution and increasing strength over long-term operation, effectively solving the inherent problems of lagging model updates and high maintenance costs in traditional security systems.

[0170] In some embodiments, the second interface further includes an export control. After the second interface is displayed, the elevator security behavior detection method may further include: In response to the fifth input to the export control, export a report of abnormal behavior.

[0171] The fifth input is the user's interactive operation on the second interface, which triggers the export control for the report generation function, such as clicking the "Export Report" button.

[0172] As an example, when maintenance personnel click the "Export Report" button at the bottom of the second interface displaying the complete inference chain of "electric vehicle entering the elevator," the system immediately responds. The system automatically synthesizes all content presented on the current interface—including the behavioral result "electric vehicle entering the elevator" at the top, with a confidence level of 98.2%, key image frames marked with red and yellow boxes on the left, the knowledge graph subgraph highlighted in the middle, the load and door status time-series curves on the right, and related metadata such as device numbers (e.g., Device-003) and event timestamps (e.g., 2024-05-20 14:35:22)—to generate a PDF inference report containing the "China Mobile IoT" watermark. This report is generated and available for download within 10 seconds. Its content is completely consistent with the interface display, and the creation time in the file attributes matches the event timestamp, constituting a complete electronic evidence file that can be used for traceability or evidence collection.

[0173] In this way, the dynamic, interactive, and visual reasoning interface is solidified into a static, complete, and tamper-proof electronic evidence report through a one-click export function. This technology achieves a seamless transformation from real-time algorithmic judgment to legally valid evidence materials. It not only meets the rigid requirements of post-event traceability, audit compliance, and operation and maintenance archiving for elevator safety incidents, but also, due to its embedded complete evidence chain and anti-counterfeiting information, it becomes a technically substantial proof that can be directly used in judicial or rights protection scenarios. Thus, while improving the efficiency of operation and maintenance management, it constructs a key closed loop that integrates technology demonstration, process traceability, and rights protection.

[0174] In some embodiments, this application also includes GDPR compliance design, namely, privacy protection in elevator scenarios, with specific implementation examples as follows: First, zero storage of raw data is performed, including: after the ECMAFN network extracts visual features, the original elevator images / videos are immediately deleted using the SHA-256 irreversible hash algorithm, retaining only structured descriptions such as "electric vehicle.wheel.quantity=2"; Then, personal information is desensitized. In the image displayed on the visualization interface, the faces of elevator passengers are automatically blurred, for example, by using Gaussian blur with a blur radius of ≥10px, and personal information such as "employee number" and "fingerprint" is deleted. Next, data minimization is performed, storing only structured data related to behavior recognition, such as "number of wheels" and "load changes," while excluding irrelevant information, such as the color of clothing worn by passengers. Finally, traceability must be ensured. Under the conditions of complying with Article 15 of the GDPR on data access rights, operations and maintenance personnel can query the inference path, but cannot obtain the original image, thus balancing security and privacy.

[0175] In this way, through a systematic privacy protection design, compliance with strict data privacy regulations is achieved at the technical architecture level for smart elevator security. By adhering to the principles of "zero storage of raw data" and "data minimization," the system proactively destroys sensitive data such as original images after feature extraction, fundamentally eliminating the source risk of privacy leaks. Through "personal information anonymization" technology, individual identity information is automatically hidden in the visual interface, meeting the anonymization requirements for data display. Simultaneously, by preserving a traceable structured reasoning path, the system, while refusing access to raw data, still satisfies the user's "right to know" and "right to interpretation" as mandated by regulations. This series of measures collectively constructs a "privacy-friendly" technical closed loop, enabling this solution to ensure efficient and precise security capabilities while possessing the legal qualifications and core competitiveness to enter strictly regulated markets such as the EU.

[0176] In summary, the elevator security behavior detection method, apparatus, device, and computer storage medium of this application can simultaneously acquire and fuse elevator monitoring images and sensor data to determine the actual operating scenario, then extract visual features, graph semantic features, and sensor data features, and dynamically calculate the weights of each modality based on the scenario before performing attention fusion, ultimately generating a fused feature vector for abnormal behavior detection. Thus, this application can perform deep fusion of multimodal data features based on scene weights and attention mechanisms, and detect elevator security behavior based on these deeply fused features, thereby improving the accuracy of elevator security behavior detection.

[0177] Based on the elevator security behavior detection method described above, this application also provides an elevator security behavior detection device.

[0178] Figure 5 This is a schematic diagram of a device structure provided in an embodiment of this application. Figure 5 As shown, the device may include: The data acquisition module 510 is used to acquire monitoring images of the elevator and sensor data collected by the elevator sensors. The scene determination module 520 is used to determine the actual operating scene of the elevator based on monitoring images and sensor data. The feature extraction module 530 is used to extract target visual features from real-time elevator monitoring images and determine visual features; extract graph semantic features related to the actual operation scenario from the elevator scene behavior knowledge graph; encode sensor data and determine sensor data features; the elevator scene behavior knowledge graph includes multiple entity nodes and the relationships between different entity nodes. The multiple entity nodes include elevator component nodes, behavior subject nodes, sensor data nodes, and rule event nodes; the relationships include structural relationships connecting multiple entity nodes, behavior rule relationships, and sensor linkage relationships; The weight determination module 540 is used to determine the scene weights of visual features, map semantic features, and sensor data features based on the actual operating scenario. The feature vector fusion module 550 is used to perform attention fusion on visual features, map semantic features and sensor data features based on scene weights to generate a fused feature vector. The detection module 560 is used to detect whether abnormal behavior occurs inside the elevator based on the fused feature vector and obtain the detection result; Display module 570 is used to display a first interface including the detection results when the detection results indicate that abnormal behavior has occurred inside the elevator.

[0179] In this way, by constructing a collaborative device consisting of seven core modules, a complete technical closed loop is achieved, from automated processing of multi-source data to behavior recognition and result traceability. This design creatively integrates automatic knowledge graph construction, scenario-based dynamic cross-modal fusion, and deep 3D visualization reasoning. This not only significantly improves the accuracy, efficiency, and cross-scenario generalization ability of elevator abnormal behavior recognition, but also, through strong coupling of data and logic between modules, forms an integrated solution for automated knowledge construction, adaptive multimodal fusion, and observable and verifiable output. Technically, this simultaneously overcomes the three major pain points of existing solutions: high knowledge acquisition costs, poor fusion effects, and uninterpretable results. It possesses both high engineering practicality and significant commercial protection value.

[0180] In some embodiments, the feature extraction module 530 can also be used for: For each entity node in the elevator scene behavior knowledge graph, the features of the K-order neighbors are aggregated to obtain the embedded representation of the entity node; The embedded representations related to the actual operating scenario are combined to form graph semantic features.

[0181] In some embodiments, the feature vector fusion module 550 can also be used for: Visual-graph feature pairs are constructed based on different visual features and graph semantic features. Calculate the feature similarity between each visual-spectral feature pair to obtain the initial attention matrix; Increase the attention weights in visual-map feature pairs in the initial attention matrix whose feature similarity is greater than a preset similarity to obtain the adjusted attention matrix; Based on scene weights and the adjusted attention matrix, attention fusion is performed on visual features, graph semantic features and sensor features to generate a fused feature vector.

[0182] In some embodiments, the elevator security behavior detection device can also be used for: Acquire multi-source heterogeneous data on elevators, including elevator-related document data and historical visual data on abnormal elevator behavior; Analyze multi-source heterogeneous data to obtain multi-source parsing information, which includes key-value pairs, text description information, and associated visual attribute information; By using a pre-trained language model enhanced with an elevator domain dictionary, we can identify and extract elevator-specific entities and their attributes from multi-source parsed information. Based on the pre-defined elevator scene relationships and the extracted entities and their attributes, the association relationships between entities are established to form an initial graph structure; Based on preset priority rules, conflicting knowledge from different data sources is adjudicated, and questionable knowledge is verified based on multimodal verification rules, outputting an elevator scene behavior knowledge graph.

[0183] In some embodiments, the elevator security behavior detection device can also be used for: Extract structured data from elevator-related document data to obtain key-value pairs; Information features are extracted from the visual data of historical elevator abnormal behavior to obtain visual attribute features.

[0184] In some embodiments, the visual data of historical elevator abnormal behavior includes image data and video data of historical elevator abnormal behavior.

[0185] Elevator security behavior detection devices can also be used for: Optical character recognition is performed on historical elevator abnormal behavior image data to obtain the visual attribute features of the target in the historical elevator abnormal behavior image data. Extract the temporal information from historical elevator abnormal behavior video data to obtain the temporal characteristics of elevator abnormal behavior events.

[0186] In some embodiments, the elevator security behavior detection device can also be used for: Receive the first input to the first interface; In response to the first input, a second interface is displayed, which includes a visual evidence area associated with the abnormal behavior, a map rule area, and a sensor data area. The visual evidence area includes at least one of the following: image frames and / or video clips in the surveillance images corresponding to the abnormal behavior, bounding boxes in the image frames used to mark entities corresponding to the abnormal behavior, and entity feature information; The graph rule unit includes entity nodes related to abnormal behavior and the relationships between nodes. The sensing data unit includes at least one of the following: a curve showing the change in sensor data, a graph showing the relationship between elevator door status and time, and smoke concentration data.

[0187] In some embodiments, the elevator security behavior detection device can also be used for: In response to a second input to the bounding box, the attention weights of entity feature information in the attention matrix are displayed; In response to the third input of the target node in the node, search for the association relationship corresponding to the target node from the elevator scene behavior knowledge graph; display the association relationship; In response to the fourth input in the target time point of the sensor data change curve, the attention weight of the sensor data at the target time in the attention matrix is ​​displayed.

[0188] In some embodiments, the elevator security behavior detection device can also be used for: In response to the fifth input to the export control, export a report of abnormal behavior.

[0189] In some embodiments, the elevator security behavior detection device can also be used for: In response to the sixth input to the feedback control, the detection result is recorded as a detection error case; Based on the detection of error cases, the elevator cross-modal attention fusion network is updated.

[0190] Figure 6 A schematic diagram of the hardware structure for elevator security behavior detection provided in an embodiment of this application is shown.

[0191] The elevator security behavior detection device may include a processor 601 and a memory 602 storing computer program instructions.

[0192] Specifically, the processor 601 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0193] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 602 may include removable or non-removable (or fixed) media, or memory 602 may be non-volatile solid-state memory. Memory 602 may be internal or external to the integrated gateway disaster recovery device.

[0194] Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0195] The processor 601 reads and executes computer program instructions stored in the memory 602 to achieve... Figure 2 The elevator security behavior detection method in the illustrated embodiment.

[0196] In one example, the elevator security behavior detection device may also include a communication interface 603 and a bus 604. For example, Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 604 and complete communication with each other.

[0197] The communication interface 603 is mainly used to realize communication between various modules, systems, units and / or devices in the embodiments of this application.

[0198] Bus 604 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0199] This elevator security behavior detection device can execute the online data traffic billing method described in this application embodiment based on currently blocked spam text messages and text messages reported by users, thereby achieving a combination of... Figure 1 and Figure 2 The method described is for detecting elevator security behaviors.

[0200] Furthermore, in conjunction with the elevator security behavior detection method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the elevator security behavior detection methods in the above embodiments.

[0201] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the elevator security behavior detection methods described in the above embodiments.

[0202] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0203] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0204] It should also be noted that the exemplary embodiments mentioned in this application describe methods or apparatuses based on a series of steps or devices. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0205] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0206] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0207] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0208] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for detecting security behavior of an elevator, characterized by, The method comprises the following steps: obtaining monitoring images of an elevator and sensing data collected by an elevator sensor; determining an actual running scenario of the elevator according to the monitoring images and the sensing data; extracting target visual features of the monitoring images of the elevator in real time to determine visual features; extracting graph semantic features related to the actual running scenario from an elevator scene behavior knowledge graph; encoding the sensing data to determine sensing data features; the elevator scene behavior knowledge graph comprises a plurality of entity nodes and relationships between different entity nodes, and the plurality of entity nodes comprise elevator component nodes, behavior subject nodes, sensing data nodes and rule event nodes; the relationships comprise structural relationships, behavior rule relationships and sensing linkage relationships connecting the plurality of entity nodes; determining scenario weights of the visual features, the graph semantic features and the sensing data features according to the actual running scenario; performing attention fusion on the visual features, the graph semantic features and the sensing data features based on the scenario weights to generate a fusion feature vector; detecting whether an abnormal behavior occurs in the elevator according to the fusion feature vector to obtain a detection result; displaying a first interface comprising the detection result in a case where the detection result indicates that an abnormal behavior occurs in the elevator.

2. The elevator security behavior detection method according to claim 1, characterized by, The method further comprises the following steps: for each entity node in the elevator scene behavior knowledge graph, aggregating features of K-order neighbors to obtain an embedding representation of the entity node; K is a positive integer; combining the embedding representations related to the actual running scenario to form graph semantic features.

3. The elevator security behavior detection method of claim 1, wherein, The method further comprises the following steps: based on different visual features and graph semantic features, constructing visual-graph feature pairs respectively; calculating feature similarities between each of the visual-graph feature pairs to obtain an initial attention matrix; increasing attention weights in the visual-graph feature pairs in the initial attention matrix where the feature similarities are greater than a preset similarity to obtain an adjusted attention matrix; performing attention fusion on the visual features, the graph semantic features and the sensor features based on the scenario weights and the adjusted attention matrix to generate a fusion feature vector.

4. The method of claim 1, wherein, The method further comprises the following steps: obtaining multi-source heterogeneous data of the elevator, wherein the multi-source heterogeneous data comprises elevator domain document data and historical elevator abnormal behavior visual data; parsing the multi-source heterogeneous data to obtain multi-source parsed information, wherein the multi-source parsed information comprises key-value pairs, text description information and associated visual attribute information; using a pre-trained language model enhanced by an elevator domain dictionary to identify and extract elevator scene-specific entities and their attributes from the multi-source parsed information; based on a preset elevator scene relationship and the extracted entities and their attributes, establishing an association relationship between the entities to form an initial graph structure; The conflicting knowledge from different data sources is judged based on preset priority rules, and suspicious knowledge is checked based on multi-modal verification rules, and an elevator scene behavior knowledge graph is output.

5. The elevator security behavior detection method according to claim 4, characterized by, The multi-source heterogeneous data is parsed to obtain multi-source parsed information, including: Extracting structured data in the elevator field document data to obtain key-value pairs; Extracting structured data in the elevator field document data to obtain key-value pairs; 6. The elevator security behavior detection method according to claim 5, characterized by, Extracting structured data in the elevator field document data to obtain key-value pairs; The historical elevator abnormal behavior visual data includes historical elevator abnormal behavior image data and historical elevator abnormal behavior video data; the information features of the historical elevator abnormal behavior visual data are extracted to obtain visual attribute features, including: Performing optical character recognition on the historical elevator abnormal behavior image data to obtain the visual attribute features corresponding to the target in the historical elevator abnormal behavior image data; 7. The elevator security behavior detection method according to any one of claims 1-6, characterized by, Extracting the time sequence information of the historical elevator abnormal behavior video data to obtain the time sequence features of the elevator abnormal behavior event. The display further includes: Receiving a first input on the first interface; In response to the first input, a second interface is displayed, which includes a visual evidence area, a graph rule area, and a sensor data area associated with the abnormal behavior; The visual evidence area includes at least one of the following: image frames and / or video segments corresponding to the abnormal behavior in the monitoring image, and the image frames have a marking box for marking an entity corresponding to the abnormal behavior and entity feature information; The graph rule area includes entity nodes related to the abnormal behavior and relationships between the nodes; 8. The elevator security behavior detection method according to claim 7, characterized by, The sensor data area includes at least one of the following: a change curve of sensor data, a relationship diagram of elevator door state and time, and smoke concentration data. The method further includes at least one of the following: In response to a second input on the marking box, display the attention weight of the entity feature information in the attention matrix; In response to a third input on a target node in the node, find the associated relationship corresponding to the target node from the elevator scene behavior knowledge graph; Display the associated relationship; 9. The elevator security behavior detection method according to claim 7, characterized by, In response to a fourth input on a target time point in the change curve of the sensor data, display the attention weight of the sensor data at the target time in the attention matrix. The second interface further includes an export control, and after displaying the second interface, the method includes:

10. The elevator security behavior detection method of claim 7, wherein, In response to a fifth input on the export control, export a report of the abnormal behavior. The second interface further includes a feedback control, which is used to indicate that the detection result is incorrect; The abnormal behavior detection is obtained by an elevator cross-modal attention fusion network, and after displaying the second interface, the method includes: In response to a sixth input on the feedback control, record the detection result as a detection error case; 11. An elevator security behavior detection apparatus, characterized by comprising: Based on the detection error case, update the elevator cross-modal attention fusion network. including: A data acquisition module for acquiring monitoring images of an elevator and sensor data collected by elevator sensors; A scene determination module is configured to determine an actual operation scene of the elevator according to the monitoring image and the sensing data; A feature extraction module is configured to extract a target visual feature of the monitoring image of the elevator in real time, and determine a visual feature; A graph semantic feature related to the actual operation scene is extracted from an elevator scene behavior knowledge graph; the sensing data is encoded to determine a sensing data feature; the elevator scene behavior knowledge graph includes a plurality of entity nodes and relationships between different entity nodes, the plurality of entity nodes including elevator component nodes, behavior subject nodes, sensing data nodes, and rule event nodes; the relationships include structural relationships, behavior rule relationships, and sensing linkage relationships connecting the plurality of entity nodes; A weight determination module is configured to determine scene weights of the visual feature, the graph semantic feature, and the sensing data feature according to the actual operation scene; A feature vector fusion module is configured to perform attention fusion on the visual feature, the graph semantic feature, and the sensing data feature based on the scene weights, and generate a fused feature vector; A detection module is configured to detect whether an abnormal behavior occurs in the elevator according to the fused feature vector, and obtain a detection result; A display module is configured to display a first interface including the detection result in a case where the detection result indicates that an abnormal behavior occurs in the elevator.

12. An elevator security behavior detection device, characterized by, The device includes a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the elevator security behavior detection method according to any one of claims 1-10.

13. A computer storage medium, characterized in that The computer storage medium stores computer program instructions; the computer program instructions are executed by the processor to implement the elevator security behavior detection method according to any one of claims 1-10.

14. A computer program product, characterised in that, The computer program is executed by the processor to implement the elevator security behavior detection method according to any one of claims 1-10.

Images