An elevator multi-modal risk intelligent identification and disposal method

Abstract

This application provides a method for intelligent identification and handling of multimodal risks in elevators, relating to the field of elevator safety monitoring technology. The method includes: acquiring multi-source data from the elevator operating environment, the multi-source data including visual data, depth information, audio data, and user identity information; extracting and fusing features from the multi-source data based on an intelligent identification model to obtain fused multimodal features; performing risk assessment based on the multimodal features to generate a risk assessment result; wherein, during the risk assessment process, a dynamic weight vector is used to weight each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results; and triggering a corresponding tiered response mechanism based on the risk assessment result.

Description

Technical Field

[0001] This application belongs to the field of elevator safety monitoring technology, specifically relating to a method for intelligent identification and handling of multimodal risks in elevators. Background Technology

[0002] In existing elevator safety management systems, video surveillance is commonly used as the primary technology. However, this approach, relying on a single visual modality, has several limitations. First, it is difficult to accurately identify objects inside the elevator car based solely on image information. For example, children's electric bikes and illegally parked electric bicycles, which are highly similar in appearance, can easily lead to misjudgments by the system. Second, traditional solutions lack a deep understanding of behavioral intent and cannot effectively establish a causal link between "object entry" and "potential safety risk." For instance, the system cannot know whether an electric bicycle entering the elevator belongs to a resident of the building, or its historical behavioral patterns, thus making it impossible to make accurate risk assessments based on context.

[0003] Furthermore, existing risk assessment models are typically static and fixed, unable to self-optimize based on actual operational results. When dealing with complex and ever-changing elevator scenarios (such as changes in lighting, object occlusion, and different user behaviors), the system's recognition accuracy and response efficiency significantly decrease. Although some solutions incorporate multi-sensor data, these solutions often simply stack data, lacking deep feature fusion and collaborative analysis, and even more so, the ability to dynamically adjust decision priorities based on historical performance. This results in limited system intelligence, low warning accuracy, and an inability to meet the proactive prevention and control needs of modern elevator safety management. Summary of the Invention

[0004] This application provides a method for intelligent identification and handling of multimodal risks in elevators to solve one of the aforementioned technical problems.

[0005] The technical solution adopted in this application is as follows:

[0006] This application provides a method for intelligent identification and handling of multimodal risks in elevators, including:

[0007] Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information;

[0008] Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features;

[0009] Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0010] Based on the risk assessment results, a corresponding tiered response mechanism is triggered.

[0011] According to one embodiment of this application, the user identity information is obtained by reading the UID of the elevator card and associating it with the homeowner information in the property management system, wherein the homeowner information includes registered electric vehicle information.

[0012] According to one embodiment of this application, the intelligent recognition model performs feature fusion by employing a multimodal Transformer structure, and its visual feature extraction branch adopts an improved YOLOv7 model and introduces an attention mechanism.

[0013] According to one embodiment of this application, the adaptive adjustment of the dynamic weight vector includes:

[0014] Based on historical risk assessment results, calculate the contribution of each characteristic dimension to the high-risk determination result;

[0015] Based on the contribution, the weight values ​​of each feature in the dynamic weight vector are updated.

[0016] According to one embodiment of this application, the hierarchical response mechanism includes at least:

[0017] Level 1 response, triggering a voice prompt;

[0018] A level-two response sends an alarm message to the property management system.

[0019] A Level 3 response will be implemented, including delaying elevator door closing and activating emergency ventilation.

[0020] According to one embodiment of this application, after triggering the tiered response mechanism, the method further includes:

[0021] Based on the response effect data, the weight allocation strategy in the intelligent identification model and / or risk assessment process is optimized.

[0022] A second aspect of this application provides an intelligent identification and handling device for multimodal risks in elevators, comprising:

[0023] The data acquisition module is used to acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information.

[0024] The feature processing module is used to extract and fuse features from the multi-source data based on the intelligent recognition model to obtain fused multimodal features;

[0025] The risk assessment module is used to perform risk assessment based on the multimodal features and generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculation on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0026] The decision response module is used to trigger the corresponding tiered response mechanism based on the risk assessment results.

[0027] A third aspect of this application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps described in the method.

[0028] A fourth aspect of this application provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described.

[0029] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0030] This application acquires and integrates multi-source data, including visual, depth, audio, and user identity information, enabling the system to cross-verify target information from multiple dimensions. For example, combining user identity information (such as the registered electric vehicle information of the homeowner) can assist visual judgment, effectively distinguishing between children's electric vehicles and illegal electric bicycles, thereby significantly improving the accuracy of target recognition.

[0031] By introducing a mechanism that uses a dynamic weight vector for weighted calculations, and this vector can adaptively adjust based on historical results, the system acquires the ability to continuously learn. The system can automatically discover and strengthen key features that have been repeatedly verified as effective in real-world scenarios (such as "visual features of the battery area" or "vehicle entry behavior of a specific homeowner"), while weakening insensitive features, thereby dynamically optimizing the evaluation model and improving the system's robustness and the accuracy of risk assessment in different scenarios.

[0032] By triggering a tiered response mechanism based on risk assessment results, the system achieves precise and tiered handling measures. It can match differentiated response strategies (such as a progressive approach from voice prompts to delayed door closure) according to risk levels (e.g., low, medium, high), avoiding a "one-size-fits-all" approach that could lead to a decline in user experience or insufficient prevention and control, thus achieving an optimal balance between safety and efficiency. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 A flowchart illustrating an intelligent identification and handling method for multimodal risks in elevators, provided as an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0036] Figure label:

[0037] 810, Processor; 820, Communication interface; 830, Memory; 840, Communication bus. Detailed Implementation

[0038] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0039] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0040] In this application, unless otherwise expressly specified and limited, the "above" or "below" of the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0041] Example 1

[0042] like Figure 1 As shown, a method for intelligent identification and handling of multimodal risks in elevators includes:

[0043] Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information.

[0044] As described above, this step involves simultaneously collecting different types of environmental and user data through various sensing devices and information systems deployed inside and around the elevator car, forming a multi-source data set. Visual data refers to images or video streams captured by high-definition cameras (such as wide-angle or fisheye lenses) for real-time monitoring of objects, people, and behaviors within the elevator car; depth information is acquired by depth sensors (such as time-of-flight cameras or structured light devices) to provide the three-dimensional spatial dimensions and positional relationships of target objects, compensating for the shortcomings of two-dimensional vision in distinguishing occlusions or similar appearances; audio data is collected through microphone arrays to capture specific acoustic events (such as the sound of electric vehicle motors or abnormal alarm sounds); user identity information is obtained by reading the user card UID through the elevator card swiping system and securely associating it with the homeowner registration information in the property management system (such as the number and type of electric vehicles in the home), thereby injecting contextual semantics into risk assessment. These data are collected using hardware synchronization mechanisms (such as timestamp-based alignment) to ensure consistency of multi-source data in the time dimension, laying the foundation for subsequent feature fusion and risk analysis. The aim is to enhance the system's adaptability to complex scenarios (such as changes in lighting and object occlusion) by leveraging the complementarity of multimodal data, and to provide rich input for deep semantic understanding.

[0045] For example, an embedded device integrating a visual camera and a ToF depth sensor is deployed on the top of the elevator car. The visual camera captures a video stream at 25 frames per second to detect objects suspected of being electric vehicles. Simultaneously, the ToF sensor generates depth point cloud data in real time to calculate the object's three-dimensional dimensions (length, width, and height) to distinguish between children's electric vehicles (smaller in size) and illegal electric bicycles (standard in size). A microphone array is installed on the inner wall of the car to continuously monitor ambient audio. When a continuous motor humming sound is detected, it can help confirm that the vehicle is in operation. In addition, the elevator access control system reads the user's UID information when swiping their card. The backend server associates this UID with the property database to query whether the resident has registered an electric bicycle—for example, if the resident's information shows that they have two electric bicycles registered in their home, the system can use this as supplementary evidence for high-risk determination. Through this multi-source data collaboration, the system can cross-verify from multiple dimensions such as object appearance, spatial attributes, acoustic features, and user background, significantly improving recognition accuracy.

[0046] It should be noted that, in specific implementation scenarios, based on the above solutions, an infrared thermal imaging sensor can be integrated at the data acquisition end to detect abnormal battery heating characteristics, thereby adding thermodynamic modes to the multi-source data; or an inertial measurement unit (IMU) can be introduced to collect elevator vibration data to help determine the dynamic behavior of vehicles entering. Regarding user identity information, this can be extended to data interaction with other intelligent building systems (such as parking management systems or smart home platforms) to obtain users' vehicle usage habits or charging records, enriching the contextual information. Furthermore, the data acquisition strategy itself can be designed as an adaptive mode: dynamically adjusting acquisition parameters based on feedback from subsequent risk assessment results—for example, when the system frequently identifies high-risk events, automatically increasing the resolution of the visual camera or the audio sampling rate to obtain more refined data; or temporarily shutting down some sensor modules during low-risk periods to save energy.

[0047] Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features.

[0048] As described above, this step involves using an intelligent recognition model architecture to perform deep feature extraction and cross-modal information integration on the aforementioned multi-source data. Feature extraction refers to extracting the most discriminative feature representations for different types of input data using specialized neural network sub-modules: for visual data, a convolutional neural network-based object detection model (such as an improved variant of YOLOv7) is used to extract appearance, texture, and spatial features, and attention mechanisms can be used to enhance focus on key parts (such as batteries and wheels); for depth information, point cloud processing or 3D convolutional networks are used to extract the geometric structure and spatial relationship features of the object; for audio data, the spectrogram after time-frequency analysis is used to extract acoustic feature sequences through convolutional recurrent networks; for user identity information, it is converted into a dense feature vector through an embedding layer. Feature fusion is not a simple feature concatenation, but rather a fusion strategy based on a cross-attention mechanism. For example, visual features are used as query vectors to actively "ask" features from other modalities to obtain supplementary information, thereby achieving cross-modal semantic alignment and information complementarity at the feature level, ultimately generating a unified fusion feature representation rich in multi-dimensional information. This process addresses the semantic gap between multi-source heterogeneous data, providing a comprehensive and accurate feature base for subsequent risk assessment.

[0049] For example, when the system detects a two-wheeled object in an elevator, the intelligent recognition model initiates multi-path feature extraction in parallel: the visual branch identifies the object's appearance features of "two wheels" and a "long, narrow shape"; the depth branch calculates its three-dimensional dimensions, which conform to the typical range of an adult electric bicycle; the audio branch analyzes the characteristic pattern of low-frequency motor operation sounds in the environment; and the user identity branch uses the card swipe UID to find that an electric bicycle is registered in the resident's home. Subsequently, in the fusion stage, the model is guided by visual features. For example, when the confidence of visual classification is low, it automatically increases the weight of user identity features and audio features. Through cross-validation between features, a strengthened fused feature is finally formed. This feature can clearly represent "the current object is very likely the resident's electric bicycle and is powered on," rather than just a simple collection of various modal features.

[0050] It should be noted that, in specific implementation scenarios, a dynamic feature selection mechanism can be introduced on top of the above solutions. This mechanism can automatically adjust the contribution weights of each modality feature in the fusion process based on the real-time scenario context (such as elevator load status and time period). Alternatively, a hierarchical fusion strategy can be adopted, first fusing high- and low-level features within the same modality, and then performing deep fusion across modalities. Furthermore, graph neural networks can be introduced to model the relationships between multimodal features. For example, people, vehicles, and environmental elements within the elevator can be constructed as a heterogeneous graph, and the interaction relationships between them can be learned to enrich the representational capabilities of the fused features. Another approach is based on a meta-learning framework, enabling the model to quickly adapt to new violation behavior patterns based on a small number of samples, dynamically adjusting its feature extraction and fusion strategies.

[0051] Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results.

[0052] As described above, this step involves quantifying risk and determining risk level based on the aforementioned fused multimodal features using a comprehensive evaluation model. This evaluation process does not simply involve adding fixed weights to each feature dimension; instead, it introduces a dynamic weight vector mechanism. Each weight value in this vector corresponds to the importance coefficient of a feature dimension (such as battery identification confidence in visual features, object size matching in depth features, motor sound intensity in audio features, and vehicle registration information in user features) in the risk assessment. The core characteristic of this dynamic weight vector is its adaptability, enabling it to dynamically adjust based on feedback loops from historical risk assessment results. The system calculates the contribution of each feature dimension in historical high-risk cases using attribution analysis techniques (such as gradient backpropagation or contribution analysis based on Shapley values). It then increases the weight of feature dimensions that have been verified as highly indicative of high-risk identification, while appropriately decreasing the weight of feature dimensions with lower contribution or those prone to introducing noise. This dynamic adjustment mechanism allows the risk assessment model to continuously learn from practical experience, gradually optimizing its decision boundaries and thus improving its adaptability to complex and changing scenarios. Finally, the model outputs a quantified risk score through weighted calculation, and maps it to discrete risk levels (such as low, medium, and high) according to preset thresholds, thereby generating the final risk assessment result.

[0053] For example, in its initial state, the system might assign high weights to visual features (such as object shape classification). However, after running for a period of time, analysis of historical data revealed that in several successfully identified high-risk events (such as unauthorized electric bicycles entering elevators), the feature contributions (i.e., the contribution values ​​derived from interpretable analysis) of the dimensions "user identity information" (e.g., the homeowner not having registered any electric bicycles) and "specific audio features" (e.g., a continuous motor humming sound) remained consistently high. Meanwhile, purely "visual features" sometimes led to misjudgments (e.g., for a new model of self-balancing scooter). Based on this analysis, the system initiated dynamic weight adjustment: automatically increasing the weights of "user identity information" and "audio features" in the dynamic weight vector, and correspondingly decreasing the weight of "visual features." When the system encounters a novel two-wheeled vehicle again, even if its visual classification confidence is low, the adjusted evaluation model, due to the strong signals from these high-weighted features and the detection of a noticeable motor sound and the card-swiping user being an unregistered electric bicycle owner, is more inclined to output a high-risk judgment, thus significantly improving the accuracy and robustness of the identification.

[0054] It should be noted that, in specific implementation scenarios, the adjustment of the dynamic weight vector can be based on more complex strategies, such as introducing a reinforcement learning framework to model the weight adjustment process as a policy optimization problem, using long-term risk identification accuracy as the reward signal, thereby learning the optimal weight allocation strategy. Furthermore, the granularity of adaptive adjustment can be further refined. For example, multiple scenario-based dynamic weight vector configuration files can be maintained and used according to different time periods (e.g., higher risk weights for moving electric vehicles at night), different elevator locations (e.g., high violation rates in specific buildings), or different user groups, to achieve more refined risk assessment. Another approach is to introduce weight adjustment based on prediction uncertainty. When the model has high uncertainty regarding a particular risk assessment, a more conservative weight strategy can be automatically triggered, or more modal data can be sought for validation.

[0055] Based on the risk assessment results, a corresponding tiered response mechanism is triggered.

[0056] As mentioned above, this step involves mapping the aforementioned risk assessment results into specific, tiered response actions, constructing a complete "perception-decision-execution" closed loop. The core of this tiered response mechanism lies in establishing a dynamic mapping relationship between risk levels and response measures. Its design follows the "proportionality principle," meaning the intensity of response measures is proportional to the risk level, ensuring a balance between the effectiveness of the response and the user experience. Specifically, the system predefines multiple risk level thresholds (e.g., low risk, medium risk, high risk) and configures differentiated response strategy combinations for each level. These strategies include not only immediate on-site interventions (e.g., voice alarms, device control) but also subsequent management processes (e.g., information push notifications, record archiving). The triggering of the response mechanism is not a simple conditional judgment but an intelligent decision-making process that can be dynamically adjusted based on real-time context. For example, for the same risk level, the response level may be automatically upgraded during nighttime hours, or stronger measures may be taken against repeat offenders. Furthermore, the system establishes a response effectiveness evaluation mechanism, verifying the effectiveness of the response by collecting post-response scenario change data (e.g., whether the target has left, whether the risk has been eliminated), providing feedback for subsequent strategy optimization.

[0057] For example, when the system identifies a resident pushing a registered electric bicycle into the elevator based on multimodal features, it might classify it as "low risk," triggering a Level 1 response: a voice prompt will announce, "For elevator safety, please do not bring electric bicycles into the elevator car," and a friendly reminder will be displayed on the interface. If the user ignores this and the vehicle remains in the car for more than a preset time, the system will upgrade the risk level to "medium risk," triggering a Level 2 response: an alarm message containing time, location, and image data will be automatically pushed to the property management personnel's terminal, and local video recording will be initiated. If the system further detects an abnormally high battery temperature through infrared data, the risk level will be immediately upgraded to "high risk," triggering a Level 3 response: while maintaining the voice alarm, the elevator door closing will be delayed, the emergency ventilation system will be activated, and an emergency response notification will be sent to the community security center. This layered and progressive response method ensures the effectiveness of security control while avoiding overreaction.

[0058] It should be noted that, in specific implementation scenarios, the response strategy library can incorporate more collaborative handling methods based on the above solutions. For example, it can be linked with building access control systems to temporarily restrict elevators from stopping on the same floor when high-risk conditions are detected; or it can interact with community charging station management systems to guide users to designated parking areas. The response threshold can be designed with an adaptive adjustment mode, dynamically optimizing the triggering conditions for each risk level based on historical handling effect data, such as appropriately lowering the high-risk threshold during certain periods to improve sensitivity. Furthermore, a reinforcement learning-based response strategy optimizer can be introduced to autonomously learn the optimal combination of response strategies with the goal of maximizing long-term safety benefits. Another approach is to establish a cross-elevator collaborative response network. When a particular elevator frequently exhibits a specific risk pattern, preventative warning measures can be initiated in adjacent elevators in advance.

[0059] According to one embodiment of this application, the user identity information is obtained by reading the UID of the elevator card and associating it with the homeowner information in the property management system, wherein the homeowner information includes registered electric vehicle information.

[0060] As described above, the process of obtaining the user's identity information is as follows: When a user operates the elevator card, the system reads the unique identification code (UID) stored in the card through a card reader. This UID serves as a key index, and is used for secure matching and association queries with the resident information database pre-established in the property management system.

[0061] The property management system's resident information database contains comprehensive resident profiles, including a dedicated field for electric vehicle registration information. These fields record specific details about the electric vehicles registered under a resident's name, including but not limited to the number of vehicles, vehicle type (such as electric bicycles, balance scooters, etc.), and vehicle brand and model.

[0062] By establishing an accurate correspondence between UIDs and homeowner information, the system can obtain the identity background of current elevator users and the status of their electric vehicles in real time. This association mechanism provides important contextual information for subsequent risk assessment, enabling the system to combine visual recognition results with actual user registration information to more accurately identify and assess the risks of vehicles entering the elevator.

[0063] According to one embodiment of this application, the intelligent recognition model performs feature fusion by employing a multimodal Transformer structure, and its visual feature extraction branch adopts an improved YOLOv7 model and introduces an attention mechanism.

[0064] As described above, the intelligent recognition model employs a multimodal Transformer structure as its core fusion framework. This structure, through its inherent cross-attention mechanism, achieves deep interaction and semantic alignment between features from different modalities. Specifically, visual features, deep features, audio features, and user features are mapped to a unified feature space, where features from any modality can serve as query vectors to actively retrieve and aggregate relevant key information from other modalities.

[0065] In this model, the visual feature extraction branch adopts an improved YOLOv7 model architecture. The improvement mainly lies in the embedding of an attention mechanism module, which calculates the importance weights of different spatial locations in the feature map, enabling the model to adaptively focus on key regions related to elevator risk identification, such as discriminative feature regions like the battery area and wheel structure of an electric bicycle, while suppressing background interference information.

[0066] Through this structural design, the model can not only retain the unique characteristics of each modality, but also establish cross-modal semantic associations during the fusion stage, ultimately generating a unified multimodal feature representation rich in contextual information.

[0067] According to one embodiment of this application, the adaptive adjustment of the dynamic weight vector includes:

[0068] Based on historical risk assessment results, calculate the contribution of each characteristic dimension to the high-risk determination result;

[0069] Based on the contribution, the weight values ​​of each feature in the dynamic weight vector are updated.

[0070] As described above, the adaptive adjustment process of the dynamic weight vector specifically includes the following two steps:

[0071] First, the system performs attribution analysis on historical risk assessment results, calculating the contribution of each feature dimension to the high-risk determination result through model interpretability techniques. Specifically, the system collects multimodal feature data from historical high-risk events and uses methods such as gradient backpropagation or class activation mapping to quantify the relative importance of each feature dimension in the final risk determination decision.

[0072] Secondly, the system iteratively updates the dynamic weight vector based on the calculated contribution. For feature dimensions that have shown a high contribution in historical high-risk assessments, their weight values ​​in the weight vector are increased accordingly; while for feature dimensions with consistently low contributions, their weights are appropriately reduced. This update process is implemented through preset weight update rules, ensuring that the weight allocation accurately reflects the actual discriminative ability of each feature dimension.

[0073] Through the aforementioned adjustment mechanism, the dynamic weight vector can continuously optimize itself as the system operates, enabling the risk assessment model to continuously adapt to changes in user behavior patterns and scenarios, thereby improving the accuracy and adaptability of risk identification.

[0074] According to one embodiment of this application, the hierarchical response mechanism includes at least:

[0075] Level 1 response, triggering a voice prompt;

[0076] A level-two response sends an alarm message to the property management system.

[0077] A Level 3 response will be implemented, including delaying elevator door closing and activating emergency ventilation.

[0078] As described above, the hierarchical response mechanism specifically includes three progressively escalating response levels:

[0079] The Level 1 response is activated when a low-risk situation is identified. It outputs preset warning voice messages through the voice broadcasting equipment deployed in the elevator car to provide initial reminders and dissuasion to the user.

[0080] The Level 2 response is triggered by medium-risk events. The system automatically generates structured alarm information containing the event time, location, and image evidence, and pushes the information to the property management's monitoring platform or mobile terminal in real time through the network transmission interface.

[0081] A Level 3 response is activated immediately upon determining a high risk. On one hand, it extends the door closing delay time by controlling the elevator door operator system. On the other hand, it activates forced ventilation by linking the emergency ventilation device in the car. Simultaneously, it maintains voice alerts and sends the highest priority notification to security personnel.

[0082] According to one embodiment of this application, after triggering the tiered response mechanism, the method further includes:

[0083] Based on the response effect data, the weight allocation strategy in the intelligent identification model and / or risk assessment process is optimized.

[0084] As described above, after triggering the tiered response mechanism, the method further includes a system optimization process based on the response effect data. The system continuously collects scenario state data after the response action is executed, including whether the target behavior has terminated, whether the risk has been eliminated, and whether false alarms or missed alarms have occurred. This response effect data, together with the corresponding multimodal features and risk assessment results, constitutes the training samples.

[0085] Based on these samples, the system is optimized in two ways: for the intelligent recognition model, the collected sample data, especially misjudgment and missed judgment cases, are used to fine-tune and update the network parameters of the model to improve the accuracy of feature extraction and fusion; for the risk assessment process, the weight distribution of each feature in the weight vector is dynamically adjusted according to the difference in contribution of feature dimensions in correct and incorrect judgments to strengthen the decision influence of key features.

[0086] This optimization process forms a complete closed-loop learning system, enabling the intelligent recognition model and risk assessment mechanism to continuously iterate and improve based on actual operational results.

[0087] A second aspect of this application provides an intelligent identification and handling device for multimodal risks in elevators, comprising:

[0088] The data acquisition module is used to acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information.

[0089] The feature processing module is used to extract and fuse features from the multi-source data based on the intelligent recognition model to obtain fused multimodal features;

[0090] The risk assessment module is used to perform risk assessment based on the multimodal features and generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculation on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0091] The decision response module is used to trigger the corresponding tiered response mechanism based on the risk assessment results.

[0092] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the first aspects above.

[0093] Figure 2An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logical instructions in the memory 830 to execute the method in any of the embodiments of the first aspect described above, the method including:

[0094] Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information;

[0095] Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features;

[0096] Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0097] Based on the risk assessment results, a corresponding tiered response mechanism is triggered.

[0098] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0099] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to perform the methods provided by the above methods, the method comprising:

[0100] Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information;

[0101] Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features;

[0102] Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0103] Based on the risk assessment results, a corresponding tiered response mechanism is triggered.

[0104] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided by the above methods, the method comprising:

[0105] Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information;

[0106] Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features;

[0107] Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results;

[0108] Based on the risk assessment results, a corresponding tiered response mechanism is triggered.

[0109] Example 2

[0110] This embodiment aims to construct an elevator safety system capable of sensing, cognition, decision-making, and self-evolution. Its core lies in breaking away from the traditional assembly-line processing model and introducing a dynamic feedback loop, enabling the system's risk assessment, data collection, and feature extraction capabilities to continuously optimize based on actual operational results.

[0111] 1. Enhanced multi-source data acquisition and semantic association

[0112] Based on existing video, depth, and audio data, identity and contextual data are deeply integrated:

[0113] Identity data: The elevator card reader system reads the user card UID and performs a secure match with the property database (containing homeowner information, the number of registered electric vehicles, vehicle models, etc.) to create a user profile vector. .

[0114] Contextual data: introduce timestamp t (to distinguish between peak hours and nighttime), elevator load w, and operating status s (going up, going down, or stopped).

[0115] Data synchronization and correlation: All data streams are time-aligned via a hardware synchronization chip and synchronized with timestamp t and elevator. Use it as the key to construct a unified data frame .

[0116] 2. Fine-grained multimodal feature extraction and fusion

[0117] 2.1 Visual Features :

[0118] The improved YOLOv7-Elevator model is used, and its loss function incorporates Focal Loss to address the class imbalance issue between electric vehicles, children's vehicles, and luggage.

[0119]

[0120] Embedding the CBAM attention module in the model's neck allows the model to focus on key areas such as the battery and wheel structure, outputting fine-grained classification results and bounding boxes.

[0121] 2.2 Deep Features :

[0122] Based on the ToF depth map, the 3D point cloud of the target object is calculated, and its minimum bounding cube is fitted to obtain the physical size features (L, W, H).

[0123] Construct a size-type prior library, and output a type confidence vector by calculating the Mahalanobis distance between the current target size and the size distribution of each category (electric bicycle, balance bike, children's bike) in the library.

[0124] 2.3 Audio Features :

[0125] Using a CRNN model, the audio signal is converted into a Mel spectrogram. Local features are extracted through convolutional layers, and then temporal dependencies are learned by RNN (such as LSTM) layers to finally identify specific events (such as motor humming or alarm sounds).

[0126] 2.4 User and Context Features :

[0127] User profiles (UU) and scene data (t, w, s) are embedded and encoded to form structured feature vectors. .

[0128] 2.5 Multimodal Feature Fusion:

[0129] A Transformer fusion model based on cross-attention is employed. As a query, , , As both Key and Value, visual features can "question" information from other modalities. For example, when visual classification is ambiguous, they can proactively retrieve information from user features, such as "whether the householder has registered an electric vehicle," to aid decision-making. The fused unified feature representation is as follows: .

[0130] 3. Dynamic Risk Assessment and Weight Adaptive Mechanism

[0131] 3.1 Risk Assessment Model:

[0132] Construct a multi-task learning neural network, whose input is... The output layer contains three tasks:

[0133] Task_det: Object detection (classification + localization)

[0134] Task_act: Behavior recognition (e.g., moving objects, prolonged stay).

[0135] Task_risk: Risk assessment (a scalar score R between 0 and 1)

[0136] The total loss function is: Through joint training, the three tasks can share features and promote each other.

[0137] 3.2 Feature Contribution Analysis and Dynamic Weight Adjustment:

[0138] Attribution algorithms such as SHAP or Integrated Gradients are used to back-calculate each risk assessment result R, thus calculating each input feature. The contribution of "battery visual characteristics", "number of electric vehicles owned by users", and "motor audio characteristics" to the risk score. .

[0139] Maintain a dynamic weight vector Its initial value is the benchmark weight. .

[0140] Weight update rule: The weights of each feature are dynamically adjusted based on its historical contribution in recent assessments (e.g., the past 100 assessments). Features that consistently contribute highly in high-risk (R>0.7) events should have their weights increased.

[0141]

[0142] Where η is the learning rate. Simultaneously, weight normalization is performed to ensure... .

[0143] Updated weights

[0144] This will be used in the next batch of risk assessment calculations, allowing the system to increasingly focus on those characteristics that truly indicate high risk.

[0145] 4. Intelligent Decision-Making and Perception - Decision-Making Closed Loop

[0146] 4.1 Tiered Response Mechanism: Dynamically triggered based on Risk Score (RR):

[0147] 0.3≤R<0.6 (Low to medium risk): Voice prompt.

[0148] 0.6≤R<0.8 (Medium-high risk): Alarm is sent to property management, and video recording is initiated and uploaded.

[0149] R≥0.8 (High Risk): Delay closing the door, activate emergency ventilation, and simultaneously increase the sampling frequency of the local depth sensor and infrared thermal imager.

[0150] 4.2 Perception-Decision Closed Loop: Decision results guide data acquisition strategies, forming a closed loop. For example, when the system frequently enters high-risk states, edge devices will automatically adjust: "To reduce response latency, temporarily disable high-energy-consuming audio feature extraction and concentrate computing power on visual and depth modal analysis." This is an adaptive perception optimization based on resource constraints and real-time risk levels.

[0151] 5. Continuous model evolution and feedback optimization closed loop

[0152] 5.1 Feedback Collection: The system records the "quadruple" for each event: (multimodal data, predicted risk value, actions taken, and final result). The final result is either manually confirmed by property management or automatically inferred based on whether a failure occurs within a subsequent period.

[0153] 5.2 Model Optimization:

[0154] Weight calibration based on the confusion matrix: The confusion matrix of the model is periodically statistically analyzed. If an increase is found in cases where "children's electric vehicles" are misclassified as "electric bicycles," a penalty term for that category pair is automatically added to the loss function, or the global weight of visual features is reduced during feature fusion. And relatively improve user characteristics The weight (because user information may help distinguish).

[0155] Reinforcement learning optimizes decision-making strategies: The Proximal Policy Optimization (PPO) algorithm models the decision-making process as a Markov decision process. The agent, based on the state (fused features), optimizes the decision-making strategy. The system selects an action (response level) and updates its decision policy π(a|s) based on the final result (successful blocking is a positive reward, and false alarms are a negative reward), thereby achieving long-term optimization of the response policy.

[0156] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0157] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0158] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for intelligent identification and handling of multimodal risks in elevators, characterized in that, include: Acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information; The depth information is acquired by a depth sensor and is used to provide the three-dimensional spatial dimensions and positional relationship of the target object; Based on the intelligent recognition model, feature extraction and fusion are performed on the multi-source data to obtain fused multimodal features; Risk assessment is performed based on the multimodal features to generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculations on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results; Based on the risk assessment results, a corresponding tiered response mechanism is triggered; The user identity information is obtained by reading the elevator card's UID and associating it with the homeowner information in the property management system. The homeowner information includes registered electric vehicle information. The intelligent recognition model uses a multimodal Transformer structure for feature fusion, and its visual feature extraction branch adopts an improved YOLOv7 model and introduces an attention mechanism. The adaptive adjustment of the dynamic weight vector includes: Based on historical risk assessment results, calculate the contribution of each characteristic dimension to the high-risk determination result; Based on the contribution, update the weight values ​​of each feature in the dynamic weight vector; The graded response mechanism includes at least: Level 1 response, triggering a voice prompt; A level-two response sends an alarm message to the property management system. A Level 3 response includes delaying elevator door closing and activating emergency ventilation. After triggering the tiered response mechanism, the method further includes: Based on the response effect data, the weight allocation strategy in the intelligent identification model and / or risk assessment process is optimized.

2. An intelligent identification and handling device for multimodal risks in elevators, used to implement the intelligent identification and handling method for multimodal risks in elevators as described in claim 1, characterized in that, include: The data acquisition module is used to acquire multi-source data in the elevator operating environment, including visual data, depth information, audio data, and user identity information. The feature processing module is used to extract and fuse features from the multi-source data based on the intelligent recognition model to obtain fused multimodal features; The risk assessment module is used to perform risk assessment based on the multimodal features and generate risk assessment results; wherein, during the risk assessment process, a dynamic weight vector is used to perform weighted calculation on each feature dimension, and the dynamic weight vector can be adaptively adjusted according to historical risk assessment results; The decision response module is used to trigger the corresponding tiered response mechanism based on the risk assessment results.

3. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 1.

4. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method as described in claim 1.

Images