Method and system for monitoring behaviors in elevator

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, and provides a method and system for monitoring behaviors in an elevator. The method comprises the steps that an original frame image of an elevator is input into a target detection model, a target detection result is obtained, the target detection result comprises a target detection frame, a region of interest is determined based on the detection frame, the region of interest is input into a resolution reconstruction model, a high-resolution image is output, and the high-resolution image is the region of interest with the enhanced resolution. Fusing the region of interest with enhanced resolution with the original frame image to obtain a fused image, performing histogram equalization on the fused image to generate a target frame image, inputting the target frame image into a behavior recognition model to obtain a behavior recognition result, and obtaining a behavior monitoring result according to a target detection result and the behavior recognition result. Through automatic target detection and behavior recognition, an accurate, efficient and robust elevator monitoring solution can be provided.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology, and in particular to a method and system for monitoring behavior in an elevator. Background Art

[0002] Traditional video surveillance systems have obvious limitations in elevator safety monitoring. These video surveillance systems often rely on a single technical means, such as using only target detection technology or behavior recognition technology, which leads to problems such as low recognition accuracy or high missed detection rate. Summary of the invention

[0003] The embodiments of the present application provide a method and system for monitoring behavior in an elevator, which can solve the problem of inaccuracy of the method for monitoring behavior in an elevator in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a method for monitoring behavior in an elevator, comprising: Get the original frame image of the elevator; Inputting the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target; Determining a region of interest based on a detection frame of the target; Inputting the region of interest into a preset resolution reconstruction model to output a high-resolution image; wherein the high-resolution image is a region of interest with enhanced resolution; The resolution-enhanced region of interest is merged with the original frame image, and the original resolution of the original frame image of the non-interest region is retained to obtain the merged image; wherein the non-interest region is the region in the original frame image other than the region of interest; Performing histogram equalization on the fused image to generate the target frame image; Inputting the target frame image into a preset behavior recognition model to obtain a behavior recognition result; A behavior monitoring result is obtained according to the target detection result and the behavior recognition result.

[0005] Optionally, the loss function of the target detection model is for: ;in, represents the classification loss; represents the positioning loss; represents the occlusion loss.

[0006] Optionally, the method further includes: Updating the region of interest based on a preset formula; The preset formula is: ;in, is the region of interest for the next frame of image, is the region of interest of the current frame image, is the offset caused by the target movement.

[0007] Optionally, the determining the region of interest based on the detection frame of the target includes: Determine a scene state based on the original frame image; If the scene state is a complex scene, the step of determining the region of interest based on the detection frame of the target is performed.

[0008] In a second aspect, an embodiment of the present application provides an elevator behavior monitoring system, including: An acquisition module, used for acquiring the original frame image of the elevator; A target detection module, used to input the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target; A determination module, configured to determine a region of interest based on a detection frame of the target; a resolution reconstruction module, configured to input the region of interest into a preset resolution reconstruction model and output a high-resolution image; wherein the high-resolution image is the region of interest with enhanced resolution; fuse the region of interest with enhanced resolution with the original frame image, retain the original resolution of the original frame image of the region of non-interest, and obtain the fused image; wherein the region of non-interest is the region of the original frame image other than the region of interest; perform histogram equalization on the fused image to generate the target frame image; A behavior recognition module, used to input the target frame image into a preset behavior recognition model to obtain a behavior recognition result; The result output module is used to obtain the behavior monitoring result according to the target detection result and the behavior recognition result.

[0009] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for monitoring behavior in an elevator as described in any one of the first aspects above is implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for monitoring behavior in an elevator described in any one of the first aspects above is implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer device, the computer device executes the elevator behavior monitoring method described in any one of the above-mentioned first aspects.

[0012] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by processing the original frame image of the elevator using a preset target detection model, the target (such as a person, electric vehicle, battery, etc.) in the image can be accurately identified to generate a target detection result. The region of interest is determined based on the target detection result, and the key part of the image, that is, the area where the target is located, can be focused on. The resolution of the region of interest is reconstructed, the region of interest is input into the resolution reconstruction model, and a high-resolution image is output. The high-resolution image is the region of interest with enhanced resolution. The region of interest with enhanced resolution is fused with the original frame image to obtain a fused image. The fused image is subjected to histogram equalization, and a higher-quality target frame image can be generated, which helps to improve the accuracy of behavior recognition. The higher-quality target frame image after resolution reconstruction is input into the behavior recognition model, and the target behavior can be more accurately identified. Combining the target detection results and the behavior recognition results, a more comprehensive behavior monitoring result can be obtained. Through automated target detection and behavior recognition, an accurate, efficient and robust elevator monitoring solution can be provided.

[0013] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 It is a flowchart of a method for monitoring behavior in an elevator provided in an embodiment of the present application; Figure 2 It is a structural diagram of an elevator behavior monitoring system provided by an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, systems, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0017] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0018] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0019] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0020] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0022] With the continuous advancement of smart city construction, elevators, as an important part of urban vertical transportation, are increasingly used and relied on. Elevator safety issues have also become the focus of public attention, especially when electric vehicles, batteries and other items that may cause safety risks enter the elevator. These issues become more prominent. These items may not only damage elevator equipment, but also cause serious safety accidents such as fires, threatening the lives of passengers. Therefore, it is particularly important to develop an elevator monitoring system that can monitor in real time and effectively identify improper behavior. Such an elevator monitoring system can detect and respond to potential safety threats in a timely manner, thereby ensuring the safe operation of the elevator.

[0023] However, traditional video surveillance systems have obvious limitations in elevator safety monitoring. These video surveillance systems often rely on a single technical means, such as using only target detection technology or behavior recognition technology, which leads to problems such as low recognition accuracy and high missed detection rate.

[0024] In order to solve the above problems, this application proposes a behavior monitoring method in an elevator, which aims to improve the recognition accuracy of specific behaviors by integrating multimodal technologies such as target detection, behavior recognition and resolution reconstruction, ensure the safe operation of the elevator, and meet the needs of modern intelligent monitoring.

[0025] Figure 1 A schematic flow chart of a method for monitoring behavior in an elevator provided by the present application is shown.

[0026] S101, obtaining an original frame image of an elevator.

[0027] The video stream of the elevator is obtained, and then the video stream is decoded to obtain a video frame (ie, an original frame image).

[0028] The step of obtaining the video stream of the elevator includes: obtaining the real-time video stream in the elevator by configuring a camera or by a video stream service (such as a real-time streaming protocol RTSP, a real-time messaging protocol RTMP, etc.).

[0029] S102, inputting the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target.

[0030] The target detection results include: the target detection box, category label and confidence level. Targets include but are not limited to: electric vehicles, batteries and people. Each detected target is assigned a category label, which indicates the predicted target category.

[0031] Among them, the preset target detection model is preferably the YOLOv5 model. The YOLOv5 model is an ideal choice for elevator monitoring scenarios due to its efficient real-time detection capability and high accuracy. The YOLOv5 model is based on deep learning technology and can quickly identify and locate targets in complex backgrounds.

[0032] Among them, before the original frame image is input into the target detection model, the original frame image needs to be preprocessed, including but not limited to: image size adjustment, image normalization processing, etc.

[0033] Image resizing: The YOLOv5 model usually requires the input image to be of size 640x640, so the original frame image needs to be scaled to that size.

[0034] Image normalization: Normalize the pixel values ​​in the original frame image to ensure that the pixel values ​​of the original frame image are between 0 and 1, which meets the input requirements of the YOLOv5 model.

[0035] Through the above preprocessing, the input quality of the target detection model can be ensured.

[0036] Among them, the target detection model performs target detection through the following algorithm formula: ;in, Refers to the input image, that is, the original frame image after preprocessing; Refers to the target category predicted by the target detection model, such as identifying electric vehicles in an image; is the conditional probability, indicating that given an input image In the case of target category, the target detection model The predicted probability of represents the activation function; Represents the weight matrix of the target detection model, which are learned during the training process; Represents the bias term, which is also a parameter learned during training and is used to adjust the output of the target detection model.

[0037] Among them, the target detection model training process specifically includes the following contents: Data collection and annotation: Collect images containing electric vehicles and batteries. Annotate the collected images and mark the locations of electric vehicles and batteries. Usually, a true box (also called annotated box or true annotated box) is used to annotate the location and size of the target.

[0038] Data format conversion: Convert the labeled images into YOLO format.

[0039] Model configuration: Configure the YOLOv5 model, including specifying the model structure, number of categories, etc.

[0040] Model training: Use the prepared dataset to train the YOLOv5 model. During the training process, the model will learn how to identify electric vehicles and batteries from images.

[0041] Set the detection threshold: The detection threshold is a hyperparameter that determines the confidence of the model's prediction. A detection box will only be retained if the model's confidence in the detection box is higher than the detection threshold.

[0042] Non-maximum suppression (NMS): NMS is a technique for reducing duplicate detection frames. NMS reduces duplicate detection by selecting the detection frame with the highest confidence and removing other detection frames that overlap with it.

[0043] Parameter adjustment and optimization: During the training process, hyperparameters such as learning rate, batch size, and number of iterations need to be adjusted to optimize the performance and detection accuracy of the model.

[0044] Model evaluation and deployment: After training is complete, the model is evaluated to check its performance on unseen data. Once the evaluation results meet the preset requirements, the model is deployed to actual applications for real-time detection of targets (e.g., electric vehicles, batteries, etc.).

[0045] Optionally, confidence in target detection models usually refers to the model's certainty or prediction probability of its prediction results. The formula is as follows: ;in, is the confidence of the i-th detection box; is the predicted probability of the target category in the detection box; The intersection-over-union ratio between the detection box and the true box. When it is greater than the set detection threshold, the detection boxes that meet this condition will be retained, and other detection boxes below the detection threshold will be discarded.

[0046] Optionally, in multi-target detection, multiple overlapping detection frames are often generated. These frames may represent different detection results of the same target. In order to remove duplicate frames, NMS is used to merge overlapping detection frames. The specific steps are as follows: Calculate the IoU value of each pair of detection frames. The IoU formula is ; Where A and B represent two different detection frames; ∩ represents the intersection, and ∪ represents the union. From all the detection frames, select the detection frame with the highest confidence and use it as the retained frame. For each detection frame that has not been deleted, calculate the IoU with the retained frame. If the IoU exceeds the set detection frame threshold, delete the detection frame. Repeat the above steps until all detection frames are processed. Through NMS, duplicate detection frames can be effectively removed and only one optimal detection frame can be retained, thereby improving the accuracy and efficiency of detection.

[0047] In an optional embodiment, in order to further improve the accuracy and robustness of target detection, the loss function of the target detection model is improved in this application. The loss function of the traditional YOLOv5 model is mainly composed of the classification loss , positioning loss On this basis, occlusion loss is introduced , in order to solve the problem of decreased detection accuracy when the target is blocked in complex environments.

[0048] Among them, the loss function of the target detection model is for: ;in, represents the classification loss; represents the positioning loss; represents the occlusion loss.

[0049] In the embodiment of the present application, the improved loss function consists of three parts: .in, Classification Loss , which is used to measure the model's prediction accuracy for the target category. The cross entropy loss is calculated using the formula: ,in, It is The true category label of each target; is the predicted probability of the target class.

[0050] Positioning loss , which is used to measure the deviation between the detection box and the real box. The generalized IoU (GIoU) ​​is used as the metric. ;in, and are the coordinates of the detection box and the true box respectively.

[0051] Occlusion Loss , designed for complex scenes in elevators, and mainly used to solve the detection problem when electric vehicles or batteries are partially blocked. The calculation formula is: ;in, is the occlusion coefficient, ranging from [0,1], which is used to control the weight of the occlusion loss; is the target importance function, which indicates the importance of the current target and is defined as: , is the area of ​​the real box, is the area of ​​the entire image; is the occlusion degree function, which indicates the degree to which the target is occluded and is defined as: , is the area of ​​the obscured part.

[0052] Among them, by introducing occlusion loss The improved YOLOv5 target detection model can better adapt to the target occlusion in the elevator scene. During the training process, for samples with high occlusion, the occlusion coefficient is appropriately increased. , in order to enhance the robustness of the model for such scenarios. The model optimization steps are as follows: Data enhancement: Add simulated occlusion samples to the training set to improve the model's generalization ability for occlusion scenes.

[0053] Parameter adjustment: Dynamically adjust according to the occlusion sample ratio The value range of (for example, when the occlusion is severe, it is set to 0.8, and when the occlusion is slight, it is set to 0.4).

[0054] Convergence analysis of loss function: In each iteration, monitor , and , ensuring that the optimization direction of the loss function is in line with expectations.

[0055] S103: Determine a region of interest based on the detection frame of the target.

[0056] The detection box in the above text is expressed in the form of a rectangle. The coordinates are expressed as: ; , Indicates the center coordinates of the detection box; Indicates the width of the detection box; Indicates the height of the detection frame. According to the detection frame, the positions of the electric vehicle and the battery are marked as regions of interest (ROI). These regions of interest will be used as the core areas for subsequent processing to ensure that the target is focused and avoid processing background information that is not related to the target, thereby improving processing efficiency.

[0057] Among them, the coordinates of the upper left corner and the lower right corner of the area of ​​interest can be calculated by the following formula:

[0058] in,( , ) represents the coordinates of the upper left corner of the region of interest, ( , ) represents the coordinates of the lower right corner of the region of interest. The above formula ensures that the region of interest is a region that only contains the target.

[0059] In an optional embodiment, in order to adapt to the dynamic changes of electric vehicles and batteries in the elevator, the target detection model will re-detect and adjust the region of interest in each frame. This dynamic adjustment ensures that the position of the target can be updated in real time, so that no matter how the target moves, important information will not be missed, thereby improving the effectiveness of overall monitoring. The region of interest is updated based on a preset formula; the preset formula is ;in, is the region of interest of the next frame image (it can also be called the region of interest after the current frame image is updated). is the region of interest of the current frame image, is the offset caused by the target movement; Refers to the index of the time frame, such as, can represent the current frame, and The dynamic adjustment ensures the real-time tracking of moving targets and the real-time update of the region of interest.

[0060] S104, inputting the region of interest into a preset resolution reconstruction model, and outputting a high-resolution image; wherein the high-resolution image is a resolution-enhanced region of interest; fusing the resolution-enhanced region of interest with the original frame image, retaining the original resolution of the original frame image of the non-interest region, to obtain the fused image; wherein the non-interest region is the region in the original frame image other than the region of interest; performing histogram equalization on the fused image, and generating the target frame image.

[0061] The above S104 can be summarized as reconstructing the resolution of the region of interest to generate a target frame image.

[0062] The resolution reconstruction is super-resolution reconstruction, that is, reconstructing a low-resolution image into a high-resolution image. The definitions of low in a low-resolution image and high in a high-resolution image refer to the prior art.

[0063] Among them, super-resolution reconstruction aims to reconstruct a high-resolution image from one or more low-resolution images. Its basic goal is to improve the spatial resolution of the image and make the details of the image clearer and richer.

[0064] In the embodiment of the present application, a super resolution residual network (SRResNet) is selected as the resolution reconstruction model. SRResNet is a deep learning model based on a residual network, which has good performance and can effectively restore the high-frequency details of the image. The formula is as follows: ;in, For the output high-resolution image; is the input image (here is a low-resolution image, i.e. the region of interest above); is the output of the residual block, representing the high-frequency information recovered from it; is the weight parameter of the residual block in the network.

[0065] Before super-resolution reconstruction, the region of interest needs to be preprocessed, including adjusting the image size, normalizing, and converting the data format, etc., to meet the input requirements of the resolution reconstruction model. Specifically: adjust the region of interest (also called the region of interest image) to the input size required by the resolution reconstruction model (such as 128×128), and perform pixel value normalization so that the pixel value range is between 0 and 1. The normalization formula is as follows: ;in, is the pixel value of the region of interest, is the normalized pixel value. This preprocessing step ensures that the input image meets the input requirements of the resolution reconstruction model, thereby improving the image reconstruction effect.

[0066] After that, the preprocessed region of interest is input into the resolution reconstruction model, and super-resolution reconstruction is performed using the forward propagation mechanism of the resolution reconstruction model. The resolution reconstruction model will analyze and process the input image through the features learned during its training process to generate a high-resolution image (the high-resolution image is the resolution-enhanced region of interest).

[0067] After the high-resolution image is generated, the high-resolution image needs to be post-processed. The resolution-enhanced region of interest is fused with the original frame image (the original frame image here is preferably the pre-processed original frame image), and the original resolution of the original frame image of the non-interest region is retained to obtain the fused image; wherein the non-interest region is the other region in the original frame image except the interest region.

[0068] Here, retaining the original resolution of the original frame image of the non-interest region means that the resolution of the non-interest region does not change, and is the same as the resolution of the original frame image.

[0069] Afterwards, the fused image is post-processed, such as using a histogram equalization technique, to increase the contrast between the region of interest and the region of no interest in the fused image, making the details more prominent.

[0070] Among them, the formula for histogram equalization is: ;in, is the pixel value of the original frame image, and are the minimum and maximum pixel values ​​of the original frame image, respectively. is the pixel value after equalization.

[0071] In the embodiment of the present application, the reconstructed and post-processed target frame images will be used as input to the behavior recognition module. The detailed information contained in these target frame images will help in subsequent recognition and improve the accuracy and reliability of recognition.

[0072] S105, inputting the target frame image into a preset behavior recognition model to obtain a behavior recognition result.

[0073] Among them, the behavior recognition model adopts the SlowFast network model. The SlowFast model can be used to capture the spatiotemporal features in the video for behavior recognition. There are two key components in the SlowFast network model: the slow path SlowPath and the fast path Fast Path. They are responsible for processing different information in the video respectively. Slow Path focuses on capturing spatial semantic information in the video. It runs at a lower frame rate and can capture key information provided by images or sparse frames. It is usually used to analyze static content in the video. Fast Path is used to capture rapidly changing behaviors. It runs at a higher frame rate and has a high temporal resolution, and can capture dynamic content in the video, such as details of fast motion.

[0074] Among them, the target area in the target frame image input to the behavior recognition model has been marked.

[0075] In an optional embodiment, in order to solve the problem of large computing resource consumption of the SlowFast model when processing high-resolution images, the present application introduces a scene state analysis scheme. The scene state is determined based on the original frame image; wherein the scene state includes complex scenes and non-complex scenes; if the scene state is a complex scene, the step of determining the region of interest based on the detection frame of the target is executed; if the scene state is a non-complex scene, the non-super-resolution processing strategy, the time step adjustment strategy and the frame selection strategy are executed.

[0076] Among them, non-complex scenes refer to scenes with little static or dynamic changes (such as only one static person or an electric vehicle without significant movement, etc.). Complex scenes refer to scenes with fast movements or complex behaviors in the elevator (such as pushing an electric vehicle into the elevator or multiple people interacting with each other).

[0077] Among them, there is no super-resolution processing strategy: the above original frame image is directly input into the SlowFast model, eliminating the resolution reconstruction stage.

[0078] Time step adjustment strategy: Increase the time step of Slow Path and Fast Path to reduce the number of frames processed per second. For example, adjust the time step of Slow Path from the default 8 to 16, and the time step of Fast Path from 2 to 4.

[0079] Frame selection strategy: Reduce the amount of computation by reducing the frequency of frame sampling, while optimizing performance without significantly affecting the accuracy of behavior recognition.

[0080] Among them, for complex scenes, it is necessary to optimize computing resource consumption while ensuring high-precision recognition, and resolution reconstruction strategies and model optimization strategies are needed.

[0081] Resolution reconstruction strategy: only perform resolution reconstruction on key frames and regions of interest, reducing high-resolution enhancement of the entire video frame.

[0082] Model optimization strategy: Reduce the computational complexity by reducing the number of network layers or channels. For example, the number of Slow Path channels is reduced from 64 to 32, and the number of Fast Path channels is reduced from 128 to 64.

[0083] Among them, let the time sequence of the video frame be T, the spatial feature be S, and for the Slow Path path, the time step is Larger, for Fast Path, the time step Small. Then the processing of the SlowFast model can be expressed as:

[0084]

[0085] in, Indicates the result after Slow Path processing; Indicates the result after Fast Path processing; Conv3D represents a three-dimensional convolution operation, which is used to process the spatiotemporal features in the video; and are the time steps of Slow Path and Fast Path respectively; represents the input video frame; S represents the spatial feature.

[0086] The characteristics of these two paths enable the SlowFast model to capture both long-span background information and rapidly changing behavior information in the time dimension, thereby more accurately identifying the target's behavior in the elevator.

[0087] The SlowFast model extracts spatiotemporal features from the video by performing convolution operations on spatiotemporal data, and recognizes each behavior through a classification network. The output of the model is expressed as: ;in, Represents a video frame given an input Under the condition of The probability distribution of Represents the weight matrix of the spatiotemporal convolutional layer. In the spatiotemporal convolutional network, these weights are used to learn spatiotemporal features from video frames; Represents the video frame Features obtained after spatiotemporal convolution processing; Represents the activation function.

[0088] S106, obtaining a behavior monitoring result according to the target detection result and the behavior recognition result.

[0089] Among them, according to the target detection results and behavior recognition results, combined with the preset rules, it is judged whether there is any violation or other abnormal situation that needs to be alarmed in the elevator. The specific implementation steps are as follows: Behavior classification and target association: In the behavior recognition stage, the behavior recognition model classifies the behavior of the target (preferably a person) in the elevator and outputs the behavior category and confidence level of each target. For example, the behavior recognition model may output behavior categories such as "pushing an electric vehicle" or "taking a battery". Combined with the target detection results (including the location information of the electric vehicle and battery, etc.), the behavior category is associated with the target to determine whether a violation has occurred.

[0090] For each detected target, the behavior is determined according to the following formula: ; Among them, B represents the final behavior monitoring result (illegal behavior or normal behavior); A is the behavior category output by the SlowFast model, which includes the behavior label of each target (such as "pushing an electric car") and the confidence of the behavior; G is the target detection result (including the location information of the electric car or battery and the area of ​​interest, etc.).

[0091] For ease of understanding, an example is given here. If a person is detected pushing an electric bike in an elevator, and the electric bike is located in the area of ​​interest (i.e., the location of the electric bike meets the preset rules), the following logic can be used to determine that the behavior is in violation: The meaning of this logic is: if A identifies "pushing an electric car" and G shows that the electric car is within the elevator's area of ​​interest, then B will be judged as a violation.

[0092] Among them, the preset rules are a series of standards or conditions used to determine whether a behavior violates the rules. They combine the target detection results and behavior recognition results to determine whether a violation has occurred or an abnormal situation that requires an alarm has occurred.

[0093] Here are a few examples of preset rules: Electric vehicle entering the elevator: The preset rule can be defined as "If an electric vehicle is detected in the elevator, and the location information of the electric vehicle is within the interest area of ​​the elevator, it is judged as a violation." Carrying flammable items: If the target detection results include flammable items (such as gasoline barrels), and the behavior recognition model identifies the behavior of "carrying flammable items", combined with the preset rules, it can be judged as a violation.

[0094] Elevator overload: The preset rules can set the maximum load weight or number of people in the elevator. If the target detection result shows that the number of people or total weight in the elevator exceeds this limit, it will be judged as a violation.

[0095] Smoking in the elevator: If the behavior recognition model identifies the "smoking" behavior and the target detection result shows that the smoker is in the elevator, this will violate the preset rules and be judged as a violation.

[0096] Pet is not on a leash: If the target detection result shows that there is a pet in the elevator, and the behavior recognition model does not recognize the "leash" behavior, this may violate the preset rules and be judged as a violation.

[0097] Inappropriate behavior: If the behavior recognition model identifies inappropriate behavior (such as fighting, harassing others), and the target detection results show that these behaviors occur in the elevator, this will violate the preset rules and be judged as a violation.

[0098] Optionally, use the following logic to determine whether the behavior is illegal: . Among them, A is the behavior category output from the SlowFast model; G is the target detection result (including the location information of the electric vehicle or battery and the area of ​​interest, etc.); C is the behavior context information, analyzing the environment and conditions in which the behavior occurs to determine whether the behavior is reasonable; T is the time series information, considering the time series of the behavior, analyzing whether the behavior conforms to the normal pattern; S is the behavior intensity assessment result, assessing the intensity of the behavior, such as the strength and speed of pushing the electric vehicle, to determine whether the behavior is abnormal. C, T and S are combined with A and G to form a comprehensive feature vector. A multi-layer neural network is used to process this comprehensive feature vector, and the hidden layer of the network can learn the complex relationship between these features. The output layer of the network gives the final behavior monitoring result B, indicating whether the behavior is illegal. In this way, the behavior in the elevator can be more comprehensively understood and judged, and the accuracy and reliability of behavior recognition can be improved.

[0099] When a violation is identified, an alarm mechanism will be triggered and the event information will be recorded in the database. This ensures that when a violation occurs, a timely response can be made and provides a basis for subsequent analysis and processing. The specific implementation steps are as follows: Trigger alarm mechanism: define corresponding alarm conditions for different types of violations. For example, when electric vehicles enter the elevator area or prohibited items appear, these behaviors will be automatically judged as violations and the alarm mechanism will be triggered.

[0100] Event storage: When a violation is identified and an alarm is triggered, relevant event information will be recorded and stored in the database immediately. Relevant event information includes: violation type (such as electric vehicles entering the elevator, objects falling, personnel not behaving in accordance with regulations), time and location of the violation (record the specific timestamp of the violation, and the elevator number or floor information where the violation occurred), and store the relevant event information in the database.

[0101] In the embodiment of the present application, the above method effectively improves the problems of low accuracy and missed detection of behavior recognition in elevator monitoring, improves the real-time recognition capability of specific behaviors, and ensures the safety and effectiveness of elevator use.

[0102] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0103] Figure 2 A structural block diagram of an elevator behavior monitoring system provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0104] Reference Figure 2 , the elevator behavior monitoring system includes: An acquisition module, used for acquiring the original frame image of the elevator; A target detection module, used to input the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target; A determination module, configured to determine a region of interest based on a detection frame of the target; a resolution reconstruction module, configured to input the region of interest into a preset resolution reconstruction model and output a high-resolution image; wherein the high-resolution image is the region of interest with enhanced resolution; fuse the region of interest with enhanced resolution with the original frame image, retain the original resolution of the original frame image of the region of non-interest, and obtain the fused image; wherein the region of non-interest is the region of the original frame image other than the region of interest; perform histogram equalization on the fused image to generate the target frame image; A behavior recognition module, used to input the target frame image into a preset behavior recognition model to obtain a behavior recognition result; The result output module is used to obtain the behavior monitoring result according to the target detection result and the behavior recognition result.

[0105] Among them, the elevator behavior monitoring system is the video monitoring system mentioned above.

[0106] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with external devices through a network connection. When the computer program is executed by the processor, a method for monitoring behavior in an elevator is implemented.

[0108] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a computer device is provided, comprising: a memory, a processor, and an elevator behavior monitoring method program stored in the memory and executable on the processor, wherein the elevator behavior monitoring method program implements the steps of the elevator behavior monitoring method described in any of the above embodiments when executed by the processor.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a program for monitoring behavior in an elevator is stored. When the program for monitoring behavior in an elevator is executed by a processor, the steps of the method for monitoring behavior in an elevator as described in any of the above embodiments are implemented.

[0111] In one embodiment, a computer program product is provided, the computer program product comprising an elevator behavior monitoring method program, and when the elevator behavior monitoring method program is executed by a processor, the steps of the elevator behavior monitoring method described in any of the above embodiments are implemented.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for monitoring behavior in an elevator, characterized in that: include: Get the original frame image of the elevator; Inputting the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target; Determining a region of interest based on a detection frame of the target; Inputting the region of interest into a preset resolution reconstruction model to output a high-resolution image; wherein the high-resolution image is a region of interest with enhanced resolution; The resolution-enhanced region of interest is merged with the original frame image, and the original resolution of the original frame image of the non-interest region is retained to obtain the merged image; wherein the non-interest region is the region in the original frame image other than the region of interest; Performing histogram equalization on the fused image to generate the target frame image; Inputting the target frame image into a preset behavior recognition model to obtain a behavior recognition result; A behavior monitoring result is obtained according to the target detection result and the behavior recognition result.

2. The method for monitoring behavior in an elevator according to claim 1, characterized in that: The loss function of the target detection model for: ;in, represents the classification loss; represents the positioning loss; represents the occlusion loss.

3. The method for monitoring behavior in an elevator according to claim 1, characterized in that: The method further comprises: Updating the region of interest based on a preset formula; The preset formula is: ;in, is the region of interest for the next frame of image, is the region of interest of the current frame image, is the offset caused by the target movement.

4. The method for monitoring behavior in an elevator according to claim 1, characterized in that: The determining of the region of interest based on the detection frame of the target comprises: Determine a scene state based on the original frame image; If the scene state is a complex scene, the step of determining the region of interest based on the detection frame of the target is performed.

5. An elevator behavior monitoring system, characterized in that: include: An acquisition module, used for acquiring the original frame image of the elevator; A target detection module, used to input the original frame image into a preset target detection model to obtain a target detection result; wherein the target detection result includes a detection frame of the target; A determination module, configured to determine a region of interest based on a detection frame of the target; a resolution reconstruction module, configured to input the region of interest into a preset resolution reconstruction model and output a high-resolution image; wherein the high-resolution image is the region of interest with enhanced resolution; fuse the region of interest with enhanced resolution with the original frame image, retain the original resolution of the original frame image of the region of non-interest, and obtain the fused image; wherein the region of non-interest is the region of the original frame image other than the region of interest; perform histogram equalization on the fused image to generate the target frame image; A behavior recognition module, used to input the target frame image into a preset behavior recognition model to obtain a behavior recognition result; The result output module is used to obtain the behavior monitoring result according to the target detection result and the behavior recognition result.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A computer program product, characterized in that When the computer program product is executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 4.

Images