Abnormal behavior detection method and device based on four-feature fusion
By preprocessing, target detection, and tracking the elevator car video stream, and combining hand key point information to calculate multiple confidence levels, the problems of misjudgment and missed judgment in the detection of abnormal behavior in elevator cars are solved, achieving higher detection accuracy and reliability.
Patent Information
- Application Number
- CN202511531061.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-24
- Publication Date
- 2025-12-12
AI Technical Summary
Existing methods for detecting abnormal behavior in elevator cars suffer from misjudgments and omissions, especially in densely populated areas where it is difficult to accurately identify abnormal behavior, resulting in a high false alarm rate.
An abnormal behavior detection method based on four-feature fusion is adopted. By acquiring the video stream of the elevator car, preprocessing, target detection and tracking are performed. Combined with the hand key point localization and passenger geometric information, the confidence of gesture grasping, confidence of hand attribution and direction consistency are calculated to determine whether the hand is pointing to the waist warning area and the duration. The total confidence is calculated to determine the abnormal behavior.
It effectively reduces false alarms caused by accidental proximity and non-malicious contact, and improves the accuracy and reliability of abnormal behavior detection.
Smart Images
Figure CN121121870A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and particularly relates to an abnormal behavior detection method and device based on four-feature fusion, a storage medium and an electronic device. BACKGROUND
[0002] As an important facility for vertical transportation, elevators affect many people's lives. However, as a relatively closed space, the elevator car is prone to become a place where abnormal behaviors are committed by lawbreakers. Especially in places with a large number of people such as shopping malls and hospitals, abnormal behaviors in the elevator car occur frequently, seriously endangering the property safety of the public.
[0003] Traditional elevator monitoring and detection methods mainly rely on a template library, and this method is prone to misjudgment and missed judgment. Although some methods currently use artificial intelligence algorithms based on computer vision to identify behaviors in videos, these algorithms mostly use 3D neural networks to directly extract features from videos, which has problems such as complex calculation, complex network framework, and high requirements for hardware devices.
[0004] Currently, some methods identify abnormal behaviors through image processing, but this method still has the problem of insufficient accuracy in identification, and is prone to false positives when facing normal interactive behaviors. SUMMARY
[0005] The present application provides an abnormal behavior detection method and device based on four-feature fusion, and a storage medium and an electronic device, which can improve the accuracy of abnormal behavior identification.
[0006] The present application provides an abnormal behavior detection method based on four-feature fusion, comprising: Obtaining a passenger video stream in an elevator car, decoding the passenger video stream to obtain video frames, and preprocessing the video frames; Performing target detection on the preprocessed video frames to obtain multiple categories of targets; Performing target tracking on the multiple categories of targets to form target motion trajectories and mark target identities; Performing key point positioning and feature extraction on the target passengers to obtain hand information and passenger geometric information; the hand information includes hand center and hand detail information, and the passenger geometric information includes passenger geometric center; Calculating a hand gesture grabbing confidence based on the hand detail information; Calculating a hand attribution confidence based on the hand center and the passenger geometric center; The waist warning area is determined based on passenger geometric information, the hand actual motion direction is obtained by spatio-temporal kinematic analysis on the hand motion trajectory, the direction consistency weight between the hand actual motion direction and the theoretical direction is calculated, and it is judged whether the hand points to the waist warning area and the duration after entering the waist warning area; The total confidence is calculated based on the gesture grabbing confidence, the hand attribution confidence, the direction consistency weight and the duration, and when the total confidence exceeds the confidence threshold, it is determined that the abnormal behavior occurs.
[0007] Further, according to the above abnormal behavior detection method based on four feature fusion, wherein the plurality of categories of targets include target objects and target passengers; Target tracking is performed on the plurality of categories of target objects, target motion trajectories are formed, and target identities are marked, including: Feature extraction and recognition are performed on each target object and target passenger to obtain initial position information of the target object and the target passenger; The position of the target in the current frame is predicted based on Kalman filtering and matched and updated; The identity and current position information of all tracked target passengers and target objects are recorded to obtain target motion trajectories.
[0008] Further, according to the above abnormal behavior detection method based on four feature fusion, wherein the hand detail information includes finger flexion, palm opening degree and thumb opposition angle The gesture grabbing confidence is calculated based on the hand detail information, which is calculated by the following formula:
[0009] Wherein g represents the grabbing gesture grabbing confidence, u represents the finger bending degree normalization, a represents the palm opening degree score, and s represents the thumb opposition angle score.
[0010] Further, according to the above abnormal behavior detection method based on four feature fusion, wherein the hand attribution confidence is calculated based on the hand center and the passenger geometric center, including: The normalized distance of the hand center and the passenger geometric center is calculated, which is represented by the following formula:
[0011] Wherein t represents the normalized distance of the hand center and the passenger geometric center, r represents the actual spatial distance, represents the maximum value function, w represents the width of the passenger in the image, and h represents the height of the passenger in the image; The hand attribution confidence is calculated based on the normalized distance, which is represented by the following formula:
[0012] wherein f represents hand attribution confidence, and t represents normalized distance between hand center and passenger geometric center.
[0013] Further, according to the abnormal behavior detection method based on four-feature fusion, wherein the direction consistency weight between the actual motion direction of the hand and the theoretical direction is calculated, comprising: calculating the cosine similarity between the actual motion direction of the hand and the theoretical direction towards the target, and obtaining the direction consistency weight based on the cosine similarity.
[0014] Further, according to the abnormal behavior detection method based on four-feature fusion, wherein the total confidence is calculated based on the gesture grabbing confidence, the hand attribution confidence, the direction consistency weight and the duration, and the total confidence is calculated by the following formula:
[0015] wherein Z represents the total confidence, d represents the direction consistency weight, and c represents the time duration.
[0016] Further, according to the abnormal behavior detection method based on four-feature fusion, wherein the method further comprises: judging whether the hand points to the waist warning area by calculating the included angle between the actual motion direction of the hand and the ideal direction pointing to the waist.
[0017] The application also provides an abnormal behavior detection device based on four-feature fusion, comprising: an acquisition and preprocessing module, configured to acquire a passenger video stream in an elevator car, decode the passenger video stream to obtain video frames, and pre-process the video frames; a target detection module, configured to perform target detection on the pre-processed video frames to obtain targets of multiple categories; a target tracking module, configured to perform target tracking on the targets of multiple categories to form target motion trajectories and mark target identities; an abnormal behavior analysis module, configured to perform key point positioning and feature extraction on target passengers to obtain hand information and passenger geometric information; the hand information comprises a hand center and hand detail information, and the passenger geometric information comprises a passenger geometric center; and calculate a gesture grabbing confidence based on the hand detail information; and calculate a hand attribution confidence based on the hand center and the passenger geometric center; And, based on passenger geometric information, a waist warning area is determined, a spatiotemporal kinematic analysis is performed on the hand movement trajectory to obtain an actual hand movement direction, a direction consistency weight between the actual hand movement direction and a theoretical direction is calculated, and it is judged whether the hand is pointing to the waist warning area and the duration after entering the waist warning area. And, based on the gesture grabbing confidence, the hand attribution confidence, the direction consistency weight and the duration, a total confidence is calculated, and when the total confidence exceeds a confidence threshold, it is determined that an abnormal behavior occurs.
[0018] The application also provides a computer readable storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute any one of the above abnormal behavior detection methods based on four feature fusions.
[0019] The application also provides an electronic device, which comprises a processor and a memory, the processor is electrically connected with the memory, the memory is used for storing instructions and data, and the processor is used for the steps in any one of the above abnormal behavior detection methods based on four feature fusions.
[0020] The application provides an abnormal behavior detection method, device, storage medium and electronic device based on four feature fusions, the application obtains video frames by framing a passenger video stream, pre-processes the video frames, and then detects targets such as people, wallets and mobile phones; tracks the recognized targets to obtain the movement trajectories of the targets; then obtains the results of gesture grabbing confidence, hand attribution confidence, direction consistency and time duration; calculates a total confidence according to the results of gesture grabbing confidence, hand attribution confidence, direction consistency and time duration and corresponding weights; if the total confidence is greater than or equal to a threshold value, it is determined that an abnormal behavior occurs, otherwise it is normal. The application effectively reduces false positives caused by accidental approaching and non-malicious contact, and improves the accuracy and reliability of abnormal behavior detection. BRIEF DESCRIPTION OF DRAWINGS
[0021] The technical scheme and other beneficial effects of the application will be apparent from the following detailed description of the specific embodiments of the application with reference to the accompanying drawings.
[0022] Figure 1 The flowchart of the abnormal behavior detection method based on four feature fusions provided by the embodiments of the application.
[0023] Figure 2 The schematic diagram of the waist warning area provided by the embodiments of the application.
[0024] Figure 3 The structure schematic diagram of the abnormal behavior detection device based on four feature fusions provided by the embodiments of the application.
[0025] Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0026] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.
[0027] The embodiment of the present application provides an abnormal behavior detection method and device based on four-feature fusion, a storage medium and an electronic device. The abnormal behavior detection device based on four-feature fusion provided by the embodiment of the present application can be integrated in an electronic device, which can be a terminal, a server or the like. The terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box or other devices.
[0028] Please refer to Figure 1 , Figure 1 The flowchart of the abnormal behavior detection method based on four-feature fusion provided by the embodiment of the present application is applied to an electronic device. It should be noted that the abnormal behavior in the present application refers to the pickpocketing behavior in an elevator car. The abnormal behavior detection method based on four-feature fusion includes the following steps: S1, obtaining a passenger video stream in an elevator car, decoding the passenger video stream to obtain a video frame, and pre-processing the video frame.
[0029] Specifically, video stream data in the elevator car is collected from monitoring devices installed at four corners of the top of the elevator car. In a specific embodiment, the monitoring device uses a high-definition network camera with a resolution of 1920x1080.
[0030] In a specific embodiment, the extraction of the video frame can be realized by using the OpenCV library.
[0031] In one embodiment, the pre-processing of the video frame in step S1 includes: performing resolution normalization processing on the video frame, so that the resolutions of all video frames are unified to 640x480; performing brightness and contrast robustness processing on the video frame to enhance the definition and detail performance of the image. A histogram equalization algorithm is used to adjust the brightness and contrast threshold of the image; noise suppression processing is performed on the video frame to improve the definition and detail performance of the image. Morphological filtering is used with a filter kernel size of 5x5 or 7x7.
[0032] The resolution normalization processing, the histogram equalization algorithm, and the morphological filtering can be realized through the OpenCV library.
[0033] S2, target detection is performed on the preprocessed video frame to obtain a plurality of categories of targets.
[0034] Specifically, a YOLOv8 algorithm based on deep learning is used to detect targets in the video frame, and all targets that can be "people" in the picture are framed out.
[0035] S3, target tracking is performed on the plurality of categories of targets to form target motion trajectories and mark target identities.
[0036] The plurality of categories of targets include target objects and target passengers.
[0037] In an embodiment, step S2 includes the following steps: S21, feature extraction and recognition are performed on each target object and target passenger to obtain initial position information of the target objects and target passengers.
[0038] Each target is assigned a unique identity for subsequent behavior analysis, for example, in the style of person (ID: 1), person (ID: 2), and person (ID: 3).
[0039] S22, the position of the current frame of the target is predicted based on Kalman filtering and matched and updated.
[0040] Specifically, for each passenger (such as Zhang San with ID 1) that has been tracked in the previous frame, the tracker will predict the position information of this frame based on the speed and direction of each passenger before. The tracker will pair all the detection boxes that are recognized as passengers in this frame with all the positions it predicts. The pairing criteria are: how close is the position? (IoU), how similar is it? (ReID feature, DeepSORT).
[0041] Update: Once a successful pairing is found (for example, it is found that frame A is Zhang San with ID 5), the tracker will use the real position of the detection box to correct the prediction just made, making the trajectory smoother and more accurate.
[0042] S23, the identity and current position information of all tracked target passengers and target objects are recorded to obtain target motion trajectories.
[0043] If a target does not appear for several consecutive frames, the tracker considers that the target has left the picture and stops tracking the target. The ID corresponding to the target is temporarily not used.
[0044] Store all the information of the current tracking target in a list, traverse the list, get the ID and current position of each tracked target. Then, store the center point position of each target passenger (the center point of the solid rectangular frame) in the dictionary. Thus, each ID corresponds to a line composed of historical position points, which is his motion trajectory.
[0045] S4, key point positioning and feature extraction are performed on the target passenger to obtain hand information and passenger geometric information; the hand information includes hand center and hand detail information, and the passenger geometric information includes passenger geometric center.
[0046] Specifically, step S4 includes the following steps: Based on the MediaPipe library, 21 key point positions of the target passenger's hand are located. A key point detection model based on a convolutional neural network is used, and the key points include finger tips, finger roots, wrists, and palm centers (i.e., hand centers); The hand detail information includes finger flexion, palm opening, and thumb opposition angle information, the finger flexion range is pixels, the joint angle range is 0°-180°, the palm opening range is pixels, and the thumb opposition angle range is 0°-90°.
[0047] In specific embodiments, the judgment method of the finger flexion is as follows: the finger bending is judged by two methods. The first method is to see if the position of the finger tip is lower than the joint in the middle of the finger, if so, it is considered that the finger is bent, and finally the number of fingers that are bent in this way is counted. The second method is to directly calculate the real angle formed by the three consecutive joints (palm connection, middle joint, and finger tip joint) of the finger. The smaller the angle, the more bent the finger is.
[0048] In specific embodiments, the judgment method of the palm opening is as follows: the distances between the finger tips of the index finger, middle finger, ring finger, and little finger are calculated, and then an average distance value is calculated. The larger the average distance, the more open the palm is; the smaller the distance, the more the palm is closing or has been clenched.
[0049] In specific embodiments, the judgment method of the thumb opposition is as follows: the average distance from the tip of the thumb to the tips of the other four fingers is calculated. If the average distance is very small, it means that the thumb is leaning towards the palm and cooperating with the other fingers to prepare for the "pinch" or "grab" action. The smaller the distance, the higher the degree of opposition.
[0050] S5, calculate the gesture grabbing confidence based on the hand detail information.
[0051] The gesture grabbing confidence is calculated by the following formula:
[0052] wherein g represents the grab gesture grabbing confidence, u represents the finger curvature normalization, a represents the palm opening degree score, and s represents the thumb opposition angle score.
[0053] It should be noted that the finger curvature normalization is obtained by normalizing the extracted finger flexion data, the palm opening degree score is obtained by comparing the extracted palm opening degree characteristic value with the preset opening degree threshold and normalizing in proportion, and the thumb opposition angle score is obtained by normalizing the extracted thumb opposition angle characteristic value with reference to the reference angle.
[0054] S6, a hand attribution confidence is calculated based on the hand center and the passenger geometric center.
[0055] A normalized distance of the hand center and the passenger geometric center is calculated, the normalized distance ranges from 0 to 1, and is represented by the following formula:
[0056] wherein t represents the normalized distance of the hand center and the passenger geometric center, and r represents the actual spatial distance. represents a maximum value function, w represents the width of the passenger in the image, and h represents the height of the passenger in the image.
[0057] A hand attribution confidence is calculated based on the normalized distance, and is represented by the following formula:
[0058] wherein f represents the hand attribution confidence, and t represents the normalized distance of the hand center and the passenger geometric center.
[0059] S7, a waist alert area is determined based on passenger geometric information, a hand actual motion direction is obtained by performing a space-time kinematics analysis on a hand motion trajectory, a direction consistency weight between the hand actual motion direction and a theoretical direction is calculated, and it is determined whether the hand points to the waist alert area and the duration after entering the waist alert area.
[0060] Figure 2 A waist alert area schematic diagram provided by the embodiment of the present application is shown in the figure, wherein the solid rectangular frame represents a person, and the dashed rectangular frame represents a waist alert area ROI.
[0061] In a specific embodiment, the waist alert area ROI is determined according to the passenger geometric information in proportion to the height, preferably in the height proportion interval of 40%–70%, and is expanded in proportion in the horizontal direction and the vertical direction.
[0062] In the sliding event window, the hand movement trajectory is analyzed in space-time motion, the actual hand movement direction is obtained, the direction consistency weight between the actual hand movement direction and the theoretical direction is calculated, and specifically, the direction consistency weight is obtained based on the cosine similarity between the actual hand movement direction and the theoretical direction towards the target.
[0063] In an embodiment, it is judged whether the hand points to the waist warning area, and the specific judgment standard is: The directions are extremely consistent: the angle between the "actual moving direction" and the "ideal direction pointing to the waist" is calculated. If the angle is very small (close to 0 degrees), it means that the hand is moving accurately towards the waist. (Note: The ideal direction pointing to the waist: the vector direction of the palm center pointing to the center of the warning area ROI, which is dynamically changing) The distance is shortening: not only the direction is correct, but also the position of the hand must be approaching the waist area. If the direction is correct, but the hand is far away, the weight will be lower; if the direction is correct, and the hand is very close to the waist, the weight will be sharply increased.
[0064] For example, if the motion end point of the hand falls into the waist area, the weight is 1.5. If the motion direction of the hand points to the waist area (the angle is less than 30 degrees), the weight is 1.2. Otherwise, the weight is 0.8.
[0065] Further, the method further comprises: performing space-time kinematic analysis on the hand trajectory in the sliding time window, calculating the average speed, the maximum speed relative sudden increase, the average acceleration, and the number of movement direction changes, and combining the depth direction change to determine whether it is a large swing.
[0066] Specifically, the hand trajectory is analyzed in time series. The time series length is 60 frames, and the sliding time window length is 0.5 seconds. The average speed, maximum speed change rate, acceleration, and direction change number of the trajectory are calculated. The speed range is 0-15 pixels / s, the acceleration range is -15-15 pixels / (s*s), and the direction change number range is 0-15 times. Based on the depth change information, it is determined whether it is a large swing behavior. The depth change amplitude range is -25-25 pixels.
[0067] S8, calculating the total confidence based on the gesture grabbing confidence, the hand attribution confidence, the direction consistency weight, and the duration, and when the total confidence exceeds the confidence threshold, determining that the abnormal behavior occurs.
[0068] The total confidence is calculated by the following formula:
[0069] Wherein, Z represents the total confidence, d represents the direction consistency weight, c represents the time duration, 0.3 is the weight of the grabbing gesture confidence in the total confidence, 0.25 represents the weight of the hand attribution confidence in the total confidence, 0.15 represents the weight of the direction consistency in the total confidence, and 0.3 represents the weight of the event duration in the total confidence.
[0070] When the total confidence exceeds the confidence threshold and meets the minimum duration threshold, it is determined that pickpocketing occurs.
[0071] According to the method described in the above embodiment, this embodiment will be further described from the perspective of the four-feature fusion-based abnormal behavior detection device, which can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, a server or the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices, etc.
[0072] Please refer to Figure 3 , Figure 3 The four-feature fusion-based abnormal behavior detection device provided by the embodiment of the application is specifically described, which is applied to an electronic device and can include: An acquisition and preprocessing module is configured to acquire a passenger video stream in an elevator car, decode the passenger video stream to obtain video frames, and pre-process the video frames; A target detection module is configured to perform target detection on the pre-processed video frames to obtain targets of multiple categories; A target tracking module is configured to perform target tracking on the targets of multiple categories to form target motion trajectories and mark target identities; An abnormal behavior analysis module is configured to perform key point positioning and feature extraction on a target passenger to obtain hand information and passenger geometric information; the hand information includes a hand center and hand detail information, and the passenger geometric information includes a passenger geometric center; and calculate a gesture grabbing confidence based on the hand detail information; and calculate a hand attribution confidence based on the hand center and the passenger geometric center; and determine a waist alert area based on the passenger geometric information, perform spatio-temporal kinematic analysis on a hand motion trajectory to obtain an actual hand motion direction, calculate a direction consistency weight between the actual hand motion direction and a theoretical direction, and determine whether the hand points to the waist alert area and the duration after entering the waist alert area; And, based on the gesture grasp confidence, the hand attribution confidence, the direction consistency weight, and the duration, a total confidence is calculated, and when the total confidence exceeds a confidence threshold, it is determined that the abnormal behavior occurs.
[0073] In a practical implementation, the above various modules and / or units can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various modules and / or units can refer to the method embodiments above. The beneficial effects that can be achieved can also refer to the beneficial effects in the method embodiments above, and will not be repeated here.
[0074] In addition, the embodiment of the present application also provides an electronic device, which can be a computer, a tablet computer and the like. The electronic device can implement the steps in any embodiment of the abnormal behavior detection method based on four-feature fusion provided by the embodiment of the present application, and thus can achieve the beneficial effects of any abnormal behavior detection method based on four-feature fusion provided by the embodiment of the present application. Details can be found in the above embodiments, and will not be repeated here.
[0075] Figure 4 The specific structure block diagram of the electronic device provided by the embodiment of the present application is shown, which can be used to implement the abnormal behavior detection method based on four-feature fusion provided in the above embodiments. The electronic device 500 can be a terminal, a server and the like, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a micro processing box, or other devices and the like.
[0076] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments, and the processor 580 can execute various functional applications and data processing by running the software programs and modules stored in the memory 520, i.e. to realize the functions of photographing by the front camera, processing the photographed images, switching the display color of the display content on the display screen and the like. The memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory 520 can further include a memory remotely arranged with respect to the processor 580, which can be connected to the electronic device 500 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.
[0077] The input unit 530 can be used to receive input digital or character information, and generate a keyboard, a mouse and the like related to user settings and function control. The display unit 540 can be used to display information input by a user or provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. The display unit 540 can include a display panel 541, which can optionally be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.
[0078] The electronic device 500 can help a user receive requests, send information, and the like through a transmission module 570 (e.g., a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown, it is understood that it does not necessarily constitute the electronic device 500 and can be omitted as needed without changing the essence of the application.
[0079] The processor 580 is the control center of the electronic device 500, connecting various parts of the entire mobile phone through various interfaces and lines, executing various functions of the electronic device 500 and processing data by running or executing software programs and / or modules stored in the memory 520 and calling data stored in the memory 520, thereby monitoring the entire electronic device. Optionally, the processor 580 can include one or more processing cores; in some embodiments, the processor 580 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It is understood that the above-mentioned modem processor can also not be integrated into the processor 580.
[0080] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to various components, and in some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can realize functions such as management of charging, discharging, and power consumption management. The power supply 590 can also include one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and any other components.
[0081] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, and the like, which are not described here in detail. In this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory and one or more programs stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include instructions for: Obtaining a passenger video stream in an elevator car, decoding the passenger video stream to obtain video frames, and pre-processing the video frames; Performing target detection on the pre-processed video frames to obtain targets of multiple categories; Performing target tracking on the targets of multiple categories to form target motion trajectories and mark target identities; Performing key point positioning and feature extraction on the target passengers to obtain hand information and passenger geometric information; the hand information includes hand center and hand detail information, and the passenger geometric information includes passenger geometric center; Calculating a gesture grabbing confidence based on the hand detail information; Calculating a hand attribution confidence based on the hand center and the passenger geometric center; Determining a waist warning area based on the passenger geometric information, performing spatio-temporal kinematic analysis on the hand motion trajectory to obtain an actual motion direction of the hand, calculating a direction consistency weight between the actual motion direction of the hand and a theoretical direction, and judging whether the hand points to the waist warning area and a duration after entering the waist warning area; Calculating a total confidence based on the gesture grabbing confidence, the hand attribution confidence, the direction consistency weight, and the duration, and determining that an abnormal behavior occurs when the total confidence exceeds a confidence threshold.
[0082] In specific implementation, the above various modules can be implemented as independent entities, or can be combined as the same or several entities, and the specific implementation of the above various modules can be referred to the method embodiments above, which will not be described here.
[0083] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor. Therefore, the embodiments of the present application provide a storage medium in which a plurality of instructions are stored, which can be loaded by a processor to execute the steps of any embodiment of the abnormal behavior detection method based on four feature fusion provided by the embodiments of the present application.
[0084] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0085] Since the instructions stored in the storage medium can execute the steps in any embodiment of the abnormal behavior detection method based on four-feature fusion provided by the embodiments of the present application, the beneficial effects that can be achieved by any abnormal behavior detection method based on four-feature fusion provided by the embodiments of the present application can be achieved. For details, see the foregoing embodiments, which will not be repeated here.
[0086] The above describes in detail the abnormal behavior detection method, device, storage medium and electronic equipment based on four-feature fusion provided by the embodiments of the present application. The principles and implementation manners of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.
Claims
1. An abnormal behavior detection method based on four-feature fusion, characterized in that, The method includes: The system acquires the passenger video stream inside the elevator car, decodes the passenger video stream to obtain video frames, and preprocesses the video frames. Target detection is performed on the preprocessed video frames to obtain targets of multiple categories; Track targets of multiple categories, generate target trajectories, and identify targets. Key point localization and feature extraction are performed on the target passenger to obtain hand information and passenger geometric information; the hand information includes the hand center and hand detail information, and the passenger geometric information includes the passenger geometric center; Calculate gesture grasping confidence based on detailed hand information; Calculate the hand attribution confidence score based on the hand center and the passenger geometric center; The waist warning zone is determined based on passenger geometric information. Spatiotemporal kinematic analysis of the hand movement trajectory is performed to obtain the actual direction of hand movement. The directional consistency weight between the actual direction of hand movement and the theoretical direction is calculated, and it is determined whether the hand is pointing to the waist warning zone and the duration after entering the waist warning zone. The total confidence score is calculated based on gesture capture confidence score, hand attribution confidence score, orientation consistency weight, and duration. When the total confidence score exceeds the confidence score threshold, abnormal behavior is determined to have occurred.
2. The abnormal behavior detection method based on four-feature fusion according to claim 1, characterized in that, The multiple categories of targets include target objects and target passengers; Track targets from multiple categories, generate target trajectories, and identify targets, including: Feature extraction and identification are performed on each target object and target passenger to obtain the initial location information of the target object and target passenger; Predict the target's position in the current frame based on Kalman filtering and then perform matching and updating. Record the identification and current location information of all tracked target passengers and objects to obtain the target's movement trajectory.
3. The abnormal behavior detection method based on four-feature fusion according to claim 2, characterized in that, The detailed hand information includes finger flexion, palm opening, and thumb opposition angle. The confidence score for gesture grasping is calculated based on detailed hand information using the following formula: Where g represents the grasping confidence score, u represents the normalized finger flexion score, a represents the palm opening score, and s represents the thumb-to-palm angle score.
4. The abnormal behavior detection method based on four-feature fusion according to claim 2, characterized in that, The confidence score for hand attribution is calculated based on the hand center and the passenger's geometric center, including: The normalized distance between the center of the hand and the geometric center of the passenger is calculated using the following formula: Where t represents the normalized distance between the center of the hand and the geometric center of the passenger, and r represents the actual spatial distance. Let w represent the width of the passenger in the image, and h represent the height of the passenger in the image. The confidence score for hand attribution is calculated based on the normalized distance and expressed by the following formula: Where f represents the confidence level of hand attribution, and t represents the normalized distance between the center of the hand and the geometric center of the passenger.
5. The abnormal behavior detection method based on four-feature fusion according to claim 1, characterized in that, Calculate the directional consistency weight between the actual and theoretical directions of hand movement, including: Calculate the cosine similarity between the actual direction of hand movement and the theoretical direction toward the target, and obtain the direction consistency weight based on the cosine similarity.
6. The abnormal behavior detection method based on four-feature fusion according to claim 1, characterized in that, The total confidence score is calculated based on gesture capture confidence, hand attribution confidence, orientation consistency weight, and duration, using the following formula: Where Z represents the total confidence level, d represents the directional consistency weight, and c represents the time duration.
7. The abnormal behavior detection method based on four-feature fusion according to claim 1, characterized in that, The method further includes: By calculating the angle between the actual direction of hand movement and the ideal direction pointing towards the waist, it can be determined whether the hand is pointing towards the waist warning area.
8. An abnormal behavior detection device based on four-feature fusion, characterized in that, include: The acquisition and preprocessing module is used to acquire the passenger video stream inside the elevator car, decode the passenger video stream to obtain video frames, and preprocess the video frames. The target detection module is used to detect targets in the preprocessed video frames to obtain targets of multiple categories; The target tracking module is used to track targets of multiple categories, generate target motion trajectories, and label target identities. The abnormal behavior analysis module is used to locate key points and extract features from target passengers to obtain hand information and passenger geometric information; the hand information includes the hand center and hand detail information, and the passenger geometric information includes the passenger geometric center; In addition, the confidence level of gesture grasping is calculated based on detailed hand information; In addition, the confidence level of hand attribution is calculated based on the hand center and the passenger geometric center; Furthermore, based on passenger geometric information, the waist warning area is determined, the spatiotemporal kinematic analysis of the hand movement trajectory is performed to obtain the actual direction of hand movement, the directional consistency weight between the actual direction of hand movement and the theoretical direction is calculated, and it is determined whether the hand is pointing to the waist warning area and the duration after entering the waist warning area. Furthermore, the total confidence score is calculated based on gesture capture confidence score, hand attribution confidence score, orientation consistency weight, and duration. When the total confidence score exceeds the confidence score threshold, abnormal behavior is determined to have occurred.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the abnormal behavior detection method based on four-feature fusion as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, The method includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the abnormal behavior detection method based on four-feature fusion as described in any one of claims 1 to 7.