Abnormal behavior detection method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an abnormal behavior detection method and device, equipment and a storage medium. The method comprises the steps that a trained target detection model is utilized, and based on video data collected in a target environment, recognition for an operator and / or a vehicle is executed; determining a motion track of the operator based on an identification result of the operator; determining whether a predetermined behavior is detected based on the movement track of the operator; if it is determined that the predetermined behavior is detected, an abnormal detection result of the predetermined behavior is determined based on the recognition result of the operator and / or the vehicle, and the abnormal detection result indicates whether the predetermined behavior is an abnormal behavior or not. Therefore, through the target detection model and the trajectory analysis technology, the abnormal behavior of the operator can be rapidly identified. And the spatial relationship is comprehensively considered based on the identification result, so that the condition of normal operation can be prevented from being identified as an abnormal behavior, and the misjudgment rate is effectively reduced.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of information technology, and in particular, to methods, devices, equipment, and computer-readable storage media for abnormal behavior detection. Background Art

[0002] With the continuous increase in logistics demand, the movement patterns of workers and vehicles in logistics park scenarios have become more complex, and safety management issues in logistics parks have become increasingly prominent. In particular, accurate monitoring and timely response to abnormal behaviors such as workers crossing the boundary of the area have become key challenges.

[0003] In related technologies, the detection of abnormal behavior of operators has problems such as low detection accuracy and high false alarm rate. It cannot effectively cope with the complex behavior patterns and dynamically changing environments of logistics parks, and has limitations in personnel and vehicle behavior analysis and anomaly identification.

[0004] Therefore, a more efficient and accurate method is needed for abnormal behavior detection. Summary of the invention

[0005] In a first aspect of the present disclosure, a method for abnormal behavior detection is provided. The method comprises: using a trained target detection model, based on video data collected in a target environment, performing recognition of an operator and / or a vehicle; based on the recognition result of the operator, determining the movement trajectory of the operator; based on the movement trajectory of the operator, determining whether a predetermined behavior is detected; and if it is determined that the predetermined behavior is detected, determining an abnormal detection result of the predetermined behavior based on the recognition result of the operator and / or the vehicle, the abnormal detection result indicating whether the predetermined behavior is an abnormal behavior.

[0006] In a second aspect of the present disclosure, a device for abnormal behavior detection is provided. The device includes: a recognition module configured to use a trained target detection model to perform recognition of an operator and / or a vehicle based on video data collected in a target environment; a trajectory determination module configured to determine the motion trajectory of the operator based on the recognition result of the operator; a first detection module configured to determine whether a predetermined behavior is detected based on the motion trajectory of the operator; and a second detection module configured to determine an abnormal detection result of the predetermined behavior based on the recognition result of the operator and / or the vehicle if the predetermined behavior is determined to be detected, and the abnormal detection result indicates whether the predetermined behavior is an abnormal behavior.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method according to the first aspect of the present disclosure.

[0010] It should be understood that the content described in this content section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0013] Figure 2 A flowchart showing a process of abnormal behavior detection according to some embodiments of the present disclosure;

[0014] Figure 3A A schematic diagram showing video data collected from a target environment according to some embodiments of the present disclosure;

[0015] Figure 3B A schematic diagram showing a marking operation area in some embodiments of the present disclosure;

[0016] Figure 3C A schematic diagram showing some embodiments of the present disclosure for determining anomaly detection results of a first video frame;

[0017] Figure 3D A schematic diagram showing a method of determining an abnormality detection result of a second video frame according to some embodiments of the present disclosure is shown;

[0018] Figure 3EA schematic diagram showing the determination of anomaly detection results in some embodiments of the present disclosure is shown;

[0019] Figure 3F A schematic diagram showing a process of training a target detection model according to some embodiments of the present disclosure;

[0020] Figure 4 A block diagram showing an apparatus for abnormal behavior detection according to some embodiments of the present disclosure; and

[0021] Figure 5 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0029] As briefly mentioned above, with the continuous increase in logistics demand, the movement patterns of workers and vehicles in logistics park scenarios have become more complex, and safety management issues in logistics parks have become increasingly prominent. In particular, accurate monitoring and timely response to abnormal behaviors such as workers crossing the work area have become key challenges.

[0030] For example, in a logistics park, the platform is an important interface and operation area for the handover of goods inside and outside the warehouse. The safety management of the platform is crucial to ensure the safety of front-line workers and goods. Usually, the platform area is divided into two areas, inside and outside the warehouse, with strict regulations on the paths and security inspection procedures for people entering and leaving the warehouse, aiming to prevent dangerous goods from entering the warehouse and prevent smuggled goods from leaving the warehouse. However, in actual operations, operators may choose to jump directly from the platform due to the short path or other factors, bypassing the formal safety management process. Although this behavior of crossing the boundary of the operation area may save time in the short term, it brings serious safety hazards and risks.

[0031] In order to effectively manage and prevent the safety and operational risks brought about by such abnormal behaviors, logistics parks can use cameras and other equipment to collect video data to monitor and detect such abnormal behaviors.

[0032] In the related art, there are mainly two types of methods for detecting abnormal behavior of operators:

[0033] Video classification method: Based on surveillance video, abnormal behavior detection is defined as a binary classification problem, that is, judging whether abnormal behavior occurs in the video. Deep learning models are usually used to predict whether there is platform jumping behavior in the video through training, and the output is a binary probability value. This method requires a large amount of labeled video data as a training data set, but the publicly available data sets are very limited, resulting in a large amount of manpower required for labeling. And because the output result is only a binary probability value, there is a lack of detailed explanation and understanding of the detection results, and it is easy to be disturbed by false alarms of similar actions, such as misjudging normal operating behavior on the platform as abnormal behavior.

[0034] Trajectory analysis method: Determine whether abnormal behavior has occurred through personnel detection and trajectory analysis. For example, the position relationship between the operator's action trajectory and the operating area is detected to determine whether the operator has crossed the boundary. However, this method requires real-time analysis and processing of a large amount of video data, which has high requirements for computing resources and algorithms. In addition, for scenes with standardized operating actions, such as operators entering the carriage from the platform to load and unload goods, false alarms are likely to occur.

[0035] Therefore, in order to address the problems faced by related technologies in abnormal behavior detection, such as large data requirements, high false alarm rates, and limited applicability, a more efficient and accurate solution is needed that can more accurately distinguish abnormal behaviors from normal operating behaviors, thereby effectively managing and ensuring the safety and efficiency of logistics parks.

[0036] In an embodiment of the present disclosure, a scheme for abnormal behavior detection is provided. According to an embodiment of the present disclosure, a trained target detection model is used to perform recognition of an operator and / or a vehicle based on video data collected in a target environment; based on the recognition result of the operator, a motion trajectory of the operator is determined; based on the motion trajectory of the operator, it is determined whether a predetermined behavior is detected; and if it is determined that the predetermined behavior is detected, an abnormal detection result of the predetermined behavior is determined based on the recognition result of the operator and / or the vehicle, and the abnormal detection result indicates whether the predetermined behavior is an abnormal behavior.

[0037] Therefore, through the target detection model and trajectory analysis technology, it is possible to monitor and analyze surveillance video frames in real time and quickly identify abnormal behaviors of operators. And at least based on the identification results of operators and vehicles, the spatial relationship and movement direction are comprehensively considered to avoid identifying normal operations as abnormal behaviors, thereby solving the problem that the method of determining whether a boundary is crossed based on a single trajectory may produce misjudgment due to technical limitations or the complexity of specific scenarios, and improving the detection accuracy.

[0038] Figure 1A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, after the electronic device 110 receives the video data 120 collected in the target environment, it performs recognition for the operator and / or the vehicle based on the video data 120 to obtain a recognition result (collectively or individually referred to as a recognition result 130). In some embodiments, the recognition result 130 may include a recognition result 130-1 for the operator and / or a recognition result 130-2 for the vehicle. In other embodiments, the recognition result 130 may also include recognition results for more, fewer, or different other objects in the target environment. The electronic device 110 may determine whether a predetermined behavior 140 is detected based on the recognition result 130. The electronic device 110 may also determine an abnormality detection result 150 for the predetermined behavior 140 based on the recognition result 130. The abnormality detection result 150 indicates whether the detected predetermined behavior 140 is an abnormal behavior.

[0039] In the environment 100, the electronic device 110 can be any type of device with computing capabilities, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0040] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

[0041] Figure 2 1 is a flowchart of a process 200 of video processing according to some embodiments of the present disclosure. The process 200 may be implemented in the environment 100, for example, the process 200 may be implemented at the electronic device 110. In the following, for ease of discussion, reference will be made to Figure 1 The process 200 is described below.

[0042] In block 210 , the electronic device 110 utilizes the trained target detection model to perform recognition of a worker and / or a vehicle based on video data collected in a target environment.

[0043] In some embodiments, the electronic device 110 obtains video data collected in the target environment and performs target recognition on the video data to detect various behaviors in the target environment. For example, the electronic device 110 can obtain video data collected by a camera installed at a key location in a logistics park (such as the edge of a platform, a cargo loading and unloading area, etc.).

[0044] In some embodiments, after the electronic device 110 acquires the video data collected in the target environment, it samples video frames from the video data at predetermined time intervals.

[0045] In some embodiments, the electronic device 110 may periodically sample video frames from the video stream at set time intervals for subsequent target detection and analysis.

[0046] In some embodiments, the electronic device 110 inputs each sampled video frame into a target detection model to determine a recognition result, where the recognition result at least indicates a bounding box for defining a worker and / or a bounding box for defining a vehicle in each video frame.

[0047] As an example, the electronic device 110 can extract a single video frame from a continuous video stream at a predetermined time interval (such as every second or every few seconds). Each sampled video frame is input into the trained target detection model, which means that a static picture is captured from the video stream for target recognition. Among them, the target detection model is based on a deep learning algorithm and can identify specific target objects such as workers and vehicles in the picture. After the target detection model processes the video frame, it outputs the recognition result, that is, outputs the position of the workers and vehicles identified in the video frame. Specifically, the recognition result may include bounding boxes of workers and vehicles, which can be used to define the position and size of the identified targets in the video frame.

[0048] In the disclosed embodiment, after acquiring the video data, the video frames are periodically sampled and input into the target detection model to identify the workers and vehicles in the video frames, and generate a bounding box for defining the target location, thereby providing basic data for subsequent abnormal behavior detection and analysis.

[0049] In some embodiments, due to the complexity of the target environment and the accuracy requirements of behavior detection, the electronic device 110 can obtain video data from different directions to complement each other, reduce missed detections or false detections due to occlusion, and improve the accuracy of target detection and behavior analysis.

[0050] In some embodiments, the video data collected in the target environment includes: first video data collected from the first direction to the target environment and / or second video data collected from the second direction to the target environment. In other embodiments, the video data may also include video data collected from more, fewer or different other directions in the target environment.

[0051] Figure 3A FIG. 3 is a schematic diagram showing video data 310 collected from a target environment according to some embodiments of the present disclosure. Figure 3A As shown, the video data may include first video data 311 collected from a first direction of the target environment and second video data 312 collected from a second direction of the target environment.

[0052] In some embodiments, the first direction refers to the horizontal axis direction in the picture of the target environment, and the second direction refers to the vertical axis direction in the picture of the target environment. In other embodiments, the first direction and / or the second direction may also be other directions for collecting video data from the target environment, such as a top-down direction, a diagonal direction, etc. The electronic device 110 may also receive video data collected from the target environment from a third, fourth, or other directions to perform abnormal behavior detection.

[0053] As an example, a device for collecting first video data from a first direction can be set on the side of a predetermined work area (such as a platform) to collect video data along the cargo loading and unloading channel or the direction of personnel passage. This setting can capture the lateral movement of personnel and vehicles on the side of the work area or the cargo loading and unloading area. By analyzing the video data in the first direction, it can be detected whether a person crosses from one side of the work area to the other side, and the trajectory of the person or vehicle during the lateral movement.

[0054] As another example, the device for collecting the second video data from the second direction can also be set in front of the working area (such as a platform) to collect video data along the direction of the goods passing through the predetermined working area or the personnel getting on and off the passage. This setting can capture the movement of people and vehicles in front of and behind the platform or in the vertical direction. By analyzing the video data in the second direction, it can be detected whether the operator crosses the predetermined working area or whether the cargo is loaded and unloaded through the normal area.

[0055] In some embodiments, the electronic device 110 can annotate the sampled video frames to define the area where the operator normally works in the picture. This step can clarify the monitoring range, so as to accurately determine whether there is abnormal behavior in subsequent recognition processing and behavior analysis.

[0056] As an example, Figure 3B FIG. 3 is a schematic diagram showing a marking operation area 320 of some embodiments of the present disclosure. Figure 3BAs shown, video frames in different directions may be annotated to obtain video frames that determine the annotated area, such as a first video frame 321 that determines the annotated area and a second video frame 322 that determines the annotated area.

[0057] In some embodiments, the electronic device 110 can receive annotated video frames to delineate the area where the operator normally works. Usually, the position of the device that obtains the video data is fixed, so only one manual calibration is required, and the electronic device 110 can delineate the area where the operator normally works based on the received annotated video frames. For example, the electronic device 110 determines the area boundary 321-1 and / or the area boundary 322-1 of the normal operation of the operator through the received annotated video frame 321 and / or the video frame 322.

[0058] In some embodiments, the electronic device 110 may also use a deep learning algorithm to automatically mark and demarcate the area where the operator normally works by predicting the depth difference of the ground. Because the working areas such as platforms, cargo storage areas, loading and unloading areas have obvious depth differences with the ground in the display of the video frame, the boundaries of these areas can be determined by using depth prediction algorithms and other methods.

[0059] Return to reference Figure 2 In box 220, the electronic device 110 determines the movement trajectory of the operator based on the recognition result of the operator.

[0060] In some embodiments, in order to monitor and analyze the behavior of the operator in real time so as to detect and identify whether abnormal behavior has occurred in a timely manner, the electronic device 110 can determine the movement trajectory of the operator through the bounding box of the operator output by the target detection model.

[0061] In some embodiments, the electronic device 110 may use a target tracking algorithm to determine the movement trajectory of the operator based on the bounding box of the operator indicated by the recognition result.

[0062] In some embodiments, the electronic device 110 can use Simple Online and Realtime Tracking (SORT) to determine the movement trajectory of the operator based on the bounding box of the operator. The SORT algorithm is a tracking algorithm based on Kalman filtering and Hungarian algorithm, which has the characteristics of low computational complexity and fast running speed. By associating the bounding boxes of the same operator in consecutive video frames, a path describing the movement trajectory is generated. This method does not rely on GPU devices and can achieve real-time tracking.

[0063] As an example, the target detection model processes the video frame, identifies the position of the operator, and generates a rectangular bounding box (detection box) that marks the position of the operator in the video frame. In the generated bounding box of the operator, the electronic device 110 can select the point in the lower right corner as the key point, and record the position of the lower right corner point of the personnel detection box in each frame of the video. Then, through the SORT algorithm or other tracking algorithms, the electronic device 110 associates the position of the point in each frame to obtain the motion trajectory of the operator. The lower right corner point is well representative of the overall motion trajectory of the operator, and can usually avoid partial occlusion of the personnel, especially in logistics operation environments (such as when personnel enter a car).

[0064] In the disclosed embodiment, the movement trajectory of the operator is determined by a target tracking algorithm, and the position of the operator can be updated and recorded in real time and continuously. Even when the operator moves at a fast speed or there is a brief occlusion, the continuity and accuracy of tracking can be maintained, thereby more accurately executing the process of determining the subsequent abnormal behavior detection results.

[0065] Return to reference Figure 2 In box 230, the electronic device 110 determines whether a predetermined behavior is detected based on the movement trajectory of the operator.

[0066] In some embodiments, the electronic device 110 can monitor in real time whether the operator performs a predetermined behavior by accurately tracking the operator's movement trajectory.

[0067] In some embodiments, the predetermined behavior, as a preliminary detection result, refers to the cross-border behavior of the operator crossing the preset normal operation area. If it is determined that the predetermined behavior is detected, the operator may have performed abnormal behavior that does not conform to the operation specification.

[0068] In some embodiments, abnormal behavior as the final detection result means that after excluding all normal operating conditions, the predetermined behavior is identified as a behavior that does not comply with the operating specifications. This is the result after further analysis and confirmation, indicating that the behavior does violate the operating specifications and may bring safety hazards.

[0069] In some embodiments, if it is determined that the motion trajectory overlaps with the first area marked in the video frame, the electronic device 110 can determine the positional relationship between the start and end points of the motion trajectory and the first area. If the positional relationship indicates that the start point of the motion trajectory is within the first area and the end point of the motion trajectory is outside the first area, the electronic device 110 determines that the predetermined behavior is detected.

[0070] In some embodiments, the electronic device 110 determines the motion trajectory of the operator based on the bounding box of the operator in each video frame. Then, the electronic device 110 can detect whether the motion trajectory of the operator overlaps with the first area, for example, a geometric algorithm can be used to determine whether the motion trajectory intersects the boundary of the first area. Through this detection, it can be determined whether the motion path of the operator crosses the boundary of the first area.

[0071] In some embodiments, false alarms may result from simply determining whether the motion trajectory overlaps with the first area. For example, the motion trajectory of a person may have an intersection with the first area, but in fact they may have just passed by the area boundary without actually crossing it. Therefore, the electronic device 110 can also determine whether a predetermined behavior has occurred by determining the positional relationship between the starting point and the end point of the motion trajectory and the first area. If the positional relationship indicates that the starting point of the motion trajectory is within the first area and the end point of the motion trajectory is outside the first area, it can be determined that the motion trajectory of the operator has actually crossed the boundary from the inside of the first area to the outside of the first area, thereby avoiding misjudgments due to only overlaps or intersections.

[0072] In some embodiments, the first area is marked as the area where the operator works in the target environment, that is, the electronic device 110 demarcates the area where the operator normally works through the deep learning algorithm and / or the video frame received and marked. The boundary of the first area in the video frame can be one or more line segments, which define the scope of the first area.

[0073] In the disclosed embodiment, by tracking the positional relationship between the operator's movement trajectory and the first area, it is possible to more accurately identify whether the operator has performed a predetermined behavior of crossing the first area, thereby effectively avoiding potential safety risks such as personal injury, cargo loss, etc.

[0074] In some embodiments, the predetermined behavior also includes some normal operation behaviors that comply with the operation specifications.

[0075] Return to reference Figure 2 In block 240 , if it is determined that a predetermined behavior is detected, the electronic device 110 determines an abnormal detection result of the predetermined behavior based on the identification result for the operator and / or the vehicle.

[0076] In some embodiments, after determining that a predetermined behavior has been detected, the electronic device 110 also needs to determine whether the predetermined behavior is a normal operation, thereby determining the abnormal detection result of the predetermined behavior. If only relying on the detection of predetermined behaviors, normal operating behaviors may be misjudged as abnormal behaviors. For example, in a logistics park scenario, operators enter the carriage from the platform area or load and unload goods on the pedals. These behaviors may cause personnel to cross the platform boundary, but these operations are in compliance with regulations. Misjudging this normal operation as abnormal behavior may lead to unnecessary intervention and increased management costs.

[0077] In some embodiments, the electronic device 110 determines whether a predetermined behavior occurs within a predetermined area of ​​the target environment based at least on the recognition result of the operator and / or the vehicle.

[0078] In some embodiments, the electronic device 110 may analyze the spatial relationship between the operator and the vehicle and the direction of movement of the operator based on the recognition results of the operator and / or the vehicle to determine whether the predetermined behavior is a normal operating condition.

[0079] In some embodiments, the predetermined behavior occurring in the predetermined area may refer to a situation where an operator enters a carriage to load and unload cargo.

[0080] In some embodiments, if the electronic device 110 determines that the recognition result indicates that a worker and a vehicle are recognized in the first video frame captured from the first direction, and the overlap ratio between the bounding box used to define the worker and the bounding box used to define the vehicle in the first video frame is greater than a threshold ratio, then it is determined that the predetermined behavior occurs within the predetermined area.

[0081] As an example, Figure 3C FIG. 3 is a schematic diagram showing a method of determining an abnormality detection result 331 of a first video frame in some embodiments of the present disclosure. Figure 3C As shown, in the first video frame acquired in the first direction, the operator may cross the boundary 331-1 from the platform area to enter the carriage to perform the predetermined behavior of loading and unloading cargo, and such behavior may be misjudged as abnormal behavior. Therefore, the electronic device 110 can determine the proportion of the intersection of the vehicle area and the human body area to the human body area through the operator's boundary box 331-2 and the vehicle's boundary box 331-3 in the recognition result. If the intersection ratio is greater than the set threshold (such as 0.75), the electronic device 110 can determine that the predetermined behavior occurs in the predetermined area of ​​the target environment (such as personnel entering the carriage normally for operation), that is, normal operation. Conversely, if it is lower than the set threshold ratio, the predetermined behavior can be determined to be abnormal behavior.

[0082] In some embodiments, if the electronic device 110 determines that the recognition result indicates that the worker and the vehicle are recognized in the second video frame captured from the second direction, and the distance between the end point of the worker's motion trajectory and the calibrated first area is less than the threshold distance, the electronic device 110 can determine that the predetermined behavior occurs within the predetermined area.

[0083] In some embodiments, the first area is marked as the area where the workers perform their work in the target environment, that is, the electronic device 110 defines the area where the workers normally work through a deep learning algorithm and / or receives the marked video frames.

[0084] As an example, Figure 3D FIG. 3 is a schematic diagram showing a method of determining an abnormality detection result 332 of a second video frame in some embodiments of the present disclosure. Figure 3D As shown, in the second video frame collected in the second direction, the operator may cross the boundary 332-1 from the platform area to enter the carriage to perform the predetermined behavior of loading and unloading goods. In the video frame of the second direction, due to the occlusion of the carriage, the boundary box 332-2 of the operator may disappear, causing such behavior to be misjudged as abnormal behavior. Therefore, the electronic device 110 can determine whether the predetermined behavior still occurs in the predetermined area by evaluating the distance from the boundary of the first area to the end point of the trajectory after the operator boundary box 332-2 disappears. If this distance is less than a preset threshold (for example, the threshold is set to one-fourth of the total length of the operator boundary box 332-2), the electronic device 110 can assume that the person enters the carriage normally, thereby determining that the predetermined behavior occurs in the predetermined area. Through this judgment method, the electronic device 110 can effectively distinguish normal working scenes from abnormal behaviors that may be potentially dangerous (such as jumping on the platform), reduce the false alarm rate, and improve the accuracy and reliability of detection.

[0085] In some embodiments, if the electronic device 110 determines that the recognition result indicates that the operator, the vehicle, and the vehicle pedal are recognized, and there is an overlap between the bounding box used to define the operator and the bounding box used to define the vehicle pedal in the video frame, it is determined that the predetermined behavior occurs within the predetermined area.

[0086] As an example, Figure 3E FIG. 3 is a schematic diagram showing a method of determining anomaly detection results 333 in some embodiments of the present disclosure. Figure 3EAs shown, the operator may cross the boundary 333-1 from the platform area to perform the predetermined behavior of loading and unloading goods. However, at this time, the operator is performing normal operations across the platform area through the vehicle pedal. Therefore, the electronic device 110 can identify the operator, the vehicle, and the vehicle pedal, and determine whether there is an overlap between the operator's boundary box 333-2 and the vehicle pedal's boundary box 333-3 (such as determining whether the lower left corner of the operator's boundary box 333-2 is within the vehicle pedal's boundary box 333-3), thereby determining whether the operator is always performing normal operations on the vehicle pedal.

[0087] In some embodiments, if it is determined that the predetermined behavior does not occur within the predetermined area, then it is determined that the abnormality detection result indicates that the predetermined behavior is an abnormal behavior.

[0088] In some embodiments, if the electronic device 110 determines that the predetermined behavior is not a normal working behavior occurring in the predetermined area based on the spatial relationship between the operator and the vehicle and the direction of the operator's movement, the electronic device 110 determines that the abnormal detection result indicates that the predetermined behavior is an abnormal behavior, that is, determines that an abnormal behavior is detected.

[0089] In the disclosed embodiment, by combining the recognition results for at least the operating personnel and the vehicle and the predefined areas and thresholds, it is possible to effectively distinguish abnormal behaviors from normal operating behaviors, thereby improving the accuracy of abnormal behavior detection and reducing the occurrence of false positives and false negatives.

[0090] Figure 3F FIG. 3 is a schematic diagram of a process 300A for training a target detection model according to some embodiments of the present disclosure. The process 300A may be implemented at the electronic device 110 .

[0091] In some embodiments, the electronic device 110 locates various target objects in the target environment based on a deep learning target detection algorithm. Such algorithms require a large amount of labeled data to train the target detection model so that it can accurately detect the target object.

[0092] In block 341 , the electronic device 110 acquires a first training data set.

[0093] In some embodiments, the first training data set includes labeled training data. For example, open source data that has been labeled with various target objects such as people and vehicles is used as training data. These data sets usually contain pictures of various scenes and angles to help the model learn different situations.

[0094] In block 342 , the electronic device 110 uses a semi-supervised learning method to determine a second training data set based on unlabeled training data.

[0095] In some embodiments, because the annotated first training data set includes data of various scenarios, and the training data for the target environment in the present disclosure is relatively small, a small amount of training data needs to be manually annotated. This can enhance the generalization ability of the model in specific scenarios and make it perform better in practical applications.

[0096] In box 343, based on the first training data set and / or the second training data set, the electronic device 110 trains the target detection model.

[0097] In some embodiments, the electronic device 110 uses a semi-supervised learning method to determine a second training data set. For example, the electronic device 110 can use the initial model trained based on the first training data set to perform target detection on unlabeled training data. If the detection result is correct, the electronic device 110 adds the training data to the second training data set. If the detection result is wrong, the electronic device 110 can receive data for manual correction of the detection result, add it to the second training data set and continue training the model. Repeat this process until the detection accuracy of the target detection model reaches a certain threshold (for example, 95%).

[0098] In the disclosed embodiment, semi-supervised data labeling and deep learning model training are used to reduce the labor cost of data labeling, and the generalization ability of the model is improved to obtain more accurate recognition results, which helps to improve the accuracy of abnormal behavior detection.

[0099] In some embodiments, if the electronic device 110 determines the predetermined behavior as an abnormal behavior, the detection result corresponding to the video frame in which the abnormal behavior is detected in the video data is output. The detection result indicates the area where the operator is located in the video frame.

[0100] In the disclosed embodiment, the electronic device 110 can not only detect abnormal behavior, but also clearly identify the area where the person who has committed the abnormal behavior is located (for example, by outputting a bounding box). This method of outputting specific area annotations as detection results is easier to explain the decision-making process than a classification model that only outputs probability values, and can facilitate management personnel to quickly locate specific people who have committed abnormal behavior and perform necessary intervention and processing.

[0101] Figure 4 A block diagram of an apparatus 400 for abnormal behavior detection according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or include the electronic device 110.

[0102] The device 400 includes an identification module 410, which is configured to use a trained target detection model to perform identification of workers and / or vehicles based on video data collected in a target environment; a trajectory determination module 420, which is configured to determine the movement trajectory of the workers based on the identification results of the workers; a first detection module 430, which is configured to determine whether a predetermined behavior is detected based on the movement trajectory of the workers; and a second detection module 440, which is configured to determine an abnormal detection result of the predetermined behavior based on the identification results of the workers and / or vehicles if it is determined that the predetermined behavior is detected, and the abnormal detection result indicates whether the predetermined behavior is an abnormal behavior.

[0103] In some embodiments, the recognition module 410 is further configured to sample video frames from the video data at predetermined time intervals; and input each sampled video frame into a target detection model to determine a recognition result, wherein the recognition result at least indicates a bounding box for defining a worker and / or a bounding box for defining a vehicle in each video frame.

[0104] In some embodiments, the video data collected in the target environment includes: first video data collected from a first direction of the target environment and / or second video data collected from a second direction of the target environment.

[0105] In some embodiments, the second replacement module is configured to replace the first video data with the target video data based on the modified application programming interface of the second interface version if it is determined that the operating system version is greater than or equal to a threshold version.

[0106] In some embodiments, the trajectory determination module 420 is further configured to use a target tracking algorithm to determine the movement trajectory of the operator based on the bounding box of the operator indicated by the recognition result.

[0107] In some embodiments, the first detection module 430 is further configured to determine the positional relationship between the starting point and the end point of the motion trajectory and the first area if it is determined that the motion trajectory overlaps with the first area marked in the video frame, and the first area is marked as the area where the operator performs operations in the target environment; and if the positional relationship indicates that the starting point of the motion trajectory is within the first area and the end point of the motion trajectory is outside the first area, it is determined that the predetermined behavior is detected.

[0108] In some embodiments, the second detection module 440 is further configured to determine whether the predetermined behavior occurs within a predetermined area of ​​the target environment based at least on the identification results of the operating personnel and / or the vehicle; if it is determined that the predetermined behavior does not occur within the predetermined area, then the abnormal detection result indicates that the predetermined behavior is an abnormal behavior.

[0109] In some embodiments, the second detection module 440 is further configured to determine that the predetermined behavior occurs in the predetermined area if the recognition result indicates that the operator and the vehicle are recognized in the first video frame collected from the first direction, and the overlap ratio between the bounding box used to define the operator and the bounding box used to define the vehicle in the first video frame is greater than a threshold ratio; if the recognition result indicates that the operator and the vehicle are recognized in the second video frame collected from the second direction, and the distance between the end point of the operator's motion trajectory and the calibrated first area is less than the threshold distance, then the predetermined behavior is determined to occur in the predetermined area, and the first area is marked as the area where the operator performs work in the target environment.

[0110] In some embodiments, the second detection module 440 is further configured to determine that the predetermined behavior occurs within a predetermined area if the recognition result indicates that the operator, the vehicle, and the vehicle pedal are recognized, and there is an overlap between the bounding box used to define the operator and the bounding box used to define the vehicle pedal in the video frame.

[0111] In some embodiments, the device 400 includes a training module 450, which is configured to obtain a first training data set, the first training data set including labeled training data; determine a second training data set based on unlabeled training data using a semi-supervised learning method; and train a target detection model based on the first training data set and / or the second training data set.

[0112] In some embodiments, the device 400 includes an output module 460, which is configured to output a detection result corresponding to a video frame in which the abnormal behavior is detected in the video data if the predetermined behavior is determined to be an abnormal behavior, and the detection result indicates the area where the operator is located in the video frame.

[0113] The modules included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0115] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0116] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0117] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0118] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0119] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0120] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0121] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0124] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0125] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the implementations disclosed herein.

Claims

1. A method for abnormal behavior detection, include: Using the trained target detection model, perform identification of operating personnel and / or vehicles based on video data collected in the target environment; Determining a movement trajectory of the operator based on the recognition result of the operator; Based on the movement trajectory of the operator, determining whether a predetermined behavior is detected; as well as If it is determined that the predetermined behavior is detected, an abnormal detection result of the predetermined behavior is determined based on the identification result of the operator and / or the vehicle, and the abnormal detection result indicates whether the predetermined behavior is the abnormal behavior.

2. The method according to claim 1, wherein the trained target detection model is used to perform recognition of the operator and / or vehicle based on the video data collected in the target environment. include: sampling video frames from the video data at predetermined time intervals; as well as Each sampled video frame is input into the target detection model to determine a recognition result, wherein the recognition result at least indicates a bounding box for defining the operator and / or a bounding box for defining the vehicle in each video frame.

3. The method according to claim 1, wherein the video data collected in the target environment include: First video data collected from a first direction for the target environment and / or second video data collected from a second direction for the target environment.

4. The method according to claim 1, wherein the movement trajectory of the operator is determined based on the recognition result of the operator. include: The target tracking algorithm is used to determine the movement trajectory of the operator based on the boundary box of the operator indicated by the recognition result.

5. The method according to claim 1, wherein based on the movement trajectory of the operator, it is determined whether a predetermined behavior is detected. include: If it is determined that the motion trajectory overlaps with a first area marked in the video frame, a positional relationship between a start point and an end point of the motion trajectory and the first area is determined, and the first area is marked as an area where the operator performs work in the target environment; as well as If the position relationship indicates that the starting point of the motion trajectory is within the first area and the ending point of the motion trajectory is outside the first area, it is determined that the predetermined behavior is detected.

6. The method according to claim 1, wherein the abnormal detection result of the predetermined behavior is determined based on the recognition result of the operator and / or the vehicle. include: At least based on the recognition result of the operator and / or the vehicle, determining whether the predetermined behavior occurs within a predetermined area of ​​the target environment; If it is determined that the predetermined behavior does not occur within the predetermined area, then it is determined that the abnormality detection result indicates that the predetermined behavior is the abnormal behavior.

7. The method according to claim 6, wherein it is determined whether the predetermined behavior occurs in a predetermined area of ​​the target environment based at least on the recognition result of the operator and / or the vehicle. include: If the recognition result indicates that the operator and the vehicle are recognized in the first video frame acquired from the first direction, and the overlap ratio between the bounding box used to define the operator and the bounding box used to define the vehicle in the first video frame is greater than a threshold ratio, it is determined that the predetermined behavior occurs within the predetermined area; If the recognition result indicates that the operator and the vehicle are identified in the second video frame captured from the second direction, and the distance between the end point of the operator's motion trajectory and the calibrated first area is less than a threshold distance, it is determined that the predetermined behavior occurs within the predetermined area, and the first area is marked as the area where the operator performs work in the target environment.

8. The method according to claim 6, wherein at least based on the recognition result of the operator and / or the vehicle, it is determined whether the predetermined behavior occurs in the predetermined area of ​​the target environment. include: If the recognition result indicates that the operator, the vehicle, and the vehicle pedal are recognized, and there is an overlap between a bounding box used to define the operator and a bounding box used to define the vehicle pedal in the video frame, it is determined that the predetermined behavior occurs within the predetermined area.

9. The method according to claim 1, wherein the target detection model is trained by the following operations: Acquire a first training data set, where the first training data set includes labeled training data; Determine a second training data set based on unlabeled training data using a semi-supervised learning method; as well as The target detection model is trained based on the first training data set and / or the second training data set.

10. The method according to claim 1, further comprising: include: If the predetermined behavior is determined to be the abnormal behavior, a detection result corresponding to a video frame in which the abnormal behavior is detected in the video data is output, wherein the detection result indicates an area where the operator is located in the video frame.

11. A device for abnormal behavior detection, include: The recognition module is configured to use the trained target detection model to perform recognition of operating personnel and / or vehicles based on video data collected in the target environment; A trajectory determination module is configured to determine a movement trajectory of the operator based on a recognition result of the operator; A first detection module is configured to determine whether a predetermined behavior is detected based on the movement trajectory of the operator; as well as The second detection module is configured to determine an abnormal detection result of the predetermined behavior based on the identification result of the operator and / or the vehicle if it is determined that the predetermined behavior is detected, and the abnormal detection result indicates whether the predetermined behavior is the abnormal behavior.

12. An electronic device, include: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 10.

Images