Method, apparatus, device and storage medium for anomaly operation detection

Abstract

According to embodiments of the present disclosure, a method, device, equipment and storage medium for abnormal operation detection are provided. The method comprises: identifying a first target object and a second target object in a target scene based on a plurality of image data of the target scene; in response to identifying the first target object and the second target object, determining a first motion trajectory of the first target object and a second motion trajectory of the second target object based on the image data; determining an interaction trajectory corresponding to an operation of the first target object on the second target object based on the first motion trajectory and the second motion trajectory; and determining an abnormal operation of the first target object on the second target object in a process of performing a predetermined task based on at least a moving distance between the first target object and the second target object and a moving speed of the second target object indicated by the interaction trajectory. Thus, the interaction trajectory of the object is analyzed based on a predetermined rule, and the abnormal operation can be accurately identified.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of information technology, and particularly to methods, apparatus, devices, and computer-readable storage media for detecting abnormal operations. Background Technology

[0002] In various operating environments, especially those involving logistics, manufacturing, and warehouse management, monitoring and timely response to abnormal operations are of great importance to ensuring operational safety and task execution efficiency.

[0003] In the detection of abnormal operations by workers, related technologies generally suffer from low detection accuracy and high false alarm rates. Especially in dynamic and complex environments, such as logistics parks, they often cannot cope with the diversity and complexity of various objects and the dynamic changes in the scene, thus failing to accurately identify and detect abnormal operations.

[0004] Therefore, a more efficient and accurate method for detecting abnormal operations is needed. Summary of the Invention

[0005] In a first aspect of this disclosure, a method for detecting abnormal operations is provided. The method includes: identifying a first target object and a second target object in a target scene based on multiple image data of the target scene, wherein the first target object includes an object performing a predetermined task in the target scene, and the second target object includes an object being operated on in the predetermined task; in response to identifying the first target object and the second target object, determining a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points; determining an interaction trajectory corresponding to an operation by the first target object on the second target object based on the first motion trajectory and the second motion trajectory; and determining, at least based on the moving distance between the first target object and the second target object indicated by the interaction trajectory and the moving speed of the second target object, an abnormal operation by the first target object on the second target object during the execution of the predetermined task.

[0006] In a second aspect of this disclosure, an apparatus for detecting abnormal operations is provided. The apparatus includes: an identification module configured to identify a first target object and a second target object in a target scene based on multiple image data of the target scene, wherein the first target object includes an object performing a predetermined task in the target scene, and the second target object includes an object being operated on in the predetermined task; a motion trajectory determination module configured to, in response to the identification of the first target object and the second target object, determine a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points; an interaction trajectory determination module configured to determine an interaction trajectory corresponding to an operation by the first target object on the second target object based on the first motion trajectory and the second motion trajectory; and an abnormal operation determination module configured to determine, at least based on the moving distance between the first target object and the second target object indicated by the interaction trajectory and the moving speed of the second target object, an abnormal operation by the first target object on the second target object during the execution of the predetermined task.

[0007] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0009] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this disclosure.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0013] Figure 2 A schematic diagram illustrating an example process for detecting abnormal operations according to some embodiments of the present disclosure is shown;

[0014] Figure 3 A schematic diagram illustrating the identification results of some embodiments of this disclosure is shown;

[0015] Figure 4 A flowchart illustrating an abnormal operation detection process according to some embodiments of the present disclosure is shown;

[0016] Figure 5 A block diagram of an apparatus for detecting abnormal operation according to some embodiments of the present disclosure is shown; and

[0017] Figure 6 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0018] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0019] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0025] As briefly mentioned above, in various operating environments, especially those involving logistics, manufacturing, and warehouse management, the monitoring and timely response to abnormal operations are of great significance to ensuring operational safety and task execution efficiency.

[0026] For example, in logistics park scenarios, with the continuous increase in logistics demand, sorting centers face unprecedented pressure in handling packages. Under such high-intensity working conditions, in order to complete the processing of a large number of packages in a very short time, some personnel may adopt non-standard methods such as throwing packages to improve work efficiency.

[0027] To effectively manage and prevent the safety and operational risks posed by such abnormal operations, traditional methods rely on manual monitoring or sensor data collection for detection. However, manual monitoring has a very limited scope, often covering only a small area. Furthermore, manual monitoring is constrained by human resources and the time required for focused attention, making continuous and comprehensive monitoring of the entire process impossible. Sensor-based methods, on the other hand, can only collect relatively single data dimensions, resulting in insufficient accuracy and making precise behavioral identification and effective judgment difficult.

[0028] In related technologies, methods for detecting abnormal operations on target objects mainly fall into two categories. Deep learning-based recognition methods possess strong automatic feature extraction capabilities and can learn to identify complex abnormal behavior patterns through large amounts of data, achieving high accuracy in more scenarios. However, this method requires a large amount of labeled data as a training dataset, but publicly available datasets are very limited, leading to a significant need for manual annotation. Furthermore, since the output is usually just a binary probability value, it lacks detailed interpretation and understanding of the detection results, making it susceptible to false alarms from similar actions.

[0029] Furthermore, methods based on simple, pre-set rules struggle to maintain high accuracy in trajectory detection and behavior recognition in dynamic scenarios such as complex logistics environments. In complex or unstable environments, they are easily affected by external factors, impacting the accuracy of target detection and consequently affecting the accuracy of detecting abnormal operations.

[0030] Therefore, in order to address the problems of insufficient accuracy and limited applicability of related technologies in abnormal operation detection, a more efficient and accurate solution is needed to detect abnormal operations more accurately.

[0031] In embodiments of this disclosure, a scheme for detecting abnormal operations is provided. According to embodiments of this disclosure, a first target object and a second target object in a target scene are identified based on multiple image data of the target scene, wherein the first target object includes an object performing a predetermined task in the target scene, and the second target object includes an object being operated on in the predetermined task; in response to the identification of the first target object and the second target object, a first motion trajectory of the first target object and a second motion trajectory of the second target object are determined based on a set of image data acquired at consecutive time points; based on the first motion trajectory and the second motion trajectory, an interaction trajectory corresponding to the operation of the first target object on the second target object is determined; and at least based on the moving distance between the first target object and the second target object indicated by the interaction trajectory and the moving speed of the second target object, an abnormal operation of the first target object on the second target object during the execution of the predetermined task is determined.

[0032] Therefore, by combining a rule-based abnormal behavior identification method with the precise real-time location identification of a target detection model, efficient and accurate anomaly detection of interactive operations on target objects is achieved. Furthermore, analyzing interaction trajectories and utilizing rules for real-time detection of these interactive operations demonstrates significant real-time performance and efficiency, enabling rapid response and accurate identification of abnormal operations. This approach not only improves efficiency in real-time detection but also ensures the accuracy of anomaly detection, effectively reducing false alarms and false negatives, and providing strong support for security monitoring in various application scenarios.

[0033] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, electronic device 110 performs identification of multiple target objects in a target scene based on image data 120. In some embodiments, electronic device 110 may determine a detection result 140 based on the identification result. Detection result 140 indicates whether the detected predetermined behavior 140 detected abnormal operation of the target object during the performance of the predetermined task.

[0034] In environment 100, electronic device 110 can be any type of computing device, including terminal devices or server devices. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0035] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0036] Figure 2 A schematic diagram of an example process 200 for detecting abnormal operation according to some embodiments of the present disclosure is shown. Process 200 can be implemented in environment 100, for example, process 200 can be implemented at electronic device 110. In the following discussion, reference will be made to... Figure 1 To describe process 200.

[0037] In box 210, electronic device 110 identifies a first target object and a second target object in the target scene based on multiple image data 205 of the target scene. The first target object includes an object that performs a predetermined task in the target scene, and the second target object includes an object that is operated on in the predetermined task.

[0038] For example, the multiple image data 205 can be video data continuously acquired by the electronic device 110 in the target environment. For instance, the electronic device 110 can acquire real-time video data acquired by a camera installed in a logistics park. The multiple image data 205 can also be keyframes or sampled frames obtained by sampling the video data at predetermined time intervals after acquiring the video data collected in the target environment.

[0039] The first target object refers to the object of the operation that performs the predetermined task, that is, the subject performing the operation. For example, in a logistics scenario, the first target object can be the staff who handle or sort packages; in a warehouse management scenario, the first target object can be the personnel or automated equipment that move goods. The second target object refers to the object being operated on in the predetermined task, that is, the object of the operation, such as packages or goods.

[0040] In some embodiments, the electronic device 110 determines the recognition results for a first target object and a second target object by inputting multiple image data 205 into a trained target detection model. The recognition results at least indicate the location information of the first target object and the second target object.

[0041] As an example, electronic device 110 can input multiple image data 205 (e.g., real-time monitoring video frames or image frames extracted from videos) into a trained object detection model to identify and locate a first target object and a second target object in a target scene. The object detection model is based on a deep learning algorithm and is capable of identifying specific target objects such as workers and packages in the image data.

[0042] In some embodiments, the object detection model may include a single machine learning model based on the You Only Look Once (YOLO) algorithm. YOLO is an object detection method based on convolutional neural networks, whose main feature is its ability to simultaneously predict the location and category of multiple objects (such as people, packages, etc.). Unlike traditional object detection algorithms, YOLO-based models can transform the detection task into a regression problem, predicting the location and category of multiple objects in an image through network inference using only a single model.

[0043] In some embodiments, location information indicates bounding boxes used in image data 205 to define the first target object and the second target object, as well as body calibration points of the first target object and contour calibration points of the second target object.

[0044] As an example, Figure 3 A schematic diagram illustrating the identification result 310 of some embodiments of this disclosure is shown. For example... Figure 3 As shown, after the target detection model performs recognition on image data 205, it outputs recognition result 310, which is the positional information of the first target object and the second target object identified in image data 205. Specifically, recognition result 310 may include bounding boxes 311 and 312 used to define the first target object and the second target object. These bounding boxes can be used to define the position and size of the target object in each image data, thereby enabling the identification and tracking of positional changes of each target object (e.g., a person or a package) in the target scene.

[0045] Continue to refer to Figure 3The recognition result 310 may also include the body landmarks of the first target object. The YOLO model can identify the key points (also known as skeletal key points) and key point coordinates of the first target object's body. These key points typically include parts such as the head, shoulders, elbows, wrists, spine, hips, knees, and ankles. These key points can help understand the posture and movement state of the first target object. For example, by analyzing the key points of a person's shoulders and wrists, it is possible to analyze whether the person performed an action such as throwing a package or other specific movements.

[0046] The recognition result 310 may also include the contour calibration points of the second target object. For the second target object, the YOLO model can identify the key points of the outer contour of the second target object. For example, these key points may include the edge corner points of the object (such as the four corner points of the object). These key points are used to calibrate the position, orientation, and size of the second target object, helping subsequent processing to accurately determine the position and shape of the second target object in the target scene, and providing a basis for subsequent abnormal operation analysis.

[0047] In some examples, the recognition result 310 may also include the category of the identified target object and the confidence level that the category of the target object was correctly identified. For example, "package, 0.9" indicates that the confidence level for the recognition result of the package category object is 0.9.

[0048] In other embodiments, the identification result 310 may also include more, less, or different information that can be used for the detection of operations on the target object.

[0049] In the embodiments of this disclosure, by utilizing a single YOLO model, the recognition processes of key points on the human skeleton and key points on the outer contour of the package are coupled together, rather than being performed separately, which reduces the computational load of recognition. This approach can accelerate processing speed and improve the real-time performance of recognition.

[0050] In some embodiments, the electronic device 110 may also incorporate hardware acceleration technology to accelerate the processing of computational tasks, thereby improving the efficiency and performance of recognition. Hardware acceleration refers to using specialized hardware, such as leveraging the parallel computing capabilities of a graphics processing unit (GPU), to accelerate the computation of the object detection model. Especially in complex image processing and machine learning tasks, hardware acceleration technology can significantly improve computational speed and reduce latency.

[0051] Return to reference Figure 2 In box 220, in response to identifying a first target object and a second target object, the electronic device 110 determines a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points.

[0052] In some examples, the motion trajectory includes the path of the target object's position change across consecutive image frames. The electronic device 110 can perform the operation of identifying image data using a target detection model for each frame of image data. The target detection model can output the position of the target object and possible calibration points in each frame of image data. Since multi-target tracking often encounters situations such as target object loss, occlusion, or addition / removal, the electronic device 110 can match the target object in the current frame with the target object in the previous frame, thereby ensuring the continuity of the identity and position of each target object across multiple frames. This method ensures that the motion trajectory of each target object is continuous and not lost.

[0053] In some embodiments, the electronic device 110 determines the moving speed and appearance characteristics of the first and second target objects based on the recognition results of the first and second target objects. The recognition results output by the target detection model of the electronic device 110 include the position information of each target object. By tracking the target position in consecutive image frames and calculating the target displacement in each frame, the moving speed of the target can be obtained.

[0054] Appearance features typically refer to the visual characteristics of a target object, such as color, texture, and shape. Appearance features can be obtained by extracting feature representations from intermediate network layers of an object detection model. Using appearance features, it is possible to further ensure that the target object can be correctly tracked in subsequent image frames even if its spatial location changes or it becomes occluded.

[0055] In some embodiments, the electronic device 110 determines a first motion trajectory by matching position information, movement speed, and appearance features corresponding to a first target object in a set of image data using a predetermined matching algorithm.

[0056] The predetermined matching algorithm may include the Hungarian matching algorithm, which is an algorithm used to solve the optimal matching problem. The matching algorithm performs an optimal match between the target object in the current frame and the target object in the previous frame, thereby ensuring that the identity of each target object is consistent.

[0057] As an example, electronic device 110 can construct an association cost matrix for the first target object. Each element in the matrix represents the "matching cost" between a target object in the current frame and a target object in the previous frame. By constructing the association cost matrix, the association cost can be calculated by comprehensively considering multiple dimensions of factors, such as the target object's position information (e.g., center point coordinates, and target width and height), velocity characteristics (e.g., center point movement speed, and target width and height change speed), and appearance characteristics (e.g., color, shape, etc.), thereby determining the minimum cost matching combination, that is, finding the best matching relationship between the target object in the previous frame and the target object in the current frame.

[0058] In some embodiments, the electronic device 110 can determine a second motion trajectory by using a matching algorithm to match the position information, movement speed, and appearance features corresponding to the second target object in a set of image data. Similar to the process of determining the first motion trajectory described above, the electronic device 110 can also use a matching algorithm to match the second target object. For example, by matching the position information, movement speed, and appearance features of a package, the motion trajectory of the package can be accurately determined. This process ensures that the electronic device 110 can accurately identify the target object in each frame of image data, maintaining consistent recognition even in complex environments (such as target occlusion, target disappearance, or the appearance of a new target).

[0059] Return to reference Figure 2 In box 230, electronic device 110 determines the interaction trajectory corresponding to the operation of the first target object on the second target object based on the first motion trajectory and the second motion trajectory.

[0060] The interaction trajectory can include the temporal and spatial trajectory data of the interaction between the first target object and the second target object. For example, the interaction trajectory includes the positional and distance changes of the first target object in the target scene as it performs a predetermined task on the second target object, as well as the time sequence of the operation. Through the interaction trajectory, the electronic device 110 can analyze a series of interactive behaviors (such as contact, carrying, and putting down) between the first and second target objects.

[0061] Since the first and second motion trajectories indicate the motion paths of the first and second target objects in a continuous set of images (e.g., continuously captured real-time video frames), they can include information such as the position and velocity of the first and second target objects. By analyzing the first and second motion trajectories, it can be determined whether an interaction has occurred between the first and second target objects, thereby determining the corresponding interaction trajectory.

[0062] The following describes an example process for determining an interaction trajectory. In some embodiments, the electronic device 110 can determine the distance between a first target object and a second target object based on a first motion trajectory and a second motion trajectory. The process of determining the interaction trajectory is performed by analyzing the motion trajectories between the first target object and the second target object, as well as the changes in their relative distance. The electronic device 110 can first calculate the distance between the first target object and the second target object.

[0063] As an example, electronic device 110 can calculate the Euclidean distance between people and packages using their location information (typically the coordinates of the center point of a bounding box). The calculation results are stored as a dataset that records the distance value for each pair of people and packages at each time point, serving as the basis for subsequent matching. This distance data is continuously updated as the image data is updated in real time to track changes in their relative distances as the target objects continue to move.

[0064] In some embodiments, the electronic device 110, in response to a distance less than a first threshold distance and a duration of less than the threshold distance greater than a first threshold time, determines a candidate interaction trajectory based on a first motion trajectory and a second motion trajectory. If the distance between the first target object and the second target object is less than a preset first threshold distance, they can be considered to be physically in contact or very close, and an interaction operation may exist. If the duration of the distance being less than the preset first threshold distance is greater than the first threshold time, a valid interaction operation can be considered to have occurred.

[0065] As an example, electronic device 110 can iterate through the distance dataset within a certain time range to search for situations where the distance between a person and a package is less than a preset first threshold distance and persists for a predetermined first threshold time. If a matching person-package combination is found, it is marked as a potential interaction pair. Electronic device 110 can create candidate interaction trajectories corresponding to this potential interaction pair (i.e., the person-package combination) and add them to an interaction trajectory list, thereby initiating monitoring of this interaction process. Each candidate interaction trajectory in the interaction trajectory list is assigned a unique identifier and records the location and speed information of the corresponding person and package, the start and end times of the interaction, and key distance changes during the process.

[0066] In some embodiments, the electronic device 110 can determine the interaction trajectory based on the changing trends of candidate interaction trajectories and distance over a subsequent second threshold time period. For example, in subsequent monitoring, the electronic device 110 can update the candidate interaction trajectories in the interaction trajectory list based on real-time image data. When the changes in distance and speed meet preset conditions, the electronic device 110 can determine the candidate interaction trajectory as the interaction trajectory, thereby confirming that the first target object has performed a complete operation of a predetermined task on the second target object in the target scene.

[0067] In some embodiments, the electronic device 110 determines a candidate interaction trajectory as an interaction trajectory in response to determining that the distance is greater than a second threshold distance and the rate of change of the distance is greater than a threshold rate within a subsequent second threshold time period.

[0068] As an example, electronic device 110 can analyze the distance changes of potential interaction pairs in candidate interaction trajectories. If the distance between a person and a package initially decreases from less than a first threshold distance to gradually increase to greater than a second threshold distance, and the rate of change meets preset conditions (e.g., meets a certain rate range), then electronic device 110 can consider this a complete interaction operation. Electronic device 110 can then determine this candidate interaction trajectory as the interaction trajectory representing the entire process from when the operator touches the package to when they leave the package.

[0069] In some examples, if the time of an interaction track in the interaction track list has expired or the interaction track no longer meets the determination conditions due to data correction (such as the duration exceeding a predetermined threshold), the electronic device 110 can delete the interaction track from the interaction track list, thereby ensuring the accuracy and validity of the data in the interaction track list.

[0070] In the embodiments of this disclosure, by accurately determining the relative distance between target objects and their changing trends, the corresponding interaction trajectories can be efficiently determined. This process can effectively identify the interaction operations performed by the first target object on the second target object in the target scene, providing reliable data support for subsequent behavior analysis and anomaly identification.

[0071] Return to reference Figure 2 In box 240, electronic device 110 determines whether the interaction between the first target object and the second target object is an abnormal operation based on the interaction trajectory. For example, electronic device 110 can determine whether the abnormal operation of the first target object on the second target object during the execution of a predetermined task is based at least on the moving distance between the first and second target objects indicated by the interaction trajectory and the moving speed of the second target object.

[0072] In some embodiments, the electronic device 110 determines a complete interaction trajectory, which means that the electronic device 110 has identified a complete interaction operation between the first target object and the second target object (e.g., the interaction process from a person touching a package to a person leaving the package). At this time, the electronic device 110 can determine whether the interaction operation is an abnormal operation based on multiple preset rules.

[0073] In some embodiments, the electronic device 110 may preset judgment rules based on conditions such as moving distance and moving speed. For example, the electronic device 110 may determine the sum of the relative moving distances between the first target object and the second target object during operation based on the positions of the first target object and the second target object indicated by the interaction trajectory. Alternatively or consequentially, the electronic device 110 may determine the number of times the moving speed of the second target object is greater than a threshold speed during operation based on the moving speed of the second target object indicated by the interaction trajectory.

[0074] The sum of relative movement distances, also known as the relative cumulative movement distance, refers to the total displacement of the second target object relative to the first target object during a single interaction. This represents the cumulative value of the relative positional change of the first target object during its interaction with the second target object within the target scene.

[0075] For example, in logistics handling scenarios, during normal package handling operations, the package and the personnel typically maintain a relatively stable and regular relative displacement. In other words, the package's displacement is usually continuous and uniform, and the range of movement is relatively stable. However, in cases of abnormal handling (such as violent throwing), the package will exhibit drastic and irregular displacement changes. For instance, the package might suddenly be thrown very far or rapidly change direction, which contrasts sharply with normal, smooth handling. During the interaction, if the relative displacement of the package changes very drastically, the change in its relative cumulative movement distance will differ significantly from normal operation. To determine if abnormal behavior exists, this distance can be calculated, and upper and lower thresholds can be set for filtering. If the calculated relative cumulative movement distance is within the threshold range, it is considered to meet the conditions for abnormal behavior.

[0076] The situation where the second target object's movement speed exceeds the threshold speed can also be understood as the second target object undergoing high-speed movement. This type of movement is typically a characteristic of abnormal operations such as throwing.

[0077] For example, in logistics handling scenarios, during normal package handling operations, the package typically moves at a constant speed or with slow acceleration, resulting in a relatively stable overall movement. However, in cases of abnormal operations (such as violent throwing), the package will experience high-speed movement for a short period, which is significantly different from normal handling operations. By traversing the interaction trajectory data between personnel and packages, the electronic device 110 can count the number of high-speed movements of the package during the interaction process. When the counted number of high-speed movements exceeds a predetermined lower threshold, the electronic device 110 can consider it to meet the criteria for abnormal operation.

[0078] In some embodiments, the electronic device 110 determines that the interaction operation performed by the first target object on the second target object in the target scene is an abnormal operation in response to the sum of the relative movement distances satisfying a threshold range and the number of times the movement speed of the second target object is greater than the threshold speed being greater than the threshold number of times.

[0079] For example, if the relative cumulative distance the package travels is within a set range, and the number of high-speed movements exceeds a set threshold, this can be confirmed as an abnormal operation of violent throwing. By combining these two conditions, the electronic device 110 can more accurately detect and identify abnormal operations.

[0080] In some examples, the electronic device 110 can also determine corresponding judgment rules for different abnormal operations to determine whether the interaction operation performed by the first target object on the second target object in the target scene is an abnormal operation. For example, it can determine whether the package is lost or not handled according to regulations based on the disappearance or abnormal stop of the package, or determine whether there is boundary crossing behavior based on the location information of the first target object, etc.

[0081] In the embodiments of this disclosure, the rule-based identification method directly determines abnormal operations by calculating features such as the distance and speed between the first target object and the second target object. Compared with deep learning model-based identification methods, this approach requires less computation and is more efficient, enabling rapid identification, processing, and early warning in scenarios requiring real-time monitoring. Furthermore, by combining the real-time and accurate location and appearance recognition results provided by the target detection model, the interactions between target objects can be analyzed more comprehensively and accurately, thereby providing more adaptive and accurate detection results.

[0082] Return to reference Figure 2In box 250, in response to determining an abnormal operation by the first target object on the second target object during the execution of a predetermined task, the electronic device 110 can output an abnormal interaction trajectory for early warning. For example, the electronic device 110 can generate an abnormal interaction trajectory based on the interaction trajectories of the first and second target objects. This trajectory records detailed information related to the interaction between the first and second target objects, such as position information, speed changes, and relative distance changes, as well as the time sequence of occurrence. The electronic device 110 can also provide alerts for detected abnormal behavior, such as through a visual interface or audible alarms, to notify relevant personnel of the detected abnormal operation. The electronic device 110 can also generate a detailed abnormal operation report, recording the abnormal behavior, time, target information, etc., for subsequent reference and review.

[0083] Figure 4 A flowchart of an abnormal operation detection process 400 according to some embodiments of the present disclosure is shown. Process 400 can be implemented in environment 100, for example, process 400 can be implemented at electronic device 110.

[0084] In box 410, electronic device 110 identifies a first target object and a second target object in the target scene based on multiple image data of the target scene, wherein the first target object includes an operation object performing a predetermined task in the target scene, and the second target object includes an operation object in the predetermined task.

[0085] In box 420, in response to identifying a first target object and a second target object, electronic device 110 determines a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points.

[0086] In box 430, electronic device 110 determines an interaction trajectory corresponding to the operation of the first target object on the second target object based on the first motion trajectory and the second motion trajectory.

[0087] In box 440, electronic device 110 determines, at least based on the moving distance between the first target object and the second target object indicated by the interaction trajectory and the moving speed of the second target object, an abnormal operation of the first target object on the second target object during the execution of a predetermined task.

[0088] In some embodiments, identifying a first target object and a second target object includes: determining identification results for the first target object and the second target object by inputting multiple image data into a trained target detection model, wherein the identification results at least indicate the location information of the first target object and the second target object.

[0089] In some embodiments, location information indicates bounding boxes in image data used to define the first target object and the second target object, as well as body calibration points of the first target object and contour calibration points of the second target object.

[0090] In some embodiments, determining the first motion trajectory and the second motion trajectory includes: determining the moving speed and appearance features of the first target object and the second target object based on the recognition results of the first target object and the second target object, wherein the appearance features indicate at least one of color, texture, and shape; determining the first motion trajectory by matching the position information, moving speed, and appearance features corresponding to the first target object in a set of image data using a predetermined matching algorithm; and determining the second motion trajectory by matching the position information, moving speed, and appearance features corresponding to the second target object in a set of image data using a matching algorithm.

[0091] In some embodiments, determining the interaction trajectory includes: determining the distance between a first target object and a second target object based on a first motion trajectory and a second motion trajectory; determining a candidate interaction trajectory based on the first motion trajectory and the second motion trajectory in response to a distance being less than a first threshold distance and the duration of the distance being less than the threshold distance being greater than a first threshold time; and determining the interaction trajectory based on the changing trend of the candidate interaction trajectory and the distance in a subsequent second threshold time.

[0092] In some embodiments, determining an interaction trajectory includes: in response to determining that, within a subsequent second threshold time period, the distance is greater than a second threshold distance and the rate of change of the distance is greater than a threshold rate, determining a candidate interaction trajectory as an interaction trajectory.

[0093] In some embodiments, determining an abnormal operation includes: determining the sum of the relative movement distances between the first target object and the second target object during the operation based on the positions of the first target object and the second target object indicated by the interaction trajectory; determining the number of times the movement speed of the second target object is greater than a threshold speed during the operation based on the movement speed of the second target object indicated by the interaction trajectory; and determining the operation as an abnormal operation in response to the sum of the relative movement distances satisfying a threshold range and the number of times being greater than a threshold number of times.

[0094] Figure 5 A block diagram of an apparatus 500 for detecting abnormal operation according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or include electronic device 110.

[0095] The device 500 includes an identification module 510 configured to identify a first target object and a second target object in a target scene based on multiple image data of the target scene, wherein the first target object includes an operation object performing a predetermined task in the target scene, and the second target object includes an operation object being operated on in the predetermined task; a motion trajectory determination module 520 configured to determine a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points in response to the identification of the first target object and the second target object; an interaction trajectory determination module 530 configured to determine an interaction trajectory corresponding to the operation of the first target object on the second target object based on the first motion trajectory and the second motion trajectory; and an abnormal operation determination module 540 configured to determine an abnormal operation of the first target object on the second target object during the execution of the predetermined task, based at least on the moving distance between the first target object and the second target object indicated by the interaction trajectory and the moving speed of the second target object.

[0096] In some embodiments, the identification module 510 is further configured to identify the first target object and the second target object in some embodiments by: inputting multiple image data into a trained target detection model to determine the identification result for the first target object and the second target object, wherein the identification result at least indicates the location information of the first target object and the second target object.

[0097] In some embodiments, location information indicates bounding boxes in image data used to define the first target object and the second target object, as well as body calibration points of the first target object and contour calibration points of the second target object.

[0098] In some embodiments, the motion trajectory determination module 520 is further configured to determine a first motion trajectory and a second motion trajectory by: determining the moving speed and appearance features of the first target object and the second target object based on the recognition results of the first target object and the second target object, wherein the appearance features indicate at least one of color, texture, and shape; determining the first motion trajectory by matching the position information, moving speed, and appearance features corresponding to the first target object in a set of image data using a predetermined matching algorithm; and determining the second motion trajectory by matching the position information, moving speed, and appearance features corresponding to the second target object in a set of image data using a matching algorithm.

[0099] In some embodiments, the interaction trajectory determination module 530 is further configured to determine the distance between a first target object and a second target object based on a first motion trajectory and a second motion trajectory; in response to a distance being less than a first threshold distance and the duration of the distance being less than the threshold distance being greater than a first threshold time, to determine a candidate interaction trajectory based on the first motion trajectory and the second motion trajectory; and to determine the interaction trajectory based on the changing trend of the candidate interaction trajectory and the distance within a subsequent second threshold time.

[0100] In some embodiments, the interaction trajectory determination module 530 is further configured to determine the candidate interaction trajectory as an interaction trajectory in response to determining that the distance is greater than a second threshold distance and the rate of change of the distance is greater than a threshold rate within a subsequent second threshold time period.

[0101] In some embodiments, the abnormal operation determination module 540 is further configured to determine the sum of the relative movement distances between the first target object and the second target object during the operation based on the positions of the first target object and the second target object indicated by the interaction trajectory; determine the number of times the movement speed of the second target object is greater than a threshold speed during the operation based on the movement speed of the second target object indicated by the interaction trajectory; and determine the operation as an abnormal operation in response to the sum of the relative movement distances satisfying the threshold range and the number of times being greater than the threshold number of times.

[0102] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0103] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0104] like Figure 6As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0105] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 600.

[0106] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0107] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0108] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0109] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0110] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0111] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0112] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0114] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for detecting abnormal operations, comprising: Based on multiple image data of the target scene, a first target object and a second target object in the target scene are identified, wherein the first target object includes an operation object performing a predetermined task in the target scene, and the second target object includes an operation object in the predetermined task; In response to the identification of the first target object and the second target object, a first motion trajectory of the first target object and a second motion trajectory of the second target object are determined based on a set of image data acquired at consecutive time points; Based on the first motion trajectory and the second motion trajectory, determine the interaction trajectory corresponding to the operation of the first target object on the second target object; as well as Based at least on the movement distance between the first target object and the second target object indicated by the interaction trajectory and the movement speed of the second target object, it is determined that the first target object performed an abnormal operation on the second target object during the execution of the predetermined task.

2. The method according to claim 1, wherein the identification of the first target object and the second target object comprises: By inputting the plurality of image data into a trained target detection model, identification results are determined for the first target object and the second target object, the identification results indicating at least the location information of the first target object and the second target object.

3. The method according to claim 2, wherein the location information indicates the bounding box in the image data used to define the first target object and the second target object, as well as the body calibration point of the first target object and the contour calibration point of the second target object.

4. The method according to claim 1, wherein determining the first motion trajectory and the second motion trajectory comprises: Based on the identification results of the first target object and the second target object, the moving speed and appearance characteristics of the first target object and the second target object are determined, wherein the appearance characteristics indicate at least one of color, texture and shape; By using a predetermined matching algorithm, the position information, movement speed and appearance features corresponding to the first target object are matched in the set of image data to determine the first motion trajectory. as well as By using the matching algorithm, the second motion trajectory is determined by matching the position information, movement speed, and appearance features corresponding to the second target object in the set of image data.

5. The method according to claim 1, wherein determining the interaction trajectory comprises: Based on the first motion trajectory and the second motion trajectory, determine the distance between the first target object and the second target object; In response to the distance being less than a first threshold distance and the duration of the distance being less than the threshold distance being greater than a first threshold time, a candidate interaction trajectory is determined based on the first motion trajectory and the second motion trajectory; as well as The interaction trajectory is determined based on the changing trend of the candidate interaction trajectory and the distance within a subsequent second threshold time period.

6. The method of claim 5, wherein determining the interaction trajectory comprises: In response to determining that, in a subsequent second threshold time period, the distance is greater than a second threshold distance and the rate of change of the distance is greater than a threshold rate, the candidate interaction trajectory is determined as the interaction trajectory.

7. The method of claim 1, wherein determining the abnormal operation comprises: Based on the positions of the first target object and the second target object indicated by the interaction trajectory, the sum of the relative movement distances between the first target object and the second target object during the operation is determined; Based on the movement speed of the second target object indicated by the interaction trajectory, determine the number of times the movement speed of the second target object is greater than a threshold speed during the operation; as well as In response to the sum of the relative movement distances meeting a threshold range and the number of times exceeding the threshold number, the operation is determined to be an abnormal operation.

8. An apparatus for detecting abnormal operation, comprising: The recognition module is configured to recognize a first target object and a second target object in the target scene based on multiple image data of the target scene, wherein the first target object includes an operation object performing a predetermined task in the target scene, and the second target object includes an operation object in the predetermined task; The motion trajectory determination module is configured to determine a first motion trajectory of the first target object and a second motion trajectory of the second target object based on a set of image data acquired at consecutive time points in response to the identification of the first target object and the second target object; The interaction trajectory determination module is configured to determine the interaction trajectory corresponding to the operation of the first target object on the second target object based on the first motion trajectory and the second motion trajectory. as well as An abnormal operation determination module is configured to determine, at least based on the movement distance between the first target object and the second target object indicated by the interaction trajectory and the movement speed of the second target object, an abnormal operation performed by the first target object on the second target object during the execution of the predetermined task.

9. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 7.

11. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 7.

Images