Anomaly detection method and server

Abstract

The application discloses an abnormality detection method and a server. The method obtains a video to be detected, and performs target detection on a plurality of continuous images in the video to be detected to obtain at least one target contained in each of the plurality of images. The running condition of the at least one target is determined according to the positions of the at least one target in the plurality of continuous images. In the case that the running condition indicates that there is a stationary target, the placement posture detection of the stationary target is performed by using a posture detection model according to the positions of the stationary target in the plurality of continuous images to obtain a placement posture detection result of the stationary target. In the case that the running condition indicates that there are at least two moving targets, and the at least two moving targets include a clamping device and a clamped article, whether the clamping device illegally clamps the clamped article is determined according to the positions of the clamping device and the clamped article in the plurality of continuous images and the type of the clamped article. The abnormality detection accuracy of a complex operation scene can be improved.

Description

Technical Field

[0001] This application relates to the field of warehousing and logistics technology, and in particular to an anomaly detection method and server. Background Technology

[0002] With the rapid development of warehousing and logistics, traditional warehousing and logistics sectors face numerous safety and economic problems, such as improper personnel attire, violations of regulations, and non-standard operations. Manual inspections are extremely labor-intensive and difficult to implement 24 / 7 monitoring. This has led to the development of visual expert models for abnormal behavior detection; however, these models suffer from low accuracy in complex operational scenarios. Summary of the Invention

[0003] This application provides an anomaly detection method and server to solve the problem of low anomaly detection accuracy in complex operation scenarios.

[0004] In a first aspect, some embodiments provide an anomaly detection method, the method comprising:

[0005] The video to be detected is acquired, and target detection is performed on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images;

[0006] Based on the position of at least one target in multiple consecutive images, determine the running status of at least one target;

[0007] When the running status indicates the presence of a stationary target, the placement posture is detected by using a posture detection model based on the position of the stationary target in multiple consecutive images, and the placement posture detection result of the stationary target is obtained.

[0008] If the operation status indicates the presence of at least two moving targets, and the at least two moving targets include a clamping device and a clamped object, determine whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images and the type of the clamped object.

[0009] Technical Effects: Some embodiments provide an anomaly detection method that acquires a video to be detected and performs target detection on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images; based on the position of at least one target in the multiple consecutive images, the running status of at least one target is determined, thereby identifying stationary targets and moving targets; when the running status indicates the presence of a stationary target, a pose detection model is used to detect the placement pose of the stationary target based on its position in the multiple consecutive images, obtaining the placement pose detection result of the stationary target, reducing detection misjudgments caused by a single image; when the running status indicates the presence of at least two moving targets, and at least... When two moving targets include a clamping device and a clamped object, the system determines whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images, as well as the type of the clamped object. Since the posture of the moving object changes over time, the clamping specifications may differ for different types of clamped objects. Therefore, determining whether illegal clamping occurs by analyzing the positional changes in multiple consecutive images and the type of clamped object helps improve the accuracy of illegal clamping detection. Using different detection logics for stationary and moving targets, focusing on the placement posture detection of stationary targets and illegal clamping detection of moving targets respectively, helps improve the anomaly detection accuracy in complex work scenarios.

[0010] Secondly, some embodiments also provide a server including a processor configured to:

[0011] The video to be detected is acquired, and target detection is performed on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images;

[0012] Based on the position of at least one target in multiple consecutive images, determine the running status of at least one target;

[0013] When the running status indicates the presence of a stationary target, the placement posture is detected by using a posture detection model based on the position of the stationary target in multiple consecutive images, and the placement posture detection result of the stationary target is obtained.

[0014] If the operation status indicates the presence of at least two moving targets, and the at least two moving targets include a clamping device and a clamped object, determine whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images and the type of the clamped object.

[0015] Technical Effects: Some embodiments provide a server that acquires a video to be detected and performs target detection on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images; based on the position of at least one target in the multiple consecutive images, the running status of at least one target is determined, thereby identifying stationary and moving targets; when the running status indicates the presence of a stationary target, a pose detection model is used to detect the placement pose based on the position of the stationary target in the multiple consecutive images, obtaining the placement pose detection result of the stationary target, reducing detection misjudgments caused by a single image; when the running status indicates the presence of at least two moving targets, and at least two In cases where a moving target includes a clamping device and a clamped object, the system determines whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images, as well as the type of the clamped object. Since the posture of the moving object changes over time, the clamping specifications may differ for different types of clamped objects. Therefore, determining whether illegal clamping occurs by analyzing the positional changes in multiple consecutive images and the type of clamped object helps improve the accuracy of illegal clamping detection. Furthermore, employing different detection logics for stationary and moving targets, focusing on the placement posture detection of stationary targets and illegal clamping detection of moving targets respectively, helps improve the anomaly detection accuracy in complex work scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0018] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the hardware configuration of the control device provided in some embodiments of this application;

[0020] Figure 4 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0021] Figure 5 A flowchart illustrating an anomaly detection method provided in some embodiments of this application;

[0022] Figure 6A schematic diagram illustrating target screening provided for some embodiments of this application;

[0023] Figure 7 Schematic diagram of image samples matching new requirements provided in some embodiments of this application;

[0024] Figure 8 Schematic diagram of image samples provided for some embodiments of this application;

[0025] Figure 9 Schematic diagram of image samples provided for other embodiments of this application;

[0026] Figure 10 Illustrations of image samples provided for some embodiments of this application;

[0027] Figure 11 A schematic diagram of the overall flow of the anomaly detection method provided in some embodiments of this application;

[0028] Figure 12 A schematic diagram of the anomaly detection process for moving targets provided in some embodiments of this application;

[0029] Figure 13 This is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0030] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0031] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0032] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0033] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0034] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0035] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0036] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0037] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0038] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0039] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0040] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0041] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0042] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0043] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0044] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0045] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0046] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0047] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0048] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0049] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0050] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0051] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0052] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0053] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0054] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0055] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0056] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC module 133.

[0057] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0058] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0059] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0060] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0061] In some embodiments, to enable user interaction, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can (control the display device) provide a user interface, allowing users to interact with the display device 200 and supporting the running of various applications.

[0062] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0063] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0064] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run the applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0065] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0066] like Figure 4 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0067] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0068] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0069] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 4 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0070] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0071] With the rapid development of warehousing and logistics, traditional warehousing and logistics sectors face numerous safety and economic losses. These include issues such as improper personnel attire, violations of regulations, and non-standard operations. Manual inspections are extremely labor-intensive and can only be conducted on a spot basis or for post-event traceability, making 24 / 7 monitoring difficult. This has led to the development of visual expert models for anomaly detection. However, while visual expert models can detect some anomalies, they rely on post-processing rules and suffer from numerous false positives and low accuracy in complex operational scenarios. In practical applications, manual re-inspection is still necessary, resulting in a significant workload and high time and effort commitment.

[0072] Based on this, this application proposes an anomaly detection method. It acquires a video to be detected and performs target detection on multiple consecutive images within the video, obtaining at least one target contained in each image. Based on the position of at least one target in the multiple consecutive images, it determines the running status of at least one target, thereby identifying stationary and moving targets. When the running status indicates the presence of a stationary target, it uses a pose detection model to detect the placement pose based on the position of the stationary target in the multiple consecutive images, obtaining the placement pose detection result of the stationary target, reducing detection errors caused by a single image. When the running status indicates the presence of at least two moving targets, and at least... When two moving targets include a clamping device and a clamped object, the system determines whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images, as well as the type of the clamped object. Since the posture of the moving object changes over time, the clamping specifications may differ for different types of clamped objects. Therefore, determining whether illegal clamping occurs by analyzing the positional changes in multiple consecutive images and the type of clamped object helps improve the accuracy of illegal clamping detection. Using different detection logics for stationary and moving targets, focusing on the placement posture detection of stationary targets and illegal clamping detection of moving targets respectively, helps improve the anomaly detection accuracy in complex work scenarios.

[0073] In one exemplary embodiment, such as Figure 5 As shown, an anomaly detection method is provided, which can be applied to... Figure 1 Taking server 400 as an example, the explanation includes steps 502 to 508. Wherein:

[0074] Step 502: Obtain the video to be detected, and perform target detection on multiple consecutive images in the video to be detected, so as to obtain at least one target contained in each of the multiple images.

[0075] The video to be detected refers to video data for which anomaly detection is required. For example, it can originate from surveillance cameras. The video to be detected can be video captured over a historical period or video captured in real-time. Multiple consecutive images refer to images with a temporal relationship, which can be obtained by decoding the video to be detected. For example, decoding the video to be detected yields an image sequence, which can be processed by frame skipping to obtain multiple consecutive images, thus improving the algorithm's frame rate. For instance, target detection can be performed every other frame.

[0076] Optionally, the image sequence can be further processed to extract the Region of Interest (ROI) to obtain an ROI image sequence. The image sequence can also be preprocessed, for example, through image enhancement or deblurring, to obtain a preprocessed image sequence.

[0077] Object detection refers to the identification of targets in an image using visual processing algorithms. Machine learning algorithms, deep learning algorithms based on CNN or Transformer architectures, etc., can be used; the type of algorithm is not limited, taking YOLO as an example. In the warehousing and logistics field, detected targets can include workers, equipment (e.g., forklifts, clamp trucks, etc.), and items (e.g., display devices with packaging boxes, refrigerators, washing machines, air conditioners, etc.). Detected targets can be classified into one category, with each category including at least one target. For example, target number 0 represents workers, number 1 represents equipment, and number 2 represents items. Detected targets can also be classified into multiple categories, with each category having at least one target. For example, target number 0 represents workers, number 1 represents forklifts, number 2 represents clamp trucks, number 3 represents display device packaging boxes, and number 4 represents refrigerator packaging boxes.

[0078] Step 504: Determine the running status of at least one target based on the position of at least one target in multiple consecutive images.

[0079] Specifically, object detection can output the location of each object in the image. For example, the location of each object in the image can be marked using bounding boxes.

[0080] Motion status indicates the change in the target's position, categorized as either stationary or moving. Based on the position of each target across multiple consecutive images, its stationary or moving status can be precisely determined. For each target, the distance it has moved is determined based on its position across multiple consecutive images; if the moving distance is less than a preset distance, the target is identified as stationary; if the moving distance is not less than the preset distance, the target is identified as moving.

[0081] For example, the position of each target in the image is the coordinate of its center point. For each target, obtain the center point coordinates of the target in N consecutive images: (x1, y1), (x2, y2), ..., (x N ,y N ), calculate the distance D that the target moves across N images: If D is less than the preset distance T D If the target is stationary and D is not less than the preset distance T, then the target is stationary. D If so, then the target is a moving target.

[0082] In the warehousing and logistics field, anomaly detection primarily targets personnel and equipment. Therefore, targets can be filtered based on the location of at least one objective or a preset spatial relationship. Specifically, targets within a preset area are selected as targets to be detected, or targets conforming to a preset spatial relationship are selected as targets to be detected. The operational status of at least one target is determined based on its position in multiple consecutive images. For example... Figure 6 The diagram shows target filtering in some embodiments. The dashed boxes in the diagram represent target boxes, and the items in target boxes 1-4 are the filtered targets to be detected.

[0083] The preset region can be a specified polygonal area, and items outside the preset region are filtered out. The preset spatial relationship can be the intersection or proximity of the bounding boxes of multiple targets. The preset spatial relationship can be represented as follows:

[0084]

[0085] Where boundingbox1 represents the bounding box of target 1, and boundingbox2 represents the bounding box of target 2, if IOU <T IOU If target 1 and target 2 do not conform to the preset spatial relationship, then IOU ≥ T IOU If so, then target 1 and target 2 conform to the preset spatial relationship.

[0086] Step 506: When the running status indicates the presence of a stationary target, the placement posture is detected using a posture detection model based on the position of the stationary target in multiple consecutive images, and the placement posture detection result of the stationary target is obtained.

[0087] The pose detection model refers to a machine learning model used to analyze the placement pose of a stationary target, such as MobileNet, VIT, and ShuffleNet. Placement pose includes abnormal placement poses and normal placement poses. For example, abnormal placement poses include vertical placement, horizontal placement, and suspension, while normal placement poses include horizontal placement.

[0088] The placement pose detection result refers to the detection result of the placement pose of a stationary target output by the pose detection model, which can include the specific type of placement pose and the confidence level. The placement pose detection result for each stationary target in each image output by the pose detection model can include multiple placement poses and the confidence level for each pose, with the sum of the confidence levels for each pose being 1. For example, the placement pose detection result includes {(vertical, 70%), (horizontal, 10%), (suspended, 10%), (flat, 10%)}.

[0089] In some embodiments, the placement posture with a confidence level greater than a preset value, or the posture with the highest confidence level, can be used as the placement posture of the stationary target in the image. The placement posture detection result can include the placement posture of the stationary target in each of multiple consecutive images. In other embodiments, if the confidence level of any placement posture of the target is greater than a preset value, and the number of corresponding images exceeds a preset number, then that placement posture is used as the placement posture of the target. The placement posture detection result can include the placement posture of the stationary target in one or more images. For example, in 10 images, 6 are vertically placed, 3 are horizontally placed, and 1 is flat.

[0090] By using a pose detection model to detect placement posture, abnormally placed targets can be detected in a timely manner. In some embodiments, when the placement posture detection result indicates that the placement of a stationary target is abnormal, an abnormal placement warning message for the stationary target matching the placement posture detection result can be output, and the bounding box position of the stationary target in the image can also be output.

[0091] Step 508: If the operation status indicates the presence of at least two moving targets, and the at least two moving targets include a clamping device and a clamping object, determine whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamping object in multiple consecutive images and the type of the clamping object.

[0092] In the warehousing and logistics sector, moving objects primarily include clamping equipment and clamped goods. Clamping equipment refers to devices used for gripping and moving goods, such as forklifts and clamping vehicles. Clamped goods refer to items that are gripped and moved by the clamping equipment. Clamped goods may vary in shape, weight, characteristics, purpose, and physical properties. For example, based on physical properties, clamped goods can be categorized into box types and storage basket types. Based on purpose, clamped goods can be categorized into display device packaging boxes and refrigerator packaging boxes.

[0093] Different types of items require different clamping operation specifications. Improper clamping refers to the clamping equipment's operation of clamping items without conforming to preset specifications. This can be caused by positional deviations, improper force, or incompatibility with the type of item being clamped, potentially leading to damage or loss of the item. Therefore, the presence of improper clamping can be determined by examining the clamping equipment, the location of the clamped item, and the type of item being clamped.

[0094] In the above anomaly detection method, the video to be detected is acquired, and target detection is performed on multiple consecutive images in the video to obtain at least one target contained in each image. Based on the position of at least one target in the multiple consecutive images, the running status of at least one target is determined, thereby identifying stationary and moving targets. When the running status indicates the presence of a stationary target, a pose detection model is used to detect the placement pose based on the position of the stationary target in the multiple consecutive images, obtaining the placement pose detection result of the stationary target, reducing detection errors caused by a single image. When the running status indicates the presence of at least two moving targets, and at least two moving targets... When the target includes both clamping equipment and clamped objects, the system determines whether the clamping equipment is illegally clamping the object based on the positions of the clamping equipment and the clamped object in multiple consecutive images, as well as the type of the clamped object. Since the posture of moving objects changes over time, the clamping specifications for different types of clamped objects may differ. Therefore, determining whether illegal clamping occurs by analyzing positional changes and the type of clamped object in multiple consecutive images helps improve the accuracy of illegal clamping detection. Furthermore, employing different detection logics for stationary and moving targets, focusing on the placement posture detection of stationary targets and illegal clamping detection of moving targets respectively, helps improve the anomaly detection accuracy in complex work scenarios.

[0095] In one exemplary embodiment, determining whether the clamping device is illegally clamping the item based on the respective positions of the clamping device and the clamped item in multiple consecutive images and the type of the clamped item includes: determining the clamping state of the clamping device on the clamped item based on the respective positions of the clamping device and the clamped item in multiple consecutive images; and, if the clamping state indicates that the clamping device is clamping the item, determining whether the clamping device is illegally clamping the item based on the type of the clamped item.

[0096] The clamping state refers to the state in which the clamping device clamps or does not clamp the item. The clamping state can be determined based on the positional changes of the clamping device and the clamped item across multiple consecutive images. For example, if the relative position between the clamping device and the clamped item remains unchanged (or changes less than a preset value) across multiple consecutive images, the clamping state is determined to be that the clamping device is clamping the item. If, after the clamping device is determined to be clamping the item, the relative position between the clamping device and the clamped item continuously increases in subsequent images, the clamping state is determined to be that the clamping device changes from clamping the item to not clamping it.

[0097] The clamping status indicator shows that when a clamping device clamps an item, the clamping status includes either normal clamping or illegal clamping. Since clamping specifications may differ for different types of items, the type of item being clamped can determine whether the clamping device is illegally clamping it. For example, clamping a display device packaging box is prohibited; therefore, if the clamping device clamps a display device packaging box, it is considered illegal clamping.

[0098] In this embodiment, the clamping state is determined by the positions of the clamping device and the clamped object in multiple consecutive images, which can avoid misjudgment caused by a single image and improve the detection accuracy of the clamping state. Combining the type of the clamped object further determines whether it is an illegal clamping, improving the matching degree between anomaly detection and the actual scene.

[0099] In an exemplary embodiment, determining the clamping state of the clamping device on the clamped item based on the respective positions of the clamping device and the clamped item in multiple consecutive images includes: for each image, determining the distance between the clamping device and the clamped item in the image based on their respective positions, and determining the clamping state of the clamping device on the clamped item based on the distance between the clamping device and the clamped item in multiple consecutive images; or, determining a horizontal correlation coefficient based on the horizontal position of the clamping device and the clamped item in multiple consecutive images, determining a vertical correlation coefficient based on the vertical position of the clamping device and the clamped item in multiple consecutive images, and determining the clamping state of the clamping device on the clamped item based on the horizontal and vertical correlation coefficients; or, weighting the distance between the clamping device and the clamped item in multiple consecutive images, the horizontal correlation coefficient, and the vertical correlation coefficient, and determining the clamping state of the clamping device on the clamped item based on the weighted processing result.

[0100] In this context, when the clamping device clamps an item, the relative distance between the clamping device and the item remains constant or changes less than a preset value across multiple consecutive images. Therefore, the distance d between the clamping device and the item in each image is calculated. i Based on the distance d corresponding to multiple consecutive images i The mean, median, total number, and maximum number determine the clamping state.

[0101] For example, the trajectory of the center point of the clamping device in N consecutive images is: vehicle_track = [(vx1, vy1), (vx2, vy2), ..., (vx N ,vy N The trajectory of the center point of the object being held is: box_track = [(bx1,by1),(bx2,by2),…,(bx1,by1),…,(bx2,by2 ... N by NThe distance d between the clamping device and the clamped item. i It can be represented as:

[0102] Based on the distance d corresponding to N consecutive images i Let's take the average distance to determine the clamping state as an example. The average distance can be expressed as:

[0103] For objects whose center point is obscured or that are large in size, the distance d may be affected. i The fluctuation is relatively large; therefore, this application embodiment proposes to utilize the area of ​​the clamped object to adjust the distance d. i Normalize:

[0104]

[0105] The average distance can be expressed as:

[0106] If D mean If the distance is less than the preset threshold, the clamping device is determined to be clamping the item; otherwise, the clamping device is determined not to be clamping the item.

[0107] The clamping state can also be determined based on trajectory correlation. Trajectory correlation includes horizontal correlation coefficient and vertical correlation coefficient, which can be determined using the following correlation coefficient calculation formula:

[0108]

[0109] Where, p i q i These are the variable values ​​of the clamping device v and the clamped item b in the i-th image; is the mean of variables v and b; n is the number of images. The value of corr ranges from -1 to 1, where 1 indicates perfect positive correlation, 0 indicates no correlation, and -1 indicates perfect negative correlation.

[0110] Horizontal position of the clamping device v: vehicle_track_x = [vx1, vx2, ..., vx] N The vertical position of the clamping device v: vehicle_track_y = [vy1, vy2, ..., vy N The horizontal position of the object b being held: box_track_x = [bx1, bx2, ..., bx] N The vertical position of the object b being held: box_track_y = [by1,by2,…,by] N Then the horizontal correlation coefficient is: r x =corr(vehicle)x ,box x Vertical correlation coefficient: r y =corr(vehicle) v ,box y ).

[0111] The clamping state is determined by the horizontal correlation coefficient and the vertical correlation coefficient. For example, if r x and r y All are greater than the threshold T r If the movement trends of the clamping device v and the clamped item b are consistent and they move synchronously, then the clamping device v is clamping the item b. Conversely, if the movement trends are not consistent, then the clamping device v is not clamping the item b.

[0112] Distance-based methods for determining clamping status are simple and direct, filtering out parallel or accidentally approaching targets. They are suitable for small targets or scenarios with accurate center points, but are prone to errors when the center point of large targets is offset. Trajectory-based methods are robust to large targets and scenarios with offset center points, but are prone to errors with parallel or accidentally approaching targets. Therefore, to improve accuracy, an improved method combining distance and trajectory correlation can be used. This involves iterating through each clamping device v and the clamped item b, and calculating a weighted average distance and trajectory correlation. Where w1 and w2 represent weighting coefficients, corr i Trajectory correlation can be represented by r x and r y The weighted average is then used to obtain the results. Combinations with high relevance and small distances are selected, where the clamping device (v) clamps the item (b) within the combination.

[0113] In this embodiment, determining the clamping state by observing distance changes in multiple consecutive images is beneficial for accurately identifying the clamping state in scenarios with small targets or accurate center points. Using horizontal and vertical correlation coefficients to reflect the synchronicity of movement between targets is beneficial for accurately identifying the clamping state in scenarios with large targets or offset center points. The method of combining distance and trajectory correlation is beneficial for improving the accuracy of clamping state recognition in complex operation scenarios.

[0114] In an exemplary embodiment, determining whether a clamping device is illegally clamping an item based on the type of the item being clamped includes: if the type of the item being clamped is a display device packaging box and the confidence level is not less than a first confidence level threshold, or if the type of the item being clamped is a refrigerator packaging box, determining the visible character type of the item being clamped; and if the visible character type corresponding to multiple consecutive images is a preset character type and the confidence level is not less than a second confidence level threshold, determining that the clamping device is illegally clamping the item.

[0115] The type of item being carried can be identified using a pre-trained machine learning model. The identification results can include multiple types and a confidence level for each type, with the sum of the confidence levels for all types being 1. For example, the identification results could be {(display device packaging box type, 90%), (refrigerator packaging box type, 10%)}.

[0116] If the identification result indicates that the item being clamped is a display device packaging box, and the confidence level corresponding to the display device packaging box type is not less than the first confidence level threshold, it indicates that the clamping device is highly likely to clamp the display device packaging box type. Since the display device is a fragile item and cannot be clamped, it can be directly determined that the clamping device is illegally clamping the display device packaging box.

[0117] Other types of items are allowed to be clamped, but different types of items have different clamping rules. For refrigerator boxes, clamping is not allowed on the side with the printed preset characters. Therefore, if the recognition result indicates that the clamped item is a refrigerator box, and the confidence level corresponding to the refrigerator box type is not less than the first confidence threshold, it indicates that the clamping device has a high probability of clamping the refrigerator box type. Further, a character recognition algorithm is used to identify the visible character type of the clamped item. For example, the character recognition algorithm can be a multimodal large model (such as qwenvl, internvl, etc.) or an OCR small model (such as paddleocr, etc.). The results identified by the character recognition algorithm include multiple visible character types and the confidence level of each type, with the sum of the confidence levels of multiple types being 1. For example, the recognition result is {(Chinese character type, 90%), (English character type, 10%)}. For example, if the visible character type corresponding to multiple consecutive images is Chinese character type, and the confidence level corresponding to the Chinese character type is not less than the second confidence threshold, it is determined that the clamping device is illegally clamping the refrigerator box.

[0118] In this embodiment, different violation clamping detection rules are applied to the display device packaging box and the refrigerator packaging box to accurately determine whether there is any violation clamping; the confidence level condition is used to further improve the reliability of identification.

[0119] In an exemplary embodiment, determining whether a clamping device is illegally clamping an item based on the type of the item being clamped includes: if the type of the item being clamped is a display device packaging box and the confidence level is less than a first confidence threshold; or if the type of the item being clamped is a display device packaging box in a first preset number of images and the confidence level is not less than the first confidence threshold; or if the type of the item being clamped is a refrigerator packaging box and the confidence level of the visible surface character type is a preset character type is less than a second confidence threshold; or if the type of the item being clamped is a refrigerator packaging box and the visible surface character type is a preset character type in a second preset number of images and the confidence level is not less than the second confidence threshold, then determining whether the clamping device is illegally clamping an item based on multiple consecutive images using a large model.

[0120] Specifically, for cases where the confidence level of the model recognition result is low, a large model is used for re-judgment of illegal clamping. Specifically, for the display device packaging box type, if the confidence level of the display device packaging box type in the model recognition results of multiple consecutive images is less than a first confidence threshold, a large model is used to re-judge illegal clamping for all consecutive images. If only a portion of the images (e.g., a first preset number) in multiple consecutive images represent the display device packaging box type, and the confidence level of the display device packaging box type is not less than the first confidence threshold, a large model is used to re-judge illegal clamping for images with a confidence level less than the first confidence threshold.

[0121] For refrigerator packaging boxes, if the visible character type of multiple consecutive images is a preset character type, and the confidence level of the preset character type is less than the second confidence threshold, a large model is used to re-determine the violation of clamping in multiple consecutive images. If only a portion of the multiple consecutive images (such as a second preset number) have the visible character type of the preset character type, and the confidence level of the preset character type is not less than the second confidence threshold, a large model is used to re-determine the violation of clamping in images with a confidence level less than the second confidence threshold.

[0122] The large model performs a second review of violations involving clamping, and can construct review prompts based on the violation judgment rules. For example, the review prompts could be: Analyze the type of the clamped item; the violation judgment rules are: if the clamped item is a display device packaging box, then it is determined to be a violation; if the clamped item is a refrigerator packaging box and the visible surface character type is a preset character type, then it is determined to be a violation.

[0123] In this embodiment, when the confidence level of the model's type recognition result is low, and in complex scenarios where the type recognition is reliable but the details are vague, the strong understanding ability of the large model is used to compensate for the limitations of the model recognition, reduce false judgments, and improve the accuracy of anomaly detection. Different re-judgment trigger conditions are adopted for different types of clamped objects to adapt to the characteristics of different objects and the complexity of the scenario, thereby improving the reliability of the detection results.

[0124] In an exemplary embodiment, the anomaly detection method further includes: when the clamping state indication of the clamping device on the clamped item changes from clamping the clamped item to not clamping, determining the speed changes of the clamping device and the clamped item respectively based on their respective positions in multiple consecutive images, and determining whether the clamped item has fallen from a high position based on the speed changes.

[0125] The clamping status indicator changes from clamping an item to not clamping it. This indicates that the clamping device may have placed the item normally on the ground, or that the item was not completely placed on the ground during transport but fell to the ground in free fall at a certain distance, i.e., a high-altitude fall. The processor can determine the speed change based on the positional changes of the clamping device and the item, thereby determining whether the item has experienced a high-altitude fall.

[0126] The velocity change can be represented by the velocity in the vertical direction. In some embodiments, the vertical velocity v of the clamping device... vehicle And the vertical velocity v of the object being gripped box It can be represented as follows:

[0127]

[0128] If the clamping device changes from clamping an item to not clamping, v box =0, v vehicle If the change indicates that the item being held is properly placed on the ground; if v box Change, v vrhicle Change, and v box >v vrhicOe If this result is present in multiple consecutive images, then it is determined that the object being held fell from a high position.

[0129] When it is determined that the object being held has fallen from a height, a high-rise fall warning message is generated and sent to the terminal.

[0130] In this embodiment, when the clamping device changes from clamping the object to not clamping it, the speed changes of the clamping device and the object being clamped can automatically and accurately identify high-level fall scenarios, reduce labor costs, improve the efficiency and accuracy of anomaly detection, and enhance the ability to manage safe operations.

[0131] In an exemplary embodiment, the anomaly detection method further includes: if a preset number of images indicate that the clamped item has fallen from a high position, determining whether the clamped item has fallen from a high position using a large model based on multiple consecutive images.

[0132] In cases where only a subset of images (e.g., a predetermined number) in a series of consecutive images indicate a high-level drop, a large model is used to re-determine the high-level drop across the multiple images. After the initial algorithm's judgment, there may be misjudgments due to blurring, occlusion, or angular deviation. Therefore, using a large model for re-judgment can correct misjudgments or reduce missed judgments.

[0133] In this embodiment, by leveraging the strong semantic understanding and complex scene parsing capabilities of the large model, the system outputs a re-judgment result of whether a fall from a high position has occurred through the contextual correlation of multiple consecutive images, and uses this result as the final judgment. This approach can compensate for the limitations of the initial algorithm in recognizing blurred or occluded scenes. This detection method, which combines initial judgment with re-judgment by the large model, improves the accuracy of high-position fall detection.

[0134] In an exemplary embodiment, the anomaly detection method further includes: when the placement posture detection result of the stationary target indicates an abnormal placement posture and the confidence level of the abnormal placement posture is less than a third confidence threshold, or when the placement posture detection result corresponding to a third preset number of images indicates an abnormal placement posture and the confidence level of the abnormal placement posture is not less than the third confidence threshold, using a large model to re-determine the placement posture of multiple consecutive images to obtain the placement posture re-determination result of the stationary target.

[0135] Among them, the placement posture detection result indicates abnormal placement posture, indicating that the model detects abnormal placement postures such as vertical or horizontal placement of the stationary target; the confidence of the abnormal placement posture is less than the third confidence threshold, indicating that the reliability of the detection result is low. A large model is used to re-judge the placement posture of the images with confidence less than the third confidence threshold to obtain the re-judgment result of the placement posture of the stationary target.

[0136] If only a portion of multiple consecutive images (such as a third preset number) have abnormal placement postures, and the confidence level of the abnormal placement postures is not less than the third confidence threshold, the detection results may contain misjudgments or missed detections. A large model is used to re-judge the placement postures of multiple consecutive images to obtain the re-judge results of the placement postures of stationary targets.

[0137] In this embodiment, when the confidence level of the model detection results is low or only some of them are highly reliable, a large model is used for re-judgment to avoid model misjudgment and missed judgment, thereby improving the accuracy of abnormal placement detection. For complex scenes such as blurry or special angles, this solves the problem of poor detection accuracy of traditional models in complex scenes.

[0138] In an exemplary embodiment, the anomaly detection method further includes: upon receiving a new anomaly detection request, acquiring an image sample matching the new request and an instruction sample corresponding to the image sample; the instruction sample includes at least one of a global image description question-and-answer pair, an instruction expression description question-and-answer pair, and a thought chain prompt question-and-answer pair; performing performance verification on an initial large model based on the image sample and the corresponding instruction sample; if the performance verification passes, using the initial large model for anomaly detection; if the performance verification fails, fine-tuning the initial large model based on the image sample and the corresponding instruction sample to obtain an adjusted large model, and using the adjusted large model for anomaly detection.

[0139] Among them, the new demand for anomaly detection refers to new requirements arising from business expansion, risk escalation, etc., where existing anomaly types do not cover the abnormal scenarios in the new demands. For example, existing anomaly types mainly include abnormal placement of stationary targets, unauthorized clamping of moving targets, and falls from heights, while new demands may include smoking detection, not wearing work clothes, not wearing safety helmets, etc.

[0140] The new requirement is that the image samples matched should contain targets exhibiting smoking postures. For example... Figure 7 The diagram shows image samples for matching new requirements in some embodiments. The instruction samples corresponding to the image samples include at least one of the following: global image description question-and-answer pairs, instruction expression description question-and-answer pairs, and thought chain prompt question-and-answer pairs. Global image description question-and-answer pairs refer to question-and-answer pairs that provide a global description of the image sample; instruction expression description question-and-answer pairs refer to question-and-answer pairs that describe and analyze the target type marked by the target box in the image; and thought chain prompt question-and-answer pairs refer to question-and-answer pairs that guide the large model to gradually analyze and reason to obtain the reasoning result through a thought chain approach.

[0141] The initial large model refers to a large model with basic image recognition capabilities. When a new requirement is received, the initial large model can be used to verify its performance using image samples matching the new requirement and the corresponding instruction samples. If the performance indicators meet the preset conditions, the performance verification is passed, and the initial large model can be directly used for anomaly detection, which is beneficial for the rapid deployment of new algorithms. For example, if the new requirement is smoke detection, using the initial large model for smoke detection can meet the performance requirements.

[0142] In the warehousing and logistics field, anomaly detection scenarios are quite complex, and the initial large-scale model may fail performance validation. To improve the recognition accuracy of the initial large-scale model, it can be fine-tuned using image samples matching new requirements and corresponding instruction samples. Using the adjusted large-scale model can enhance the anomaly detection capability in specific scenarios. For existing anomaly types, the large-scale model is also fine-tuned using the aforementioned image samples and instruction samples.

[0143] like Figure 8 The diagram shown is a schematic representation of image samples in some embodiments. Exemplarily, Figure 8 The corresponding global image description question-and-answer pair could be: Q1: Please describe this image in detail. A1: This image shows a scene of a warehouse or logistics center. A forklift is moving a large cardboard box, which has the letter "xxx" on it, indicating it might be some kind of household appliance. In front of the forklift, there are two upright rectangular cardboard boxes on the ground, with blue markings. More cardboard boxes can be seen neatly stacked together in the background. There are also some scattered items on the ground, including a blue package and some white items. The overall environment appears to be an area used for storing and moving goods.

[0144] like Figure 9 The diagram shown is a schematic representation of image samples in other embodiments. Exemplarily, Figure 9 The corresponding Q&A pair can be: Q2: Please describe the product type in the boxes marked 1-4 in the image. Product types include display devices and air conditioners. Please first answer the product type, and then give the reason. A2: 1. Product type: Air conditioner. Reason: The box in rectangle 1 is long and vertical, a shape commonly used for air conditioner packaging. 2. Product type: Air conditioner. Reason: The box in rectangle 2 is also long and vertical, consistent with typical air conditioner packaging. 3. Product type: Display device. Reason: The box in rectangle 3 is relatively flat and rectangular, marked with "xxx". This shape and marking are commonly used for display devices. 4. Product type: Display device. Reason: The box in rectangle 4 is flat and rectangular, marked with "xxx". This shape and marking are commonly used for display device packaging.

[0145] like Figure 10 The image shown is a schematic diagram of image samples in some other embodiments. Figure 10 The three images a, b, and c in the image are images extracted from the video. For example, a thought-chain prompt question-and-answer pair could be: Q3: These images were extracted from the video and sent in chronological order. Please first analyze the type of item being clamped by the clamping device, and then, combining this with the violation clamping judgment rules (clamping a display device packaging box constitutes violation clamping), determine whether violation clamping exists in the video. Please first answer the product type, then answer whether violation clamping exists with "yes" or "no," and give a reason. A3: Product type: TV, yes, reason: The item clamped by the clamping device is marked with "xxx" and is flat and rectangular in shape, typically used for display device packaging. According to the violation clamping judgment rules, clamping a display device packaging box is a violation, therefore violation clamping exists.

[0146] In this embodiment, by fine-tuning the initial model using image samples and instruction samples to meet new requirements for anomaly detection, a fine-tuned large model is obtained. This allows the large model to quickly grasp the judgment logic of new anomalies and adapt to new requirements. It also enhances the large model's understanding of specific scenarios and improves the accuracy of anomaly detection in complex work scenarios.

[0147] To illustrate the anomaly detection method and server in this solution in detail, the following is a detailed embodiment:

[0148] The anomaly detection method proposed in this application can be applied to various fields. Taking its application in the logistics and warehousing field as an example, the method first detects the workers and machinery in the monitored area. When workers or machinery perform operations such as handling goods, a visual expert model is used to determine whether there are abnormal postures, such as vertical / horizontal placement, falling from a height, or illegal clamping. Then, a large model is used to further analyze one frame or multiple consecutive frames of video images. Based on the image / video understanding capabilities of the large model, it is determined whether an anomaly has occurred. When an anomaly is determined, the anomaly data is reported to the system. The system can perform manual review as needed and push the alert to the terminal.

[0149] like Figure 11 The diagram illustrates the overall flow of anomaly detection methods in some embodiments. The process involves reading the video to be detected, extracting the detection area, and performing target detection within that area to obtain at least one target. Targets are then filtered based on location or preset spatial relationships, and the movement of at least one of the filtered targets is tracked. For stationary targets, a posture detection model is used to detect placement posture. If an abnormal placement posture is detected and the confidence level is greater than a preset threshold, an alarm is output; otherwise, a larger model is used for re-judgment, and if the re-judgment result indicates an abnormal placement posture, an alarm is output. For moving targets, the location of the clamping device and the object being clamped, as well as the type of the object being clamped, are used to determine whether there is illegal clamping or a fall from a high position. If an anomaly is detected and the confidence level is greater than a preset threshold, an alarm is output; otherwise, a larger model is used for re-judgment, and if the re-judgment result indicates illegal clamping or a fall from a high position, an alarm is output.

[0150] The processing of the video to be detected mainly includes: decoding the video to obtain an image sequence, and extracting the Regions of Interest (ROIs) from the images according to the set detection regions, resulting in the ROI sequence img1, img2, img3, ..., imgn... During the actual algorithm execution, frame skipping is usually performed to increase the algorithm's frame rate; for example, detection is performed every other frame, resulting in the final image sequence to be detected as image1, image2, ..., imagen... Optionally, the image sequence can be preprocessed, such as image enhancement or deblurring.

[0151] Object detection primarily refers to the real-time detection of multiple consecutive images using visual expert models. Visual expert models can refer to machine learning algorithms, deep learning algorithms based on CNN or transformer architectures, etc., with no limitation on algorithm type. Taking YOLO as an example, at least one detected object includes workers / equipment / items. Workers refer to personnel working within the area; equipment includes forklifts, clamp trucks, and other goods transport equipment; and items refer to the actual transported products, such as display devices with packaging boxes, refrigerators, washing machines, and air conditioners. In the actual algorithm, equipment / items can be classified as a single category (e.g., target number 0-workers, 1-equipment, 2-items), or multiple categories (0-workers, 1-forklift, 2-clamp truck, 3-display device packaging box, 4-refrigerator packaging box, etc.).

[0152] Target filtering refers to selecting targets to be detected based on location or preset spatial relationships. Target filtering based on location involves detecting the state of items within a specified area, which can be a defined polygonal region, filtering out items outside this area. Target filtering based on preset spatial relationships involves filtering groups of people / items that are close to or intersect with the selected item. Preset spatial relationships can be represented as follows:

[0153]

[0154] Where boundingbox1 represents the bounding box of target 1, and boundingbox2 represents the bounding box of target 2, if IOU <T IOU If target 1 and target 2 do not conform to the preset spatial relationship, then IOU ≥ T IoU If so, then target 1 and target 2 conform to the preset spatial relationship.

[0155] Target tracking is primarily used to determine whether a target is stationary or moving. Tracking algorithms such as DeepSORT, ByteTrack, and FairMOT can be used. If the target's position remains unchanged for N1 consecutive frames (N1>=1), it is considered a stationary target; if the target's position changes, it is considered a moving target. The determination method is as follows: For example, the position of each target in the image is the coordinates of its center point. For each target, obtain the center point coordinates of the target in N consecutive images: (x1,y1), (x2,y2), ..., (x... N ,y N ), calculate the distance D that the target moves across N images: If D is less than the preset distance T D If the target is stationary and D is not less than the preset distance T, then the target is stationary. D If so, then the target is a moving target.

[0156] For anomaly detection of stationary targets, specifically for stationary targets such as display device packaging boxes, target classification algorithms, such as MobileNet, VIT, and ShuffleNet, are used to determine whether the item has an abnormal posture such as being placed vertically, horizontally, or suspended, or a normal posture (laid flat).

[0157] a) If the above-mentioned abnormal posture exists, the confidence level is greater than the threshold T, and the abnormal posture exists for N2 consecutive frames (N2>=1), then the stationary target is determined to be abnormally placed, and the warning information and target box position are output.

[0158] b) If the above-mentioned abnormal posture exists, but the confidence level is less than the threshold T, or the confidence level is greater than the threshold T, but only some frames in N2 consecutive frames (N2>=1) exist, then the image is input into the large model for re-judgment.

[0159] c) If the large model analyzes the image sequence and determines that there is an abnormal pose, it outputs a warning message and the location of the target box; otherwise, it determines that the image is normal and does not output a warning.

[0160] d) If the confidence level of the normal posture is greater than the threshold T, then the stationary target is directly determined to be normally placed.

[0161] For stationary targets such as refrigerator packaging boxes, target classification algorithms, such as MobileNet, VIT, and ShuffleNet, are used to determine whether the target has a vertical or suspended posture.

[0162] a) If the above abnormal posture exists, the confidence level is greater than the threshold T, and the abnormal posture exists for N2 consecutive frames (N2>=1), then the stationary target is determined to be abnormally placed, and the warning information and target box position are output.

[0163] b) If the above-mentioned abnormal posture exists, but the confidence level is less than the threshold T, or the confidence level is greater than the threshold T, but only some frames in N2 consecutive frames (N2>=1) exist, then the image is input into the large model for re-judgment.

[0164] c) If the large model analyzes the image sequence and determines that there is an abnormal pose, it outputs a warning message and the location of the target box; otherwise, it determines that the image is normal and does not output a warning.

[0165] d) If the confidence level of the normal posture is greater than the threshold T, then the stationary target is directly determined to be normally placed.

[0166] For other types, abnormal placement is usually not an issue, and the posture can be directly determined to be normal (for special scenarios where posture determination is required, methods a) and b) can be used).

[0167] Anomaly detection of moving targets primarily determines whether there are any abnormal acts such as unauthorized grabbing or falls from heights. For example... Figure 12The diagram illustrates the anomaly detection process for moving targets in some embodiments. First, the clamping state is determined based on the position of the clamping device and the clamped object. Then, based on the type of the clamped object and the type of characters on the visible surface, it is determined whether there is illegal clamping or a fall from a height.

[0168] Clamping state determination: For example, the trajectory of the center point of the clamping device in N consecutive images is: vehicle_track = [(vx1, vy1), (vx2, vy2), ..., (vx N ,vy N The trajectory of the center point of the object being held is: box_track = [(bx1,by1),(bx2,by2),…,(bx1,by1),…,(bx2,by2 ... N by N The determination of the clamping state can be achieved using distance-based judgment methods, trajectory correlation-based judgment methods, and combined judgment methods.

[0169] Among them, the distance-based judgment method is based on the distance d between the clamping device and the clamped item. i It can be represented as: Based on the distance d corresponding to N consecutive images i Let's take the average distance to determine the clamping state as an example. The average distance can be expressed as:

[0170] For objects whose center point is obscured or that are large in size, the distance d may be affected. i The fluctuation is relatively large; therefore, this application embodiment proposes to utilize the area of ​​the clamped object to adjust the distance d. i Normalize: The average distance can be expressed as:

[0171] If D mean If the distance is less than the preset threshold, the clamping device is determined to be clamping the item; otherwise, the clamping device is determined not to be clamping the item.

[0172] Trajectory-based determination method: Horizontal position of the clamping device v: vehicle_track_x = [vx1, vx2, ..., vx] N The vertical position of the clamping device v: vehicle_track_y = [vy1, vy2, ..., vy N The horizontal position of the object b being held: box_track_x = [bx1, bx2, ..., bx] N The vertical position of the object b being held: box_track_y = [by1,by2,…,by] N Then the horizontal correlation coefficient is: rx =corr(vehicle) x ,box x Vertical correlation coefficient: r y =corr(vehicle) y ,box y ).

[0173] The clamping state is determined by the horizontal correlation coefficient and the vertical correlation coefficient. For example, if r x and r y All are greater than the threshold T r If the movement trends of the clamping device v and the clamped item b are consistent and they move synchronously, then the clamping device v is clamping the item b. Conversely, if the movement trends are not consistent, then the clamping device v is not clamping the item b.

[0174] Combined judgment method: Iterate through each clamping device v and the clamped item b, and calculate the weighted average distance and trajectory correlation. Where w1 and w2 represent weighting coefficients, corr i Trajectory correlation can be represented by r x and r y The weighted average is then used to obtain the results. Combinations with high relevance and small distances are selected, where the clamping device (v) clamps the item (b) within the combination.

[0175] If the clamping status indicator shows that the clamping device is clamping an item, further determination will be made as to whether there is any illegal clamping.

[0176] For display device packaging box type: a) If the confidence level of the packaging box type is greater than the threshold T, and abnormal consecutive N3 frames (N3>=1) occur, it is judged as illegal clamping, and the warning information and target box position are output.

[0177] b) If the confidence level for the packaging box type as a display device packaging box is less than the threshold T, or the confidence level is greater than the threshold T, but only some frames in N3 consecutive frames (N3>=1) show anomalies, then N2 consecutive frames are input into the large model for re-evaluation. The large model can employ multimodal large models such as the qwenvl series, internvl series, and GPT4-O, and fine-tuning can be performed based on the constructed instruction dataset.

[0178] c) If the large model analyzes the image sequence and determines that there is a violation of the bounding box rule, it will output a warning message and the location of the target box; otherwise, it will determine that the image is normal and will not output a warning message.

[0179] d) If the above two steps do not apply, then the condition is considered normal.

[0180] For refrigerator packaging boxes: Identify whether the visible character type of the item being clamped matches a preset character type to determine if clamping is illegal. The visible character type recognition algorithm can use a multimodal large model (such as qwenvl, internvl, etc.) or an OCR small model (such as paddleocr, etc.).

[0181] a) If the visible character type is not the default character type, it is considered a normal wrap.

[0182] b) If the visible character type is the default character type, it is considered an illegal wrapping.

[0183] c) For the illegal hugging scenario in b), if the confidence level is greater than the threshold T1 and it occurs for N3 consecutive frames (N3>=1), it is judged as illegal hugging, and the warning information and target box position are output.

[0184] d) For the illegal hugging scenario in b), if the confidence level is less than the threshold T1, or the confidence level is greater than the threshold T1, but only some frames in the consecutive N3 frames (N3>=1) have anomalies, then the consecutive N3 frames of images are input into the large model for re-judgment.

[0185] e) If the large model analyzes the image sequence and determines that there is illegal clamping, it will output a warning message and the location of the target box; otherwise, it will determine that the image is normal and will not output a warning.

[0186] If the clamping status indicator changes from clamping an item to not clamping it, further investigation is needed to determine if a fall from a height has occurred.

[0187] Track the ID of the gripping device / item, and the vertical velocity v of the gripping device. vehicle And the vertical velocity v of the object being gripped box It can be represented as follows:

[0188]

[0189] a) If v box =0, v vehicle If there is a change, it indicates that the item being held is properly placed on the ground;

[0190] b) If v box Change, v vehicle Change, and v box >v vehicle If this result is present in multiple consecutive images, then it is determined that the object being held fell from a high position.

[0191] c) If only some frames in a series of Nc frames (Nc>=1) have anomalies, then input the images of the series of Nc frames into the large model for re-judgment.

[0192] d) If the large model analyzes the image sequence and determines that there is a high-level drop, it will output a warning message and the target box position; otherwise, it will determine that the image is normal and will not output a warning.

[0193] Construction of large-scale model instruction samples: Large-scale models refer to multimodal large models such as qwenvl and internvl. In actual business applications, the models need to be fine-tuned according to business data. Based on the temporal video understanding and image understanding capabilities of multimodal large models, and combined with various methods such as global image description, instruction expression understanding, and video question answering, an instruction dataset is constructed using the mind chain prompting method.

[0194] Rapid Response to New Requirements Based on Large Models: When new requirements arise, prompt word engineering methods can be used to quickly deploy new algorithms based on existing large models. 1) Verify the current performance of the model using a test dataset. If the performance metrics meet user requirements, the algorithm can be deployed directly. 2) If the performance metrics do not meet the requirements, the model can be fine-tuned based on a small amount of data, and then the model performance can be verified. Once the requirements are met, the algorithm can be deployed.

[0195] Some embodiments provide an anomaly detection method and server, which acquire a video to be detected and perform target detection on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images; determine the running status of at least one target based on the position of at least one target in the multiple consecutive images, thereby identifying stationary targets and moving targets; when the running status indicates the presence of a stationary target, perform placement posture detection using a posture detection model based on the position of the stationary target in the multiple consecutive images to obtain the placement posture detection result of the stationary target, reducing detection misjudgments caused by a single image; when the running status indicates the presence of at least two moving targets, and at least When two moving targets include a clamping device and a clamped object, the system determines whether the clamping device is illegally clamping the object based on the positions of the clamping device and the clamped object in multiple consecutive images, as well as the type of the clamped object. Since the posture of the moving object changes over time, the clamping specifications may differ for different types of clamped objects. Therefore, determining whether illegal clamping occurs by analyzing the positional changes in multiple consecutive images and the type of clamped object helps improve the accuracy of illegal clamping detection. Using different detection logics for stationary and moving targets, focusing on the placement posture detection of stationary targets and illegal clamping detection of moving targets respectively, helps improve the anomaly detection accuracy in complex work scenarios.

[0196] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0197] Based on the same inventive concept, this application also provides an anomaly detection device for implementing the anomaly detection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more anomaly detection device embodiments provided below can be found in the limitations of the anomaly detection method described above, and will not be repeated here.

[0198] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores the video to be detected, abnormal placement posture detection results, abnormal placement posture re-judgment results, abnormal motion posture detection results, and abnormal motion posture re-judgment results. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an anomaly detection method.

[0199] Those skilled in the art will understand that Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0200] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An anomaly detection method, characterized in that, The method includes: The video to be detected is acquired, and target detection is performed on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images; Based on the position of at least one target in multiple consecutive images, determine the running status of at least one target; When the running status indicates the presence of a stationary target, the placement posture is detected by using a posture detection model based on the position of the stationary target in multiple consecutive images, and the placement posture detection result of the stationary target is obtained. If the operation status indicates that there are at least two moving targets, and the at least two moving targets include a clamping device and a clamping object, determine whether the clamping device illegally clamps the clamping object based on the position of the clamping device and the clamping object in multiple consecutive images and the type of the clamping object. The step of determining whether the clamping device illegally clamps the clamped item based on the respective positions of the clamping device and the clamped item in multiple consecutive images, as well as the type of the clamped item, includes: The clamping state of the clamping device on the clamped item is determined based on the positions of the clamping device and the clamped item in multiple consecutive images. When the clamping state indicates that the clamping device should clamp the clamped item, it is determined whether the clamping device has violated regulations by clamping the clamped item, based on the type of the clamped item. The step of determining whether the clamping device illegally clamps the clamped item based on the type of the clamped item includes: If the type of the clamped item is a display device packaging box and the confidence level is less than the first confidence level threshold, or if the type of the clamped item is a display device packaging box in a first preset number of images and the confidence level is not less than the first confidence level threshold, or if the type of the clamped item is a refrigerator packaging box and the confidence level of the visible surface character type is a preset character type is less than the second confidence level threshold, or if the type of the clamped item is a refrigerator packaging box and the visible surface character type is a preset character type in a second preset number of images and the confidence level is not less than the second confidence level threshold, a large model is used based on multiple consecutive images to determine whether the clamping device illegally clamps the clamped item. The method further includes: If the placement posture detection result of the stationary target indicates an abnormal placement posture and the confidence level of the abnormal placement posture is less than the third confidence level threshold, or if the placement posture detection result of the third preset number of images indicates an abnormal placement posture and the confidence level of the abnormal placement posture is not less than the third confidence level threshold, a large model is used to re-determine the placement posture of multiple consecutive images to obtain the placement posture re-determination result of the stationary target.

2. The method according to claim 1, characterized in that, Determining the clamping state of the clamping device on the clamped item based on the positions of the clamping device and the clamped item in multiple consecutive images includes: For each image, the distance between the clamping device and the clamped item in the image is determined based on their respective positions in the image. Based on the distance between the clamping device and the clamped item in multiple consecutive images, the clamping state of the clamping device on the clamped item is determined. Alternatively, based on the horizontal positions of the clamping device and the clamped item in multiple consecutive images, a horizontal correlation coefficient is determined; based on the vertical positions of the clamping device and the clamped item in multiple consecutive images, a vertical correlation coefficient is determined; and based on the horizontal and vertical correlation coefficients, the clamping state of the clamping device on the clamped item is determined. Alternatively, the distance, horizontal correlation coefficient, and vertical correlation coefficient between the clamping device and the clamped item in multiple consecutive images can be weighted, and the clamping state of the clamping device on the clamped item can be determined based on the weighted processing result.

3. The method according to claim 1, characterized in that, The method further includes: If the type of the clamped item is a display device packaging box and the confidence level is not less than the first confidence level threshold, or if the type of the clamped item is a refrigerator packaging box, the visible surface character type of the clamped item is determined. If the visible surface character type of multiple consecutive images is a preset character type and the confidence level is not less than the second confidence level threshold, it is determined that the clamping device is illegally clamping the clamped item.

4. The method according to claim 1, characterized in that, The method further includes: When the clamping device changes from clamping the object to not clamping it, the speed changes of the clamping device and the object are determined based on their respective positions in multiple consecutive images. Based on the speed changes, it is determined whether the object has fallen from a high position.

5. The method according to claim 4, characterized in that, The method further includes: If a preset number of images indicate that the clamped item has fallen from a high position, a large model is used to determine whether the clamped item has fallen from a high position based on multiple consecutive images.

6. The method according to claim 1, characterized in that, The method further includes: Upon receiving a new requirement for anomaly detection, obtain image samples matching the new requirement and instruction samples corresponding to the image samples; the instruction samples include at least one of global image description question-and-answer pairs, instruction expression description question-and-answer pairs, and thought chain prompt question-and-answer pairs; based on the image samples and the corresponding instruction samples, perform performance verification on the initial large model, and if the performance verification passes, use the initial large model for anomaly detection; If the performance verification fails, the initial large model is fine-tuned based on the image samples and corresponding instruction samples to obtain an adjusted large model, which is then used for anomaly detection.

7. A server, characterized in that, include: processor; The processor is configured to: The video to be detected is acquired, and target detection is performed on multiple consecutive images in the video to obtain at least one target contained in each of the multiple images; Based on the position of at least one target in multiple consecutive images, determine the running status of at least one target; When the running status indicates the presence of a stationary target, the placement posture is detected by using a posture detection model based on the position of the stationary target in multiple consecutive images, and the placement posture detection result of the stationary target is obtained. If the operation status indicates that there are at least two moving targets, and the at least two moving targets include a clamping device and a clamping object, determine whether the clamping device illegally clamps the clamping object based on the position of the clamping device and the clamping object in multiple consecutive images and the type of the clamping object. The processor is also configured to: The clamping state of the clamping device on the clamped item is determined based on the positions of the clamping device and the clamped item in multiple consecutive images. When the clamping state indicates that the clamping device should clamp the clamped item, it is determined whether the clamping device has violated regulations by clamping the clamped item, based on the type of the clamped item. The processor is also configured to: If the type of the clamped item is a display device packaging box and the confidence level is less than the first confidence level threshold, or if the type of the clamped item is a display device packaging box in a first preset number of images and the confidence level is not less than the first confidence level threshold, or if the type of the clamped item is a refrigerator packaging box and the confidence level of the visible surface character type is a preset character type is less than the second confidence level threshold, or if the type of the clamped item is a refrigerator packaging box and the visible surface character type is a preset character type in a second preset number of images and the confidence level is not less than the second confidence level threshold, a large model is used based on multiple consecutive images to determine whether the clamping device illegally clamps the clamped item. The processor is also configured to: If the placement posture detection result of the stationary target indicates an abnormal placement posture and the confidence level of the abnormal placement posture is less than the third confidence level threshold, or if the placement posture detection result of the third preset number of images indicates an abnormal placement posture and the confidence level of the abnormal placement posture is not less than the third confidence level threshold, a large model is used to re-determine the placement posture of multiple consecutive images to obtain the placement posture re-determination result of the stationary target.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images