Method, apparatus and automated device for detecting an item

Abstract

The present disclosure relates to a kind of detection method, device and automated equipment of article, relate to the field of automation technology.The detection method comprises: using machine learning model, respectively to the article to be detected in multiple frames of images is classified, to determine the state information of article to be detected in each frame of image, state information includes has been separated from article pickup device or not separated from article pickup device;According to whether there is state information for multiple frames of time continuous images that have been separated from article pickup device, whether the article to be detected falls from article pickup device is judged.The technical scheme of the present disclosure can improve detection accuracy and reduce cost.

Description

Technical Field

[0001] This disclosure relates to the field of automation technology, and in particular to a method for detecting articles, an apparatus for detecting articles, automated equipment, and a non-volatile computer-readable storage medium. Background Technology

[0002] With the rapid development of e-commerce, online shopping, and the logistics industry, the application of automated equipment in the basic parts of logistics, such as warehousing, handling, depalletizing, and sorting, is becoming increasingly common. The use of robotic arms to replace manual labor in handling goods (such as industrial parts, commodities, and boxes) is becoming more and more widespread (e.g., grasping, moving, and placing).

[0003] However, during the process of a robotic arm grasping and moving goods, problems such as unstable grasping and goods falling often occur due to improper selection of the grasping point, vibration of the picked goods, and difficulty in grasping the picked goods. Therefore, in order to monitor the problem of goods falling during the robotic arm's grasping process, there is an urgent need for goods falling detection technology to identify and analyze falling events, falling trajectories, and falling points.

[0004] In related technologies, for industrial production scenarios, the difference between a standard template image and the image to be detected is used to calculate whether there is fallen material; for smart retail terminal scenarios, pressure sensors and vibration sensors are used to detect whether there is fallen goods. Summary of the Invention

[0005] The inventors of this disclosure have discovered the following problems in the aforementioned related technologies: over-reliance on standard template images leads to poor detection accuracy; and the need to deploy pressure sensors and vibration sensors results in high costs.

[0006] In view of this, this disclosure proposes a technical solution for the detection of articles, which can improve detection accuracy and reduce costs.

[0007] According to some embodiments of this disclosure, a method for detecting an item is provided, comprising: using a machine learning model to classify the item to be detected in multiple frames of images to determine the state information of the item to be detected in each frame of images, the state information including whether it has been removed from the item picking device or not; and determining whether the item to be detected has fallen from the item picking device based on whether there are multiple frames of time-continuous images with the state information indicating that it has been removed from the item picking device.

[0008] In some embodiments, determining whether an item to be detected has fallen from the item pickup device includes: determining that the item to be detected has fallen from the item pickup device if the number of multiple time-continuous images is greater than a number threshold.

[0009] In some embodiments, the detection method further includes: generating a state trajectory of the object to be detected according to the state information corresponding to each frame image in the time sequence of each frame image; determining that the object to be detected has fallen from the object picking device includes: when there are multiple consecutive trajectory points on the state trajectory whose corresponding state information is that they have been removed from the object picking device, determining whether the number of multiple consecutive trajectory points is greater than a number threshold; when the number of multiple consecutive trajectory points is greater than the number threshold, determining that the object to be detected has fallen from the object picking device.

[0010] In some embodiments, determining whether the number of multiple consecutive trajectory points is greater than a quantity threshold includes: determining a state change trajectory point on the state trajectory, wherein the state information corresponding to the state change trajectory point is that it has been removed from the item picking device, and the state information corresponding to the previous trajectory point of the state change trajectory point is that it has not been removed from the item picking device; taking the state change trajectory point as the starting point, counting the number of multiple consecutive trajectory points whose corresponding state information is that it has been removed from the item picking device; and determining whether the number of multiple consecutive trajectory points is greater than a quantity threshold.

[0011] In some embodiments, generating the state trajectory of the item to be detected includes: determining the state information corresponding to the trajectory point as a first value when the item to be detected has been removed from the item picking device in the image corresponding to the trajectory point; determining the state information corresponding to the trajectory point as a second value when the item to be detected has not been removed from the item picking device in the image corresponding to the trajectory point; determining whether the number of multiple consecutive trajectory points is greater than a number threshold includes: performing median filtering on the state information of each trajectory point in the state trajectory; determining whether there are multiple consecutive trajectory points in the state trajectory after median filtering that correspond to state information indicating that the item has been removed from the item picking device.

[0012] In some embodiments, generating the state trajectory of the object to be detected includes: performing target tracking on the object to be detected in multiple frames of images to determine the motion information of the object to be detected in each frame of images; generating the state trajectory based on the motion information; the detection method further includes: when it is determined that the object to be detected has fallen from the object picking device, determining at least one of the falling trajectory, the starting position of falling, or the last falling position of the object to be detected based on the motion information.

[0013] In some embodiments, target tracking of objects to be detected in multiple frames of images includes: the D-IOU (Distance-Intersection over Union) method and the SORT (Simple Online and Realtime Tracking) method, to track objects to be detected in multiple frames of images.

[0014] In some embodiments, classifying objects to be detected in multiple frames of images using a machine learning model includes: determining multiple candidate regions in the current frame image where moving objects exist; classifying the moving objects in the multiple candidate regions using a machine learning model to determine whether each candidate region contains an object to be detected; merging the multiple candidate regions containing the object to be detected into a target region where the object to be detected is located when there are multiple candidate regions in the current frame image; and classifying the objects to be detected in the target region using a machine learning model to determine the state information of the objects to be detected.

[0015] In some embodiments, merging multiple candidate regions containing an item to be detected into a target region containing the item to be detected includes: calculating the distance between each candidate region containing the item to be detected; merging candidate regions whose distance is less than a distance threshold to determine the target region.

[0016] In some embodiments, determining multiple candidate regions in an image where a moving object exists includes: determining a moving foreground region in the current frame image based on the difference between the current frame image and the previous frame image, wherein multiple moving objects exist in the moving foreground region; and determining multiple candidate regions in the moving foreground region.

[0017] In some embodiments, determining multiple candidate regions in a moving foreground region includes: performing connected component calculation on the moving foreground region to determine multiple connected components in the image; and determining multiple candidate regions based on the multiple connected components.

[0018] In some embodiments, determining multiple candidate regions in the moving foreground region includes: performing morphological processing on the moving foreground region, the morphological processing including at least one of morphological opening operation based on cross-shaped structuring elements, morphological closing operation based on rectangular structuring elements, or morphological opening operation based on cross-shaped structuring elements; and determining multiple candidate regions in the morphologically processed moving foreground region.

[0019] In some embodiments, classifying the objects to be detected in multiple frames of images using a machine learning model includes classifying moving objects in each frame of images as objects that have been removed from the object picking device, objects that have not been removed from the object picking device, or non-objects to be detected, wherein non-objects to be detected include the object picking device.

[0020] According to some other embodiments of this disclosure, an item detection device is provided, comprising: a classification unit, configured to classify the item to be detected in multiple frames of images using a machine learning model, so as to determine the state information of the item to be detected in each frame of images, the state information including whether it has been removed from the item picking device and whether it has not been removed from the item picking device; and a judgment unit, configured to determine whether the item to be detected has fallen from the item picking device based on whether there are multiple frames of time-continuous images corresponding to the state information that have been removed from the item picking device.

[0021] In some embodiments, the determination unit determines that the object to be detected has fallen from the object picking device if the number of images in multiple consecutive frames exceeds a number threshold.

[0022] In some embodiments, the detection device further includes: a generation unit, configured to generate a state trajectory of the object to be detected according to the time sequence of each frame image and the state information corresponding to each frame image; and a judgment unit, in the case where the state information corresponding to multiple consecutive trajectory points on the state trajectory is that the object has been removed from the object picking device, to determine whether the number of multiple consecutive trajectory points is greater than a quantity threshold, and in the case where the number of multiple consecutive trajectory points is greater than the quantity threshold, to determine that the object to be detected has fallen from the object picking device.

[0023] In some embodiments, the determination unit determines state change trajectory points on the state trajectory. The state information corresponding to the state change trajectory point is that the item has been removed from the item picking device, and the state information corresponding to the previous trajectory point of the state change trajectory point is that the item has not been removed from the item picking device. Starting from the state change trajectory point, the unit counts the number of multiple consecutive trajectory points whose corresponding state information is that the item has been removed from the item picking device, and determines whether the number of multiple consecutive trajectory points is greater than the number threshold.

[0024] In some embodiments, when the object to be detected has been removed from the object picking device in the image corresponding to the trajectory point, the generation unit determines the state information corresponding to the trajectory point as a first value; when the object to be detected has not been removed from the object picking device in the image corresponding to the trajectory point, the generation unit determines the state information corresponding to the trajectory point as a second value; and performs median filtering on the state information of each trajectory point in the state trajectory; the judgment unit determines whether there are multiple consecutive trajectory points in the state trajectory after median filtering that correspond to state information indicating that the object has been removed from the object picking device.

[0025] In some embodiments, the generation unit performs target tracking on the object to be detected in multiple frames of images to determine the motion information of the object to be detected in each frame of images, and generates a state trajectory based on the motion information; when the determination unit determines that the object to be detected has fallen from the object picking device, it determines at least one of the falling trajectory, the starting position of falling, or the last falling position of the object to be detected based on the motion information.

[0026] In some embodiments, the determination unit performs target tracking on the object to be detected in multiple frames of images based on the D-IOU method and the SORT method.

[0027] In some embodiments, the classification unit determines multiple candidate regions in the current frame image where there are moving objects, uses a machine learning model to classify the moving objects in the multiple candidate regions to determine whether each candidate region contains the object to be detected. If there are multiple candidate regions in the current frame image that contain the object to be detected, the multiple candidate regions containing the object to be detected are merged into the target region where the object to be detected is located. The machine learning model is then used to classify the object to be detected in the target region to determine the state information of the object to be detected.

[0028] In some embodiments, the classification unit calculates the distance between each candidate region containing the item to be detected, and merges candidate regions whose distance is less than a distance threshold to determine the target region.

[0029] In some embodiments, the classification unit determines the moving foreground region in the current frame image based on the difference between the current frame image and the previous frame image. There are multiple moving objects in the moving foreground region, and multiple candidate regions are determined in the moving foreground region.

[0030] In some embodiments, the classification unit performs connected component calculations on the moving foreground region to determine multiple connected components in the image, and determines multiple candidate regions based on the multiple connected components.

[0031] In some embodiments, the classification unit performs morphological processing on the moving foreground region. The morphological processing includes at least one of morphological opening operation based on cross-shaped structuring elements, morphological closing operation based on rectangular structuring elements, or morphological opening operation based on cross-shaped structuring elements. In the morphologically processed moving foreground region, multiple candidate regions are determined.

[0032] In some embodiments, the classification unit classifies moving objects in each frame image into objects that have detached from the object pickup device, objects that have not detached from the object pickup device, or non-detectable objects, where non-detectable objects include the object pickup device.

[0033] According to further embodiments of the present disclosure, an article detection apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the article detection method of any of the above embodiments based on instructions stored in the memory device.

[0034] According to further embodiments of this disclosure, an automated device is provided, comprising: an item detection device as described in any of the above embodiments; an item picking device for picking up items; and an image acquisition device for acquiring multiple frames of images including at least one of the item picking device or items.

[0035] According to further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the article detection method of any of the above embodiments.

[0036] In the above embodiments, based on the classification results of the items to be detected, changes in the state information of the items to be detected are determined, and then the occurrence of an item falling event is detected. In this way, the two important factors of grasping goods and the movement of goods falling are comprehensively considered in the scenario of robotic arm grasping, and there is no need to rely on standard template images, nor is it necessary to deploy pressure sensors and vibration sensors, thereby improving detection accuracy and reducing costs. Attached Figure Description

[0037] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0038] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description:

[0039] Figure 1 Flowcharts illustrating some embodiments of the detection methods for the articles of this disclosure;

[0040] Figures 2a-2d Schematic diagrams illustrating some embodiments of the detection method for articles of this disclosure;

[0041] Figure 3 Schematic diagrams illustrating other embodiments of the detection method for articles of this disclosure;

[0042] Figure 4 Flowcharts illustrating other embodiments of the detection methods for the articles of this disclosure;

[0043] Figure 5 Block diagrams illustrating some embodiments of the detection apparatus for articles of this disclosure;

[0044] Figure 6 Block diagrams illustrating further embodiments of the detection apparatus for articles of this disclosure;

[0045] Figure 7 Block diagrams illustrating further embodiments of the detection apparatus for articles of this disclosure;

[0046] Figure 8 Block diagrams illustrating some embodiments of the automated devices of this disclosure are shown. Detailed Implementation

[0047] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0048] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0049] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0050] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0051] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0053] As mentioned earlier, in industrial production scenarios, the method of calculating whether there has been material drop by using the difference between a standard template image and the image to be detected within the ROI (Region of Interest) relies too heavily on the standard template, is unsuitable for highly variable scenarios, and lacks flexibility. Furthermore, this method is limited by the ROI division and cannot effectively prove that the change was caused by a drop event.

[0054] In addition, in the smart retail terminal scenario, the method of relying on pressure sensors and vibration sensors requires the sensors to be installed and deployed on the area to be detected or the goods, which is costly, cumbersome to operate, and difficult to modify.

[0055] In other words, there is no mature method for detecting drops in scenarios where robotic arms or other object-grabbing devices are handling goods.

[0056] Cargo falling is a dynamic event that can be addressed using computer vision. One approach is an end-to-end model that uses video as input to a video understanding model and directly outputs labels. However, this method is highly data-dependent, struggles to output precise falling trajectories and landing locations, and lacks interpretability.

[0057] Alternatively, target detection can be used to directly detect the fallen goods. However, when there are many goods in the background, it is difficult to distinguish the fallen goods from other goods. Furthermore, this method relies on a large amount of labeled data, has poor generalization ability, and cannot guarantee the causal relationship of the falling event, i.e., that the falling event is a dynamic process of the goods falling off after being grasped by the robotic arm.

[0058] Therefore, this disclosure proposes a technical solution to address the technical problem of detecting goods falling during the process of a robotic arm grasping goods in the fields of industrial and logistics intelligence.

[0059] Regarding the aforementioned technical issues, this disclosure reveals that cargo falling is a dynamic event with two crucial factors: the robotic arm's grasping of the cargo and the cargo's movement after detaching from the grasper. Both are indispensable and should be considered comprehensively.

[0060] In addition, this disclosure utilizes moving target detection to degenerate (or simplify) the problem of detecting falling goods into a classification problem. Moving target detection provides target location information, and a deep learning classification model provides target category information with high-level semantics. Then, target tracking is used to determine the complete motion trajectory of the target goods and record the state information of each frame. Finally, based on the state information of each frame in the trajectory sequence, it is determined whether the current event belongs to the event of falling goods and the falling trajectory and the location of the falling point.

[0061] For example, the technical solution of this disclosure can be implemented through the following embodiments.

[0062] Figure 1 Flowcharts illustrating some embodiments of the methods for detecting articles according to this disclosure are shown.

[0063] like Figure 1 As shown, in step 110, a machine learning model is used to classify the objects to be detected in multiple frames of images to determine the state information of the objects to be detected in each frame. The state information includes whether the object has been removed from the object picking device or not.

[0064] In some embodiments, an image sequence from a video is acquired, the image sequence comprising multiple frames. For example, a local video file or an input video stream is decoded and frame-stripped to extract the image sequence, which is then used as input for subsequent processing.

[0065] In some embodiments, a moving foreground region in the current frame image is determined based on the difference between the current frame image and the previous frame image, wherein multiple moving objects exist in the moving foreground region; and multiple candidate regions are determined in the moving foreground region.

[0066] For example, algorithms such as frame differencing, background modeling, and optical flow can be used to calculate the difference between the current frame and the previous frame. Frame differencing and optical flow take the current frame and the previous frame as input and calculate the change information between the two frames. Then, based on an empirical threshold, the processing results of frame differencing, optical flow, or GMM (Gaussian Mixture Model) background modeling are subjected to image binary segmentation to obtain the preliminary moving foreground region.

[0067] For example, it can be done through Figures 2a-2b In the embodiments described, multiple candidate regions are identified.

[0068] Figures 2a-2b Schematic diagrams illustrating some embodiments of the article detection method of this disclosure.

[0069] like Figure 2a The image shown is a motion foreground map at a certain moment when a robotic arm grasps small box-like goods; the non-white areas are the calculated motion foreground regions.

[0070] In some embodiments, morphological processing is performed on the moving foreground region, including at least one of morphological opening operation based on cross-shaped structuring elements, morphological closing operation based on rectangular structuring elements, or morphological opening operation based on cross-shaped structuring elements; and multiple candidate regions are determined in the morphologically processed moving foreground region.

[0071] For example, morphological processing of images includes dilation, erosion, closing, and opening operations. Morphological closing operations can be used to bridge narrow gaps and long, thin grooves, eliminate small holes between objects, and fill breaks in contour lines; morphological opening operations are typically used to break narrow connections between objects, eliminate sharp protrusions along object edges, and smooth contours.

[0072] For example, different morphological operations and combinations of morphological structural elements of different sizes and types can be used for different application scenarios.

[0073] For example, firstly, morphological opening operations based on cross-shaped structuring elements are used to remove noise; then, morphological closing operations based on rectangular structuring elements are used to fill holes and fill broken contour lines; next, morphological opening operations based on rectangular structuring elements are used to disconnect narrow connections between objects, that is, to separate the gripped goods from the robotic arm moving in the same direction by utilizing the narrow connection between the robotic arm gripper and the gripped goods; finally, morphological opening operations based on cross-shaped structuring elements are used to trim and eliminate burrs in the result.

[0074] like Figure 2b As shown, this is the foreground region after morphological processing.

[0075] In some embodiments, connected component calculation is performed on the moving foreground region to determine multiple connected regions in the image; based on the multiple connected regions, multiple candidate regions are determined.

[0076] For example, for Figure 2b After morphological processing, the moving foreground region is used to calculate connected regions, and the largest bounding rectangle is calculated for the outermost edge of each connected region as a candidate moving target bounding box. Furthermore, the connected region area S can be filtered out. roi Moving object boxes smaller than the area threshold are used to filter out noise, thus obtaining... Figure 2b The six moving target boxes in the image are used as candidate regions.

[0077] In some embodiments, for Figure 2b Moving targets in six candidate regions are classified. For example, moving objects in each frame are classified as objects that have detached from the object pickup device, objects that have not detached from the object pickup device, or non-detectable objects, with non-detectable objects including the object pickup device.

[0078] For example, firstly, using Figure 2b The process involves cropping multiple local images to be classified from the original image using moving bounding boxes; then, a machine learning model (such as a deep learning model) is used to identify and classify these images. For example, a deep learning classification model can be used to classify the images.

[0079] For example, when a robotic arm grasps a piece of goods, both the robotic arm and the goods move, therefore Figure 2b The bounding box for moving targets may include moving targets other than the grasped goods. Furthermore, goods falling is a dynamic event, requiring both elements: the process of the robotic arm grasping the goods and the movement of the goods after they fall from the grasper. Therefore, the category labels for the classification model can be set as follows: a first value of 1, representing an item that has not detached from the item pickup device; a second value of 2, representing an item that has detached from the item pickup device; and a third value of 3, representing other moving objects and disturbances (such as the robotic arm, noise, etc.).

[0080] For example, the category of other moving objects and interference can be further subdivided based on the actual scenario.

[0081] like Figure 2b As shown, based on the classification results, the target of interest is filtered out, retaining only items that have not left the item pickup device and items that have left the item pickup device, while other items are filtered out.

[0082] For example, the dashed rectangle represents the robotic arm's movement area and other motion-related noise; the solid rectangle represents items that remain attached to the item pickup device. Other identified moving goods (including items that have detached from the item pickup device) can also be retained.

[0083] In some embodiments, multiple candidate regions containing moving objects are determined in the current frame image; the moving objects in the multiple candidate regions are classified using a machine learning model to determine whether each candidate region contains an item to be detected; if there are multiple candidate regions containing items to be detected in the current frame image, the multiple candidate regions containing items to be detected are merged into a target region where the item to be detected is located; the item to be detected in the target region is classified using a machine learning model to determine the state information of the item to be detected.

[0084] For example, it can be done through Figures 2a-2b The embodiments in the text determine the status information of the item to be detected.

[0085] Figures 2c-2d Schematic diagrams illustrating some embodiments of the article detection method of this disclosure.

[0086] like Figure 2c As shown, due to the characteristics of cargo movement in certain situations, the calculation results of the motion foreground often only segment a part of the cargo. For example, when the cargo moves from the dashed line state to the solid line state, the movement amplitude of the area near the center of the cargo is small, which makes it easy to miss the area near the center of the cargo when calculating the motion foreground region.

[0087] like Figure 2d As shown, mask region 1 and mask region 2 are the obtained moving foreground regions. Due to the morphological processing above, mask regions belonging to the same cargo may be separated in some cases.

[0088] In some embodiments, for the above situation, the distance between each candidate region containing the item to be detected is calculated; candidate regions with a distance less than a distance threshold are merged to determine the target region.

[0089] In this way, a moving target merging strategy is adopted to merge the separated cargo targets to ensure that complete target box location information is obtained.

[0090] For example, first, calculate the distance (such as d-iou) between each pair of goods (moving target boxes), and merge goods whose distance is less than the distance threshold; then, update the position and category information of the merged moving target boxes.

[0091] like Figure 2dAs shown, the motion target box of mask region 1 is Box B, the motion target box of mask region 2 is Box A, and the merged motion target box is Box C.

[0092] In some embodiments, the target region to be merged can be determined based on the smallest region that can cover multiple candidate regions to be merged.

[0093] For example, taking the top left corner of the image as the origin, the smallest coordinate among the four vertices of BoxA is (A xmin A ymin The maximum coordinate is (A). xmax A ymax The smallest coordinate among the four vertices of Box B is (B). xmin B ymin The maximum coordinate is (B). xmax B ymmax If ), then the coordinates of the top left corner of Box C (C) xmin C ymin )for:

[0094] C xmin =min(A xmin B xmin );

[0095] C ymin =min(A ymin B ymin )

[0096] The coordinates of the bottom right corner of Box C are:

[0097] C xmax =max(A xmax B xmax );

[0098] C ymax =min(A ymax B ymmax )

[0099] In some embodiments, since the classification result of the moving target before merging is the local classification result of the goods, a classification process is required after merging; the result of the reclassification is then used to update the status information of the merged goods.

[0100] In the above embodiments, based on the combined morphological processing strategy and the idea of ​​separation followed by merging, the noise of the original moving foreground segmentation results is eliminated, so that the grasped cargo target can be effectively separated from other interfering motion areas (such as the movement of the robotic arm itself); then, the filtered and retained moving cargo targets are merged to obtain a complete moving cargo target box.

[0101] After determining the target area, it can be done through... Figure 1 The remaining steps in the algorithm determine when a fall event has occurred.

[0102] In step 120, based on whether there are multiple time-continuous images with status information indicating that the item has been removed from the item pickup device, it is determined whether the item to be detected has fallen from the item pickup device.

[0103] In some embodiments, if the number of consecutive multi-frame images with status information indicating that the item has not been removed from the item pickup device is greater than a first threshold, and the number of consecutive multi-frame images with status information indicating that the item has been removed from the item pickup device is greater than a second threshold, it is determined that the item to be detected has fallen from the item pickup device.

[0104] For example, the time corresponding to a multi-frame temporally continuous image whose status information is still within the object pickup device is earlier than the time corresponding to a multi-frame temporally continuous image whose status information is already within the object pickup device.

[0105] In some embodiments, the state trajectory of the object to be detected is generated according to the time sequence of each frame image and the state information corresponding to each frame image.

[0106] For example, target tracking methods such as nearest neighbor matching, sort, and deep-sort can be used to track the trajectory of goods.

[0107] The sort tracking method first calculates the distance metric for the detection boxes, and then performs Hungarian matching. That is, the sort method calculates the Intersection over Union (IoU) between boxes. However, falling goods often move very quickly, and their positions at different times may not intersect. In this case, the IoU equals 0, making it impossible to measure the degree of matching between target boxes.

[0108] For example, a sorted tracking method based on D-IOU matching can be used. That is, the D-IOU is used to calculate the matching degree (i.e., distance) between the moving target boxes corresponding to the candidate regions; then, based on the matching degree, the sorted tracking method is used to perform trajectory tracking and generate a state trajectory.

[0109] Therefore, in order to address the above-mentioned technical problems, the above embodiments use the d-iou matching metric to replace IOU. Even when the boxes do not intersect, the distance metric is still considered, and the degree of matching between target boxes can still be measured.

[0110] In some embodiments, the tracking results are recorded, such as Track(i,t). i represents the target identifier (ID) being tracked, and t represents the latest tracking status at frame t (including the latest displacement, cumulative distance, maximum speed, average speed, acceleration, number of lost frames, trajectory target length, etc.). Each time a new target is matched, it is recorded in the trajectory target list (such as the status trajectory) with the corresponding ID.

[0111] In some embodiments, the obtained state trajectories are calculated one by one to determine whether the state trajectory is the trajectory of the goods falling after being grasped by the robotic arm and the location of the falling point. For example, if the number of multiple time-continuous images exceeds a threshold, it is determined that the item to be detected has fallen from the item picking device.

[0112] In the above embodiments, based on moving target detection, the target detection problem of the goods to be analyzed is degenerated (or simplified) into a classification problem. Moving target detection provides target location information, and the classification model provides target category information. Compared with target detection methods that require simultaneous annotation of target bounding box position and category label, this overcomes the technical problems of high difficulty and cost of target bounding box annotation, susceptibility to background interference, and reliance on a large amount of labeled data.

[0113] Furthermore, by reducing (or simplifying) the object detection problem to a classification problem, the analysis and identification can be focused on the region of interest, reducing the learning and annotation difficulties of the model. Since the classification problem only needs to distinguish between the moving target of interest and other moving targets, the dependence on the amount of labeled data is also greatly reduced, resulting in better generalization performance.

[0114] In some embodiments, if the object to be detected has been removed from the object picking device in the image corresponding to the trajectory point, the state information corresponding to the trajectory point is determined to a first value (e.g., 1); if the object to be detected has not been removed from the object picking device in the image corresponding to the trajectory point, the state information corresponding to the trajectory point is determined to a second value (e.g., 0). This can be achieved through... Figure 4 In the example provided, information about an item being dropped is determined.

[0115] Figure 3 Schematic diagrams illustrating some other embodiments of the article detection method of this disclosure.

[0116] like Figure 3 As shown, the first row is the original state trajectory, and the second row is the state trajectory after median filtering. In the state trajectory, 0 indicates that the robotic arm is grasping the goods (i.e., it has not detached from the item picking device), and 1 indicates that the goods are moving normally (i.e., they have detached from the item picking device).

[0117] In some embodiments, the state information of each trajectory point in the state trajectory is subjected to median filtering; it is determined whether there are multiple consecutive trajectory points in the state trajectory after median filtering whose state information indicates that they have been removed from the item picking device.

[0118] For example, median filtering is used to process the classification state results identified in each frame of the trajectory. Based on the state information of adjacent frames, the state information of the current frame is corrected, thereby mitigating the state noise caused by the limited prediction accuracy of the classification model itself.

[0119] In the above embodiments, median filtering is used to process the prediction result sequence (state trajectory) of the classification model for the video image sequence. Utilizing the prediction results of preceding and following frames can correct and mitigate the noise caused by the limited prediction accuracy of the classification model itself, reducing the dependence on the deep model. Furthermore, when a frame is missed, the state of that frame can be reconstructed using the results of preceding and following frames.

[0120] In some embodiments, if multiple consecutive trajectory points on the state trajectory correspond to state information indicating that the item has been removed from the item pickup device, it is determined whether the number of these multiple consecutive trajectory points exceeds a quantity threshold. If the number of these multiple consecutive trajectory points exceeds the quantity threshold, it is determined that the item to be detected has fallen from the item pickup device.

[0121] For example, the calculation includes the number of items that have detached from the item picking device, the time when the fall began, and the post-fall status information, which is the number of items that have detached from the item picking device (i.e., the duration of movement after the fall). When the number of times the robotic arm has grasped goods exceeds a quantity threshold, and the post-fall status information shows that the number of items detached from the item picking device exceeds the quantity threshold, this status trajectory is determined to be the goods falling trajectory when the robotic arm grasps the goods. This dynamic process is the goods falling event when the robotic arm grasps the goods.

[0122] In the above embodiments, the trajectory of the cargo of interest is obtained using moving target detection and tracking. Based on the changes in the complete target state information within the trajectory, it is determined whether it is a cargo drop event during the robotic arm's grasping of the cargo. This comprehensively considers two indispensable factors in this event: the robotic arm grasping the cargo and the cargo drop motion. Ultimately, a complete drop trajectory and drop point location can be output, offering strong interpretability and flexibility. Subsequent drop event handling mechanisms can be flexibly configured based on this result.

[0123] In some embodiments, a state change trajectory point (i.e., the arrow indicating the start of falling in the figure) is determined on the state trajectory. The state information corresponding to the state change trajectory point is that the item has been removed from the item pickup device, and the state information corresponding to the previous trajectory point is that the item has not been removed from the item pickup device. Starting from the state change trajectory point, the number of multiple consecutive trajectory points with the corresponding state information of "removed from the item pickup device" is counted. It is then determined whether the number of multiple consecutive trajectory points is greater than a quantity threshold.

[0124] In some embodiments, target tracking is performed on the object to be detected in multiple frames of images to determine the motion information of the object to be detected in each frame of images; a state trajectory is generated based on the motion information; and if it is determined that the object to be detected has fallen from the object picking device, at least one of the falling trajectory, the starting position of falling, or the last falling position of the object to be detected is determined based on the motion information.

[0125] For example, target tracking of objects to be detected in multiple frames of images includes the D-IOU method and the SORT method.

[0126] For example, the location of the cargo at the last moment of the state trajectory can be determined as the final landing point of the cargo.

[0127] Figure 4 Flowcharts illustrating some other embodiments of the detection method for the articles disclosed herein.

[0128] like Figure 4 As shown, in step 410, an image sequence from the video is acquired, which includes multiple frames. For example, a local video file or an input video stream is decoded and frame-stripped to extract the image sequence, which is then used as input for subsequent processing.

[0129] In step 420, algorithms such as frame difference, background modeling, and optical flow can be used to calculate the difference between the current frame and the previous frame. Frame difference and optical flow use the current and previous frames as inputs to calculate the change information between the two frames. Then, based on an empirical threshold, the processing results of frame difference, optical flow, or GMM background modeling are subjected to image binary segmentation to obtain the preliminary moving foreground region.

[0130] like Figure 2a The image shown is a motion foreground map at a certain moment when a robotic arm grasps small box-like goods; the non-white areas are the calculated motion foreground regions.

[0131] In step 430, the morphological processing of the image includes dilation, erosion, closing, and opening operations. Morphological closing operations can be used to bridge narrow gaps and long, thin grooves, eliminate small holes between objects, and fill breaks in contour lines; morphological opening operations are typically used to break narrow connections between objects, eliminate sharp protrusions along object edges, and smooth contours.

[0132] For example, different morphological operations and combinations of morphological structural elements of different sizes and types can be used for different application scenarios.

[0133] For example, firstly, morphological opening operations based on cross-shaped structuring elements are used to remove noise; then, morphological closing operations based on rectangular structuring elements are used to fill holes and fill broken contour lines; next, morphological opening operations based on rectangular structuring elements are used to disconnect narrow connections between objects, that is, to separate the gripped goods from the robotic arm moving in the same direction by utilizing the narrow connection between the robotic arm gripper and the gripped goods; finally, morphological opening operations based on cross-shaped structuring elements are used to trim and eliminate burrs in the result.

[0134] like Figure 2b As shown, this is the foreground region after morphological processing.

[0135] In step 440, connected component calculation is performed on the morphologically processed foreground region from step 430, and the largest bounding rectangle is calculated for the outermost edge of each connected region as a candidate moving target bounding box. Furthermore, connected region areas S can be filtered out. roi Moving object boxes smaller than the area threshold are used to filter out noise, thus obtaining... Figure 2b The six moving target boxes in the image are used as candidate regions.

[0136] In step 450, moving targets in the candidate regions obtained in step 440 are classified. First, using the moving target bounding boxes obtained in step 440, multiple local images to be classified are cropped from the original image; then, a machine learning model (such as a deep learning model) is used to identify and classify the images to be classified. For example, a deep learning classification model is used to classify the images to be classified.

[0137] For example, when a robotic arm grasps a piece of goods, both the robotic arm and the goods move, therefore Figure 2bThe bounding box for moving targets may include moving targets other than the grasped goods. Furthermore, goods falling is a dynamic event, requiring both elements: the process of the robotic arm grasping the goods and the movement of the goods after they fall from the grasper. Therefore, the category labels for the classification model can be set as follows: a first value of 1 indicates an item that has not detached from the item pickup device; a second value of 2 indicates an item that has detached from the item pickup device; and a third value of 3 indicates other moving objects and disturbances (such as the robotic arm, noise, etc.).

[0138] For example, the category of other moving objects and interference can be further subdivided based on the actual scenario.

[0139] like Figure 2b As shown, based on the classification results, the target of interest is filtered out, retaining only items that have not left the item pickup device and items that have left the item pickup device, while other items are filtered out.

[0140] For example, the dashed rectangle represents the robotic arm's movement area and other motion-related noise; the solid rectangle represents items that remain attached to the item pickup device. Other identified moving goods (including items that have detached from the item pickup device) can also be retained.

[0141] In step 460, due to the characteristics of cargo motion in certain situations, the motion foreground calculation results often only segment a local part of the cargo. For example... Figure 2c As shown, when the goods move from the dashed line state to the solid line state, the movement amplitude of the area near the center of the goods is relatively small, which makes it easy to overlook the area near the center of the goods when calculating the foreground region. Figure 2d As shown, mask region 1 and mask region 2 are the obtained motion foreground regions.

[0142] Additionally, due to the morphological processing in step 430, masked areas belonging to the same cargo may be separated in some cases.

[0143] Therefore, a moving target merging strategy is adopted to merge the separated cargo targets in order to ensure that complete target box location information is obtained.

[0144] First, calculate the distance (e.g., d-iou) between each pair of goods (moving target boxes). For goods whose distance is less than the distance threshold, merge them pairwise. Then, update the position and category information of the merged moving target boxes.

[0145] like Figure 2d As shown, the motion target box of mask region 1 is Box B, the motion target box of mask region 2 is Box A, and the merged motion target box is Box C.

[0146] For example, taking the top left corner of the image as the origin, the smallest coordinate among the four vertices of Box A is (A... xmin A ymin The maximum coordinate is (A). xmax A ymax The smallest coordinate among the four vertices of Box B is (B). xmin B ymin The maximum coordinate is (B). xmax B ymax If ), then the coordinates of the top left corner of Box C (C) xmin C ymin )for:

[0147] C xmin =min(A xmin B xmin );

[0148] C ymin =min(A ymin B ymin )

[0149] The coordinates of the bottom right corner of Box C are:

[0150] C xmax =max(A xmax B xmax );

[0151] C ymax =min(A ymax B ymax )

[0152] Since the classification result of the moving targets before merging is the local classification result of the goods, a classification process is required after merging; the result of the reclassification is then used to update the status information of the merged goods.

[0153] In step 470, target tracking methods such as nearest neighbor matching, sorting, and deep-sorting can be used to track the trajectory of the goods obtained in step 460. The tracking results are recorded, such as Track(i, t). i represents the target identifier (ID) being tracked, and t represents the latest tracking status at frame t (including the latest displacement, cumulative distance, maximum speed, average speed, acceleration, number of lost frames, trajectory target length, etc.). Each time a new target is matched, it is recorded in the trajectory target list with the corresponding ID.

[0154] In step 480, the state trajectories obtained in step 470 are calculated one by one to determine whether the state trajectory is the trajectory of the goods falling after being grasped by the robotic arm and the location of the falling point.

[0155] For any given trajectory to be processed, median filtering is first used to process the classification state results identified in each frame of the trajectory. Based on the state information of adjacent frames, the state information of the current frame is corrected, thereby mitigating the state noise caused by the limited prediction accuracy of the classification model itself.

[0156] like Figure 3 As shown, the first row is the original state trajectory, and the second row is the state trajectory after median filtering. In the state trajectory, 0 indicates that the robotic arm is grasping the goods (i.e., it has not detached from the item picking device), and 1 indicates that the goods are moving normally (i.e., they have detached from the item picking device).

[0157] Then, the state information is calculated as the number of items that have detached from the item picking device, the time when the fall started, and the state information after the fall as the number of items that have detached from the item picking device (i.e., the duration of movement after the fall). When the number of states in which the robotic arm grasps goods exceeds a quantity threshold, and the state information after the fall as the number of items that have detached from the item picking device exceeds a quantity threshold, this state trajectory is determined to be the goods falling trajectory when the robotic arm grasps the goods. This dynamic process is the goods falling event when the robotic arm grasps the goods.

[0158] In addition, the location of the cargo at the last moment of the state trajectory can be determined as the final drop point of the cargo.

[0159] Figure 5 Block diagrams illustrating some embodiments of the detection apparatus for articles of this disclosure are shown.

[0160] like Figure 5 As shown, the article detection device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51. The processor 52 is configured to execute the article detection method of any embodiment of this disclosure based on instructions stored in the memory 51.

[0161] The memory 51 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, a boot loader, a database, and other programs.

[0162] Figure 6 Block diagrams illustrating other embodiments of the detection apparatus for articles of this disclosure are shown.

[0163] like Figure 6 As shown, the article detection device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute the article detection method of any of the foregoing embodiments based on instructions stored in the memory 610.

[0164] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, a boot loader, and other programs.

[0165] The item detection device 6 may also include an input / output interface 630, a network interface 640, and a storage interface 650. These interfaces 630, 640, and 650, as well as the memory 610 and processor 620, can be connected via, for example, a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a monitor, mouse, keyboard, touchscreen, microphone, and speakers. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0166] Figure 7 Block diagrams illustrating further embodiments of the detection apparatus for articles of this disclosure are shown.

[0167] like Figure 7 As shown, the item detection device 7 includes: a classification unit 71, which uses a machine learning model to classify the items to be detected in multiple frames of images to determine the state information of the items to be detected in each frame of images, including whether the items have been removed from the item picking device or not; and a judgment unit 72, which determines whether the items to be detected have fallen from the item picking device based on whether there are multiple frames of images with consecutive time corresponding to the state information that the items have been removed from the item picking device.

[0168] In some embodiments, the determination unit 72 determines that the object to be detected has fallen from the object picking device if the number of images with multiple consecutive time frames is greater than a number threshold.

[0169] In some embodiments, the detection device 7 further includes: a generation unit 73, configured to generate a state trajectory of the object to be detected according to the time sequence of each frame image and the state information corresponding to each frame image; and a judgment unit 72, in the case where the state information corresponding to multiple consecutive trajectory points on the state trajectory is that the object has been removed from the object picking device, to determine whether the number of multiple consecutive trajectory points is greater than a quantity threshold, and in the case where the number of multiple consecutive trajectory points is greater than the quantity threshold, to determine that the object to be detected has fallen from the object picking device.

[0170] In some embodiments, the judgment unit 72 determines a state change trajectory point on the state trajectory. The state information corresponding to the state change trajectory point is that the item has been removed from the item picking device, and the state information corresponding to the previous trajectory point of the state change trajectory point is that the item has not been removed from the item picking device. Starting from the state change trajectory point, the number of multiple consecutive trajectory points with the corresponding state information of "removed from the item picking device" is counted, and it is determined whether the number of multiple consecutive trajectory points is greater than the number threshold.

[0171] In some embodiments, when the object to be detected has been removed from the object picking device in the image corresponding to the trajectory point, the generation unit 73 determines the state information corresponding to the trajectory point as a first value, and when the object to be detected has not been removed from the object picking device in the image corresponding to the trajectory point, the generation unit 73 determines the state information corresponding to the trajectory point as a second value, and performs median filtering on the state information of each trajectory point in the state trajectory; the judgment unit 72 determines whether there are multiple consecutive trajectory points in the state trajectory after median filtering that correspond to state information indicating that the object has been removed from the object picking device.

[0172] In some embodiments, the generation unit 73 performs target tracking on the object to be detected in multiple frames of images to determine the motion information of the object to be detected in each frame of images, and generates a state trajectory based on the motion information; when the determination unit 72 determines that the object to be detected has fallen from the object picking device, it determines at least one of the falling trajectory, the starting position of falling, or the last falling position of the object to be detected based on the motion information.

[0173] In some embodiments, the determination unit 72 performs target tracking on the object to be detected in multiple frames of images according to the D-IOU method and the SORT method.

[0174] In some embodiments, the classification unit 71 determines multiple candidate regions in the current frame image where there are moving objects, and uses a machine learning model to classify the moving objects in the multiple candidate regions to determine whether each candidate region contains the object to be detected. If there are multiple candidate regions in the current frame image that contain the object to be detected, the multiple candidate regions containing the object to be detected are merged into the target region where the object to be detected is located. The machine learning model is then used to classify the object to be detected in the target region to determine the state information of the object to be detected.

[0175] In some embodiments, the classification unit 71 calculates the distance between each candidate region containing the item to be detected, and merges candidate regions whose distance is less than a distance threshold to determine the target region.

[0176] In some embodiments, the classification unit 71 determines the moving foreground region in the current frame image based on the difference between the current frame image and the previous frame image. There are multiple moving objects in the moving foreground region, and multiple candidate regions are determined in the moving foreground region.

[0177] In some embodiments, the classification unit 71 performs connected region calculation on the moving foreground region to determine multiple connected regions in the image, and determines multiple candidate regions based on the multiple connected regions.

[0178] In some embodiments, the classification unit 71 performs morphological processing on the moving foreground region. The morphological processing includes at least one of morphological opening operation based on cross-shaped structuring elements, morphological closing operation based on rectangular structuring elements, or morphological opening operation based on cross-shaped structuring elements. In the morphologically processed moving foreground region, multiple candidate regions are determined.

[0179] In some embodiments, the classification unit 71 classifies moving objects in each frame image into objects that have detached from the object pickup device, objects that have not detached from the object pickup device, or non-detectable objects, where non-detectable objects include the object pickup device.

[0180] Figure 8 Block diagrams illustrating some embodiments of the automated devices of this disclosure are shown.

[0181] like Figure 8 As shown, the automated device 8 includes: an item detection device 81 as described in any of the above embodiments; an item picking device 82 for picking up items; and an image acquisition device 83 for acquiring multiple frames of images including at least one of the item picking device or the item.

[0182] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media containing computer-usable program code, including but not limited to disk storage, CD-ROM, optical storage, etc.

[0183] The methods for detecting articles, the apparatus for detecting articles, the automated equipment, and the non-volatile computer-readable storage media according to this disclosure have been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0184] The methods and systems of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0185] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for detecting an article, comprising: Using a machine learning model, the objects to be detected in multiple frames of images are classified to determine the state information of the objects to be detected in each frame of images. The state information includes whether the objects have been removed from the object picking device or not. Based on the presence of multiple consecutive frames of images indicating that the item has been removed from the item pickup device, it is determined whether the item to be detected has fallen from the item pickup device. The step of classifying the objects to be detected in multiple frames of images using a machine learning model includes: Moving objects in each frame of the image are classified into objects that have detached from the object pickup device, objects that have not detached from the object pickup device, or objects not to be detected. The objects not to be detected include the object pickup device. The step of determining whether the item to be detected has fallen from the item pickup device based on the presence of multiple frames of time-continuous images indicating that the item has been removed from the item pickup device includes: If the number of consecutive multi-frame images with status information indicating the item has not been removed from the item pickup device exceeds a first threshold, and the number of consecutive multi-frame images with status information indicating the item has been removed from the item pickup device exceeds a second threshold, then it is determined that the item to be detected has fallen from the item pickup device. The step of classifying the objects to be detected in multiple frames of images using a machine learning model includes: Identify multiple candidate regions in the current frame image where a moving object exists; Using the machine learning model, moving objects in the multiple candidate regions are classified to determine whether each candidate region contains the item to be detected; If there are multiple candidate regions containing the object to be detected in the current frame image, the multiple candidate regions containing the object to be detected are merged into the target region where the object to be detected is located; The machine learning model is used to classify the items to be detected in the target area to determine the state information of the items to be detected.

2. The detection method according to claim 1, wherein, The step of determining whether the item to be detected has fallen from the item pickup device includes: If the number of consecutive frames of images exceeds a threshold, it is determined that the object to be detected has fallen from the object picking device.

3. The detection method according to claim 2 further includes: Based on the temporal order of the images, and according to the state information corresponding to each image, the state trajectory of the object to be detected is generated. The step of determining that the item to be detected has fallen from the item pickup device includes: If multiple consecutive trajectory points on the state trajectory correspond to a state information indicating that the item has been removed from the item picking device, it is determined whether the number of the multiple consecutive trajectory points is greater than the number threshold. If the number of consecutive trajectory points exceeds the threshold, it is determined that the item to be detected has fallen from the item picking device.

4. The detection method according to claim 3, wherein, The step of determining whether the number of the plurality of consecutive trajectory points is greater than the number threshold includes: A state change trajectory point is determined on the state trajectory. The state information corresponding to the state change trajectory point is that the item has been removed from the item picking device, and the state information corresponding to the previous trajectory point of the state change trajectory point is that the item has not been removed from the item picking device. Starting from the state change trajectory point, count the number of consecutive trajectory points that have been removed from the item picking device, corresponding to the state information. Determine whether the number of the multiple consecutive trajectory points is greater than the number threshold.

5. The detection method according to claim 3, wherein, The generation of the state trajectory of the item to be detected includes: If the object to be detected has been removed from the object picking device in the image corresponding to the trajectory point, the state information corresponding to the trajectory point is determined as the first value. If the object to be detected has not left the object picking device in the image corresponding to the trajectory point, the state information corresponding to the trajectory point is determined as the second value; The step of determining whether the number of the plurality of consecutive trajectory points is greater than the number threshold includes: The state information of each trajectory point in the state trajectory is subjected to median filtering. Determine whether there are multiple consecutive trajectory points in the state trajectory after median filtering that correspond to the state information of having detached from the item pickup device.

6. The detection method according to claim 3, wherein, The generation of the state trajectory of the item to be detected includes: Target tracking is performed on the objects to be detected in the multi-frame images to determine the motion information of the objects to be detected in each frame image; The state trajectory is generated based on the motion information; Also includes: If it is determined that the item to be detected has fallen from the item picking device, at least one of the following is determined based on the motion information: the falling trajectory of the item to be detected, the starting position of the falling, or the last falling position.

7. The detection method according to claim 6, wherein, The target tracking of the object to be detected in the multi-frame images includes: The target object in the multi-frame images is tracked using the Distance Intersection over Union (D-IOU) method and the Simple Online and Real-Time Tracking (SORT) method.

8. The detection method according to claim 1, wherein, The step of merging multiple candidate regions containing the item to be detected into a target region containing the item to be detected includes: Calculate the distance between each candidate region containing the item to be detected; Candidate regions with a distance less than a distance threshold are merged to determine the target region.

9. The detection method according to claim 1, wherein, The multiple candidate regions for identifying the presence of a moving object in the image include: Based on the difference between the current frame image and the previous frame image, a moving foreground region is determined in the current frame image, and multiple moving objects exist in the moving foreground region; In the moving foreground region, the plurality of candidate regions are determined.

10. The detection method according to claim 9, wherein, Determining the plurality of candidate regions within the moving foreground region includes: Perform connected component calculation on the moving foreground region to determine multiple connected regions in the image; Based on the multiple connected regions, the multiple candidate regions are determined.

11. The detection method according to claim 9, wherein, Determining the plurality of candidate regions within the moving foreground region includes: The motion foreground region is subjected to morphological processing, which includes at least one of morphological opening operation based on cross-shaped structuring elements, morphological closing operation based on rectangular structuring elements, or morphological opening operation based on cross-shaped structuring elements. In the morphologically processed moving foreground region, the plurality of candidate regions are determined.

12. A detection device for an article, comprising: The classification unit is used to classify the objects to be detected in multiple frames of images using a machine learning model, so as to determine the state information of the objects to be detected in each frame of images, the state information including whether the objects have been removed from the object picking device or not. The judgment unit is used to determine whether the item to be detected has fallen from the item pickup device based on whether there are multiple frames of time-continuous images corresponding to state information indicating that the item has been removed from the item pickup device. The classification unit categorizes moving objects in each frame of the image into objects that have detached from the object pickup device, objects that have not detached from the object pickup device, or non-detectable objects. Non-detectable objects include the object pickup device itself. The item detection device determines that the item to be detected has fallen from the item pickup device when the number of consecutive multi-frame images with status information indicating that the item has not been removed from the item pickup device exceeds a first threshold, and the number of consecutive multi-frame images with status information indicating that the item has been removed from the item pickup device exceeds a second threshold. The classification unit determines multiple candidate regions in the current frame image where a moving object exists; uses the machine learning model to classify the moving objects in the multiple candidate regions to determine whether each candidate region contains the object to be detected; if there are multiple candidate regions containing the object to be detected in the current frame image, the multiple candidate regions containing the object to be detected are merged into a target region where the object to be detected is located; uses the machine learning model to classify the object to be detected in the target region to determine the state information of the object to be detected.

13. A detection device for an article, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the article detection method according to any one of claims 1-11 based on instructions stored in the memory.

14. An automated device, comprising: The detection device for the article as described in claim 12 or 13; An item picking device for picking up the item; An image acquisition device is used to acquire multiple frames of images including at least one of the item picking device or the item.

15. A non-volatile computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting an article according to any one of claims 1-11.

Images