Methods for tracking items during packing

Abstract

The present invention discloses a method for tracking items during a packing process. The method includes obtaining video data of a target item on a conveyor belt during the packing process, collecting image information of the target item in the video data, and tracking the position and trajectory information of the target item through a target detection and target tracking system. The method also collects RFID tag information data and weight information data of the target item during its movement on the conveyor belt during the container loading process. The image information data, weight information data, and RFID tag information data of the target item are integrated into a database and correlated. It is determined whether the image information data, weight information data, and RFID tag information data of the target item at two different frame numbers match. Based on the matching results, it is determined whether the items at different frame numbers during the tracking process are the same item. After confirmation, the trajectory of the target item is output. The beneficial effect of the present invention is that the trajectory of the target item is tracked and verified by using multiple different types of data, ensuring the accuracy of the output trajectory.

Description

Technical Field

[0001] The present invention belongs to the technical field of article tracking, and in particular relates to an article tracking method during a packing process. Background Art

[0002] In various manufacturing and logistics industries, it is essential to accurately track and record the trajectory of items during the packing process. Existing item tracking is basically achieved through computer vision tracking.

[0003] Chinese patent application number 2023104243675 discloses a method and system for target detection and tracking based on improved YOLOv5 and DeepSORT. The method includes obtaining a target image to be detected and preprocessing it, dividing the processed image into a preprocessed image set in a ratio of 6:2:2 to obtain a training set, a validation set, and a test set; constructing a target detection model by improving YOLOv5; and detecting a target vehicle in the current frame using the trained target detection model to obtain information on the helmet worn and the position of the person riding an electric bike in the current frame.

[0004] The aforementioned patented solution utilizes YOLOv5 and DeepSORT technologies to achieve target tracking. YOLOv5 is used for target detection, i.e., identifying objects in images or videos, while DeepSORT is used for target tracking, i.e., tracking the movement of these objects. Combining these two technologies enables more comprehensive visual tasks. However, these technologies have certain tracking flaws. If there is prolonged occlusion or confusion between similar objects, other objects may be mistaken for the original target object, resulting in incorrect output of the target object's trajectory. This lack of reliability is due to the inability to accurately verify whether the target object in two different frame rates is the same object. Summary of the Invention

[0005] To solve the above problems, the present invention provides a method for tracking items during packing, comprising the following steps:

[0006] S1. Item detection and tracking

[0007] Obtain video data of target items on the conveyor belt during the packing process, collect image information data of target items in the video data, and track the position and trajectory information of target items through target detection and target tracking systems;

[0008] S2. Collect verification data

[0009] RFID tag information data and weight information data of target items during the movement of the conveyor belt during container collection;

[0010] S3. Data combination verification

[0011] The image information data, weight information data, and RFID tag information data of the target object are integrated into the database and correlated. The image information data, weight information data, and RFID tag information data of the target object at two different frames are determined to determine whether they match. The final matching result is determined based on the three results. If the matching results are consistent, step S4 is executed; otherwise, step S5 is executed.

[0012] S4, output trajectory

[0013] Update the image information data, weight information data, RFID tag information and item trajectory of the current target item and output them;

[0014] S5. Create a new trace

[0015] The image information data, weight information data, and RFID tag information of the target object are stored, and the target object is tracked as a new target object.

[0016] Preferably, the target tracking system in step S1 is DeepSort, and when determining whether the image information data of the target object at two different frame numbers match in step S3, the matching methods include IOU matching and feature matching.

[0017] Preferably, the weight information data in step S2 is collected by several weighing sensors installed under the conveyor belt, the coordinates of the target object on the conveyor belt are determined by the image information data of the target object, and the corresponding weighing sensors are locked by the coordinates to obtain the weight information of the target object.

[0018] Preferably, the RFID tag information data of the target object in step S2 is read by a plurality of RFID identifiers evenly installed around the conveyor belt, and the position of the target object is locked in combination with the image information data of the target object, and read by the RFID identifier at the corresponding position.

[0019] Preferably, when collecting the image information data, RFID tag information data and weight information data of the target item in steps S1 and S2, the timestamp of the corresponding data will be collected. In step S3, before judging whether the image information data, weight information data and RFID tag information data of the target item in two different frames before and after each other match, it is necessary to judge whether the timestamps collected by the weight information data and RFID tag information data themselves are the same as the corresponding timestamps in the image information data. If they are the same, the timestamps are determined to match and a data matching judgment is performed. If they are different, the corresponding data under the corresponding timestamp of the target item is searched.

[0020] Preferably, when determining whether the timestamps collected by the weight information data and the RFID tag information data themselves are the same as the corresponding timestamps in the image information data, if the difference between the timestamps corresponding to the weight information data and the RFID tag information data and the timestamps corresponding to the image information data does not exceed a threshold, the timestamps are still determined to match.

[0021] Preferably, step S6, video backtracking, is further included, in which the top of the conveyor belt is monitored throughout the day by a camera, and the monitoring video also includes a timestamp. When the final matching result of the image information data, weight information data, and RFID tag information data of the target item at two different frames is negative, the status of the target item in the corresponding time period is observed through video backtracking.

[0022] Preferably, the method further includes step S7, sound identification, in which sound data around the conveyor belt is collected by sound sensors arranged around the conveyor belt, and the sound data is identified by a sound event model to determine whether an object has fallen. If so, an alarm is issued; if not, sound data collection continues.

[0023] Preferably, if the corresponding sound data is determined to be the sound of an object falling in step S7, the sound model needs to recognize the corresponding sound data at least twice, and the multiple recognition results are used as the final judgment result.

[0024] The advantages of the present invention are:

[0025] The present invention adopts a method of combining multiple data. When tracking a target object through computer vision technology, it judges multiple data before and after the target object, and determines whether the target object in two different frame numbers is the same object based on the final result, thereby determining whether there is tracking loss during tracking by computer vision technology. It can output the trajectory of the target object more accurately and improve the reliability of computer vision tracking.

[0026] When comparing and matching the data before and after the target object, the timestamps corresponding to each data are collected to ensure the consistency of data collection, thereby improving the accuracy of the collected data and further ensuring the reliability of computer vision tracking.

[0027] The entire process of moving packed items is monitored by video, and the time stamps of important events are collected in real time and encoded in the video, so that video playback can accurately locate the location. When the target item is lost, the video information of the corresponding time period can be viewed in time to find the cause.

[0028] A sound sensor is provided to timely monitor whether an item has fallen. When the target item is lost, it can provide information on whether the item has fallen, and assist in investigating the cause of the target item's loss.

[0029] When the target object is lost, the new target can be tracked in time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention. Example 1

[0032] Combine Figure 1 , provides a method for tracking items during packing, comprising the following steps:

[0033] S1. Item detection and tracking

[0034] Video data of target items on the conveyor belt during the packing process is acquired, and image information data of the target items in the video data is collected. The target item's position and trajectory information are tracked through target detection and target tracking systems. Target detection uses YOLOv5, a computer vision technology used for target detection. Target detection is one of the computer vision tasks. YOLOv5 can detect target items during the packing process through deep learning technology. YOLOv5 can quickly detect and identify target items in image information data. The target tracking system is DeepSort, which uses a deep neural network to extract the feature vector of the target item and uses Kalman filtering and the Hungarian algorithm for target tracking. This maintains the target item's trajectory and assigns a unique TrackID based on the target item's motion information and other features. TrackID is a unique identifier used to identify and distinguish different targets in the target tracking system so that the system can effectively track and manage these targets.

[0035] S2. Collect verification data

[0036] Real-time RFID tag data and weight data are collected from target items as they move along the conveyor belt during container loading. Items are transported along the conveyor belt before being loaded into the container. A video camera positioned above the conveyor belt captures the entire movement of the target items. The camera captures video data of the entire movement of the target items. Weight data is collected using several load cells installed beneath the conveyor belt. These load cells are evenly spaced and positioned beneath the conveyor belt support structure, ensuring a close enough gap to accurately measure weight without significant friction. The load cells are constructed of low-friction material to minimize physical contact with the conveyor belt. As the target items move along the conveyor belt, the corresponding load cells capture their weight. The target items' RFID tag data is read by several RFID readers evenly spaced around the conveyor belt.

[0037] S3. Data combination verification

[0038] The target object's image data, weight data, and RFID tag data are integrated into a database and correlated. A match is then determined between the image data, weight data, and RFID tag data for the target object in two different frames. This match is intended to detect whether the computer vision technology has lost tracking. If the target object detected by the computer vision technology in different frames is different, tracking loss is determined.

[0039] During verification, it is first necessary to determine whether the image information data of the target object in the two different frames before and after tracking matches. The matching methods include IOU matching and feature matching. IOU matching refers to the ratio of the intersection area of ​​the bounding boxes of the two detected targets to their union area. In target object tracking, IOU matching determines whether the target object in the previous frame and the target object in the current frame represent the same object by calculating the IOU. If the IOU is greater than the threshold of 0.5, the two bounding boxes are considered to be matched successfully, indicating that they represent the same object. Feature matching determines whether the target object in the previous frame and the target object in the current frame represent the same object by comparing their feature vectors. The similarity between the feature vectors is measured by distance metric. If the similarity between the two feature vectors is greater than a certain threshold, the two detected targets are considered to be matched successfully.

[0040] In this embodiment, the two different frames represent the image information of the target object in its initial state of motion, i.e., the first frame, and the image information 30 seconds later, i.e., the second frame. Only when both IOU matching and feature matching are successful at the two different frames can the image information data of the target object at the two different frames be considered matched. The two target objects are determined to be the same object based on the image information data.

[0041] Next, a determination is made as to whether the weight information data of the target item in the two different frames matches. The corresponding weighing sensors are locked using the image information of the target item in the two different frames to obtain the weight information of the target item in the different frames. In this embodiment, the weight information of the target item in its initial moving state is determined, i.e., the weight information in the first frame and the corresponding weight information in the second frame. The coordinates of the target item on the conveyor belt can be determined based on the image information data in the corresponding frames. The corresponding weighing sensors are locked using the coordinates to obtain the weight information of the target item in the corresponding frames. Alternatively, the distance moved by the target item can be determined based on the speed of the conveyor belt and the time corresponding to the image information in the corresponding frames. The weighing sensors in the corresponding positions are determined based on the distance and the corresponding weight information is obtained. Only when the weight information data of the target item in the two different frames does not exceed a threshold value (typically 5g), can the weight information data of the target item in the two different frames be determined to match and be the same item.

[0042] Finally, the RFID tag information data of the target item in the two different frames is determined to match. The RFID tag information data of the target item is read by several RFID readers evenly installed around the conveyor belt. The target item's position is locked in combination with the image information data of the target item, and the RFID reader at the corresponding position is used to read it. The coordinates of the target item on the conveyor belt can be determined based on the image information data of the corresponding frame. The corresponding RFID reader is locked by the coordinates to obtain the RFID tag information of the target item. The distance moved by the target item can also be determined based on the speed of the conveyor belt and the time corresponding to the image information of the corresponding frame. The RFID reader at the corresponding position is determined based on the distance and the corresponding RFID tag information is obtained. Only when the unique identifiers corresponding to the RFID tag information of the target item in the two different frames are consistent can the RFID tag information of the target item in the two different frames be determined to match and be the same item.

[0043] Only when the image information data, weight information data, and RFID tag information of the target object in the two different frames match can the objects in the two different frames be determined to be the same target object and the target object has not been lost during tracking, and the target object's trajectory can be output. At the same time, the image information data, weight information data, and RFID tag information of the current target object are updated, and the target object's trajectory tracking is carried out according to the above steps for the next stage.

[0044] If there is a mismatch between the image information data, weight information data, and RFID tag information of the target item in the two different frames, it is determined that the tracked target item is lost, and the new item replaces the original target item. The image information data, weight information data, and RFID tag information of the target item are stored, and the target item is tracked as the new target item according to the above tracking method.

[0045] This embodiment combines multiple data to determine multiple data before and after the target object when tracking it through computer vision technology. Based on the final result, it is determined whether the target object in two different frame numbers is the same object, thereby determining whether the object has been lost in tracking. This can more accurately output the trajectory of the target object and improve the reliability of computer vision tracking. Example 2

[0046] This embodiment shares some of the same features as Example 1. In this embodiment, when collecting the video data, RFID tag information data, and weight information data of the target object in step S1, the corresponding data timestamps are collected. In step S3, before determining whether the image information data, weight information data, and RFID tag information data of the target object in two different frames match, it is necessary to determine whether the timestamps corresponding to the collected weight information data and RFID tag information data are identical to the timestamps corresponding to the image information data. If they are identical, the timestamps are determined to match, and the corresponding data matching determination is performed. If they are different, step S3 is repeated. For example, when collecting the image information data, weight information data, and RFID tag information corresponding to the second frame, if the timestamp corresponding to the second frame is 14:30:30, then the timestamp corresponding to the corresponding weighing sensor when collecting the weight information of the target object in the corresponding frame should also be 14:30:30, where 14, 30, and 30 represent hours, minutes, and seconds, respectively. The timestamp corresponding to the RFID tag information data of the target object in the corresponding frame read by the RFID reader should also be 14:30:30. Only in this way can the temporal consistency of the collected data be guaranteed, thereby improving the accuracy of the collected data and further ensuring the reliability of computer vision tracking.

[0047] To avoid time delays caused by data transmission, if the difference between the timestamp corresponding to any of the weight information data and the RFID tag information data and the timestamp corresponding to the image information data does not exceed the threshold, which is set to 1S, the timestamps are still considered to match.

[0048] If the timestamp of the data does not match, the RFID tag information data and weight information data of the target item under the corresponding timestamp should be found to ensure the accuracy of the data. Example 3

[0049] Compared with Example 2, this embodiment adds an additional camera to monitor the top of the conveyor belt throughout the day. The monitoring video also contains a timestamp. When the final matching result of the image information data, weight information data, and RFID tag information data of the target item at two different frames is negative, that is, when the tracked item is lost, the status of the target item in the corresponding time period can be observed through video backtracking.

[0050] This embodiment also collects sound data from the conveyor belt by placing sound sensors around it. Sound sensors or microphones are deployed within the area to be monitored. The number and location of the sound sensors should be appropriately selected based on the size and shape of the monitored area. Sound sensors can be placed on the ceiling, walls, floor, or other appropriate locations, configured to continuously capture sound data from the environment. The received sound data can be analyzed and feature extracted using sound processing algorithms. Key features may include the amplitude, frequency, duration, and energy characteristics of the sound event. During the monitoring phase, the system can use machine learning algorithms or rule-based methods to build sound event models. These models are used to identify drop events. The model training data can include both dropped items and sound data from normal operating conditions. The system continuously monitors sound data during operation. When a sound event is detected, the system compares the sound data with the sound event model to determine whether a drop event has occurred. The sound event model identifies the sound data and determines whether an item has been dropped. If so, an alarm is issued; if not, sound data collection continues. If the sound data is determined to be the sound of an object being dropped, the sound model will perform at least two more recognitions on the sound data, with the final judgment based on these multiple recognition results to ensure detection accuracy. If the target object is lost, the system can provide information on whether the object has been dropped, helping to identify the cause of the loss.

[0051] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for tracking items during packing, characterized by: The following steps are included: S1. Item detection and tracking Obtain video data of target items on the conveyor belt during the packing process, collect image information data of target items in the video data, and track the position and trajectory information of target items through target detection and target tracking systems; S2. Collect verification data RFID tag information data and weight information data of target items during the movement of the conveyor belt during container collection; S3. Data combination verification The image information data, weight information data, and RFID tag information data of the target object are integrated into the database and correlated. The image information data, weight information data, and RFID tag information data of the target object at two different frames are determined to determine whether they match. The final matching result is determined based on the three results. If the matching results are all consistent, step S4 is executed. Otherwise, execute step S5; S4, output trajectory Update the image information data, weight information data, RFID tag information and item trajectory of the current target item and output them; S5. Create a new trace Storing the image information data, weight information data, and RFID tag information of the target object, and tracking the target object as a new target object; When collecting the image information data, RFID tag information data, and weight information data of the target item in steps S1 and S2, the timestamps of the corresponding data are collected. In step S3, before determining whether the image information data, weight information data, and RFID tag information data of the target item at two different frames match, it is necessary to determine whether the timestamps collected in the weight information data and RFID tag information data themselves are the same as the corresponding timestamps in the image information data. If they are the same, it is determined that the timestamps match and a data match determination is performed. If they are different, the corresponding data corresponding to the timestamp of the target item is searched. When determining whether the timestamps collected by the weight information data and the RFID tag information data themselves are the same as the corresponding timestamps in the image information data, if the difference between the timestamps corresponding to the weight information data and the RFID tag information data and the timestamp corresponding to the image information data does not exceed the threshold, the timestamps are still determined to match; The system also includes step S6, video backtracking, in which the camera monitors the area above the conveyor belt throughout the day. The monitoring video also includes a timestamp. If the final matching result of the image information data, weight information data, and RFID tag information data of the target item at two different frames is negative, the status of the target item in the corresponding time period is observed through video backtracking. The system further includes step S7, sound identification, in which sound sensors disposed around the conveyor belt collect sound data around the conveyor belt, identify the sound data using a sound event model, and determine whether an object has fallen. If so, an alarm is sounded. If not, continue collecting sound data.

2. The method for tracking items during packing according to claim 1, wherein: The target tracking system in step S1 is DeepSort. When determining whether the image information data of the target object in two different frames before and after match in step S3, the matching methods include IOU matching and feature matching.

3. The method for tracking items during packing according to claim 2, wherein: The weight information data in step S2 is collected by a number of weighing sensors installed under the conveyor belt. The coordinates of the target object on the conveyor belt are determined by the image information data of the target object, and the corresponding weighing sensors are locked by the coordinates to obtain the weight information of the target object.

4. The method for tracking items during packing according to claim 3, wherein: In step S2, the RFID tag information data of the target object is read by a number of RFID identifiers evenly installed around the conveyor belt. The position of the target object is locked in combination with the image information data of the target object and read by the RFID identifier at the corresponding position.

5. The method for tracking items during packing according to claim 1, wherein: If the corresponding sound data is determined to be the sound of an object falling in step S7, the sound model needs to recognize the corresponding sound data at least twice more, and the multiple recognition results are used as the final judgment result.

Images