A freight object tracking method based on machine vision

Abstract

This invention proposes a machine vision-based freight object tracking method, belonging to the field of visual tracking technology for logistics conveying objects. The method includes barcode recognition, volume measurement, and queue creation for freight objects. After the freight objects pass through the conveyor line, their volume is measured by a linear laser stereo camera, and then a conveyor line queue is created. Multidimensional features of the freight objects are acquired and added to the freight object queue information. Several mainline detection points are configured on the main conveyor line, and several branch line detection points are configured on several branch lines. The mainline detection points track the movement trajectory of the freight objects on the main line. When a freight object is about to transfer from the main line to a branch line, the identified freight object is removed from the mainline freight object queue and inserted into the end of the branch line freight object queue of the corresponding branch line. The branch line detection points further track the freight object. Through video measurement and tracking at multiple stages, the reliability and accuracy of the entire material object transportation process are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual tracking of logistics transport objects, and in particular to a freight object tracking method based on machine vision. Background Art

[0002] With the continuous increase in the number of shipping cargo types, traditional cargo terminal object management methods are increasingly unable to meet the needs of efficient operations. With the rapid development of the Internet of Things, cloud computing, big data, and artificial intelligence in recent years, combined with the introduction of automation technologies, cargo terminal operational efficiency can be effectively improved, labor waste can be reduced, and the accuracy and smoothness of logistics information can be enhanced, making the entire logistics process information-based, intelligent, and visualized. In fully automated sorting and palletizing systems at shipping cargo terminals, the varying sizes, weights, and shapes of cargo items can lead to misordering and slipping on the conveyor line. Furthermore, when the balance wheel operates at the sorting point, errors such as multiple or incorrectly stacked objects are unavoidable. Current terminal sorting and palletizing conveyor lines often use photoelectric switch timing to determine the position of cargo items and control the balance wheel. However, this can occasionally lead to inconsistencies between the palletized cargo items and the system records, causing system errors and ultimately requiring manual line downtime and troubleshooting, significantly reducing the efficiency of the sorting and palletizing system. Furthermore, the similarity of cargo items in shipping materials poses challenges for machine vision recognition. Existing methods typically use a single matching algorithm for cargo object recognition, which often suffers from low efficiency and insufficient accuracy.

[0003] In view of this, it is very necessary to provide a freight object tracking method based on machine vision to improve the fault tolerance of the shipping warehouse sorting and palletizing system, enhance the reliability of the system, and accelerate the overall efficiency of the warehouse cargo flow. Summary of the Invention

[0004] In view of this, the present invention proposes a machine vision-based freight object tracking method that covers the features of each surface of the freight object, realizes size and volume measurement, contour and label recognition, and then confirms the uniqueness of the freight object and the reliability of the conveying and palletizing process by continuously tracking the freight symmetry on the main line and branch line.

[0005] The technical solution of the present invention is implemented as follows: The present invention provides a method for tracking freight objects based on machine vision, comprising the following steps:

[0006] S1: Barcode recognition: A barcode scanner is installed at the starting point of the conveyor line to dynamically scan the barcodes on each surface of the freight object. The conveyor line includes a main line and several branch lines. The freight objects are transferred from the main line to different branch lines.

[0007] S2: Volume measurement and queue creation. After the freight object passes through the conveyor line's linear laser stereo camera for volume measurement, a conveyor line queue is created. The conveyor line queue is divided into the main line freight object queue, the main line event queue, the branch line freight object queue, and the branch line event queue. After the freight object completes the barcode dynamic scanning and volume measurement, it is assigned a unique ID and inserted into the end of the main line freight object queue. The freight object's queue information is then created.

[0008] S3: Obtain multidimensional features of the freight object and add the multidimensional features of the freight object to the queue information of the freight object;

[0009] S4: Several mainline detection points are configured on the main line of the conveyor line, and several branch line detection points are configured on the branch lines. When a mainline detection point detects that a freight object first appears in the detection range, it generates a first entry event and inserts it into the end of the mainline event queue. Several mainline detection points track the movement trajectory of the freight object on the main line and use a three-dimensional matching algorithm to match the freight object passing through the mainline detection point with the freight object in the mainline freight object queue.

[0010] S5: Process the freight object based on the matching results of the mainline detection points; continue to track the freight object until it leaves the detection range of several mainline detection points. When the freight object leaves the detection range of the current mainline detection point, update the time information of the freight object in the mainline freight object queue;

[0011] S6: When a freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object. The identified freight object is then removed from the main line freight object queue and inserted into the end of the branch line freight object queue of the corresponding branch line.

[0012] S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the end of the branch line event queue of the corresponding branch line;

[0013] S8: Process the freight object based on the matching results of the branch line detection point; continuously track the freight object, and when the freight object passes through each branch line detection point on the branch line, use a two-dimensional matching algorithm to verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue;

[0014] S9: Process the freight object based on the matching results of the branch line detection point; continue to track the freight pair until it leaves the detection range of each branch line detection point or is captured by the robotic arm, remove the freight object from the branch line freight object queue, and end the tracking process of the freight object.

[0015] Based on the above technical solution, preferably, the acquisition of multi-dimensional features of the freight object in step S3 specifically includes target detection, identification of feature values ​​of the freight object, and updating queue information of the freight object; wherein:

[0016] Object detection uses a previously trained deep learning object detection model to obtain the external contours of the freight object and perform image segmentation to obtain a complete image of the freight object.

[0017] To identify the feature values ​​of a freight object, image recognition technology is used to segment the areas where all labels are located on the freight object. Color space conversion is used to identify the color of each label and the color of the freight object. By identifying whether there are bright reflective areas in the non-label areas of the freight object image, the presence of stretch film on the freight object surface can be determined.

[0018] Update the queue information of the freight object, and update the obtained photos and colors of each end face of the freight object, the color and quantity of the labels, and the feature values ​​corresponding to whether there is a stretch film on the surface of the freight object to the queue information of the freight object; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object recognized by the line laser stereo camera, the information obtained by dynamic scanning of the barcode, and the number of the destination branch line.

[0019] Preferably, the several main line detection points described in step S4 track the movement trajectory of the freight object on the main line, and each main line detection point and branch line detection point is equipped with a photoelectric radiation tube, a camera and an ultrasonic component. When the freight object passes through each main line detection point and blocks the light path of the photoelectric radiation tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera of each main line detection point has a unique ID.

[0020] Further preferably, the three-dimensional matching algorithm used in step S4 to match the freight objects passing through the mainline detection point with the freight objects in the mainline freight object queue is to match the queue information of the freight objects on the mainline with the freight objects in the mainline freight object queue based on three dimensions: volume parameter matching, arrival time matching, and image matching.

[0021] Volume parameter matching: Assume that the camera's horizontal field of view is α, its vertical field of view is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasound is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, and the pixels occupied by the freight object in the vertical direction are pixel_y. Then the true height of the freight object, box_z, is box_z = h1 - h2; the horizontal length of the freight object, box_x, is box_x = 2 × pixel_x × h2 × tan(α / 2) / X; the horizontal width of the freight object, box_y, is box_y = 2 × pixel_y × h2 × tan(β / 2) / Y. Use the obtained horizontal length, box_x, horizontal width, and true height, box_z, of the freight object to match the length, width, and height of the freight objects identified by the line laser stereo camera in the queue information of the main line freight object queue to obtain a set A of freight objects on the main line that meets the error range.

[0022] Arrival time matching: Based on the time t0 when the freight object is barcode-recognized, combined with the speed v of the conveyor line, let the distance between the barcode scanner and the current mainline detection point be s, and estimate the time when the freight object arrives at the current mainline detection point as t = t0 + s / v. The set C of freight objects on the mainline that meet the error range of the arrival time at the current mainline detection point is obtained;

[0023] Image matching: Using an improved Hungarian matching algorithm, we first combine the aforementioned volume parameter matching and arrival time matching to eliminate non-compliant freight objects from Sets A and C. We then filter the freight objects in the set based on the wrapping film attributes within the freight objects' multidimensional features. Finally, we compare the color of each end face of the freight object, the number of labels on the freight object, and the label color. We eliminate freight objects on the main line with inconsistent end face color, inconsistent label quantity, or inconsistent label color, and match only the qualified freight objects.

[0024] Further preferably, the obtained horizontal length box_x, horizontal width box_y and actual height box_z of the freight object are used to match the length, width and height of the freight object recognized by the line laser stereo camera in the queue information of the main line freight object queue, that is, the set of freight objects on the main line is M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera; the length, width and height of the freight object recognized by the line laser stereo camera are x, y, and h respectively, satisfying 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, and 0.85×mh≤h≤1.15×mh.

[0025] More preferably, the error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated time when the freight object arrives at the current mainline detection point, or ±50 seconds of the estimated time when the freight object arrives at the current mainline detection point.

[0026] Further preferably, the processing of the freight object based on the matching result of the main line detection point in step S5 includes the following:

[0027] If the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completion of the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached;

[0028] If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued;

[0029] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

[0030] More preferably, the tracking of freight objects on the branch line described in step S7 is performed through several branch line detection points on the branch line. When the freight object on the branch line passes through each branch line detection point and blocks the light path of the photoelectric radiating tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each branch line detection point has a unique ID.

[0031] Still further preferably, in step S8, when the freight object passes through each branch line detection point on the branch line, a two-dimensional matching algorithm is used to verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue. The volume parameter matching and image matching in step S4 are used again at each branch line detection point to match the contents of the queue information of the freight object on the branch line with those of the freight objects in the branch line freight object queue in two dimensions.

[0032] More preferably, the freight object is processed based on the matching result of the branch line detection point in step S9, including the following contents:

[0033] If the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completion of the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached;

[0034] If the maximum matching time has passed and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued;

[0035] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a new freight object record marked as abnormal is inserted into the branch line freight object queue and a notification is issued.

[0036] The present invention provides a method for tracking freight objects based on machine vision, which has the following advantages over the prior art:

[0037] (1) The present invention uses a barcode scanning device covering all sides of the freight object to identify the barcode content of the freight object and the corresponding conveying route content; through volume measurement and queue creation, a main line freight object queue, a main line event queue, a branch line freight object queue and a branch line event queue covering each freight object on the main line and branch line are generated, which are used to record the status update information and arrival time of the freight object when entering different positions of the main line or branch line, thereby constructing a complete and traceable conveying trajectory of the freight object; after configuring a line laser stereo camera for volume measurement, a conveying line queue is created, thereby facilitating subsequent entry and exit and writing of matching results of the passing detection points, the corresponding camera ID and the time of completion of the matching, thereby realizing continuous tracking operations;

[0038] (2) For each detection point on the main line, a three-dimensional matching algorithm is used to match the freight objects passing through the main line detection point with the freight objects in the main line freight object queue. The freight objects within the error range are screened from the three dimensions of volume parameter matching, arrival time matching and image matching, and the closest one or more freight objects are selected to achieve continuous tracking of the real-time position and arrival time of the freight objects on the main line, thus realizing the position traceability function of the main line part;

[0039] (3) After the freight object arrives at the target branch line, each inspection point on the branch line further uses a two-dimensional matching algorithm to match the freight object passing through the branch line inspection point with the freight objects in the branch line freight object queue. The freight objects within the error range are screened from the two dimensions of volume parameter matching and image matching, and the real-time position of the freight object on the branch line is continuously tracked until the freight object leaves the branch line and completes the transmission process of the logistics conveyor line, thereby achieving full traceability of the freight object. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flowchart of the steps of a freight object tracking method based on machine vision of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] Currently, most terminal sorting and palletizing conveyor lines use photoelectric switch beats to determine the location of freight objects and control the balance wheel. However, since the freight objects cannot be tracked throughout the entire process, occasional inconsistencies between the palletized freight objects and the system records may occur, causing system errors. Ultimately, the line needs to be stopped for manual inspection, which greatly reduces the efficiency of the sorting and palletizing system. Figure 1 As shown, the present invention provides a method for tracking freight objects based on machine vision, comprising the following steps:

[0044] S1: Barcode recognition: A barcode scanner is installed at the start of the conveyor line to dynamically scan the barcodes on all surfaces of the freight items. The conveyor line consists of a main line and several branch lines, through which freight items are transferred. Freight items that require sorting and palletizing are typically transported from the beginning of the main line to the end of a branch line at their destination, completing the logistics conveyor line's delivery process.

[0045] Barcodes may be affixed to any surface of a freight object, and there may be more than one. Therefore, the present invention employs a six-sided barcode scanning device installed at the starting point of the conveyor line. Barcode readers are installed on the left, right, front, back, and top of the conveyor line, and a line-scanning barcode reader is installed at the bottom of the conveyor line. The line-scanning barcode reader scans the bottom of the freight object through the gap in the conveyor line, and the conveyor line drives the freight object in uniform motion to image the bottom surface. This allows for dynamic barcode scanning of the freight object's top, bottom, left, right, front, and back surfaces, identifying the barcode order number and comparing it with all the barcode information acquired in the WCS system. This allows for the acquisition of the freight object's actual barcode information and the query of the destination branch line number corresponding to the barcode.

[0046] In this embodiment, the freight objects may be box-shaped structures such as corrugated boxes, wooden crates, small cargo boxes, or packages of regular shapes, all of which can be transported by the conveyor line. Figure 1 The term box used in the process is for illustration only and is not considered to be a limitation on the freight objects in the present invention.

[0047] S2: Volume measurement and queue creation. After the freight object passes through the linear laser stereo camera on the conveyor line for volume measurement, the conveyor line queue is created. The conveyor line queue is divided into: main line freight object queue, main line event queue, branch line freight object queue and branch line event queue; the freight object that has completed the barcode dynamic scanning and volume measurement is assigned a unique ID and inserted into the end of the main line freight object queue, and the queue information of the freight object is created.

[0048] After initial barcode recognition, the freight object is fed into a line laser stereo camera on the conveyor line to collect its specifications and image information. Based on the principle of triangulation, the line laser stereo camera uses an image sensor to capture the laser line projected by a laser generator on the object's surface, reconstructing the object's surface contours and calculating the length, width, height, and volume of the freight object. This part of the invention is a conventional technique in the field and does not invent or involve improvements to the measurement capabilities of the line laser stereo camera.

[0049] S3: Acquire the multidimensional features of the freight object and add the multidimensional features of the freight object to the queue information of the freight object.

[0050] Obtaining multi-dimensional features of freight objects, including target detection, identifying feature values ​​of freight objects, and updating queue information of freight objects; wherein:

[0051] S31: Object detection uses a previously trained deep learning object detection model to obtain the external contours of the freight object and perform image segmentation to obtain a complete image of the freight object.

[0052] Specifically, a high-performance YOLO object detection model was selected and trained using a dataset of labeled box images to ensure that the model could accurately detect boxes of various shapes and sizes. After training, the model's performance on the validation set was evaluated to ensure high detection accuracy and recall. The YOLO object detection model is a commonly used technical method in this field, and its code is open source and easily available. The trained object detection model was then deployed to the video stream processing equipment on the conveyor line, and each frame of the image was detected in real time, outputting the bounding box of each freight object and its confidence score.

[0053] A fine-grained and appropriately sized UNet image segmentation model is selected in advance and trained using a dataset of box images with pixel-level annotations to ensure that the model accurately segments the outlines of freight objects. After training, the model's performance is evaluated on a validation set to ensure high segmentation accuracy. The trained UNet image segmentation model is then used to detect bounding boxes within images containing freight objects and segment the regions within those boxes. The resulting binary image is then output, where the foreground represents the freight objects and the background represents other non-freight objects.

[0054] S32: Identifying the characteristic values ​​of the freight object uses image recognition technology to segment the areas where all labels on the freight object are located, and uses color space conversion to identify the color of each label and the color of the freight object. By identifying whether there are bright reflective areas in the non-label areas of the freight object image, it is identified whether there is a stretch film on the surface of the freight object.

[0055] After segmenting the image containing the freight object, the labeled boxes are marked. The trained object detection algorithm is used to identify and segment all labels on the surface of the freight object in the photo, and the rectangular image of the label is segmented. Then, color space conversion is used to identify the color characteristics of each label. Several rectangles are segmented from the image of the surface area of ​​the freight object outside the label, and the color of the freight object itself is also identified using color space conversion. That is, the RGB color is converted to HSV space to obtain the brightness value H, the maximum and minimum RGB color values. If the maximum and minimum RGB color values ​​are equal, the freight object is monochrome, and the color of the freight object is determined based on the RGB color value. If the maximum and minimum RGB color values ​​are not equal, the saturation and color channel values ​​are calculated to determine the color combination of the freight object.

[0056] Boxes with wrap have significant reflective properties. A well-labeled deep learning model is used to detect the presence of wrap. This is determined by the presence of continuous, non-labeled, bright areas with a grayscale value of 255 in at least one surface image. Typically, reflective areas have at least one bright, linear boundary. For each segmented box image, a feature extraction model is used to extract a 1024-dimensional feature vector. The extracted feature vectors are stored in a data structure, such as a dictionary or list, to facilitate subsequent matching operations.

[0057] S33: Update the queue information of the freight object, and update the obtained photos and colors of each end face of the freight object, the color and quantity of the labels, and the feature values ​​corresponding to whether there is a stretch film on the surface of the freight object to the queue information of the freight object corresponding to the main line freight object queue; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object identified by the line laser stereo camera, the information obtained by dynamic scanning of the barcode and the number of the destination branch line.

[0058] S4: Several mainline detection points are configured on the main line of the conveyor line, and several branch line detection points are configured on the branch lines. When a mainline detection point detects that a freight object first appears in the detection range, it generates a first entry event and inserts it into the end of the mainline event queue. Several mainline detection points track the movement trajectory of the freight object on the main line and use a three-dimensional matching algorithm to match the freight object passing through the mainline detection point with the freight object in the mainline freight object queue.

[0059] Several mainline inspection points track the movement of freight objects along the mainline. Each mainline inspection point and branch line inspection point is equipped with a photoelectric tube, camera, and ultrasonic component. When a freight object passes through a mainline inspection point and blocks the light path of the photoelectric tube, the camera and ultrasonic component are activated. The ultrasonic component measures the height of the freight object, while the camera captures an image of the top of the freight object. Each camera at the mainline inspection point has a unique ID.

[0060] Among them, a three-dimensional matching algorithm is used to match freight objects passing through the main line detection point with freight objects in the main line freight object queue. The matching is carried out based on volume parameter matching, arrival time matching and image matching. The queue information of each freight object on the main line and the main line freight object queue is matched in three dimensions.

[0061] 1) Volume parameter matching: Assume that the horizontal field of view angle of the camera is α, the vertical field of view angle is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasonic wave is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, and the pixels occupied by the freight object in the vertical direction are pixel_l, y. Then the true height of the freight object, box_z, is box_z = h1-h2; the horizontal length dimension of the freight object, box_x, is xbox_x = 2×pixel_x×h2×tan(α / 2) / X; the horizontal width dimension of the freight object, box_y, is box_y = 2×pixel_y×h2×tan(β / 2) / Y; the obtained horizontal length box_x, horizontal width box_y, and true height box_z of the freight object are used to match the length, width, and height of the freight objects recognized by the line laser stereo camera in the queue information of the main line freight object queue, and obtain the set A of freight objects on the main line that meets the error range;

[0062] The freight objects on the main line that meet the error range are as follows: let the set of freight objects on the main line be M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera; let the length, width and height of the freight object recognized by the line laser stereo camera be x, y, h respectively, then the freight objects in set A simultaneously meet 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, and 0.85×mh≤h≤1.15×mh.

[0063] 2) Arrival Time Matching: Based on the time t0 at which the barcode of the freight object is recognized and the speed v of the conveyor line, let the distance between the barcode scanning device and the current mainline detection point be s. The estimated arrival time of the freight object at the current mainline detection point is t = t0 + s / v. The set C of freight objects on the mainline that meet the error range of the arrival time at the current mainline detection point is obtained.

[0064] The error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated arrival time of the freight object at the current mainline detection point, or ±50 seconds of the estimated arrival time of the freight object at the current mainline detection point.

[0065] 3) Image Matching: Using the improved Hungarian matching algorithm, we first remove non-compliant freight objects from Sets A and C, combining the aforementioned volume parameter matching and arrival time matching. We then filter the freight objects in the set based on the wrapping film attributes within the freight objects' multidimensional features. Finally, we compare the color of each end face of the freight object, the number of labels on the freight object, and the label color. We then remove freight objects on the main line with inconsistent end face color, inconsistent label quantity, or inconsistent label color, ultimately matching the qualified freight objects.

[0066] The image matching algorithm uses an improved Hungarian matching algorithm. The traditional Hungarian matching algorithm is very complex. Shipping cargo terminals have very long conveyor lines, with a large number of cargo items. There are many inspection points along the logistics conveyor lines, and the time interval between two cargo items on the conveyor lines is approximately one second. Therefore, the image matching algorithm must be completed at multiple inspection points in a very short time, which consumes a lot of computing resources and is very costly.

[0067] Therefore, the present invention makes the following improvements to the Hungarian matching algorithm:

[0068] 3.1) Combine the volume parameter matching and arrival time matching described above to filter out shipments that clearly do not match;

[0069] 3.2) Screen out the freight objects that do not match the stretch film identification;

[0070] 3.3) Use the object detection model to filter out freight objects with inconsistent colors and exclude freight objects with inconsistent label quantity, color, and location;

[0071] In summary, the cost matrix size of the Hungarian matching algorithm is reduced, and the algorithm's time consumption is shortened. Furthermore, combined with the arrival time parameter, arrival time priority sorting is introduced, and the selected freight object set is sorted according to the calculated arrival time parameter. Finally, combined with the Hungarian matching algorithm, freight objects that meet the requirements can be matched more quickly.

[0072] S5: Process the freight object based on the matching results of the mainline detection points; continue to track the freight object until it leaves the detection range of several mainline detection points. When the freight object leaves the detection range of the current mainline detection point, update the time information of the freight object in the mainline freight object queue;

[0073] Among them, based on the matching results of the main line detection points, the freight objects are processed, including the following:

[0074] If the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completion of the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached;

[0075] If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued;

[0076] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

[0077] The insertion of a new freight object record marked as abnormal here differs from the previously mentioned updating of the freight object queue information. It primarily records the calculated volume parameters and arrival time of the unmatched freight object, as well as the conclusion that the corresponding freight object could not be found in the main freight object queue. The notification is issued to remind staff to manually confirm the missing or unknown freight object. Based on the manual confirmation, it is determined to be a mechanical failure or equipment abnormality at the detection point.

[0078] S6: When a freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object. The identified freight object is removed from the main line freight object queue and inserted into the tail of the branch line freight object queue of the corresponding branch line.

[0079] S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the end of the branch line event queue of the corresponding branch line;

[0080] Among them, tracking of freight objects on the branch line is carried out through several branch line detection points on the branch line. When the freight object on the branch line passes through each branch line detection point and blocks the light path of the photoelectric radiation tube, the camera and ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each branch line detection point also has a unique ID.

[0081] S8: Process the freight object based on the matching results of the branch line detection point; continuously track the freight object, and when the freight object passes through each branch line detection point on the branch line, use a two-dimensional matching algorithm to verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue;

[0082] When a freight object passes through each branch line checkpoint, a two-dimensional matching algorithm is used to verify whether the freight object at the current checkpoint matches the freight objects in the branch line freight object queue. At each branch line checkpoint, the volume parameter matching and image matching from step S4 are repeated to match the queue information of the freight object on the branch line with that of each freight object in the branch line freight object queue. Because the transportation speed of the branch line is likely to differ from that of the main line, simply matching by arrival time may not be appropriate. Therefore, time dimension matching is not used, and a two-dimensional matching algorithm is developed here.

[0083] S9: Process the freight object based on the matching results of the branch line detection point; continue to track the freight pair until it leaves the detection range of each branch line detection point or is captured by the robotic arm, remove the freight object from the branch line freight object queue, and end the tracking process of the freight object.

[0084] Among them, based on the matching results of the branch line detection point, the freight object is processed, including the following:

[0085] If the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completion of the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached;

[0086] If the maximum matching time has passed and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued;

[0087] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a new freight object record marked as abnormal is inserted into the branch line freight object queue and a notification is issued.

[0088] Similar to step S5, a new record of a freight object marked as abnormal is inserted, which is different from the content of the queue information of the freight object updated above. The notification is also sent to remind the staff to manually confirm the lost freight object or the unknown freight object.

[0089] For unknown freight objects, the handling method can be: 1) stop the machine to remove them manually, and then manually resume the operation of the conveyor line; 2) when the unknown freight object reaches the end of the main line or branch line, the robot removes the unknown freight object and stores it separately. After manual confirmation, the tracking method process of the present invention is re-executed; 3) if multiple unknown freight objects appear continuously, such as an odd number of more than 5, it is considered that the equipment at one or more detection points is abnormal, and the operation of the main line or branch line needs to be stopped and inspected.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A freight object tracking method based on machine vision, characterized in that: The steps include: S1: Barcode recognition: A barcode scanner is installed at the starting point of the conveyor line to dynamically scan the barcodes on each surface of the freight object. The conveyor line includes a main line and several branch lines. The freight objects are transferred from the main line to different branch lines. S2: Volume measurement and queue creation. After the freight object passes through the conveyor line's linear laser stereo camera for volume measurement, a conveyor line queue is created. The conveyor line queue is divided into the main line freight object queue, the main line event queue, the branch line freight object queue, and the branch line event queue. After the freight object completes the barcode dynamic scanning and volume measurement, it is assigned a unique ID and inserted into the end of the main line freight object queue. The freight object's queue information is then created. S3: Obtain multidimensional features of the freight object and add the multidimensional features of the freight object to the queue information of the freight object; S4: Several mainline detection points are configured on the main line of the conveyor line, and several branch line detection points are configured on the branch lines. When a mainline detection point detects that a freight object first appears in the detection range, it generates a first entry event and inserts it into the end of the mainline event queue. Several mainline detection points track the movement trajectory of the freight object on the main line and use a three-dimensional matching algorithm to match the freight object passing through the mainline detection point with the freight object in the mainline freight object queue. The three-dimensional matching algorithm is used to match the freight objects passing through the main line detection point with the freight objects in the main line freight object queue. The three dimensions of volume parameter matching, arrival time matching and image matching are used to match the queue information of the freight objects on the main line with the queue information of each freight object in the main line freight object queue. S5: Process the freight object based on the matching results of the main line detection points; The freight object is continuously tracked until the freight object leaves the detection range of several main line detection points. When the freight object leaves the detection range of the current main line detection point, the time information of the freight object in the main line freight object queue is updated once; S6: When a freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object. The identified freight object is then removed from the main line freight object queue and inserted into the end of the branch line freight object queue of the corresponding branch line. S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the end of the branch line event queue of the corresponding branch line; S8: Process the freight object based on the matching results of the branch line detection point; Continuously track freight objects. When a freight object passes through each branch line inspection point on the branch line, a two-dimensional matching algorithm is used to verify whether the freight object at the current inspection point matches the freight object in the branch line freight object queue. S9: Process the freight object based on the matching results of the branch line detection point; The freight pair is continuously tracked until it leaves the detection range of each branch line detection point or is grabbed by the robotic arm, and the freight object is removed from the branch line freight object queue, ending the tracking process of the freight object.

2. The method for tracking freight objects based on machine vision according to claim 1, characterized in that: The acquisition of multi-dimensional features of the freight object in step S3 specifically includes target detection, identification of feature values ​​of the freight object, and updating of queue information of the freight object; wherein: Object detection uses a previously trained deep learning object detection model to obtain the external contours of the freight object and perform image segmentation to obtain a complete image of the freight object. To identify the feature values ​​of a freight object, image recognition technology is used to segment the areas where all labels are located on the freight object. Color space conversion is used to identify the color of each label and the color of the freight object. By identifying whether there are bright reflective areas in the non-label areas of the freight object image, the presence of stretch film on the freight object surface can be determined. Update the queue information of the freight object, and update the obtained photos and colors of each end face of the freight object, the color and quantity of the labels, and the feature values ​​corresponding to whether there is a stretch film on the surface of the freight object to the queue information of the freight object; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object recognized by the line laser stereo camera, the information obtained by dynamic scanning of the barcode, and the number of the destination branch line.

3. The method for tracking freight objects based on machine vision according to claim 2, characterized in that: The several mainline detection points described in step S4 track the movement trajectory of the freight object on the mainline, and each mainline detection point and branch line detection point is equipped with a photoelectric radiation tube, a camera and an ultrasonic component. When the freight object passes through each mainline detection point and blocks the light path of the photoelectric radiation tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera of each mainline detection point has a unique ID.

4. The method for tracking freight objects based on machine vision according to claim 3, characterized in that: The three-dimensional matching algorithm described in step S4 includes: Volume parameter matching: The camera's horizontal field of view angle is α, the vertical field of view angle is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasonic wave is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, and the pixels occupied by the freight object in the vertical direction are pixel_y. Then the true height of the freight object box_z is box_z = h1 - h2; the horizontal length dimension of the freight object box_x is box_x = 2×pixel_x×h2×tan(α / 2) / X; the horizontal width dimension of the freight object box_y is box_y = 2×pixel_y×h2×tan(β / 2) / Y; the obtained horizontal length box_x, horizontal width box_y, and true height box_z of the freight object are used to match the length, width, and height of the freight objects recognized by the line laser stereo camera in the queue information of the main line freight object queue, and obtain the set A of freight objects on the main line that meets the error range; Arrival time matching: Based on the time t0 when the freight object is barcode-recognized, combined with the speed v of the conveyor line, let the distance between the barcode scanner and the current mainline detection point be s, and estimate the time when the freight object arrives at the current mainline detection point as t = t0 + s / v. The set C of freight objects on the mainline that meet the error range of the arrival time at the current mainline detection point is obtained; Image matching: Using an improved Hungarian matching algorithm, we first combine the aforementioned volume parameter matching and arrival time matching to eliminate non-compliant freight objects from Sets A and C. We then filter the freight objects in the set based on the wrapping film attributes within the freight objects' multidimensional features. Finally, we compare the color of each end face of the freight object, the number of labels on the freight object, and the label color. We eliminate freight objects on the main line with inconsistent end face color, inconsistent label quantity, or inconsistent label color, and match only the qualified freight objects.

5. The method for tracking freight objects based on machine vision according to claim 4, characterized in that: Use the obtained horizontal length box_x, horizontal width box_y, and true height box_z of the freight object to match the length, width, and height of the freight object identified by the line laser stereo camera in the queue information of the mainline freight object queue. Let the set of freight objects on the mainline be M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera. Let the length, width, and height of the freight object recognized by the line laser stereo camera be x, y, and h, respectively, satisfying 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, and 0.85×mh≤h≤1.15×mh.

6. The method for tracking freight objects based on machine vision according to claim 5, characterized in that: The error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated arrival time of the freight object at the current mainline detection point, or ±50 seconds of the estimated arrival time of the freight object at the current mainline detection point.

7. The method for tracking freight objects based on machine vision according to claim 4, characterized in that: The matching result of the main line detection point in step S5 is used to process the freight object, including the following: If the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completion of the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point between the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached; If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued; If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

8. The method for tracking freight objects based on machine vision according to claim 7, characterized in that: The tracking of freight objects on the branch line described in step S7 is performed through several branch line detection points on the branch line. When the freight object on the branch line passes through each branch line detection point and blocks the light path of the photoelectric transmitting tube, the camera and ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each branch line detection point has a unique ID.

9. The method for tracking freight objects based on machine vision according to claim 8, characterized in that: As described in step S8, when the freight object passes through each branch line detection point on the branch line, a two-dimensional matching algorithm is used to verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue. The volume parameter matching and image matching in step S4 are used again at each branch line detection point to match the contents of the queue information of the freight object on the branch line with those of the freight objects in the branch line freight object queue.

10. The method for tracking freight objects based on machine vision according to claim 9, characterized in that: The matching result of the branch line detection point in step S9 is used to process the freight object, including the following: If the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is unique, it means the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completion of the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the two-dimensional matching result of the branch line detection point for the freight object on the branch line and the freight object in the branch line freight object queue is not unique, it means the matching has failed, and the matching needs to be repeated until a unique matching result is found or the maximum matching time is reached; If the maximum matching time has passed and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued; If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a new freight object record marked as abnormal is inserted into the branch line freight object queue and a notification is issued.

Images