Freight object tracking method based on machine vision

Abstract

The invention provides a freight object tracking method based on machine vision, and belongs to the technical field of logistics conveying object visual tracking, and the method comprises the steps: carrying out the barcode recognition, volume measurement and queue creation of a freight object, carrying out the volume measurement of the freight object through a line laser stereo camera on a conveying line, and creating a queue of the conveying line; acquiring multi-dimensional features of the freight objects and supplementing the multi-dimensional features into queue information of the freight objects; a plurality of main line detection points are configured on a main line of the conveying line, and a plurality of branch line detection points are configured on a plurality of branch lines respectively; the plurality of main line detection points track the motion trail of the freight object on the main line; when the freight object is about to be transferred from the main line to the branch line, the identified freight object is removed from the main line freight object queue and inserted into the tail part of the branch line freight object queue of the corresponding branch line; and the branch line detection point further tracks the freight object. And through video measurement and tracking of a plurality of links, the reliability and the accuracy of whole-process conveying of the material objects are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual tracking of logistics transport objects, and in particular to a freight object tracking method based on machine vision. Background Art

[0002] With the continuous increase in the types of shipping cargo, the object management method of traditional cargo stations is increasingly unable to meet the needs of efficient operation. With the rapid development of the Internet of Things, cloud computing, big data and artificial intelligence in recent years, combined with the introduction of automation technology, it can effectively improve the operating efficiency of cargo stations, reduce manpower waste, enhance the accuracy and smoothness of logistics information, and make the informatization, intelligence and visualization of the entire logistics process possible. In the fully automated sorting and palletizing system of shipping cargo stations, due to the different volumes, weights and shapes of freight objects, freight objects may be misordered or slip on the conveyor line; and when the balance wheel is operated at the gate, errors such as multiple rows of objects and wrong objects are difficult to avoid. At present, the sorting and palletizing conveyor line of the terminal mostly uses the photoelectric switch beat to determine the position of the freight object and control the balance wheel, but this will cause occasional inconsistency between the palletized freight objects and the system records, causing system errors, and ultimately requiring manual inspection of the line, greatly reducing the efficiency of the sorting and palletizing system. In addition, the similarity of the freight objects of shipping materials brings difficulties to machine vision recognition. The existing methods usually use a single matching algorithm to identify freight objects, which often has the problems of low efficiency and insufficient accuracy.

[0003] In view of this, it is very necessary to provide a freight object tracking method based on machine vision in order to improve the fault tolerance of the shipping warehouse sorting and palletizing system, enhance the reliability of the system, and accelerate the overall efficiency of the warehouse cargo flow. Summary of the invention

[0004] In view of this, the present invention proposes a freight object tracking method based on machine vision, which covers the features of each surface of the freight object, realizes size and volume measurement, contour and label recognition, and confirms the uniqueness of the freight object and the reliability of the conveying and palletizing process through continuous tracking of freight symmetry on the main line and branch line.

[0005] The technical solution of the present invention is implemented as follows: The present invention provides a method for tracking freight objects based on machine vision, comprising the following steps:

[0006] S1: Barcode recognition: a barcode scanning device is installed at the starting position of the conveyor line to dynamically scan the barcodes on each surface of the freight object; the conveyor line includes a main line and several branch lines, and the freight object is transferred to different branch lines through the main line of the conveyor line;

[0007] S2: Volume measurement and queue creation. After the freight object passes through the line laser stereo camera on the conveyor line for volume measurement, the conveyor line queue is created. The conveyor line queue is divided into the main line freight object queue, the main line event queue, the branch line freight object queue and the branch line event queue. After the barcode dynamic scanning and volume measurement, the freight object is assigned a unique ID and inserted into the end of the main line freight object queue, and the queue information of the freight object is newly created;

[0008] S3: obtaining multidimensional features of the freight object, and adding the multidimensional features of the freight object to the queue information of the freight object;

[0009] S4: several main line detection points are configured on the main line of the conveyor line, and several branch line detection points are respectively configured on several branch lines; when the main line detection point detects that the freight object appears in the detection range for the first time, a first entry event is generated and inserted into the tail of the main line event queue; several main line detection points track the movement trajectory of the freight object on the main line, and use a three-dimensional matching algorithm to match the freight object passing through the main line detection point with the freight object in the main line freight object queue;

[0010] S5: Process the freight object according to the matching result of the main line detection point; continue to track the freight object until the freight object leaves the detection range of several main line detection points, and when the freight object leaves the detection range of the current main line detection point, update the time information of the freight object in the main line freight object queue;

[0011] S6: When the freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object, and the identified freight object is removed from the main line freight object queue and inserted into the tail of the branch line freight object queue of the corresponding branch line;

[0012] S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the tail of the branch line event queue of the corresponding branch line;

[0013] S8: Process the freight object according to the matching result of the branch line detection point; continue to track the freight object, and when the freight object passes through each branch line detection point on the branch line, verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue through a two-dimensional matching algorithm;

[0014] S9: Process the freight object according to the matching result of the branch line detection point; continue to track the freight pair until it leaves the detection range of each branch line detection point or is captured by the robot arm, remove the freight object from the branch line freight object queue, and end the tracking process of the freight object.

[0015] On the basis of the above technical solution, preferably, the acquisition of multi-dimensional features of the freight object in step S3 specifically includes target detection, identification of feature values ​​of the freight object and updating of queue information of the freight object; wherein:

[0016] Object detection uses the object detection model that has been trained by deep learning to obtain the external contour of the freight object, perform image segmentation, and obtain the complete image of the freight object;

[0017] To identify the characteristic value of the freight object, image recognition technology is used to segment the area where all labels on the freight object are located, and color space conversion is used to identify the color of each label and the color of the freight object. By identifying whether there is a bright reflective area in the non-label area of ​​the freight object image, it is identified whether there is a wrapping film on the surface of the freight object;

[0018] Update the queue information of the freight object, and update the obtained photos and colors of each end surface of the freight object, the color and quantity of the labels, and the feature value corresponding to whether there is a stretch film on the surface of the freight object to the queue information of the freight object; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object recognized by the line laser stereo camera, the information obtained by dynamic scanning of the barcode, and the number of the destination branch line.

[0019] Preferably, the several main line detection points described in step S4 track the movement trajectory of the freight object on the main line, and each main line detection point and branch line detection point is equipped with a photoelectric beam tube, a camera and an ultrasonic component. When the freight object passes through each main line detection point and blocks the light path of the photoelectric beam tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera of each main line detection point has a unique ID.

[0020] Further preferably, the three-dimensional matching algorithm used in step S4 to match the freight objects passing through the main line detection point with the freight objects in the main line freight object queue is to match the contents of the queue information of the freight objects on the main line with those of the freight objects in the main line freight object queue from three dimensions: volume parameter matching, arrival time matching and image matching;

[0021] Volume parameter matching: Assuming that the horizontal field of view of the camera is α, the vertical field of view is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasonic wave is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, and the pixels occupied by the freight object in the vertical direction are pixel_y, then the real height box_z of the freight object is box_z=h1-h2; the horizontal length dimension box_x of the freight object is box_x=2×pixel_x×h2×tan(α / 2) / X; the horizontal width dimension box_y of the freight object is box_y=2×pixel_y×h2×tan(β / 2) / Y; the obtained horizontal length box_x, horizontal width box_y and real height box_z of the freight object are used to match the length, width and height of the freight object recognized by the line laser stereo camera in the queue information of the main line freight object queue, and obtain the set A of freight objects on the main line that meets the error range;

[0022] Arrival time matching: Based on the time t0 when the freight object is barcode recognized, combined with the speed v of the conveyor line, let the distance between the barcode scanning device and the current main line detection point be s, and estimate the time when the freight object arrives at the current main line detection point as t=t0+s / v, and obtain the set C of freight objects on the main line that meet the error range of the arrival time of the current main line detection point;

[0023] Image matching: Using the improved Hungarian matching algorithm, first combine the aforementioned volume parameter matching and arrival time matching to eliminate the freight objects that do not meet the requirements in set A and set C; then screen the freight objects in the set according to the wrapping film attributes in the multidimensional features of the freight objects; finally, compare the color of each end face of the freight objects, the number of all labels on the freight objects, and the label color, and eliminate the freight objects on the main line with inconsistent end face colors, inconsistent label quantities, or inconsistent label colors, and match the freight objects that meet the requirements.

[0024] Further preferably, the obtained horizontal length box_x, horizontal width box_y and actual height box_z of the freight object are used to match the length, width and height of the freight objects recognized by the line laser stereo camera in the queue information of the main line freight object queue, that is, the set of freight objects on the main line is set to M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera; the length, width and height of the freight object recognized by the line laser stereo camera are x, y, h respectively, satisfying 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, 0.85×mh≤h≤1.15×mh.

[0025] Further preferably, the error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated time when the freight object arrives at the current mainline detection point, or ±50 seconds of the estimated time when the freight object arrives at the current mainline detection point.

[0026] Further preferably, the matching result of the main line detection point in step S5 is used to process the freight object, including the following contents:

[0027] If the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means that the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completing the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means that the matching fails and needs to be re-matched until a unique result is matched or the maximum matching time is reached;

[0028] If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued;

[0029] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

[0030] Further preferably, the tracking of the freight object on the branch line in step S7 is performed through a number of branch line detection points on the branch line. When the freight object on the branch line passes through each branch line detection point and blocks the light path of the photoelectric radiating tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each branch line detection point has a unique ID.

[0031] Still further preferably, in step S8, when the freight object passes through each branch inspection point on the branch line, a two-dimensional matching algorithm is used to verify whether the freight object at the current inspection point matches the freight object in the branch freight object queue, and the volume parameter matching and image matching in step S4 are used again at each branch inspection point to match the contents of the queue information of the freight objects on the branch line with those of the freight objects in the branch freight object queue in two dimensions.

[0032] More preferably, the matching result of the branch line detection point in step S9 is used to process the freight object, including the following contents:

[0033] If the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is unique, it means that the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completing the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is not unique, it means that the matching fails, and the matching needs to be repeated until a unique result is matched or the maximum matching time is reached;

[0034] If the maximum matching time is exceeded and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued;

[0035] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a record of a new freight object marked as abnormal is inserted into the branch line freight object queue, and a notification is issued.

[0036] The present invention provides a method for tracking freight objects based on machine vision, which has the following beneficial effects compared with the prior art:

[0037] (1) The present invention uses a barcode scanning device covering all sides of the freight object to identify the barcode content of the freight object and the corresponding conveying route content; through volume measurement and queue creation, a main line freight object queue, a main line event queue, a branch line freight object queue and a branch line event queue covering each freight object on the main line and branch line are generated to record the status update information and arrival time of the freight object when entering different positions of the main line or branch line, thereby constructing a complete and traceable conveying trajectory of the freight object; after configuring a line laser stereo camera for volume measurement, a conveying line queue is created, thereby facilitating subsequent entry and exit and writing of matching results of the passing detection points, the corresponding camera ID and the time of completion of the matching, thereby realizing continuous tracking operations;

[0038] (2) For each inspection point on the main line, a three-dimensional matching algorithm is used to match the freight objects passing through the main line inspection point with the freight objects in the main line freight object queue. The freight objects within the error range are screened from the three dimensions of volume parameter matching, arrival time matching and image matching, and the closest one or more freight objects are selected to achieve continuous tracking of the real-time position and arrival time of the freight objects on the main line, and realize the position traceability function of the main line part;

[0039] (3) After the freight object arrives at the target branch line, each detection point on the branch line further uses a two-dimensional matching algorithm to match the freight objects passing through the branch line detection point with the freight objects in the branch line freight object queue. The freight objects within the error range are screened from the two dimensions of volume parameter matching and image matching, respectively, to achieve continuous tracking of the real-time position of the freight objects on the branch line until the freight objects leave the branch line and complete the transmission process of the logistics conveyor line, thereby achieving full traceability of the freight objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 The present invention is a flowchart of the steps of a freight object tracking method based on machine vision. DETAILED DESCRIPTION

[0042] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] At present, the terminal sorting and palletizing conveyor lines mostly use the photoelectric switch beat to determine the location of the freight object and control the balance wheel. However, since the freight object cannot be tracked throughout the process, it will cause occasional inconsistencies between the palletized freight object and the system record, causing system errors, and ultimately requiring the line to be stopped for manual inspection, greatly reducing the efficiency of the sorting and palletizing system. In view of this, if Figure 1 As shown, the present invention provides a method for tracking freight objects based on machine vision, comprising the following steps:

[0044] S1: Barcode recognition, install a barcode scanner at the starting position of the conveyor line to dynamically scan the barcodes on each surface of the freight object; the conveyor line includes a main line and several branch lines, and the freight objects are transferred to different branch lines through the main line of the conveyor line. Freight objects that usually need to be sorted and palletized need to be transported from the head end of the main line to the end of a destination branch line, thus completing the logistics conveyor line transportation process.

[0045] The barcode may be attached to any surface of the freight object, and the number may be more than one. Therefore, the scanning device installed at the starting position of the conveyor line of the present invention is a six-sided barcode scanning device, and barcode reading cameras are installed on the left, right, front, back, and top of the conveyor line, and a line scanning barcode reading camera is installed at the bottom of the conveyor line; the line scanning barcode reading camera scans the bottom of the freight object through the gap in the middle of the conveyor line, and the conveyor line drives the freight object to move at a uniform speed to achieve imaging of the bottom surface of the freight object. In this way, the six sides of the freight object, such as top, bottom, left, right, front and back, can be dynamically scanned, the barcode number can be identified, and compared with all the barcode information obtained in the WCS system, so as to obtain the real barcode information content of the freight object, and query the number of the destination branch line corresponding to the barcode.

[0046] In this embodiment, the freight object can be a box-shaped structure such as a corrugated box, a wooden crate, a small cargo box, or a parcel of a regular shape, all of which can be transported by a conveyor line. Figure 1 The term box used in the process is for illustration only and is not considered as a limitation on the freight objects in the present invention.

[0047] S2: Volume measurement and queue creation. After the freight object passes through the linear laser stereo camera on the conveyor line for volume measurement, the conveyor line queue is created. The conveyor line queue is divided into: main line freight object queue, main line event queue, branch line freight object queue and branch line event queue. After completing the barcode dynamic scanning and volume measurement, the freight object is assigned a unique ID and inserted into the end of the main line freight object queue, and the queue information of the freight object is created.

[0048] After the initial barcode recognition, the freight object is sent to the line laser stereo camera on the conveyor line to collect the specifications and image information of the freight object. The line laser stereo camera is based on the principle of triangulation. Through the image sensor, it captures the laser line information projected by the laser generator on the surface of the object, reconstructs the surface contour information of the object, and thus calculates the length, width, height and volume information of the measured freight object. This part of the content belongs to the conventional technical means in this field, and does not invent or involve improvements to the measurement content of the line laser stereo camera.

[0049] S3: Acquire the multidimensional features of the freight object, and add the multidimensional features of the freight object to the queue information of the freight object.

[0050] Obtaining multi-dimensional features of freight objects, including target detection, identifying feature values ​​of freight objects, and updating queue information of freight objects; wherein:

[0051] S31: Object detection uses the object detection model that has been trained by deep learning in advance to obtain the external contour of the freight object, perform image segmentation, and obtain the complete image of the freight object;

[0052] Specifically, a high-performance YOLO target detection model is selected, and the model is trained using a labeled box image dataset to ensure that the model can accurately detect boxes of various shapes and sizes. After the training is completed, the performance of the model on the validation set is evaluated to ensure that it has high detection accuracy and recall. The YOLO target detection model is a common technical means in this field, and its code is open source and easy to obtain. Then, the trained target detection model is deployed to the video stream processing equipment on the conveyor line, and each frame of the image is detected in real time, and the bounding box of each freight object and its confidence are output.

[0053] Select a fine-grained UNet image segmentation model in advance and train the model using a box image dataset with pixel-level annotations to ensure that the model can accurately segment the contours of the freight objects. After training, evaluate the performance of the model on the validation set to ensure that it has high segmentation accuracy. Then, use the trained UNet image segmentation model to detect the image containing the bounding box of the freight object, segment the area within the bounding box, and output the segmented binary image, where the foreground part represents the freight object and the background part represents other areas of the non-freight object.

[0054] S32: Identify the characteristic values ​​of the freight object by using image recognition technology to segment the area where all labels on the freight object are located, use color space conversion to identify the color of each label and the color of the freight object, and identify whether there is a bright reflective area in the non-label area of ​​the image of the freight object, and identify whether there is a wrapping film on the surface of the freight object.

[0055] After segmenting the image containing the freight object, the boxes with labels are marked, and the trained target detection algorithm is used to identify and segment all the labels on the photos of the surfaces of the freight object, segment the rectangular image of the label, and then use color space conversion to identify the color characteristics of each label; several rectangles are segmented from the image of the surface area of ​​the freight object other than the label, and the color of the freight object itself is also identified by color space conversion, that is, the RGB color is converted to the HSV space to obtain the brightness value H, the maximum and minimum values ​​of the RGB color value. If the maximum and minimum values ​​of the RGB color value are equal, the freight object is monochrome, and the color of the freight object is determined according to the RGB color value. If the maximum and minimum values ​​of the RGB color value are not equal, the saturation and color channel values ​​are calculated to determine the color combination of the freight object.

[0056] Boxes with wrapping film have obvious reflective properties. The labeled deep learning model is used to check whether there is wrapping film, that is, there is a continuous grayscale value of 255 in the image of at least one surface of the non-labeled freight object surface area. Usually the reflective area has at least one highlighted linear boundary. For each segmented box image, a feature extraction model is used to extract a 1024-dimensional feature vector, and the extracted feature vector is stored in a data structure, such as a dictionary or list, to facilitate subsequent matching operations.

[0057] S33: Update the queue information of the freight object, and update the obtained photos and colors of each end surface of the freight object, the color and number of labels, and the feature values ​​corresponding to whether there is a wrapping film on the surface of the freight object to the queue information of the freight object corresponding to the main line freight object queue; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object recognized by the line laser stereo camera, the information obtained by dynamic scanning of the barcode and the number of the destination branch line.

[0058] S4: several main line detection points are configured on the main line of the conveyor line, and several branch line detection points are respectively configured on several branch lines; when the main line detection point detects that the freight object appears in the detection range for the first time, a first entry event is generated and inserted into the tail of the main line event queue; several main line detection points track the movement trajectory of the freight object on the main line, and use a three-dimensional matching algorithm to match the freight object passing through the main line detection point with the freight object in the main line freight object queue;

[0059] Among them, several main line detection points track the movement trajectory of freight objects on the main line, and each main line detection point and branch line detection point is equipped with a photoelectric beam tube, a camera and an ultrasonic component. When the freight object passes through each main line detection point and blocks the light path of the photoelectric beam tube, the camera and ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each main line detection point has a unique ID.

[0060] Among them, a three-dimensional matching algorithm is used to match the freight objects passing through the main line detection point with the freight objects in the main line freight object queue. The freight objects on the main line are matched with the queue information of each freight object in the main line freight object queue in three dimensions, namely, volume parameter matching, arrival time matching and image matching.

[0061] 1) Volume parameter matching: Assume that the horizontal field of view angle of the camera is α, the vertical field of view angle is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasonic wave is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, the pixels occupied by the freight object in the vertical direction are pixel_l, y, then the real height of the freight object b ox_z is box_z=h1-h2; the horizontal length dimension of the freight object b o_x is xbox_x=2×pixel_x×h2×tan(α / 2) / X; the horizontal width dimension of the freight object box_y is box_y=2×pixel_y×h2×tan(β / 2) / Y; the obtained horizontal length box_x, horizontal width box_y and real height box_z of the freight object are used to match the length, width and height of the freight object recognized by the line laser stereo camera in the queue information of the main line freight object queue, and obtain the set A of freight objects on the main line that meets the error range;

[0062] The freight objects on the main line that meet the error range are as follows: let the set of freight objects on the main line be M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera; let the length, width and height of the freight object recognized by the line laser stereo camera be x, y, h respectively, then the freight objects in set A simultaneously satisfy 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, and 0.85×mh≤h≤1.15×mh.

[0063] 2) Arrival time matching: Based on the time t0 when the freight object is barcode identified, combined with the speed v of the conveyor line, let the distance between the barcode scanning device and the current main line detection point be s, and estimate the time when the freight object arrives at the current main line detection point as t=t0+s / v, and obtain the set C of freight objects on the main line that meet the error range of the arrival time of the current main line detection point;

[0064] The error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated time when the freight object arrives at the current mainline detection point, or ±50 seconds of the estimated time when the freight object arrives at the current mainline detection point.

[0065] 3) Image matching: Using the improved Hungarian matching algorithm, first combine the aforementioned volume parameter matching and arrival time matching to eliminate the freight objects that do not meet the requirements in set A and set C; then screen the freight objects in the set according to the wrapping film attributes in the multidimensional features of the freight objects; finally, compare the color of each end face of the freight objects, the number of all labels on the freight objects, and the label color, eliminate the freight objects with inconsistent end face colors on the main line, inconsistent label quantities, or inconsistent label colors, and match the freight objects that meet the requirements.

[0066] The image matching algorithm uses the improved Hungarian matching algorithm. The traditional Hungarian matching algorithm is very complex. The conveyor line of the shipping cargo terminal is very long, and the number of cargo objects is large. There are many detection points on the logistics conveyor line. The time interval between two cargo objects on the conveyor line is about 1 second. Therefore, it is necessary to complete the image matching algorithm of multiple detection points in a very short time, which consumes huge computing resources and is very costly.

[0067] Therefore, the present invention makes the following improvements to the Hungarian matching algorithm:

[0068] 3.1) Combine the volume parameter matching and arrival time matching described above to filter out freight objects that are obviously mismatched;

[0069] 3.2) Screen out the freight objects that do not match the stretch film identification;

[0070] 3.3) Use the object detection model to filter out freight objects with inconsistent colors and exclude freight objects with inconsistent label quantity, color, and location;

[0071] In summary, the cost matrix size of the Hungarian matching algorithm is reduced, and the time consumption of the algorithm is reduced; again, combined with the arrival time parameter, the arrival time priority sorting is introduced, and the screened freight object set is sorted according to the calculated arrival time parameter; finally, combined with the Hungarian matching algorithm, the freight objects that meet the requirements can be matched more quickly.

[0072] S5: Process the freight object according to the matching result of the main line detection point; continue to track the freight object until the freight object leaves the detection range of several main line detection points, and when the freight object leaves the detection range of the current main line detection point, update the time information of the freight object in the main line freight object queue;

[0073] Among them, according to the matching results of the main line detection points, the freight objects are processed, including the following:

[0074] If the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means that the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completing the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means that the matching fails and needs to be re-matched until a unique result is matched or the maximum matching time is reached;

[0075] If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued;

[0076] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

[0077] The insertion of a new freight object record marked as abnormal mentioned here is different from the content of updating the queue information of the freight object mentioned above. It mainly records the volume parameter calculation results of the unmatched freight object, the arrival time record value, and the conclusion that the corresponding freight object was not found in the main line freight object queue. The notification is issued to remind the staff to manually confirm the missing freight object or unknown freight object. According to the manual confirmation results, it is determined to be a mechanical failure or an equipment abnormality at the detection point.

[0078] S6: When the freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object, and the identified freight object is removed from the main line freight object queue and inserted into the tail of the branch line freight object queue of the corresponding branch line.

[0079] S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the tail of the branch line event queue of the corresponding branch line;

[0080] Among them, the freight objects are tracked on the branch line through several branch line detection points on the branch line. When the freight objects on the branch line pass through each branch line detection point and block the light path of the photoelectric radiating tube, the camera and ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain the image of the top of the freight object. The camera at each branch line detection point also has a unique ID.

[0081] S8: Process the freight object according to the matching result of the branch line detection point; continue to track the freight object, and when the freight object passes through each branch line detection point on the branch line, verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue through a two-dimensional matching algorithm;

[0082] When the freight object passes through each branch line detection point on the branch line, the two-dimensional matching algorithm is used to check whether the freight object at the current detection point matches the freight object in the branch line freight object queue. The volume parameter matching and image matching in step S4 are used again at each branch line detection point to match the queue information of each freight object on the branch line with that in the branch line freight object queue. Since the conveying speed of the branch line is likely to be different from that of the main line, simply using the arrival time matching may not be appropriate, so the time dimension matching is not used, so a two-dimensional matching algorithm is formed here.

[0083] S9: Process the freight object according to the matching result of the branch line detection point; continue to track the freight pair until it leaves the detection range of each branch line detection point or is captured by the robot arm, remove the freight object from the branch line freight object queue, and end the tracking process of the freight object.

[0084] Among them, according to the matching results of the branch line detection point, the freight object is processed, including the following contents:

[0085] If the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is unique, it means that the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completing the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is not unique, it means that the matching fails, and the matching needs to be repeated until a unique result is matched or the maximum matching time is reached;

[0086] If the maximum matching time is exceeded and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued;

[0087] If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a record of a new freight object marked as abnormal is inserted into the branch line freight object queue, and a notification is issued.

[0088] Similar to step S5, a record of a new freight object marked as abnormal is inserted, which is different from the content of the queue information of the aforementioned updated freight object. The notification is also issued to remind the staff to manually confirm the lost freight object or the unknown freight object.

[0089] For unknown freight objects, the processing method can be: 1) stop the machine to remove them manually, and then manually resume the action of the conveyor line; 2) when the unknown freight object reaches the end of the main line or branch line, the robot removes the unknown freight object and stores it separately, and after manual confirmation, re-executes the process of the tracking method of the present invention; 3) if multiple unknown freight objects appear continuously, such as an odd number of more than 5, it is considered that the equipment at one or more detection points is abnormal, and the operation of the main line or branch line needs to be stopped and inspected.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for tracking freight objects based on machine vision, characterized in that: The steps include: S1: Barcode recognition: a barcode scanning device is installed at the starting position of the conveyor line to dynamically scan the barcodes on each surface of the freight object; the conveyor line includes a main line and several branch lines, and the freight object is transferred to different branch lines through the main line of the conveyor line; S2: Volume measurement and queue creation. After the freight object passes through the line laser stereo camera on the conveyor line for volume measurement, the conveyor line queue is created. The conveyor line queue is divided into the main line freight object queue, the main line event queue, the branch line freight object queue and the branch line event queue. After the barcode dynamic scanning and volume measurement, the freight object is assigned a unique ID and inserted into the end of the main line freight object queue, and the queue information of the freight object is newly created; S3: obtaining multidimensional features of the freight object, and adding the multidimensional features of the freight object to the queue information of the freight object; S4: several main line detection points are configured on the main line of the conveyor line, and several branch line detection points are respectively configured on several branch lines; when the main line detection point detects that the freight object appears in the detection range for the first time, a first entry event is generated and inserted into the tail of the main line event queue; several main line detection points track the movement trajectory of the freight object on the main line, and use a three-dimensional matching algorithm to match the freight object passing through the main line detection point with the freight object in the main line freight object queue; S5: Process the freight object according to the matching results of the main line detection points; The freight object is continuously tracked until the freight object leaves the detection range of several main line detection points, and when the freight object leaves the detection range of the current main line detection point, the time information of the freight object in the main line freight object queue is updated once; S6: When the freight object is about to transfer from the main line to the branch line, the main line detection point adjacent to the branch line performs a three-dimensional matching algorithm on the freight object, and the identified freight object is removed from the main line freight object queue and inserted into the tail of the branch line freight object queue of the corresponding branch line; S7: Tracking the freight object on the branch line; after the branch line detection point detects that the freight object enters the detection range, a second entry event is generated, and the current second entry event is inserted into the tail of the branch line event queue of the corresponding branch line; S8: Processing the freight object according to the matching result of the branch line detection point; Continuously track the freight object. When the freight object passes through each branch line detection point on the branch line, use the two-dimensional matching algorithm to verify whether the freight object at the current detection point matches the freight object in the branch line freight object queue; S9: Processing the freight object according to the matching result of the branch line detection point; The freight pair is continuously tracked until it leaves the detection range of each branch line detection point or is captured by a robotic arm, and the freight object is removed from the branch line freight object queue, thus ending the tracking process of the freight object.

2. The method for tracking freight objects based on machine vision according to claim 1, characterized in that: The acquisition of multi-dimensional features of the freight object in step S3 specifically includes target detection, identifying feature values ​​of the freight object, and updating queue information of the freight object; wherein: Object detection uses the object detection model that has been trained by deep learning to obtain the external contour of the freight object, perform image segmentation, and obtain the complete image of the freight object; To identify the characteristic value of the freight object, image recognition technology is used to segment the area where all labels on the freight object are located, and color space conversion is used to identify the color of each label and the color of the freight object. By identifying whether there is a bright reflective area in the non-label area of ​​the freight object image, it is identified whether there is a wrapping film on the surface of the freight object; Update the queue information of the freight object, and update the obtained photos and colors of each end surface of the freight object, the color and quantity of the labels, and the feature value corresponding to whether there is a stretch film on the surface of the freight object to the queue information of the freight object; the queue information of the freight object also includes the time when the freight object entered the queue, the top photo, the length, width, height and volume of the freight object recognized by the line laser stereo camera, the information obtained by dynamic scanning of the barcode, and the number of the destination branch line.

3. The method for tracking freight objects based on machine vision according to claim 2, characterized in that: The several main line detection points described in step S4 track the movement trajectory of the freight object on the main line, and each main line detection point and branch line detection point is equipped with a photoelectric beam tube, a camera and an ultrasonic component. When the freight object passes through each main line detection point and blocks the light path of the photoelectric beam tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera of each main line detection point has a unique ID.

4. The method for tracking freight objects based on machine vision according to claim 3, characterized in that: The three-dimensional matching algorithm described in step S4 is used to match the freight objects passing through the main line detection point with the freight objects in the main line freight object queue, which is to match the freight objects on the main line with the queue information of each freight object in the main line freight object queue from the three dimensions of volume parameter matching, arrival time matching and image matching; Volume parameter matching: Assuming that the horizontal field of view of the camera is α, the vertical field of view is β, the number of pixels in the horizontal direction is X, the number of pixels in the vertical direction is Y, the height between the camera and the conveyor line is h1, the height measured by ultrasonic wave is h2, the pixels occupied by the freight object in the horizontal direction are pixel_x, and the pixels occupied by the freight object in the vertical direction are pixel_y, then the real height box_z of the freight object is box_z=h1-h2; the horizontal length dimension box_x of the freight object is box_x=2×pixel_x×h2×tan(α / 2) / X; the horizontal width dimension box_y of the freight object is box_y=2×pixel_y×h2×tan(β / 2) / Y; the obtained horizontal length box_x, horizontal width box_y and real height box_z of the freight object are used to match the length, width and height of the freight object recognized by the line laser stereo camera in the queue information of the main line freight object queue, and obtain the set A of freight objects on the main line that meets the error range; Arrival time matching: Based on the time t0 when the freight object is barcode recognized, combined with the speed v of the conveyor line, let the distance between the barcode scanning device and the current main line detection point be s, and estimate the time when the freight object arrives at the current main line detection point as t=t0+s / v, and obtain the set C of freight objects on the main line that meet the error range of the arrival time of the current main line detection point; Image matching: Using the improved Hungarian matching algorithm, first combine the aforementioned volume parameter matching and arrival time matching to eliminate the freight objects that do not meet the requirements in set A and set C; then screen the freight objects in the set according to the wrapping film attributes in the multidimensional features of the freight objects; finally, compare the color of each end face of the freight objects, the number of all labels on the freight objects, and the label color, and eliminate the freight objects on the main line with inconsistent end face colors, inconsistent label quantities, or inconsistent label colors, and match the freight objects that meet the requirements.

5. The method for tracking freight objects based on machine vision according to claim 4, characterized in that: The obtained horizontal length box_x, horizontal width box_y and real height box_z of the freight object are used to match the length, width and height of the freight object recognized by the line laser stereo camera in the queue information of the main line freight object queue, that is, the set of freight objects on the main line is set to M, m∈M, m={mx,my,mh}; mx is the horizontal length of the freight object recognized by the camera, my is the horizontal width of the freight object recognized by the camera, and mh is the height of the freight object recognized by the camera; let the length, width, and height of the freight object recognized by the line laser stereo camera be x, y, and h, respectively, satisfying 0.7×mx≤x≤1.3×mx, 0.7×my≤y≤1.3×my, and 0.85×mh≤h≤1.15×mh.

6. The method for tracking freight objects based on machine vision according to claim 5, characterized in that: The error range of the arrival time of the current mainline detection point is the larger of ±20% of the estimated time when the freight object arrives at the current mainline detection point, or ±50 seconds of the estimated time when the freight object arrives at the current mainline detection point.

7. The method for tracking freight objects based on machine vision according to claim 4, characterized in that: The matching result of the main line detection point in step S5 is used to process the freight object, including the following contents: If the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is unique, it means that the matching is successful, and the camera ID corresponding to the mainline detection point and the time of completing the three-dimensional matching are updated in the queue information of the freight object in the mainline freight object queue; if the three-dimensional matching result of the mainline detection point for the freight object on the mainline and the freight object in the mainline freight object queue is not unique, it means that the matching fails and needs to be re-matched until a unique result is matched or the maximum matching time is reached; If the maximum matching time is exceeded and the first entry event is not obtained, the freight object is determined to be lost, the freight object is removed from the main line freight object queue, and a notification is issued; If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the main line and the freight object in the main line freight object queue, it is determined that there is an unknown freight object on the main line, and a new freight object record marked as abnormal is inserted into the main line freight object queue, and a notification is issued.

8. The method for tracking freight objects based on machine vision according to claim 7, characterized in that: The tracking of freight objects on the branch line described in step S7 is performed through a number of branch line detection points on the branch line. When the freight object on the branch line passes through each branch line detection point and blocks the light path of the photoelectric radiating tube, the camera and the ultrasonic component are enabled. The ultrasonic component is used to measure the height of the freight object, and the camera is used to obtain an image of the top of the freight object. The camera at each branch line detection point has a unique ID.

9. The method for tracking freight objects based on machine vision according to claim 8, characterized in that: As described in step S8, when the freight object passes through each branch inspection point on the branch line, a two-dimensional matching algorithm is used to verify whether the freight object at the current inspection point matches the freight object in the branch freight object queue. The volume parameter matching and image matching in step S4 are used again at each branch inspection point to match the contents of the queue information of the freight object on the branch line with those of the freight objects in the branch freight object queue.

10. The method for tracking freight objects based on machine vision according to claim 9, characterized in that: The matching result of the branch line detection point in step S9 is used to process the freight object, including the following contents: If the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is unique, it means that the matching is successful, and the camera ID corresponding to the branch line detection point and the time of completing the two-dimensional matching are updated in the queue information of the freight object in the branch line freight object queue; if the result of the two-dimensional matching between the freight object on the branch line and the freight object in the branch line freight object queue by the branch line detection point is not unique, it means that the matching fails, and the matching needs to be repeated until a unique result is matched or the maximum matching time is reached; If the maximum matching time is exceeded and the second entry event is not obtained, it is determined that the freight object has not entered the specified branch line, the freight object is removed from the corresponding branch line freight object queue, and a notification is issued; If the maximum matching time is exceeded and there is no corresponding matching result between the freight object on the branch line and the freight object in the branch line freight object queue, it is determined that there is an unknown freight object on the branch line, and a record of a new freight object marked as abnormal is inserted into the branch line freight object queue, and a notification is issued.

Images