Baggage sorting system based on 3D vision

Abstract

The present application discloses a 3D vision-based baggage sorting system, comprising a conveyor belt, a control device, a visual camera, and a gripping device. The conveyor belt is used to place target objects, the visual camera is used to collect 3D point cloud data of the target objects, and the gripping device is used to grip the target objects based on their posture information. The control device is used to obtain 3D point cloud data, and processes the 3D point cloud data to obtain the size information and posture information of the enclosing structure frame. When the target object is a vehicle, the angle information of the target object is determined based on the height information of the long surface of the target object. When the target object is luggage, the angle information of the target object is determined based on the ratio of the amount of point cloud data of the target object in two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface. The present application avoids manual sorting and placement of luggage, thereby preventing damage to luggage due to human negligence.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a baggage sorting system based on 3D vision. Background Art

[0002] Figure 1 is the current baggage sorting scene diagram. Figure 1 As shown, after passengers check in their luggage, the luggage will be transported to the carousel via a conveyor belt. Staff will manually sort the luggage at the carousel and place it on the corresponding luggage shuttle bus.

[0003] When flights are busy or the amount of checked baggage on a flight is large, luggage can easily be damaged due to negligence during the manual sorting and placement of luggage. Summary of the Invention

[0004] The present application provides a 3D vision-based baggage sorting system, which solves the problem of baggage damage easily caused by human negligence during manual baggage sorting and placement.

[0005] The 3D vision-based baggage sorting system includes a conveyor belt, a control device, a visual camera, and a gripping device. The visual camera is located above the conveyor belt, and the visual camera and gripping device are respectively connected to the control device for communication;

[0006] The conveyor belt is used to place the target object;

[0007] The visual camera is used to collect 3D point cloud data of the target objects on the conveyor belt;

[0008] The grasping device is used to grasp the target object according to the posture information of the target object;

[0009] The control device is used to obtain 3D point cloud data collected by the visual camera, process the 3D point cloud data to obtain size information and position information of the enclosing structure frame surrounding the target object, and then obtain the size information and position information of the target object. When the target object is a vehicle, the angle information of the target object is determined based on the height information of the long surface of the target object. When the target object is luggage, the angle information of the target object is determined based on the ratio of the amount of point cloud data of the target object in two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface;

[0010] Among them, the posture information includes size information, posture information and angle information. The two detection frames are both located in the enclosing structure frame, and one detection frame corresponds to a short face of the enclosing structure frame.

[0011] Beneficial effects: The present application provides a baggage sorting system based on 3D vision, comprising a conveyor belt, a control device, a visual camera and a gripping device, wherein the visual camera is located above the conveyor belt, and the visual camera and the gripping device are respectively connected to the control device for communication. The conveyor belt is used to place the target object so that the target object is transported along the conveyor belt. The visual camera is used to collect 3D point cloud data of the target object on the conveyor belt. The control device obtains the 3D point cloud data collected by the visual camera. The control device processes the 3D point cloud data to obtain the size information and posture information of the enclosing structure frame that encloses the target object, and then obtains the size information and posture information of the target object. The control device determines the long side and the short side of the target object based on the size information of the target object. When the category of the target object is a vehicle, the control device determines the angle information of the target object based on the height information of the long side of the target object. When the category of the target object is luggage, the control device determines the angle information of the target object based on the ratio of the amount of point cloud data of the target object in the two detection frames with the center point of the short side of the enclosing structure frame as the center point of the bottom surface. Size information, position information, and angle information constitute posture information, and the gripping device is used to grip the target object based on the received posture information of the target object. In this application, the control device obtains 3D point cloud data collected by the visual camera; the 3D point cloud data is processed according to the category of the target object to obtain the posture information of the target object. The gripping device grips the target object based on the posture information of the target object, thereby avoiding manual sorting and placement of luggage, and further preventing damage to the luggage due to human negligence during the manual sorting and placement of luggage. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 This is the current baggage sorting scene diagram;

[0014] Figure 2 A schematic diagram of the structure of a 3D vision-based baggage sorting system provided in an embodiment of the present application;

[0015] Figure 3 2D images of the target object captured by the visual camera provided in the embodiment of the present application at different times;

[0016] Figure 4 A cross-sectional view of the conveyor belt provided in an embodiment of the present application in the X direction of the visual camera;

[0017] Figure 5A schematic diagram of an enclosing structure frame enclosing a target object provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of vehicle angle information provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of luggage angle information provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of the first process of the 3D vision-based baggage sorting method provided in an embodiment of the present application;

[0021] Figure 9 This is a schematic diagram of the second process of the 3D vision-based baggage sorting method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To facilitate the technical solution of the application, some concepts involved in this application are first explained below.

[0023] Figure 2 This is a schematic diagram of the structure of the baggage sorting system based on 3D vision provided in the embodiment of the present application. Figure 2 As shown, in the embodiment of the present application, the baggage sorting system based on 3D vision includes a conveyor belt 100, a control device 200, a photoelectric trigger device 300, a visual camera 400 and a gripping device 500. Specifically,

[0024] The conveyor belt 100 is used to place target objects so that the target objects are transported along the conveyor belt. The target objects include vehicles and luggage.

[0025] The target object is placed at one end of the conveyor belt 100 and is transported to the other end of the conveyor belt 100 via the conveyor belt 100 .

[0026] Photoelectric trigger device 300, located on one side of conveyor belt 100, is configured to transmit a first signal and a second signal. Specifically, photoelectric trigger device 300 includes a transmitter 301 and a receiver 302, located on either side of conveyor belt 100. Light emitted by transmitter 301 directly enters receiver 302. When a target object is between transmitter 301 and receiver 302, receiver 302 cannot receive the light emitted by transmitter 301. However, when the target object moves from between transmitter 301 and receiver 302 and moves away from the two, receiver 302 can re-receive the light emitted by transmitter 301. When receiver 302 cannot receive the light emitted by transmitter 301, photoelectric trigger device 300 transmits the first signal. When receiver 302 re-receives the light emitted by transmitter 301, photoelectric trigger device 300 transmits the second signal.

[0027] The visual camera 400 is located directly above the conveyor belt 100 and is used to collect 2D images and 3D point cloud data of target objects on the conveyor belt 100 .

[0028] As the target object is transported along the conveyor belt 100, the visual camera 400 collects the target object's 2D image and 3D point cloud data in real time. As the conveyor belt 100 moves, the target object appears at different locations in front of the visual camera 400, and the quality of the 3D point cloud data collected at different locations also varies.

[0029] Figure 3 The 2D images of the target object captured by the visual camera provided in the embodiment of the present application at different times. Figure 3 As shown, the target object appears at multiple positions of the visual camera 400: position A, position B, position C, etc., and the distances of position A, position B and position C from the bottom of the visual camera 400 in the direction of the conveyor belt decrease successively.

[0030] like Figure 3 It can be seen that the 3D point cloud data of the target object at position C is superior to the 3D point cloud data of the target object at positions A and B. Combined with the above, it can be seen that the 3D point cloud data collected by the visual camera when the target object is directly below the visual camera is the optimal 3D point cloud data of the target object collected by the visual camera.

[0031] The control device 200 is located on one side of the conveyor belt 100 and is respectively connected to the photoelectric trigger device 300, the visual camera 400 and the grasping device 500 for communication.

[0032] In order to obtain the optimal 3D point cloud data of the target object collected by the visual camera 400, the control device 200 needs to calculate the moment when the target object is transmitted to the bottom of the visual camera 400, that is, the optimal time point.

[0033] Figure 4 This is a cross-sectional view of the conveyor belt provided in the embodiment of the present application in the X direction of the visual camera. Figure 4 It can be seen that

[0034] WH=(T1-T0)*V (1);

[0035] T2=(dY-WH / 2) / V+T0 (2);

[0036] Wherein, WH is the length of the target object in the direction of the conveyor belt, T2 is the optimal time point, T0 is the time point when the control device receives the first signal, T1 is the time point when the control device receives the second signal, V is the running speed of the conveyor belt, and dY is the vertical distance between the photoelectric trigger device and the visual camera in the direction of the conveyor belt.

[0037] According to formulas (1) and (2), the control device 200 calculates the optimal time point based on the time point of receiving the first signal, the time point of receiving the second signal, and the running speed of the conveyor belt 100. Specifically,

[0038] The control device calculates the length of the target object in the running direction of the conveyor belt according to the time point of receiving the first signal, the time point of receiving the second signal and the running speed of the conveyor belt.

[0039] The control device calculates the optimal time point based on the length of the target object in the running direction of the conveyor belt, the running speed of the conveyor belt, the time point of receiving the first signal and the vertical distance between the photoelectric trigger device and the visual camera in the running direction of the conveyor belt.

[0040] The control device 200 is configured to obtain the 2D image and 3D point cloud data captured by the visual camera at the optimal time point. Specifically, since the visual camera captures the 2D image and 3D point cloud data of the target object in real time, the control device can obtain the 2D image and 3D point cloud data captured by the visual camera at the optimal time point after calculating the optimal time point.

[0041] The control device 200 is used to identify the category of the target object in the 2D image. Specifically, the control device 200 designs a target detection network model, and the control device 200 identifies the category of the target object in the 2D image according to the target detection network model.

[0042] The control device designs a target detection network model. Specifically, a luggage dataset is generated and the target detection network model is trained.

[0043] A baggage dataset was created. Specifically, since no publicly available baggage datasets exist, a baggage dataset was collected and created in the examples of this application. To improve detection accuracy in practical application scenarios, the baggage data collected in this application consists of real baggage data from a specific airport and baggage data from a laboratory simulation environment. The baggage dataset in this application contains five types of labeled information: suitcases, carriers, cartons, labels, and school bags. The original baggage data was processed using the mosaic data augmentation method, resulting in a dataset totaling 12,000 images, divided into a training set, a validation set, and a test set.

[0044] The mosaic data augmentation method is used to process the original baggage data. The method includes: randomly selecting multiple images from the original baggage data; flipping and scaling the selected images to obtain corresponding new images; randomly selecting a center point on the new image; and cropping the selected images to the size of the vertical and horizontal lines of the center point; and arranging the cropped images in a preset order, such as top left, bottom left, top right, and bottom right, to obtain the processed data set.

[0045] Train the target detection network model. Specifically, the YOLOV5 target detection network model is used as the base network model, a channel attention mechanism is added to enhance the network feature extraction capability, the target detection network model is optimized, and the above-mentioned dataset is used for training. Compared with common public datasets, the total size of the luggage dataset produced by the embodiment of the present application is relatively small. To enable it to play a higher role in complex deep learning network training, the embodiment of the present application will first pre-train the base network model on the coco dataset, and then use the luggage dataset of the embodiment of the present application for training. A deep feature extraction network is used to extract image information. After multiple tuning and training, the final target detection network model is obtained.

[0046] The control device 200 identifies the category of the target object in the 2D image based on the target detection network model. Specifically, the control device uses the final target detection network model to detect and identify the target category in the 2D image in real time.

[0047] The control device 200 is configured to process the 3D point cloud data to obtain size information and position information of a bounding box surrounding the target object. Specifically, the control device preprocesses the 3D point cloud data to obtain foreground point cloud data of the target object. The control device processes the foreground point cloud data of the target object using a convex hull algorithm to obtain a bounding box of the target object. The control device projects the bounding box of the target object onto a projection surface and fits an outer bounding rectangle to the data points on the projection surface to obtain size information and position information of the bounding box surrounding the target object.

[0048] The projection plane is the imaginary plane on which the object is projected, including the horizontal projection plane, the upright projection plane, and the side projection plane. In the embodiment of the present application, the horizontal projection plane is the xy plane, the upright projection plane is the yz plane, and the side projection plane is the xz plane, where the y direction is the direction of the conveyor belt, the x direction is parallel to the conveyor belt and perpendicular to the conveyor belt's direction of travel, and the z direction is perpendicular to both the conveyor belt and the conveyor belt's direction of travel.

[0049] The control device pre-processes the 3D point cloud data to obtain foreground point cloud data of the target object. Specifically, the control device performs filtering processing on the 3D point cloud data to obtain filtered 3D point cloud data, and the control device processes the filtered 3D point cloud data using a point cloud segmentation algorithm to obtain foreground point cloud data of the target object.

[0050] The control device filters the 3D point cloud data to obtain filtered 3D point cloud data. Specifically, the control device sets a selected range for a region of interest and performs cropping and filtering on the data outside the region of interest to obtain primary streamlined point cloud data. The control device then performs sparse filtering on the point cloud data density using a point cloud voxel algorithm to obtain secondary streamlined point cloud data. The filtered 3D point cloud data is the secondary streamlined point cloud data.

[0051] The control device processes the filtered 3D point cloud data using a point cloud segmentation algorithm to obtain foreground point cloud data of the target object. Specifically, the control device processes the secondary simplified 3D point cloud data using a point cloud segmentation algorithm to obtain foreground point cloud data of the target object.

[0052] The control device projects the bounding box of the target object onto the projection surface and fits the data points on the projection surface to an outer bounding rectangle to obtain the size information and position information of the bounding structure box surrounding the target object. Specifically, the control device projects the bounding box of the target object onto the xy plane, uses the multiple (x, y) projected onto the xy plane to construct a Mat data structure adapted to the OPENCV image computing library, and uses the OPENCV library function to calculate the first minimum bounding rectangle surrounding the multiple points. The four vertex coordinates and center point coordinates of the first minimum bounding rectangle can be obtained. Based on the four vertex coordinates of the first minimum bounding rectangle, the length and width of the bounding structure box surrounding the target object can be obtained.

[0053] The control device projects the bounding box of the target object onto the yz plane, and uses multiple (y, z) projected onto the yz plane to construct a Mat data structure adapted to the OPENCV image computing library. The OPENCV library function is used to calculate the minimum enclosing rectangle surrounding multiple points, and the four vertex coordinates and center point coordinates of the second minimum enclosing rectangle can be obtained. According to the four vertex coordinates of the second minimum enclosing rectangle, the height of the enclosing structure box surrounding the target object can be obtained.

[0054] The position information of the enclosing structure frame, the long side of the enclosing structure frame and the short side of the enclosing structure frame are obtained according to the four vertex coordinates of the first minimum enclosing rectangle and the four vertex coordinates of the second minimum enclosing rectangle.

[0055] Based on the coordinates of the four vertices of the first minimum bounding rectangle and the length, width, and height of the enclosing frame, the coordinates of all vertices of the enclosing frame are obtained, thereby obtaining the enclosing frame. The coordinates of the four vertices of the enclosing frame in the xy plane are the coordinates of the four vertices of the first minimum bounding rectangle, and the coordinates of the remaining four vertices of the enclosing frame are calculated from the coordinates of the four vertices of the first minimum bounding rectangle and the length, width, and height of the enclosing frame.

[0056] Figure 5A schematic diagram of an enclosing structure frame enclosing a target object provided in an embodiment of the present application. Figure 5 The left side of the figure is a schematic diagram of the enclosing structure frame when the target object is a suitcase, and the right side is a schematic diagram of the enclosing structure frame when the target object is a vehicle. Figure 5 As shown, a target object is set in the enclosing structure frame.

[0057] The coordinates of the center point of the short face of the enclosing structure frame are obtained according to the coordinates of the four vertices of the short face of the enclosing structure frame.

[0058] The control device uses the size information and pose information of the bounding structure frame surrounding the target object to obtain the size information and pose information of the target object. Specifically, because the bounding structure frame surrounding the target object is similar to the target object, the control device identifies the size information of the bounding structure frame surrounding the target object as the size information of the target object, and also identifies the pose information of the bounding structure frame surrounding the target object as the pose information of the target object. The size information includes length, width, and height.

[0059] Figure 6 A schematic diagram of vehicle angle information provided in an embodiment of the present application. Figure 7 A schematic diagram of luggage angle information provided in an embodiment of the present application. Figure 6 The 0 degree direction is the reference line, which is the direction of the reverse extension of x. The vehicle direction refers to the direction of the vehicle. Figure 7 The direction of the middle luggage is the direction of the wheelset. Figure 6 and 7 As shown, in the embodiment of the present application, the control device is used to determine the angle information of the target object according to the category of the target object. Specifically,

[0060] When the target object is a vehicle, the angle of the target object is determined based on the height of the long surface of the target object. When the target object is luggage, the angle of the target object is determined based on the ratio of the point cloud data volume of the target object in the two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface.

[0061] When the category of the target object is a vehicle, the control device determines the angle information of the target object based on the height information of the long side of the target object. Specifically, the control device determines the long side of the target object based on the size information of the target object. Since there is a height difference in the long side of the target object on the yz projection plane, the control device defines the direction of the target object, that is, the vehicle direction, as the direction from the high side of the long side of the target object on the yz projection plane to the low side of the long side of the target object on the yz projection plane. Since the angle between the direction of the target object and the reference line is the angle information of the target object, the control device determines the angle information of the target object using the vehicle direction and the reference line. Among them, the z value of the high side of the long side of the target object on the yz projection plane is greater than the z value of the low side of the long side of the target object on the yz projection plane, and the reference line is a reverse extension line parallel to the conveyor belt and perpendicular to the running direction of the conveyor belt, that is, a reverse extension line of x.

[0062] When the target object is luggage, the control device determines the target object's angle information based on the ratio of the target object's point cloud data within two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface. Specifically, the control device determines the short surface of the enclosing structure frame based on the size information of the enclosing structure frame and constructs two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface. The control device determines the boundary information of the detection frames based on the size information of the detection frames. The control device cuts the target object's point cloud data based on the boundary information of the detection frames to obtain point cloud data within the detection frames. The control device determines the wheelset heading of the target object, i.e., the direction of the luggage, based on the ratio of the point cloud data within the two detection frames. The control device obtains the target object's angle information based on the direction of the luggage and a reference line. The target object's angle information is the angle between the wheelset heading of the target object and the reference line.

[0063] The two detection frames are respectively set on the two short sides of the enclosing structure frame, and the center point of the bottom surface of the detection frame is the center point of the short side of the enclosing structure frame.

[0064] The control device determines the boundary information of the detection frame based on the size information of the detection frame. Specifically, based on the statistical analysis of the current size information of airline luggage, the wheelset height is mainly distributed in the range of 40 to 60 mm, the wheelset spacing is above 200 mm, the detection frame is set on the short side of the enclosing structure frame, and the bottom center point of the detection frame is the center point of the short side of the enclosing structure frame. In the embodiment of the present application, the size information of the detection frame is 120mm*60mm*60㎜. The control device determines the coordinates of all vertices of the detection frame based on the coordinates of the center point of the short side of the enclosing structure frame and the size information of the detection frame. Among them, the boundary information of the detection frame includes the coordinates of all vertices of the detection frame.

[0065] The control device determines the wheelset orientation of the target object based on the ratio of the amount of point cloud data within the two detection frames. Specifically, the control device calculates the ratio of the amount of point cloud data of the target object within the two detection frames. When the ratio of the amount of point cloud data of the target object within the two detection frames is greater than 1.5, the end of the detection frame with the lesser amount of point cloud data is selected as the location of the wheelset of the target object, thereby determining the wheelset orientation of the target object.

[0066] The gripping device 500 is in communication with the control device 200 and is configured to grasp a target object based on its posture information and place it on a corresponding logistics vehicle. Specifically, if the target object is a vehicle, the gripping device grasps the vehicle and flips it over, dumping the contents onto the corresponding logistics vehicle. If the target object is luggage, the gripping device grasps the luggage and directly places it on the corresponding logistics vehicle. The posture information includes size, position, and angle information.

[0067] The present application provides a 3D vision-based baggage sorting system, comprising a conveyor belt, a control device, a photoelectric trigger device, a visual camera, and a gripping device. The photoelectric trigger device and the control device are located on one side of the conveyor belt, and the visual camera is located above the conveyor belt. The photoelectric trigger device, the visual camera, and the gripping device are each communicatively connected to the control device. The conveyor belt is used to place a target object so that the target object is transported along the conveyor belt. The photoelectric trigger device is used to transmit a first signal and a second signal. The visual camera is used to capture 2D images and 3D point cloud data of the target object on the conveyor belt. The control device calculates the optimal time for the visual camera to capture the 2D image and 3D point cloud data based on the time of receiving the first signal, the time of receiving the second signal, and the operating speed of the conveyor belt. The control device obtains the 2D image and 3D point cloud data captured by the visual camera at the optimal time. The control device identifies the category of the target object in the 2D image. The control device processes the 3D point cloud data to obtain size information and position information of a bounding box surrounding the target object, thereby obtaining size information and position information of the target object. The control device determines the long and short sides of the target object based on the object's size information. If the target object is a vehicle, the control device determines the target object's angle based on the height information of the long side. If the target object is luggage, the control device determines the target object's angle based on the ratio of the target object point cloud data within two detection frames with the center point of the short side of the enclosing structure frame as the center point of the bottom surface. The size information, position information, and angle information constitute the posture information, and the grasping device is used to grasp the target object based on the received posture information. In the present application, the control device calculates the optimal time point for the visual camera to collect 2D images and 3D point cloud data based on the time points of the first signal and the second signal sent by the photoelectric trigger device and the running speed of the conveyor belt, and obtains the 2D image and 3D point cloud data collected by the visual camera at the optimal time point; the control device identifies the category of the target object from the 2D image, and processes the 3D point cloud data according to the category of the target object to obtain the posture information of the target object. The grasping device grasps the target object according to the posture information of the target object, avoiding manual sorting and placement of luggage, and thus avoiding damage to the luggage due to human negligence during the process of manual sorting and placement of luggage.

[0068] In addition to providing a baggage sorting system based on 3D vision, this application also provides a baggage sorting method based on 3D vision. Figure 8 This is a schematic diagram of the first process of the 3D vision-based baggage sorting method provided in an embodiment of the present application. Figure 9 This is a second flow chart of the baggage sorting method based on 3D vision provided in the embodiment of the present application. Figure 8-9 As shown, in the embodiment of the present application, the baggage sorting method based on 3D vision includes:

[0069] S100: The optimal time point is calculated based on the time point of receiving the first signal, the time point of receiving the second signal, and the running speed of the conveyor belt.

[0070] The control device calculates the optimal time point based on the time point of receiving the first signal, the time point of receiving the second signal, and the running speed of the conveyor belt. The specific steps are as follows:

[0071] The control device calculates the length of the target object in the running direction of the conveyor belt according to the time point of receiving the first signal, the time point of receiving the second signal and the running speed of the conveyor belt.

[0072] The control device calculates the optimal time point based on the length of the target object in the running direction of the conveyor belt, the running speed of the conveyor belt, the time point of receiving the first signal and the vertical distance between the photoelectric trigger device and the visual camera in the running direction of the conveyor belt.

[0073] S200: Acquire 2D images and 3D point cloud data collected by the visual camera at the optimal time point.

[0074] S300: Identify the category of the target object in the 2D image.

[0075] The control device identifies the category of the target object in the 2D image based on the target detection network model. The specific steps are as follows:

[0076] Control device designs target detection network model.

[0077] The control device identifies the category of the target object in the 2D image based on the target detection network model.

[0078] S400: Processing the 3D point cloud data to obtain size information and position information of a bounding structure frame surrounding the target object, and then obtaining size information and position information of the target object.

[0079] The control device processes the 3D point cloud data to obtain the size information and position information of the enclosing structure frame surrounding the target object, and then obtains the size information and position information of the target object. The specific steps are as follows:

[0080] The control device pre-processes the 3D point cloud data to obtain foreground point cloud data of the target object.

[0081] The control device processes the foreground point cloud data of the target object using a convex hull algorithm to obtain a bounding box of the target object.

[0082] The control device projects the bounding box of the target object onto the projection surface, and performs outer enclosing rectangle fitting on the data points on the projection surface to obtain size information and position information of the enclosing structure box enclosing the target object.

[0083] The control device recognizes the size information of the bounding structure frame surrounding the target object as the size information of the target object, and further recognizes the posture information of the bounding structure frame surrounding the target object as the posture information of the target object.

[0084] S500: Determine angle information of the target object according to the category of the target object.

[0085] S501: Determine whether the category of the target object is luggage.

[0086] S502: When the category of the target object is a vehicle, determine the angle information of the target object according to the height information of the long surface of the target object.

[0087] When the target object is a vehicle, the control device determines the angle information of the target object based on the height information of the long surface of the target object. The specific steps are as follows:

[0088] When the target object is a vehicle, the control device defines the direction of the target object, ie, the vehicle direction, as the direction where the upper side of the target object's long surface on the yz projection plane points to the lower side of the target object's long surface on the yz projection plane.

[0089] The control device determines the angle information of the target object using the vehicle direction and the reference line.

[0090] S503: When the category of the target object is luggage, determine the angle information of the target object according to the ratio of the amount of point cloud data of the target object in two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface.

[0091] When the target object is luggage, the control device determines the angle information of the target object based on the ratio of the target object point cloud data volume in the two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface. The specific steps are as follows:

[0092] The control device determines the short side of the enclosing structure frame according to the size information of the enclosing structure frame, and constructs two detection frames with the center point of the short side of the enclosing structure frame as the center point of the bottom surface.

[0093] The control device determines boundary information of the detection frame according to size information of the detection frame.

[0094] The control device cuts the target object point cloud data according to the boundary information of the detection frame to obtain the point cloud data within the detection frame.

[0095] The control device determines the wheelset orientation of the target object, that is, the direction of the luggage, based on the ratio of the point cloud data volumes in the two detection frames.

[0096] The control device obtains the angle information of the target object according to the direction of the luggage and the reference line.

[0097] It should be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a circuit structure, article, or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such circuit structure, article, or device. In the absence of further limitations, the phrase "comprising a ..." to define an element does not exclude the presence of other identical elements in the circuit structure, article, or device comprising the element.

[0098] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure of the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0099] The above-described embodiments of the present application do not constitute a limitation on the scope of protection of the present application.

Claims

1. The baggage sorting system based on 3D vision is characterized by: The invention comprises a conveyor belt (100), a control device (200), a visual camera (400) and a gripping device (500), wherein the visual camera (400) is located above the conveyor belt (100), and the visual camera (400) and the gripping device (500) are respectively connected to the control device (200) for communication; The conveyor belt (100) is used to place target objects; The visual camera (400) is used to collect 3D point cloud data of the target object on the conveyor belt (100); The grasping device (500) is used to grasp the target object according to the posture information of the target object; The control device (200) is used to obtain 3D point cloud data collected by the visual camera (400), process the 3D point cloud data to obtain size information and position information of a surrounding structure frame surrounding the target object, and further obtain size information and position information of the target object; when the category of the target object is a vehicle, the orientation of the target object is determined based on the height information of the long surface of the target object, and the angle information of the target object is obtained based on the orientation of the target object and a reference line, wherein the reference line is a reverse extension line parallel to the conveyor belt and perpendicular to the running direction of the conveyor belt; When the target object is luggage, determining the angle information of the target object based on the ratio of the amount of point cloud data of the target object in two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface; The posture information includes size information, position information and angle information. Both of the detection frames are located in the enclosing structure frame, and one detection frame corresponds to a short side of the enclosing structure frame.

2. The baggage sorting system according to claim 1, characterized in that: The control device determines the angle information of the target object according to the ratio of the amount of point cloud data in two detection frames with the center point of the short surface of the enclosing structure frame as the center point of the bottom surface, including: The control device determines the short side of the enclosing structure frame and constructs two detection frames with the center point of the short side of the enclosing structure frame as the center point of the bottom surface; Determining boundary information of the detection frame according to size information of the detection frame; Cutting the target object point cloud data according to the boundary information of the detection frame to obtain the point cloud data within the detection frame; Determine the wheelset orientation of the target object based on the ratio of point cloud data within the two detection frames; The angle information of the target object is obtained according to the wheelset orientation of the target object and the reference line.

3. The baggage sorting system according to claim 2, characterized in that: The size of the detection frame is 120mm*60mm*60㎜.

4. The baggage sorting system according to claim 2, characterized in that: The control device determines the boundary information of the detection frame according to the size information of the detection frame, including: The control device obtains the boundary information of the detection frame according to the coordinates of the center point of the bottom surface of the detection frame and the size information of the detection frame.

5. The baggage sorting system according to claim 2, characterized in that: The control device determines the wheelset orientation of the target object according to the ratio of the amount of point cloud data of the target object in the two detection frames, including: The control device calculates the ratio of the point cloud data volume of the target objects in the two detection frames; When the ratio of the target object point cloud data volume in the two detection frames is greater than 1.5, the end of the detection frame with the smaller target object point cloud data volume is selected as the location of the target object's wheelset, thereby determining the wheelset orientation of the target object.

6. The baggage sorting system according to claim 1, characterized in that: The control device is further configured to identify the category of the target object in the 2D image, including: The control device designs a target detection network model; The category of the target object in the 2D image is identified according to the target detection network model.

7. The baggage sorting system according to claim 1, characterized in that: The control device is used to process the 3D point cloud data to obtain size information and position information of a bounding structure frame surrounding the target object; The control device pre-processes the 3D point cloud data to obtain foreground point cloud data of the target object; Processing the foreground point cloud data of the target object using a convex hull algorithm to obtain a bounding box of the target object; The bounding box of the target object is projected onto the projection surface, and the data points on the projection surface are fitted with an outer bounding rectangle to obtain the size information and posture information of the bounding box surrounding the target object.

8. The baggage sorting system according to claim 1, characterized in that: The invention also includes a photoelectric trigger device (300), wherein the photoelectric trigger device (300) includes a transmitter (301) and a receiver (302), wherein the transmitter (301) and the receiver (302) are respectively located on both sides of the conveyor belt (100); when the receiver cannot receive the light emitted by the transmitter, the photoelectric trigger device (300) sends a first signal; and when the receiver receives the light emitted by the transmitter again, the photoelectric trigger device (300) sends a second signal.

9. The baggage sorting system according to claim 8, characterized in that: The control device is used to obtain 3D point cloud data collected by the visual camera at the optimal time point, wherein the control device calculates the length of the target object in the running direction of the conveyor belt based on the time point of receiving the first signal, the time point of receiving the second signal and the running speed of the conveyor belt, and calculates the optimal time point based on the length of the target object in the running direction of the conveyor belt, the running speed of the conveyor belt, the time point of receiving the first signal and the vertical distance between the photoelectric trigger device and the visual camera in the running direction of the conveyor belt.

Images