A method for identifying stacked irregular objects and a sorting workstation thereof

Abstract

The present invention provides a method for identifying stacked irregular objects and a sorting workstation thereof, wherein the method comprises the following steps: S1 acquiring image data and point cloud data of irregular objects stacked in a target scene acquired by an acquisition device; inputting the image data into a trained target recognition algorithm to acquire the recognition frame data of each irregular object; S2 taking the optical center of the acquisition device as the vertex and according to the pinhole imaging rule, back-projecting each irregular object recognition frame to establish a cone space corresponding to each irregular object, and roughly segmenting the point cloud data to acquire the point cloud cluster corresponding to each irregular object. Thus, at least the stacked irregular objects in the target scene can be identified and segmented.

Description

Technical Field

[0001] The present invention relates to machine vision recognition technology, and in particular to a method for identifying and distinguishing stacked irregular-shaped objects, and a sorting workstation constructed using the method. Background Art

[0002] As my country's consumption levels continue to rise, demand for a wide range of goods is increasing year by year. Consequently, various manufacturers have proposed various intelligent manufacturing development plans to improve production efficiency. However, in certain product production areas, mixed production lines are often designed to integrate intelligent, automated, and flexible systems to simultaneously process products of varying specifications. For example, in the cosmetics, pharmaceutical, and food production industries, products of varying specifications and shapes are often processed together. The "irregular-shaped objects" referred to in this case refer to such products of varying specifications and shapes.

[0003] Due to the uniqueness of irregular-shaped objects themselves, they are often messy or stacked on the production line, which will greatly reduce the accuracy of the visual system in identifying objects and increase the difficulty of sorting. Summary of the Invention

[0004] To this end, the main purpose of the present invention is to provide a method for identifying stacked irregular-shaped objects and a sorting workstation thereof, so as to at least realize the identification and segmentation of the irregular-shaped objects stacked in the target scene.

[0005] In order to achieve the above object, according to a first aspect of the present invention, a method for identifying stacked irregular-shaped objects is provided, the steps of which include:

[0006] S1 obtains image data and point cloud data of stacked foreign-shaped objects in a target scene collected by an acquisition device; inputs the image data into a trained target recognition algorithm to obtain recognition frame data of each foreign-shaped object;

[0007] S2 takes the optical center of the acquisition device as the vertex and back-projects the identification frames of each irregular object according to the pinhole imaging rule to establish a conical space corresponding to each irregular object, to perform coarse segmentation of the point cloud data and obtain the point cloud cluster corresponding to each irregular object.

[0008] In a possible preferred embodiment, the method for identifying stacked irregular-shaped objects further comprises:

[0009] S3 uses the region growing algorithm to finely segment the point cloud clusters obtained in step S2 to obtain target point cloud data that is close to complete for each irregular object.

[0010] In a possible preferred embodiment, in step S1, the acquisition device also acquires depth map data of the stacked foreign-shaped objects in the target scene; corrects the coordinates of the image data based on the depth map data, and further corrects the coordinates of each foreign-shaped object identification frame data in the target scene.

[0011] In a possible preferred embodiment, the method for identifying stacked irregular-shaped objects further comprises:

[0012] S4 uses the OBB algorithm to establish bounding boxes for the target point cloud data of each alien object; it determines whether there are overlapping areas among the bounding boxes. If so, it further performs support judgment to distinguish the supporting and supported objects based on the orientation of the normal vectors of each bounding box, and then identifies the relative posture of other alien objects in contact with each alien object in the target scene.

[0013] In order to achieve the above-mentioned object, according to a second aspect of the present invention, a sorting workstation is provided, which includes: a parallel robot and a visual reasoning system, wherein the visual reasoning system includes: an acquisition device, a processing unit, and a storage unit, wherein the acquisition device acquires image data and point cloud data of stacked foreign-shaped objects in a target scene and sends them to the processing unit; the processing unit retrieves the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the recognition frame data of each foreign-shaped object; the processing unit back-projects the recognition frame of each foreign-shaped object according to the pinhole imaging rule with the optical center of the acquisition device as the vertex. A conical space corresponding to each alien-shaped object is established to perform coarse segmentation on the point cloud data and obtain the point cloud cluster corresponding to each alien-shaped object; the processing unit calls the OBB algorithm in the storage unit to establish a bounding box for the point cloud cluster of each alien-shaped object, and determines whether there is an overlapping area between the bounding boxes. If so, further support judgment is performed to distinguish the supporting and supported objects according to the direction of the normal vector of each bounding box, so as to output a target relationship tree of the alien-shaped objects stacked in the target scene, and generate a grasping sequence strategy based on the parent / child nodes of the target relationship tree; the parallel robot executes the grasping sequence strategy to sort the alien-shaped objects in the target scene.

[0014] In order to achieve the above-mentioned purpose, according to the third aspect of the present invention, a sorting workstation is provided, which includes: a parallel robot, a visual reasoning system, wherein the visual reasoning system includes: an acquisition device, a processing unit, and a storage unit, wherein the acquisition device acquires image data, depth map data, and point cloud data of stacked foreign-shaped objects in the target scene, and sends them to the processing unit; the processing unit calls the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the recognition frame data of each foreign-shaped object; then corrects the coordinates of the image data according to the depth map data, and then corrects the coordinates of the recognition frame data of each foreign-shaped object in the target scene; the processing unit takes the optical center of the acquisition device as the top According to the pinhole imaging rule, the identification frames of each foreign-shaped object are back-projected to establish the conical space corresponding to each foreign-shaped object, so as to roughly segment the point cloud data and obtain the point cloud clusters corresponding to each foreign-shaped object; the processing unit calls the OBB algorithm in the storage unit to establish bounding boxes for the point cloud clusters of each foreign-shaped object, and determines whether there are overlapping areas between the bounding boxes. If so, further support judgment is performed to distinguish the supporting and supported objects according to the direction of the normal vector of each bounding box, so as to output the target relationship tree of the foreign-shaped objects stacked in the target scene, and generate a grasping sequence strategy based on the parent / child nodes of the target relationship tree; the parallel robot executes the grasping sequence strategy to sort the foreign-shaped objects in the target scene.

[0015] In order to achieve the above-mentioned purpose, according to the fourth aspect of the present invention, a sorting workstation is provided, which includes: a parallel robot, a visual reasoning system, wherein the visual reasoning system includes: an acquisition device, a processing unit, and a storage unit, wherein the acquisition device acquires image data, depth map data, and point cloud data of the stacked foreign-shaped objects in the target scene, and sends them to the processing unit; the processing unit calls the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the data of each foreign-shaped object recognition frame; corrects the coordinates of the image data according to the depth map data, and then corrects the coordinates of the foreign-shaped object recognition frame data in the target scene; the processing unit takes the optical center of the acquisition device as the vertex and back-projects each foreign-shaped object recognition frame according to the pinhole imaging rule. Shadow, to establish the cone space corresponding to each alien-shaped object, to perform coarse segmentation on the point cloud data, and obtain the point cloud cluster corresponding to each alien-shaped object; then call the region growing algorithm in the storage unit to perform fine segmentation on each point cloud cluster, and obtain the target point cloud data of each alien-shaped object; the processing unit calls the OBB algorithm in the storage unit to establish a bounding box for the target point cloud data of each alien-shaped object, and judges whether there is an overlapping area between the bounding boxes. If so, further support judgment is performed to distinguish the supporting and supported objects according to the direction of the normal vector of each bounding box, so as to output the target relationship tree of the alien-shaped objects stacked in the target scene, and generate a grasping sequence strategy according to the parent / child nodes of the target relationship tree; the parallel robot executes the grasping sequence strategy to sort the alien-shaped objects in the target scene.

[0016] The method for identifying stacked foreign-shaped objects and the sorting workstation thereof provided by the present invention can effectively and accurately identify and segment each foreign-shaped object in a target scene, and in a preferred embodiment can also accurately determine the relative posture of each foreign-shaped object in the target scene with respect to other foreign-shaped objects in contact with it, thereby providing a grasping sequence strategy for subsequent sorting procedures, and further realizing rapid grasping and accurate sorting of foreign-shaped targets through servo control of the parallel robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0018] Figure 1 A schematic diagram of the steps of the method for identifying stacked irregular-shaped objects according to the present invention;

[0019] Figure 2 This is a schematic diagram of establishing recognition frames for various different-shaped objects in the target scene in Example 1;

[0020] Figure 3 Schematic diagram of coordinate system 1 for depth map and image data in embodiment 1;

[0021] Figure 4 Schematic diagram of establishing a conical space based on the pinhole imaging principle in Example 1;

[0022] Figure 5 Schematic diagram of using the cone space to perform coarse segmentation of alien objects in the target scene in Example 1;

[0023] Figure 6 This is a schematic diagram of target point cloud data after fine segmentation using the region growing algorithm in Example 1;

[0024] Figure 7 Schematic diagram of establishing a bounding box for target point cloud data using the OBB algorithm in Example 1;

[0025] Figure 8 Schematic diagram of six pairs of contact objects after the bounding box determines the contact relationship in Example 1;

[0026] Figure 9 The support relationship tree established after the support relationship is determined by the bounding box in Example 1;

[0027] Figure 10 This is a schematic diagram of the principle of determining the support relationship of the bounding box according to the normal vector in Example 1;

[0028] Figure 11 This is a schematic diagram of the principle of determining the support relationship of the bounding box according to the normal vector in Example 1;

[0029] Figure 12 This is a flow chart of the sorting workstation in Example 2 performing sorting of irregular-shaped objects in combination with the method in Example 1. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this case are only embodiments of a part of the present invention, rather than all embodiments. It should be pointed out that, for those of ordinary skill in the art, the embodiments in this application and the features in the embodiments can be combined with each other without departing from the concept of the present invention and without conflicting with each other. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative work should fall within the disclosure and protection scope of the present invention.

[0031] In addition, the terms "first", "second", "S1", "S2", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those described herein. At the same time, the terms "including" and "having" and any variations thereof in the present invention are intended to cover non-exclusive inclusions. Unless otherwise expressly specified and limited, the terms "setting", "layout", "installation", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a communication between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in this case can be understood based on the specific circumstances and in combination with the existing technology. (one)

[0033] like Figures 1 to 6 As shown, the first aspect of the present invention provides a method for identifying stacked irregular objects, the steps of which include:

[0034] S1 obtains image data and point cloud data of stacked foreign-shaped objects in a target scene collected by an acquisition device; inputs the image data into a trained target recognition algorithm to obtain recognition frame data of each foreign-shaped object;

[0035] S2 takes the optical center of the acquisition device as the vertex and back-projects the identification frames of each irregular object according to the pinhole imaging rule to establish a conical space corresponding to each irregular object, to perform coarse segmentation of the point cloud data and obtain the point cloud cluster corresponding to each irregular object.

[0036] S3 uses the region growing algorithm to finely segment the point cloud clusters obtained in step S2 to obtain target point cloud data that is close to complete for each irregular object.

[0037] Specifically, in the design concept of the method of the present invention, the inventors proposed a fusion segmentation method, which includes two stages: the first stage is coarse segmentation, and the second stage is fine segmentation.

[0038] In the first stage of coarse segmentation, the target scene area is semantically masked by the target detection network. In this embodiment, the target detection network preferably uses Yolov4 as the underlying framework for object detection. The advantage is that the CNN framework achieves an optimal balance between input network resolution, number of convolutional layers, number of parameters, and number of layer outputs, and optimizes the optimal parameter aggregation method from the backbone layer for different detection layers through additional blocks.

[0039] It utilizes the most advanced optimization strategies in the CNN field in recent years. Compared to the previous YOLO framework, it offers varying degrees of optimization in data processing, backbone network, network training, activation functions, loss functions, and more. This CNN uses CSPDarknet53 as its backbone network and employs the Spatial Pyramid Pooling (SPP) concept to expand the neural network's receptive field. It also uses PANet instead of FPN for parameter aggregation from the backbone network layer to the detection layer, thereby enhancing the system's processing speed and accuracy.

[0040] The effectiveness of Yolov4 has been verified by numerous experiments, and a pre-trained weight file, yolov4.weights, has been obtained for two common object detection datasets, MS COCO and VOC. Compared with other datasets of the same type, Yolov4 has significant advantages in speed and accuracy. Therefore, this paper uses the dataset trained with this framework for special-shaped objects as the basis for object detection.

[0041] The second stage of fine segmentation mainly refers to the segmentation idea based on the region growing algorithm. Since the neural network for the first stage of coarse segmentation has relatively high requirements in terms of time complexity and space complexity, and considering the real-time requirements of production equipment, the second stage of fine segmentation mainly adopts the idea based on region growing.

[0042] The basic idea behind region growing is to gradually spread out points with the same or similar properties from a certain point, continuously merging them until no more points meet the requirements. Region growing is then complete, achieving segmentation of similar regions. The advantage of region growing is that it can quickly link regions with the same features, clarify boundary information, and obtain complete segmentation results with good initial values.

[0043] In addition, in an optional example, in order to facilitate the subsequent sorting process to control the robot to accurately grasp various special-shaped objects, in step S1, Figure 3 As shown, the acquisition device also acquires depth map data of the stacked foreign-shaped objects in the target scene, thereby correcting the coordinate system of the depth map data and the image data by unifying the coordinate system, and then correcting the coordinates of each foreign-shaped object identification frame data in the target scene.

[0044] To further illustrate the implementation process, Figures 2 to 6 As shown, this example simulates an identification and sorting case on a mixed production line. In the example, the acquisition device preferably uses a depth camera to collect two-dimensional RGB image data, depth map data and three-dimensional point cloud data of the stacked foreign-shaped objects in the target scene. First, the RGB image data is input into the trained yolov4 algorithm to identify the various foreign-shaped objects in the target scene, thereby obtaining the 2D recognition frames of the 8 foreign-shaped objects in the scene.

[0045] Then, the RGB image data and the depth map coordinate system are unified, and the coordinates of the eight recognition boxes in the RGB image data are transformed into the corresponding depth map (taking the watering can in the picture as an example), so as to correct the coordinates of the recognition box data of each alien object in the target scene.

[0046] Then, the coarse segmentation procedure in the fusion segmentation is executed to back-project the 8 object recognition boxes under the depth map using the camera model, as shown in the following example: Figure 4 As shown in , according to the pinhole imaging principle, the cone space with the camera optical center as the vertex can be obtained, such as Figure 5 As shown in Figure 1, the recognition boxes of the eight objects are back-projected to obtain their corresponding cone spaces. Using the cone space as a spatial constraint, the sampled point cloud data is initially filtered to obtain point cloud clusters of the eight alien objects.

[0047] However, due to the extensibility of the conical space, the objects obtained by coarse segmentation will inevitably contain non-target point clouds, which will cause greater uncertainty in the subsequent grasping pose estimation during sorting. Therefore, in the preferred embodiment, fine segmentation is required to remove point clouds other than the target.

[0048] Therefore, the key to fine segmentation is to find the boundaries of each alien object. Since the point cloud color, curvature and other information of different alien objects in the contact boundary area are quite different, this case uses the Region Growing algorithm to further process the roughly segmented point cloud clusters. Since the region growing algorithm is a known technology in the field, it will not be described here. The fine segmentation results are as follows: Figure 6 As shown in the figure, we can obtain the target point cloud data of 8 alien objects that are close to complete, thus completing the entire fusion segmentation process.

[0049] Compared to other 3D segmentation neural networks in the field, such as PointNet++, this recognition method performs planar analysis on a 2D image and then back-projects it into the segmented scene based on the camera model. This method achieves semantic segmentation of a 3D scene with only two-dimensional computational effort. This significantly reduces computational effort, training costs, and storage space usage, resulting in higher efficiency and no significant loss in accuracy. Furthermore, due to the use of region growing and a random sampling strategy for center boxes, this method achieves higher segmentation accuracy with lower time complexity.

[0050] On the other hand, although the aforementioned fusion segmentation process can already identify various irregular objects in the target scene, allowing the conventional sorting device to complete the sorting process by grasping the corresponding irregular objects according to instructions, the inventors did not stop there. In some scenarios, if stacked irregular objects are grasped in the wrong order, grasping the irregular objects on the bottom first can easily cause the objects on top to fall and be damaged.

[0051] To this end, the inventors further proposed a set of inference and prediction methods to identify the support relationships of various heterogeneous objects in the target scene, thereby controlling the sorting device to make inference predictions on the grasping order and preventing objects from falling during the grasping process as much as possible.

[0052] Therefore, in a preferred embodiment, Figure 7 As shown, the steps of the method for identifying stacked irregular objects also include: S4 using the OBB algorithm to establish a bounding box for the target point cloud data of each irregular object; using the separating axis theorem in the collision detection algorithm to determine whether the two objects are in contact, that is, to determine whether there is an overlapping area between the bounding boxes. If so, it means that there is contact, such as in Figure 8 As shown in , there are 6 pairs of contacting alien objects in the target scene.

[0053] After determining the contact relationships between the various objects, support determination can be performed on the six pairs of contacting objects to determine whether there is a support relationship between the contacting objects. For example, the inference framework in this embodiment can calculate the normal vectors of the matching surfaces of the bounding boxes of each object and distinguish between supporting objects and supported objects based on the orientation of these normal vectors.

[0054] In addition, if the support relationship is to be used in the sorting process, the target relationship tree under the scenario can be further output, such as Figure 9 shown.

[0055] The relationship tree allows us to understand the support relationships between the various objects in the scene, such as the parent node being the supporting object and the child node being the supported object. This allows us to generate a grasping order strategy from the child node to the parent node, clarifying the grasping order between the eight objects in the scene.

[0056] Specifically, to achieve orderly grasping, it's necessary to predict the support relationships between the eight odd-shaped objects in the example above and estimate the grasping pose for each object. Therefore, the inference prediction method primarily determines the support relationships of stacked objects, deriving the relationship between the supporting and supported objects in this scenario and outputting a safe and orderly grasping strategy.

[0057] Therefore, the present invention, based on the top-down grasping principle and considering production efficiency, proposes a reasoning method based on spatial vertical components. By integrating segmentation results, the oriented bounding box (OBB) algorithm, commonly used in collision detection, is used to construct 3D bounding boxes for objects. This simplifies problem analysis, allowing for the determination of contact relationships between objects based on the separating axis theorem, and the identification of support semantics between objects, such as support and proximity, based on a vertical component reasoning strategy.

[0058] like Figure 10 As shown in the figure, in order to describe the support relationship of various objects, this case divides the possible spatial relationships between objects into three types: (a) surface-to-surface contact; (b) line-to-surface contact; (c) line-to-line contact; Note: When multiple objects in the scene support the same object, this case calculates the convex hull of the overlapping areas involved in the supporting objects. Thus, the support area of ​​the object is composed of the convex hulls involved, which project all contact surfaces onto the ground. The circle represents the contact surface, and the mask on the ground represents the support area.

[0059] After establishing and simplifying the mathematical model of the stacking scenario, in order to achieve fast and robust reasoning about the support relationship of the alien objects, this case proposed a relationship reasoning method based on the vertical component of the normal vector of the vector summation. Simply put, there must be a contact surface between all rigid objects. This case simplifies the judgment of whether it is a support or a supported object through the normal vector relationship of all contact surfaces. The judgment standard is the direction of the sum of the normal vectors on all contact surfaces of the object. The object facing upward is the support, and the object facing downward is the supported object.

[0060] Specifically, through the 3D bounding box of each object obtained, according to the OBB algorithm, we know that the 3D bounding boxes established by all objects must have overlapping areas. Therefore, the relationship points in the overlapping areas are analyzed to find the overlapping point clouds of any two bounding boxes; then the overlapping point clouds are matched with the faces of the two bounding boxes; the normal vectors of all matching faces are solved, and the support relationship between objects can be determined based on the criterion that the normal vectors of all supporting faces of all supported objects are directed downward. The support relationship determination diagram is shown as follows: Figure 11 As shown, the left side shows the complete surface contact support. After matching the stacked point cloud with all surfaces of the two 3D bounding boxes, the normal vector of the supporting surface of the supported object faces downward. Figure 11 Find the relevant supporting surface on the right side in the same way. If the sum of its normal vectors is obliquely downward, then the object can be determined to be the supported object.

[0061] Afterwards, if the support relationship is to be used in existing sorting solutions in the same field, the basic child-parent relationship results can be obtained by further establishing a visual reasoning tree in the stacking scenario. Finally, the false support can be deleted using stability reasoning to generate the operation relationship tree in the scenario, and the object grasping sequence strategy in the target scenario can be output. This can ensure that the stacking stability is maintained when sorting and grasping objects, avoiding the occurrence of situations such as objects falling, so that sorting equipment such as parallel robot systems can still complete fast and efficient grasping even when the objects are stacked in a relatively messy manner.

[0062] While the inference and prediction method proposed in this case is less versatile than current mainstream neural networks, its concept is simpler and more applicable to specific industrial scenarios, meeting the real-time and reliability requirements of industry. Compared to current model-based inference methods, this method offers a simpler design concept and lower algorithmic space complexity in terms of code implementation and execution. (two)

[0064] On the other hand, the present invention also provides a sorting workstation, which includes: a parallel robot, a visual reasoning system, wherein the visual reasoning system includes: an acquisition device, a processing unit, and a storage unit, wherein the storage unit stores the method program for identifying stacked irregular objects in the above-mentioned embodiment 1, so that the processing unit can retrieve it and execute the relevant steps of the method, such as Figure 12 As shown, the processing unit instructs the acquisition device to acquire image data, depth map data and point cloud data of the stacked foreign-shaped objects in the target scene and send them to the processing unit. In this embodiment, the lower cover acquisition device preferably includes a depth camera.

[0065] Furthermore, the processing unit retrieves the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the recognition frame data of each alien object; corrects the coordinates of the image data based on the depth map data, and then corrects the coordinates of the recognition frame data of each alien object in the target scene; the processing unit uses the optical center of the acquisition device as the vertex and, according to the pinhole imaging rule, back-projects the recognition frame of each alien object to establish a conical space corresponding to each alien object, to roughly segment the point cloud data, and obtain the point cloud cluster corresponding to each alien object.

[0066] Furthermore, the processing unit calls the region growing algorithm in the storage unit to perform fine segmentation on each point cloud cluster to obtain target point cloud data of each irregular object.

[0067] Furthermore, the processing unit calls the OBB algorithm in the storage unit to establish a bounding box for the target point cloud data of each foreign-shaped object, and determines whether there is an overlapping area between the bounding boxes. If so, it further performs support judgment to distinguish the supporting and supported objects according to the orientation of the normal vector of each bounding box, so as to output a target relationship tree of the foreign-shaped objects stacked in the target scene, and accordingly generates a grasping sequence strategy according to the parent / child nodes of the target relationship tree; the parallel robot executes the grasping sequence strategy to sort the foreign-shaped objects in the target scene.

[0068] On the other hand, in a preferred embodiment, in order to ensure efficient and energy-saving sorting of the sorting workstation, in this embodiment, the sorting workstation of the present invention also includes: a loop detector to reposition the target scene to determine whether the scene is complete by detecting whether the spatial relationship between the depth camera and the object has changed.

[0069] In this embodiment, the sorting workstation uses RGB-D images to construct a semantic graph, thus transforming the relocalization problem into a graph matching problem. After each grasping action is completed, the hand-eye manipulator returns to its initial position for relocalization. If the next grasped object is successfully relocalized, the next grasping task proceeds directly. Otherwise, recognition and grasping pose estimation are repeated.

[0070] On the other hand, the unique nature of irregular objects also poses significant challenges to the grasping of the sorting workstation. Therefore, in the preferred embodiment, the GPD (GraspPose Detection) algorithm, which has a good implementation effect, is used to generate the grasping pose of the target object, targeting the specificity of irregular objects. This algorithm can more completely determine the most suitable robot grasping pose. At the same time, due to the results of fusion segmentation, the GPD computing processing load is reduced, providing a guarantee for efficient grasping at the workstation. Based on the output results of the inference prediction method, the grasping poses of the grasped target are sequentially generated. By analyzing the geometric features of the target point cloud, the grasping pose with the most stable support is selected, thereby improving the success rate of grasping.

[0071] Traditional GPD architectures, however, do not pre-segment the target scene's point cloud. Instead, they directly select appropriate grasping points from point cloud information acquired from a single perspective. This results in grasping points that are easily located on non-target objects or even unclawable objects in the environment. Therefore, this solution significantly reduces the GPD algorithm's runtime by integrating segmentation algorithms, enabling GPD to better avoid misjudging these redundant objects during processing.

[0072] The method for identifying stacked foreign-shaped objects and the sorting workstation thereof provided by the present invention can effectively and accurately identify and segment each foreign-shaped object in a target scene, and in a preferred embodiment can also accurately determine the relative posture of each foreign-shaped object in the target scene with respect to other foreign-shaped objects in contact with it, thereby providing a grasping sequence strategy for subsequent sorting procedures, and further realizing rapid grasping and accurate sorting of foreign-shaped targets through servo control of the parallel robot.

[0073] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made based on the contents of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0074] Those skilled in the art will understand that, in addition to implementing the system, device, and various modules provided by the present invention in pure computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0075] In addition, all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a single-chip microcomputer, chip, or processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0076] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for identifying stacked irregular objects, characterized in that The steps include: S1 obtains image data and point cloud data of stacked foreign-shaped objects in a target scene collected by a collection device; Input the image data into the trained target recognition algorithm to obtain the recognition frame data of each alien object; S2 takes the optical center of the acquisition device as the vertex and back-projects the identification frames of the various shaped objects according to the pinhole imaging rule to establish a cone space corresponding to each shaped object, to roughly segment the point cloud data and obtain the point cloud clusters corresponding to each shaped object; S3 uses a region growing algorithm to finely segment the point cloud clusters obtained in step S2 to obtain target point cloud data that is close to complete for each heterogeneous object; S4 uses the OBB algorithm to establish bounding boxes for the target point cloud data of each irregular object; it determines whether there are overlapping areas among the bounding boxes. If so, it further performs support judgment to distinguish the supporting and supported objects according to the direction of the normal vectors of each bounding box. The judgment standard is to determine the direction of the sum of the normal vectors on all contact surfaces of the object. The one facing upward is the supporting object, and the one facing downward is the supported object.

2. The method for identifying stacked irregular objects according to claim 1, characterized in that: In step S1, the acquisition device also acquires depth map data of the foreign-shaped objects stacked in the target scene; corrects the coordinates of the image data according to the depth map data, and further corrects the coordinates of each foreign-shaped object identification frame data in the target scene.

3. A sorting workstation, characterized in that include: A parallel robot and a visual reasoning system, wherein the visual reasoning system comprises: a collection device, a processing unit, and a storage unit, wherein the collection device collects image data and point cloud data of stacked heterogeneous objects in a target scene and sends them to the processing unit; The processing unit retrieves the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the recognition frame data of each shaped object; The processing unit takes the optical center of the acquisition device as the vertex and back-projects the identification frames of each shaped object according to the pinhole imaging rule to establish a cone space corresponding to each shaped object, so as to roughly segment the point cloud data and obtain the point cloud clusters corresponding to each shaped object; then the regional growing algorithm in the storage unit is called to perform fine segmentation on each point cloud cluster to obtain the target point cloud data of each shaped object; The processing unit calls the OBB algorithm in the storage unit to establish bounding boxes for the target point cloud data of each heteromorphic object, and determines whether there are overlapping areas between the bounding boxes. If yes, further support judgment is performed to distinguish the supporting and supported objects according to the orientation of the normal vectors of each bounding box. The judgment standard is to determine the orientation of the sum of the normal vectors on all contact surfaces of the object. The object facing upward is the supporting object, and the object facing downward is the supported object. In this way, a target relationship tree of heteromorphic objects stacked in the target scene is output, and a grasping order strategy is generated according to the parent / child nodes of the target relationship tree. The parallel robot executes the grasping sequence strategy to sort the foreign objects in the target scene.

4. A sorting workstation, characterized in that include: A parallel robot and a visual reasoning system, wherein the visual reasoning system comprises: a collection device, a processing unit, and a storage unit, wherein the collection device collects image data, depth map data, and point cloud data of stacked foreign-shaped objects in a target scene and sends them to the processing unit; The processing unit retrieves the trained target recognition algorithm in the storage unit, inputs the image data into the calculation, and obtains the recognition frame data of each alien object; the coordinates of the image data are corrected according to the depth map data, and then the coordinates of the recognition frame data of each alien object in the target scene are corrected; The processing unit takes the optical center of the acquisition device as the vertex and back-projects the identification frames of each shaped object according to the pinhole imaging rule to establish a cone space corresponding to each shaped object, so as to roughly segment the point cloud data and obtain the point cloud clusters corresponding to each shaped object; then the regional growing algorithm in the storage unit is called to perform fine segmentation on each point cloud cluster to obtain the target point cloud data of each shaped object; The processing unit calls the OBB algorithm in the storage unit to establish bounding boxes for the target point cloud data of each heteromorphic object, and determines whether there are overlapping areas between the bounding boxes. If yes, further support judgment is performed to distinguish the supporting and supported objects according to the orientation of the normal vectors of each bounding box. The judgment standard is to determine the orientation of the sum of the normal vectors on all contact surfaces of the object. The object facing upward is the supporting object, and the object facing downward is the supported object. In this way, a target relationship tree of heteromorphic objects stacked in the target scene is output, and a grasping order strategy is generated according to the parent / child nodes of the target relationship tree. The parallel robot executes the grasping sequence strategy to sort the foreign objects in the target scene.

Images