Sorting method based on visual recognition, intelligent sorting system and readable storage medium

Abstract

The present application relates to the technical field of intelligent sorting robots, and in particular to a sorting method based on visual recognition, an intelligent sorting system, and a computer-readable storage medium. The sorting method comprises: obtaining a two-dimensional image of all sorting targets before the last sorting operation in the sorting scene, and a two-dimensional image of the remaining sorting targets after the sorting operation; performing a similarity comparison between the two-dimensional image of the remaining sorting targets after the sorting operation and the two-dimensional image of all sorting targets before the last sorting operation to determine whether the actual position of the remaining sorting targets has changed; if it is determined that the actual position of at least part of the remaining sorting targets has not changed, extracting the current posture of the remaining sorting targets whose actual position has not changed, and then controlling the intelligent sorting robot to perform this sorting operation on the remaining sorting targets whose actual position has not changed according to the preset grasping strategy and the current posture. The present application reduces the workload of calculating the overall posture of the sorting targets, and the sorting efficiency is higher.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent sorting robots, and in particular to a sorting method based on visual recognition, an intelligent sorting system and a computer-readable storage medium. Background Art

[0002] For workpiece sorting in stacking scenarios, 3D sensors, such as depth cameras, are usually used to obtain three-dimensional data of each workpiece (including depth images and point clouds), and then the spatial pose of each workpiece is calculated through a pose matching algorithm (6DoF, 6 Degrees of Freedom, representing the degrees of freedom of movement along the three rectangular coordinate axes x, y, and z and the degrees of freedom of rotation around these three coordinate axes). The graspable workpiece is then determined through the spatial pose of each workpiece, thereby controlling the intelligent sorting robot to grasp the workpiece.

[0003] The inventors of the present application have discovered through long-term research that in the prior art, a depth camera usually only performs a grasping operation once each time the spatial position of each workpiece is photographed and identified. Due to the uncertainty in the grasping process, the state of the remaining objects may change. Therefore, after each workpiece is grasped, it is necessary to re-photograph and identify, and then calculate the spatial position of each workpiece and perform a sorting operation, resulting in low sorting efficiency. Summary of the invention

[0004] The purpose of this application is to provide a sorting method based on visual recognition, an intelligent sorting system and a computer-readable storage medium to solve the problem of low sorting efficiency in the prior art that each workpiece needs to be photographed and identified again after it is grabbed, and then the spatial posture of each workpiece is calculated and the sorting operation is performed.

[0005] In order to achieve the above-mentioned purpose, an embodiment of the present application provides a sorting method based on visual recognition, comprising:

[0006] Acquire two-dimensional images of all sorting targets before the last sorting operation in the scene to be sorted, and two-dimensional images of the remaining sorting targets after the sorting operation;

[0007] Comparing the two-dimensional images of the remaining sorting targets after the sorting operation with the two-dimensional images of all sorting targets before the last sorting operation for similarity, so as to determine whether the actual positions of the remaining sorting targets have changed;

[0008] If it is determined that the actual positions of at least some of the remaining sorting targets have not changed, the current postures of the remaining sorting targets whose actual positions have not changed are extracted, and then according to the preset grasping strategy and the current postures of the remaining sorting targets, the intelligent sorting robot is controlled to perform the current sorting operation on the remaining sorting targets whose actual positions have not changed; wherein the current posture of the remaining sorting targets is the most recent historical posture of the remaining sorting targets without posture update.

[0009] As an improvement of the above scheme, it also includes:

[0010] If it is determined that the actual positions of all the remaining sorting targets have changed, the current postures of all the remaining sorting targets are recalculated to update the historical postures of the remaining sorting targets, and then the intelligent sorting robot is controlled to perform the current sorting operation on the remaining sorting targets according to the preset grasping strategy and the recalculated current postures of the remaining sorting targets.

[0011] As an improvement of the above scheme, it also includes:

[0012] The recalculating the current postures of all the remaining sorting targets includes:

[0013] Acquire three-dimensional images of all the remaining sorting objects in the scene to be sorted;

[0014] Based on a pose estimation algorithm and three-dimensional images of all the remaining sorting targets, the poses of all the remaining sorting targets are calculated;

[0015] Based on the posture optimization algorithm, the postures of all the remaining sorting targets are optimized to obtain the current postures of the remaining sorting targets.

[0016] As an improvement of the above scheme, it also includes:

[0017] Obtaining the most recent historical posture of all sorting targets in the to-be-sorted scene without posture update;

[0018] Based on the most recent historical posture of all sorting targets without posture update, at least determining a region of interest corresponding to the historical posture of each of the remaining sorting targets;

[0019] Extracting a first sub-image corresponding to the region of interest of each of the remaining sorting objects from the two-dimensional images of all sorting objects before the last sorting operation;

[0020] Then, the two-dimensional images of the remaining sorting targets after the sorting operation are compared with the two-dimensional images of all sorting targets before the last sorting operation for similarity to determine whether the actual positions of the remaining sorting targets have changed, including:

[0021] Extracting, from the two-dimensional image of the remaining sorting objects after the sorting operation, a second sub-image corresponding to the region of interest of each of the remaining sorting objects;

[0022] Comparing the second sub-image corresponding to each of the remaining sorting targets with the first sub-image for similarity;

[0023] If the similarity between the second sub-image of at least part of the remaining sorting objects and the first sub-image is greater than or equal to a similarity threshold, determining that the actual positions of at least part of the remaining sorting objects are not changed;

[0024] If the similarities between the second sub-image and the first sub-image of all the remaining sorting objects are less than the similarity threshold, it is determined that the actual positions of all the remaining sorting objects are changed.

[0025] As an improvement of the above scheme, it also includes:

[0026] Obtaining a template bounding box of a sorting target in the scene to be sorted;

[0027] Based on the most recent historical posture of all sorting targets without posture update, at least an area of ​​interest corresponding to the historical posture of each of the remaining sorting targets is determined, including:

[0028] The template bounding box is framed one by one according to the most recent historical pose of each of the remaining sorting targets without pose update, and then the vertex coordinates of the template bounding box in the camera coordinate system are recorded during each frame selection process;

[0029] Convert the vertex coordinates of the template bounding box recorded in each frame selection in the camera coordinate system into vertex coordinates in the pixel coordinate system;

[0030] In the image coordinate system, construct a minimum bounding rectangle that contains the coordinates of all vertices of the template bounding box, and then determine the minimum bounding rectangle as the region of interest corresponding to the historical pose of the remaining sorting target;

[0031] Repeat all the above operations until the regions of interest corresponding to the historical positions of all the remaining sorting targets are determined.

[0032] As an improvement of the above solution, the step of comparing the second sub-image corresponding to each of the remaining sorting targets with the first sub-image for similarity includes:

[0033] Determining the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects based on a structural similarity algorithm;

[0034] It is determined whether the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects is greater than or equal to the similarity threshold.

[0035] As an improvement of the above solution, the preset grabbing strategy includes:

[0036] Determine whether the remaining sorting objects in the to-be-sorted scene are stacked;

[0037] If there is no stacking, calculating the confidence corresponding to the current posture of each of the remaining sorting targets;

[0038] Grab the remaining sorting targets in order from high to low confidence levels;

[0039] If there is stacking, extracting the vertical height value of each of the remaining sorting targets in the current posture;

[0040] The remaining sorting objects are grabbed in order from high to low according to the vertical height values.

[0041] The present application provides a visual recognition-based sorting method, comprising:

[0042] Obtaining initial three-dimensional images of all sorting objects in the scene to be sorted;

[0043] By using the initial three-dimensional images of all the sorting targets, the initial positions and poses of all the sorting targets are calculated respectively, and the initial positions and poses of all the sorting targets are used as the first historical positions and poses of all the sorting targets;

[0044] Controlling the intelligent sorting robot to grasp at least part of the sorting targets according to a preset grasping strategy, and placing at least part of the sorting targets at a target position to perform a first sorting operation;

[0045] According to the visual recognition-based sorting method as described in any of the above embodiments, determining whether the actual position of the remaining sorting targets in the to-be-sorted scene has changed, and performing the current sorting operation based on the determination result;

[0046] Repeat the last step until all sorting targets in the to-be-sorted scene are sorted.

[0047] An intelligent sorting system provided in an embodiment of the present application includes:

[0048] Intelligent sorting robot;

[0049] A camera, arranged on the intelligent sorting robot, for collecting two-dimensional images and three-dimensional images of all sorting objects in the scene to be sorted;

[0050] One or more processors, connected to the intelligent sorting robot and the camera respectively;

[0051] A memory, coupled to the processor, for storing one or more programs;

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the visual recognition-based sorting method as described in any of the above embodiments.

[0053] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the visual recognition-based sorting method as described in any of the above embodiments is implemented.

[0054] Compared with the prior art, the visual recognition-based sorting method, intelligent sorting system, and computer-readable storage medium in the embodiments of the present application have the following beneficial effects:

[0055] By obtaining the two-dimensional images of all sorting targets before the last sorting operation in the scene to be sorted, and the two-dimensional images of the remaining sorting targets after the sorting operation, and then comparing the two-dimensional images of the remaining sorting targets after the sorting operation with the two-dimensional images of all sorting targets before the last sorting operation for similarity, it is determined whether the actual position of the remaining sorting targets has changed. If not, the historical posture of the remaining sorting targets that have not been updated in the most recent time is extracted as the current posture of the remaining sorting targets, and then the remaining sorting targets are sorted based on the current posture of the remaining sorting targets and the preset grasping strategy. If changed, the current posture of the remaining sorting targets is recalculated. In this way, the present application reduces the workload of calculating the posture of the sorting targets as a whole, and the sorting efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present application will illustrate the implementation methods in conjunction with the accompanying drawings. The accompanying drawings of the present application are only used to describe the embodiments for the purpose of demonstration. Without departing from the principles of the present application, those skilled in the art can easily make other embodiments according to the steps described below through the following description.

[0057] Figure 1 It is a schematic diagram of a scene in which an intelligent sorting system in an embodiment of the present application sorts the remaining sorting targets;

[0058] Figure 2 is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0059] Figure 3 is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0060] Figure 4is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0061] Figure 5 is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0062] Figure 6 is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0063] Figure 7 is a flow chart of a sorting method based on visual recognition in an embodiment of the present application;

[0064] Figure 8 It is a schematic diagram of using a template boundary box to select each remaining sorting target in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0066] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] Some technical terms involved in this article are explained below to facilitate understanding by those skilled in the art.

[0069] Three-dimensional image: is a special form of information expression, characterized by the three-dimensional data in the expressed space, including: depth image, geometric model (created by CAD software) and point cloud model.

[0070] Depth image: also called distance image, refers to an image that uses the distance (depth) value from the image collector to each point in the scene as the pixel value. The acquisition methods include: lidar depth imaging, computer stereo vision imaging, coordinate measuring machine method, moiré fringe method and structured light method.

[0071] Point cloud: When a laser beam hits the surface of an object, the reflected laser will carry information such as direction and distance. If the laser beam is scanned along a certain trajectory, the reflected laser point information will be recorded while scanning. Since the scanning is extremely precise, a large number of laser points can be obtained, thus forming a laser point cloud. The depth image can be calculated into point cloud data after coordinate conversion; point cloud data with rules and necessary information can be inversely calculated into a depth image.

[0072] Pose: In Simultaneous Localization and Mapping (SLAM), pose is the transformation from the world coordinate system to the camera coordinate system, including rotation and translation.

[0073] Pose estimation of sorting targets: The position and posture information of sorting targets are estimated by using the correspondence between the image pixels in the camera coordinate system and the three-dimensional space object points of the sorting targets in the world coordinate system.

[0074] See also Figure 1 An intelligent sorting system 100 provided in an embodiment of the present application includes an intelligent sorting robot 10, a camera 20, one or more processors, and a memory.

[0075] The camera 20 is arranged on the intelligent sorting robot 10 and is used to collect two-dimensional images and three-dimensional images of all sorting objects in the scene to be sorted.

[0076] One or more processors are connected to the intelligent sorting robot 10 and the camera 20, respectively. A memory is coupled to the processor and is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the visual recognition-based sorting method as described in any of the above embodiments.

[0077] The processor is used to control the overall operation of the intelligent sorting system 100 to complete all or part of the steps of the above-mentioned visual recognition-based sorting method. The memory is used to store various types of data to support the operation of the intelligent sorting system 100, and these data may include, for example, instructions for any application or method operating on the intelligent sorting system 100, and application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0078] In one embodiment, see Figure 1 The intelligent sorting robot 10 includes a driving component and a mechanical arm 11. The driving component is connected to the mechanical arm 11 for driving and is connected to the processor for communication. The camera 20 is arranged on the mechanical arm 11.

[0079] In one embodiment, the camera 20 includes an RGB-D camera, which can capture a point cloud and a two-dimensional color image of the sorting target.

[0080] In one embodiment, the same processor can be connected to the intelligent sorting robot 10 and the camera 20 respectively, process the two-dimensional images and three-dimensional images of all sorting targets in the sorting scene to be sorted acquired by the camera 20, and then process the acquired results, and generate corresponding control commands according to the acquired results, so as to control the intelligent sorting robot 10 to perform corresponding sorting operations.

[0081] In another embodiment, a processor may be connected to the intelligent sorting robot 10 to control the intelligent sorting robot 10 to perform sorting operations, and another processor may process the two-dimensional images and three-dimensional images of all sorting targets in the scene to be sorted acquired by the camera 20, and then process the acquired results, and generate corresponding control commands based on the acquired results and send them to the processor of the intelligent sorting robot 10, thereby controlling the intelligent sorting robot 10 to perform corresponding sorting operations.

[0082] See also Figure 2 , a sorting method based on visual recognition provided by an embodiment of the present application comprises the following steps:

[0083] S110, obtaining an initial three-dimensional image of all sorting objects in the scene to be sorted;

[0084] S120, calculating the initial postures of all the sorting targets respectively through the initial three-dimensional images of all the sorting targets, and taking the initial postures of all the sorting targets as the first historical postures of all the sorting targets;

[0085] S130, controlling the intelligent sorting robot 10 to grasp at least part of the sorting targets according to a preset grasping strategy, and placing at least part of the sorting targets at a target position to perform a first sorting operation;

[0086] S140, in accordance with this application Figures 3 to 7 The visual recognition-based sorting method described in any embodiment of the embodiment determines whether the actual position of the remaining sorting target 200 in the to-be-sorted scene has changed, and performs the current sorting operation based on the determination result;

[0087] S150, repeating the above step S140 until all sorting objects in the scene to be sorted are sorted.

[0088] In the scene to be sorted, multiple sorting targets can run to the sorting station with the conveyor belt or other transport structure of the intelligent sorting system 100, and then the conveyor belt or other transport structure stops running. At this time, multiple sorting targets can be placed in sequence and in disorder, or they can be stacked and placed in disorder. Each time a sorting operation is performed, the intelligent sorting robot 10 of the intelligent sorting system 100 can move at least part of the sorting targets to the target position. If the intelligent sorting system 100 determines that there are remaining sorting targets 200 in the scene to be sorted, the above step S140 is repeated; if it is determined that there are no remaining sorting targets 200 in the scene to be sorted, the sorting operation of the scene to be sorted can be ended.

[0089] Before performing the first sorting operation, it is necessary to estimate the pose of all sorting targets in the sorting scene.

[0090] Specifically, by obtaining the initial three-dimensional images of all sorting objects in the scene to be sorted, and then calculating the initial poses of all sorting objects through the pose estimation algorithm, the initial poses are saved as the first historical poses of all sorting objects. In the subsequent sorting operation process, the historical poses can be updated in real time or as needed.

[0091] After obtaining the initial positions of all sorting targets, the intelligent sorting robot 10 grabs part or all of the sorting targets according to the initial positions and the preset grabbing strategy, and then places the sorting targets at the target positions, thereby realizing the first sorting operation.

[0092] In one embodiment, the pose estimation algorithm includes a model matching algorithm based on point pair features (PPF), a 6D pose estimation algorithm based on a deep convolutional neural network (CNN), a template matching algorithm, a key point detection algorithm, etc.

[0093] The core idea of ​​the PPF algorithm is overall modeling and local matching. By constructing the four-dimensional features formed by the geometric relationship between two points and their normal vectors as point pair features, a hash table is established offline to store all the four-dimensional features of the model as the overall description of the model. In the online matching stage, the global coordinate system is used to simplify the rigid body transformation degrees of freedom, making the pose calculation simple. At the same time, with the voting idea of ​​the generalized Hough transform, the parameter space is voted to obtain a reliable pose.

[0094] It can be understood that the pose estimation algorithm is a common method for calculating the pose of an object. For details, please refer to the description of the algorithm in the existing literature, and this application will not go into details here.

[0095] In one embodiment, the calculated initial pose may be iteratively optimized using a pose optimization algorithm.

[0096] The pose optimization algorithm includes the iterative closest point (ICP) algorithm.

[0097] Among them, the ICP algorithm is a method based on the EM (Expectation-maximization algorithm) idea, which uses an alternating iterative method to optimize point cloud matching and optimize motion estimation.

[0098] It can be understood that the pose estimation algorithm is a common method for calculating the pose of an object. For details, please refer to the description of the algorithm in the existing literature, and this application will not go into details here.

[0099] In one embodiment, the sorting target may be a three-dimensional regular workpiece or an irregular workpiece.

[0100] In one embodiment, the three-dimensional image comprises a depth image or a point cloud.

[0101] In order to better understand the inventive concept of the present application, the following embodiments are described using a three-dimensional image as a point cloud as an example.

[0102] In one embodiment, the preset crawling strategy includes:

[0103] Determine whether the sorting objects in the to-be-sorted scene are stacked;

[0104] If there is no stacking, then calculating the confidence corresponding to the current posture of each of the sorting targets;

[0105] Grasping the sorting targets in order from high to low confidence levels;

[0106] If there is stacking, extracting the vertical height value of each of the sorting targets in the current posture;

[0107] The sorting objects are grasped in order from high to low according to the vertical height values.

[0108] This embodiment grabs the sorting targets according to the stacking situation. If it is determined that the sorting targets are not stacked, the corresponding confidence is calculated according to the current posture of each sorting target, and then the confidence is sorted, and some or all sorting targets are grabbed in order from high to low confidence. In this way, the intelligent sorting system 100 can identify the sorting targets that are not blocked and can be sorted through the current postures of all sorting targets, and then perform the sorting operation.

[0109] The confidence is used to characterize the probability of grasping each sorting target. In one embodiment, the confidence is normalized to [0, 1], where a value close to 1 indicates a higher grasping confidence, that is, a higher chance of successful grasping. In another embodiment, the confidence is normalized to [0, 100], where a value close to 100 indicates a higher grasping confidence, that is, a higher chance of successful grasping.

[0110] In one embodiment, the confidence can be calculated by an existing confidence calculation method. For example, taking the PPF algorithm as an example, the confidence is a parameter of the PPF algorithm. The posture of each sorting target estimated by the algorithm will have a score. In this way, after the posture of each sorting target is calculated by the PPF algorithm, the confidence corresponding to each sorting target can also be obtained.

[0111] In addition, other confidence calculation methods can be specifically referred to the description of the algorithm in existing literature, and this application will not go into details here.

[0112] If it is determined that the sorting targets are stacked, the sorting targets at the higher positions in the stacked sorting targets are grasped first, that is, the vertical height values ​​in the current posture are sorted, and some or all of the sorting targets are grasped in descending order of the vertical height values.

[0113] In other embodiments, if it is determined that some sorting targets are stacked and some sorting targets are not stacked, the confidence grasping method and the vertical height value grasping method can be used in combination to sort the sorting targets in the sorting scene. For example, for stacked sorting targets, the vertical height value grasping confidence grasping method is used preferentially, and for non-stacked sorting targets, the confidence grasping method is used preferentially.

[0114] See also Figure 1 and Figure 3 , a sorting method based on visual recognition provided by an embodiment of the present application comprises the following steps:

[0115] S210, acquiring two-dimensional images of all sorting objects before the last sorting operation in the sorting scene, and two-dimensional images of the remaining sorting objects 200 after the sorting operation.

[0116] In this embodiment, each time a sorting operation is performed in the to-be-sorted scene, two-dimensional images of sorting targets before and after the sorting operation are acquired. Specifically, not only the two-dimensional images of all sorting targets before the last sorting operation need to be acquired, but also the two-dimensional images of the remaining sorting targets 200 after the sorting operation need to be acquired.

[0117] In one embodiment, the camera 20 may collect two-dimensional images of all sorting objects before and after each sorting operation of the scene to be sorted.

[0118] In one embodiment, the two-dimensional image comprises a visible light image or an infrared image.

[0119] It can be understood that each time before performing a sorting operation, it is necessary to collect two-dimensional images of all sorting objects in the scene to be sorted, and the collected two-dimensional images are stored in the memory. Before performing the next sorting operation, the two-dimensional image before the corresponding sorting operation can be retrieved from the memory, and at the same time, the two-dimensional image of the remaining sorting objects 200 in the current scene to be sorted is obtained through the camera 20.

[0120] S220: performing a similarity comparison between the two-dimensional images of the remaining sorting objects 200 after the sorting operation and the two-dimensional images of all sorting objects before the last sorting operation to determine whether the actual positions of the remaining sorting objects 200 are changed.

[0121] This embodiment compares the similarity of the two-dimensional images of the sorting targets before and after the sorting operation, so as to determine whether the sorting operation has an impact on the remaining sorting targets 200 after each sorting operation, that is, to determine whether the actual positions of the remaining sorting targets 200 are changed.

[0122] S230: If it is determined that the actual positions of at least part of the remaining sorting targets 200 have not changed, the current positions of the remaining sorting targets 200 whose actual positions have not changed are extracted, and then according to the preset grasping strategy and the current positions of the remaining sorting targets 200, the intelligent sorting robot 10 is controlled to perform the current sorting operation on the remaining sorting targets 200 whose actual positions have not changed; wherein the current positions of the remaining sorting targets 200 are the most recent historical positions of the remaining sorting targets 200 without position update.

[0123] For all the remaining sorting targets 200, if the actual positions of all the remaining sorting targets 200 have not changed, it means that the current postures of all the remaining sorting targets 200 have not changed. By extracting the current postures of the remaining sorting targets 200 whose actual positions have not changed, and then according to the current postures and the preset sorting strategies, the intelligent sorting robot 10 is controlled to perform sorting operations on all the remaining sorting targets 200.

[0124] If the actual positions of some of the remaining sorting targets 200 have not changed, it means that the current postures of these remaining sorting targets 200 have not changed. By extracting the current postures of the remaining sorting targets 200 whose actual positions have not changed, and then controlling the intelligent sorting robot 10 to perform sorting operations on these remaining sorting targets 200 according to the current postures and preset sorting strategies.

[0125] It can be understood that the actual position of the remaining sorting objects 200 changes includes a complete change or a relatively obvious change in the actual position of the remaining sorting objects 200. For example, for stacked sorting objects, after the sorting operation, some of the remaining sorting objects 200 drop from a high position to a low position, or for deformable sorting objects, after the upper sorting objects are removed, the lower sorting objects recover their deformation and increase in volume.

[0126] See also Figure 3 In one embodiment, the visual recognition-based sorting method further comprises the following steps:

[0127] S240: If it is determined that the actual positions of all the remaining sorting targets 200 have changed, the current postures of all the remaining sorting targets 200 are recalculated to update the historical postures of the remaining sorting targets 200, and then the intelligent sorting robot 10 is controlled to perform the current sorting operation on the remaining sorting targets 200 according to the preset grasping strategy and the recalculated current postures of the remaining sorting targets 200.

[0128] In this embodiment, if it is determined that the actual positions of all the remaining sorting targets 200 have changed, the original historical postures of the remaining sorting targets 200 are no longer applicable, and the current postures need to be recalculated and the historical postures of the remaining sorting targets 200 are updated. Then, the intelligent sorting system 100 performs sorting operations on these remaining sorting targets 200 according to the recalculated current postures and the preset grasping strategies.

[0129] After sorting once, the embodiment of the present application obtains a two-dimensional image of the remaining sorting target 200, and then compares the two-dimensional image with the last captured two-dimensional image of the sorting target to determine whether the posture of the remaining sorting target 200 has changed. If not, the historical posture of the remaining sorting target 200 that has not been updated in the most recent time is extracted as the current posture of the remaining sorting target 200, and then the remaining sorting target 200 is continued to be sorted based on the current posture of the remaining sorting target 200 and the preset grasping strategy. If it has changed, the current posture of the remaining sorting target 200 is recalculated.

[0130] It should be noted that similarity comparison requires comparison of two 2D images before and after the sorting operation, while extraction of the current posture of the remaining sorting target 200 is not affected by the number of intermediate sorting operations. Therefore, in this application, the last time is not equivalent to the most recent time.

[0131] Specifically, the most recent historical posture of the remaining sorting targets 200 without posture update means that the current posture of the remaining sorting targets 200 has not been recalculated to update its historical posture. Only when the actual positions of all the remaining sorting targets 200 have changed will the corresponding historical posture be updated. Whether the sorting operation is performed is not necessarily related to updating the historical posture.

[0132] For example, before performing the first sorting operation, the first historical poses of all sorting targets are determined.

[0133] If the actual positions of all the remaining sorting targets 200 are changed after the first sorting operation is performed, the current positions of all the remaining sorting targets 200 need to be recalculated to perform the second sorting operation, and the corresponding historical positions need to be updated.

[0134] If the actual positions of at least some of the remaining targets have not changed after the first sorting operation, the corresponding historical poses are extracted to perform the second sorting operation, and the original historical poses are not updated at this time. If the actual positions of all the remaining sorting targets 200 have changed after the second sorting operation, the current poses of all the remaining sorting targets 200 need to be recalculated to perform the third sorting operation, and the corresponding historical poses are updated. If the actual positions of at least some of the remaining targets have not changed after the second sorting operation, the corresponding historical poses are extracted to perform the third sorting operation, and the original historical poses are not updated at this time.

[0135] And so on, until all the sorting targets in the scene to be sorted are sorted.

[0136] It can be understood that the first historical posture and the subsequently updated historical posture can be stored in the memory. When it is necessary to extract the most recent historical posture of the remaining sorting target 200 that has not been updated, the corresponding historical posture can be retrieved from the memory.

[0137] See also Figure 4 In one embodiment, the current positions of all the remaining sorting targets 200 may be recalculated in the following manner.

[0138] Specifically, the recalculation of the current postures of all the remaining sorting objects 200 includes the following steps:

[0139] 310. Acquire three-dimensional images of all the remaining sorting objects 200 in the scene to be sorted;

[0140] 320. Calculate the poses of all the remaining sorting objects 200 based on the pose estimation algorithm and the three-dimensional images of all the remaining sorting objects 200;

[0141] 330. Based on a posture optimization algorithm, perform posture optimization on the postures of all the remaining sorting objects 200 to obtain current postures of the remaining sorting objects 200.

[0142] In one embodiment, the pose estimation algorithm includes a PPF algorithm, a CNN-based 6D pose estimation algorithm, a template matching algorithm, a key point detection algorithm, etc.

[0143] In one embodiment, the pose optimization algorithm includes an ICP algorithm.

[0144] It can be understood that the current posture of all the remaining sorting targets 200 recalculated in this embodiment is the same as the calculation method of the initial posture of all the sorting targets in the above embodiment. The detailed calculation method can be found in the above embodiment for calculating the initial posture of all the sorting targets, which will not be repeated here.

[0145] In one embodiment, the preset crawling strategy includes:

[0146] Determine whether the remaining sorting objects 200 in the to-be-sorted scene are stacked;

[0147] If there is no stacking, then calculating the confidence corresponding to the current posture of each of the remaining sorting objects 200;

[0148] Grab the remaining sorting objects 200 in order from high to low confidence levels;

[0149] If there is stacking, extracting the vertical height value of each of the remaining sorting objects 200 in the current posture;

[0150] The remaining sorting objects 200 are grabbed in sequence according to the vertical height values ​​from high to low.

[0151] In this embodiment, the remaining sorting objects 200 are grasped according to the stacking situation. If it is determined that the remaining sorting objects 200 are not stacked, the corresponding confidence is calculated according to the current posture of each remaining sorting object 200, and then the confidence is sorted, and part or all of the remaining sorting objects 200 are grasped in order from high to low confidence.

[0152] It can be understood that the preset grabbing strategy of this embodiment is the same as the preset grabbing strategy in the above embodiment. The detailed preset grabbing strategy can be found in the above embodiment and will not be described again here.

[0153] In summary, compared with the prior art that requires calculating the posture of the sorting target before each sorting operation, which has a large amount of calculation and low sorting efficiency, the embodiment of the present application obtains the two-dimensional images of all sorting targets before the last sorting operation in the sorting scene, and the two-dimensional images of the remaining sorting targets 200 after the sorting operation, and then compares the two-dimensional images of the remaining sorting targets 200 after the sorting operation with the two-dimensional images of all sorting targets before the last sorting operation for similarity to determine whether the actual position of the remaining sorting targets 200 has changed. If not, the historical posture of the remaining sorting targets 200 that has not been updated in the most recent time is extracted as the current posture of the remaining sorting targets 200, and then the remaining sorting targets 200 are sorted based on the current posture of the remaining sorting targets 200 and the preset grasping strategy. If changed, the current posture of the remaining sorting targets 200 is recalculated. In this way, the present application reduces the workload of calculating the posture of the sorting targets as a whole, and the sorting efficiency is higher.

[0154] See also Figure 5 In one embodiment, the visual recognition-based sorting method further comprises the following steps:

[0155] S250, obtaining the most recent historical posture of all sorting targets in the to-be-sorted scene without posture update;

[0156] S260, based on the most recent historical posture of all sorting targets without posture update, at least determining a region of interest corresponding to the historical posture of each of the remaining sorting targets 200;

[0157] S270: Extracting a first sub-image corresponding to the region of interest of each of the remaining sorting objects 200 from the two-dimensional images of all sorting objects before the last sorting operation.

[0158] Then, the step S220, comparing the two-dimensional images of the remaining sorting targets 200 after the sorting operation with the two-dimensional images of all sorting targets before the last sorting operation for similarity to determine whether the actual positions of the remaining sorting targets 200 have changed, includes the following sub-steps:

[0159] S221, extracting a second sub-image corresponding to the region of interest of each of the remaining sorting objects 200 from the two-dimensional image of the remaining sorting objects 200 after the sorting operation;

[0160] S222, comparing the second sub-image corresponding to each of the remaining sorting targets 200 with the first sub-image for similarity;

[0161] S223: If the similarity between the second sub-image of at least part of the remaining sorting objects 200 and the first sub-image is greater than or equal to a similarity threshold, it is determined that the actual positions of at least part of the remaining sorting objects 200 are not changed;

[0162] S224: If the similarities between the second sub-images and the first sub-images of all the remaining sorting objects 200 are less than the similarity threshold, it is determined that the actual positions of all the remaining sorting objects 200 are changed.

[0163] This embodiment determines the region of interest (ROI) of each remaining sorting target 200 based on the position and posture of each remaining sorting target 200, and then obtains the sub-image corresponding to the region of interest in the two-dimensional images before and after the sorting operation, and compares the similarity of the two sub-images to determine whether the actual position of the remaining sorting target 200 has changed. Compared with comparing the similarity of the entire two-dimensional image twice before and after the sorting operation, the amount of calculation is large and the comparison is not accurate enough. This example divides the entire two-dimensional image into one or more sub-images through the region of interest of each remaining sorting target 200, each sub-image corresponds to a remaining sorting target 200, and then compares the similarity of the sub-image corresponding to each remaining sorting target 200, which has a small amount of calculation and accurate comparison.

[0164] Specifically, the most recent historical postures of all sorting targets in the scene to be sorted in which the postures are not updated are first obtained, and then the historical postures are used to at least determine the region of interest corresponding to the historical posture of each remaining sorting target 200. It can be understood that each historical posture of the remaining sorting target 200 corresponds to a region of interest, so the number of the remaining sorting targets 200 in the scene to be sorted corresponds to the number of regions of interest.

[0165] After determining the region of interest of each remaining sorting target 200, the two-dimensional images of all sorting targets before the last sorting operation in the sorting scene are processed, so as to extract from the two-dimensional image the first sub-image corresponding to the region of interest of each remaining sorting target 200. It can be understood that the first sub-image can be a complete imaging pattern of each remaining sorting target 200, or an incomplete imaging pattern after the remaining sorting target 200 is blocked by other remaining sorting targets 200, and the number of first sub-images corresponds to the number of regions of interest.

[0166] Correspondingly, after determining the region of interest of each remaining sorting target 200, the two-dimensional image of the remaining sorting target 200 after the sorting operation also needs to be processed, so as to extract the second sub-image corresponding to the region of interest of each remaining sorting target 200 from the two-dimensional image. It can be understood that the second sub-image can be a complete imaging pattern of each remaining sorting target 200, or an incomplete imaging pattern after the remaining sorting target 200 is blocked by other remaining sorting targets 200, and the number of second sub-images corresponds to the number of regions of interest.

[0167] The second sub-image corresponding to each remaining sorting target 200 is compared with the first sub-image for similarity: if the similarity between the second sub-image of at least part of the remaining sorting targets 200 and the first sub-image is greater than or equal to the similarity threshold, it is determined that the actual positions of at least part of the remaining sorting targets 200 have not changed; if the similarity between the second sub-image of all the remaining sorting targets 200 and the first sub-image is less than the similarity threshold, it is determined that the actual positions of all the remaining sorting targets 200 have changed.

[0168] For example, see Figure 1 There are 5 remaining sorting targets 200 in the current to-be-sorted scene, and some of the 5 remaining sorting targets 200 are stacked and placed, while some are placed separately.

[0169] The second sub-image corresponding to the first remaining sorting target 200 is compared with the first sub-image for similarity: if the similarity between the second sub-image of the remaining sorting target 200 and the first sub-image is greater than or equal to the similarity threshold, it is determined that the actual position of the remaining sorting target 200 has not changed, and step S230 is subsequently performed; if the similarity between the second sub-image of the remaining sorting target 200 and the first sub-image is less than the similarity threshold, it is determined that the actual position of the remaining sorting target 200 has changed, and step S240 is subsequently performed.

[0170] The second sub-image corresponding to the second remaining sorting target 200 is compared with the first sub-image for similarity: if the similarity between the second sub-image of the remaining sorting target 200 and the first sub-image is greater than or equal to the similarity threshold, it is determined that the actual position of the remaining sorting target 200 has not changed, and step S230 is subsequently performed; if the similarity between the second sub-image of the remaining sorting target 200 and the first sub-image is less than the similarity threshold, it is determined that the actual position of the remaining sorting target 200 has changed, and step S240 is subsequently performed.

[0171] The same logic is repeated until the similarities between the second sub-images and the first sub-images corresponding to the five remaining sorting targets 200 are compared to obtain corresponding comparison results, and then step S230 or step S240 is executed according to the comparison results.

[0172] See also Figure 5 In a specific embodiment, the step S222 compares the second sub-image corresponding to each of the remaining sorting targets 200 with the first sub-image for similarity, and includes the following sub-steps:

[0173] S222a, determining the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects 200 based on a structural similarity algorithm;

[0174] S222b: Determine whether the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects 200 is greater than or equal to the similarity threshold.

[0175] This embodiment uses a structural similarity index method (SSIM) to compare the similarity between the second sub-image and the first sub-image corresponding to each remaining sorting target 200. Specifically, the similarity between the second sub-image and the first sub-image of each remaining sorting target 200 is first calculated by the SSIM algorithm, and then it is determined whether the corresponding similarity is greater than or equal to the similarity threshold, and then step S223 or S224 is executed according to the determination result.

[0176] It can be understood that the SSIM algorithm is a common existing image similarity comparison method. For details, please refer to the description of the algorithm in existing literature, and this application will not go into details here.

[0177] In other embodiments, other image similarity comparison methods may also be used to perform similarity comparison between the second sub-image and the first sub-image corresponding to each remaining sorting target 200, which is not specifically limited here.

[0178] It should be noted that steps S250 to S270 may be executed after step S210 and before step S220, or may be executed before step S210, and no specific limitation is made here.

[0179] See also Figure 7 and Figure 8 In one embodiment, the region of interest corresponding to each remaining sorting target 200 may be obtained in the following manner.

[0180] Specifically, the visual recognition-based sorting method further includes the following steps:

[0181] S280: Obtain a template boundary box 201 of the sorting target in the scene to be sorted.

[0182] In this embodiment, the shape of the template bounding box 201 (Bbox for short) may be consistent with the shape of the sorting target, and is used for subsequently identifying each remaining sorting target 200 in the three-dimensional image, so as to determine the region of interest of each remaining sorting target 200 .

[0183] Specifically, taking the three-dimensional image as a point cloud as an example, the current posture of each remaining sorting target 200 in the three-dimensional point cloud is identified through point cloud matching, and the template bounding box 201 is also a three-dimensional point cloud, which itself has 6D posture attributes and is composed of 8 three-dimensional vertices, and the relationship between the 8 three-dimensional vertices is known.

[0184] In other embodiments, the shape of the template bounding box 201 may also be a preset shape, having a 6D pose attribute and multiple feature points, and the relationship between the multiple feature points is also known.

[0185] Then, the step S260 determines at least the region of interest corresponding to the historical posture of each of the remaining sorting targets 200 based on the most recent historical posture of all sorting targets without posture update, and specifically includes the following sub-steps:

[0186] S261, the template bounding box 201 is framed one by one according to the historical posture of each of the remaining sorting targets 200 that has not been updated in the most recent posture, and the three-dimensional image corresponding to the historical posture of each of the remaining sorting targets 200 is framed, and then the vertex coordinates of the template bounding box 201 in the camera coordinate system during each frame selection process are recorded;

[0187] S262, converting the vertex coordinates of the template bounding box 201 recorded in each selection in the camera coordinate system into vertex coordinates in the pixel coordinate system;

[0188] S263, constructing a minimum bounding rectangle that contains all vertex coordinates of the template bounding box 201 in the image coordinate system, and then determining the minimum bounding rectangle as the region of interest corresponding to the historical posture of the remaining sorting target 200;

[0189] S264, repeat all the above operations until the regions of interest corresponding to the historical positions of all the remaining sorting objects 200 are determined.

[0190] Since the point cloud model formed by each remaining sorting target 200 may have complex texture information, the amount of calculation required to directly obtain the feature points of each remaining sorting target 200 through the point cloud model is large. To solve this problem, this embodiment selects the remaining sorting targets 200 one by one through the template bounding box 201, and can quickly determine the feature points of each remaining sorting target 200, and then determine the region of interest of each remaining sorting target 200 based on the minimum circumscribed rectangle method.

[0191] Specifically, when the template boundary is used to frame each remaining sorting target 200 in the point cloud one by one, since the relationship between the eight three-dimensional vertices of the template boundary box 201 is known, if the template boundary box 201 frames each remaining sorting target 200 one by one according to the current posture, it is actually to match each remaining sorting target 200, then the feature point of each remaining sorting target 200 can be converted into the eight three-dimensional vertices of the template boundary box 201, and then the coordinates of the feature points of the corresponding remaining sorting target 200 can be obtained by obtaining the coordinates of the eight three-dimensional vertices of the template boundary box 201. In this way, the amount of calculation for extracting the feature points of the remaining sorting targets 200 is reduced.

[0192] In one embodiment, Figure 8 As shown, the standard for the template bounding box 201 to select the remaining sorting targets 200 is: according to the most recent historical pose of the remaining sorting targets 200 without pose update, the 8 three-dimensional vertices of the template bounding box 201 can be sequentially converted to the position corresponding to each remaining sorting target 200 in the camera coordinate system, and the identified point cloud of each remaining sorting target 200 is wrapped in the 8 three-dimensional vertices of the template bounding box 201.

[0193] Then, the eight three-dimensional vertices of the template bounding box 201 in the camera coordinate system are mapped to the image coordinate system in the two-dimensional image through the camera intrinsic parameters of the camera 20, so as to obtain the pixel coordinate values ​​of the eight vertices, and then the minimum enclosing rectangle accommodating the eight vertices is constructed based on the minimum enclosing rectangle method and the pixel coordinate values ​​of the eight vertices, so as to determine the position and size of the region of interest corresponding to the historical posture of the remaining sorting target 200 in the two-dimensional image.

[0194] All the above operations are repeated until the regions of interest corresponding to the historical postures of all remaining sorting objects 200 are determined.

[0195] In summary, this embodiment uses a template bounding box 201 to select each remaining sorting target 200 based on the position and posture of each remaining sorting target 200, so as to obtain the region of interest corresponding to each remaining sorting target 200. Compared with the method of directly obtaining the feature points of each remaining sorting target 200 through a point cloud model, the amount of calculation is small. In addition, compared with the method of directly identifying each remaining sorting target 200 in a two-dimensional image and obtaining the corresponding region of interest, it is more difficult to directly identify each remaining sorting target 200 in a two-dimensional image, and it is also susceptible to the recognition influence caused by the stacking of the remaining sorting targets 200. In this embodiment, the template bounding box 201 is used to select each remaining sorting target 200. Since each remaining sorting target 200 has been identified when calculating the position and posture of each remaining sorting target 200, this method does not need to repeatedly identify each remaining sorting target 200 in the two-dimensional image, and it is also not easily affected by the recognition influence caused by the stacking of the remaining sorting targets 200, and the recognition accuracy is higher.

[0196] In other embodiments, the region of interest corresponding to each remaining sorting target 200 may also be acquired through a deep learning method.

[0197] In an exemplary embodiment, the present application further provides a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the visual recognition-based sorting method as described in any of the above embodiments. For example, the computer-readable storage medium may be the above-mentioned memory including the computer program, and the above-mentioned computer program may be executed by the processor of the intelligent sorting system 100 to complete the visual recognition-based sorting method as described in any of the above embodiments, and achieve the technical effect consistent with the above-mentioned method.

[0198] The above are only preferred implementations of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sorting method based on visual recognition, characterized in that: include: Acquire two-dimensional images of all sorting targets before the last sorting operation in the scene to be sorted, and two-dimensional images of the remaining sorting targets after the sorting operation; Comparing the two-dimensional images of the remaining sorting targets after the sorting operation with the two-dimensional images of all sorting targets before the last sorting operation for similarity, so as to determine whether the actual positions of the remaining sorting targets have changed; If it is determined that the actual positions of at least some of the remaining sorting targets have not changed, the current positions of the remaining sorting targets whose actual positions have not changed are extracted, and then the intelligent sorting robot is controlled to perform the current sorting operation on the remaining sorting targets whose actual positions have not changed according to the preset grasping strategy and the current positions of the remaining sorting targets; wherein the current positions of the remaining sorting targets are the most recent historical positions of the remaining sorting targets without position update; If it is determined that the actual positions of all the remaining sorting targets have changed, the current positions of all the remaining sorting targets are recalculated to update the historical positions of the remaining sorting targets, and then the intelligent sorting robot is controlled to perform the current sorting operation on the remaining sorting targets according to the preset grasping strategy and the recalculated current positions of the remaining sorting targets; The preset grasping strategy includes: judging whether the remaining sorting targets in the to-be-sorted scene are stacked; if there is no stacking, calculating the confidence corresponding to the current posture of each of the remaining sorting targets; grasping the remaining sorting targets in descending order of the confidence; if there is stacking, extracting the vertical height value in the current posture of each of the remaining sorting targets; and grasping the remaining sorting targets in descending order of the vertical height values.

2. The visual recognition-based sorting method according to claim 1, characterized in that: The recalculating the current postures of all the remaining sorting targets includes: Acquire three-dimensional images of all the remaining sorting objects in the to-be-sorted scene; Based on a pose estimation algorithm and the three-dimensional images of all the remaining sorting targets, the poses of all the remaining sorting targets are calculated; Based on the posture optimization algorithm, the postures of all the remaining sorting targets are optimized to obtain the current postures of the remaining sorting targets.

3. The visual recognition-based sorting method according to claim 1, characterized in that: Also includes: Obtaining the most recent historical posture of all sorting targets in the to-be-sorted scene without posture update; Based on the most recent historical posture of all sorting targets without posture update, at least determining a region of interest corresponding to the historical posture of each of the remaining sorting targets; Extracting a first sub-image corresponding to the region of interest of each of the remaining sorting objects from the two-dimensional images of all sorting objects before the last sorting operation; Then, the two-dimensional images of the remaining sorting targets after the sorting operation are compared with the two-dimensional images of all sorting targets before the last sorting operation for similarity to determine whether the actual positions of the remaining sorting targets have changed, including: Extracting, from the two-dimensional image of the remaining sorting objects after the sorting operation, a second sub-image corresponding to the region of interest of each of the remaining sorting objects; Comparing the second sub-image corresponding to each of the remaining sorting targets with the first sub-image for similarity; If the similarity between the second sub-image of at least part of the remaining sorting objects and the first sub-image is greater than or equal to a similarity threshold, determining that the actual positions of at least part of the remaining sorting objects are not changed; If the similarities between the second sub-image and the first sub-image of all the remaining sorting objects are less than the similarity threshold, it is determined that the actual positions of all the remaining sorting objects are changed.

4. The visual recognition-based sorting method according to claim 3, characterized in that: Also includes: Obtaining a template bounding box of a sorting target in the scene to be sorted; Based on the most recent historical posture of all sorting targets without posture update, at least an area of ​​interest corresponding to the historical posture of each of the remaining sorting targets is determined, including: The template bounding box is framed one by one according to the most recent historical pose of each of the remaining sorting targets without pose update, and then the vertex coordinates of the template bounding box in the camera coordinate system are recorded during each frame selection process; Convert the vertex coordinates of the template bounding box recorded in each frame selection in the camera coordinate system into vertex coordinates in the pixel coordinate system; In the image coordinate system, construct a minimum bounding rectangle that contains the coordinates of all vertices of the template bounding box, and then determine the minimum bounding rectangle as the region of interest corresponding to the historical pose of the remaining sorting target; Repeat all the above operations until the regions of interest corresponding to the historical positions of all the remaining sorting targets are determined.

5. The visual recognition-based sorting method according to claim 3, characterized in that: The comparing the second sub-image corresponding to each of the remaining sorting targets with the first sub-image for similarity includes: Determining the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects based on a structural similarity algorithm; It is determined whether the similarity between the second sub-image and the first sub-image of each of the remaining sorting objects is greater than or equal to the similarity threshold.

6. A sorting method based on visual recognition, characterized in that: include: Obtaining initial three-dimensional images of all sorting objects in the scene to be sorted; Using the initial three-dimensional images of all the sorting targets, respectively calculating the initial positions and poses of all the sorting targets, and using the initial positions and poses of all the sorting targets as the first historical positions and poses of all the sorting targets; Controlling the intelligent sorting robot to grasp at least part of the sorting targets according to a preset grasping strategy, and placing at least part of the sorting targets at a target position to perform a first sorting operation; According to the visual recognition-based sorting method according to any one of claims 1 to 5, determining whether the actual position of the remaining sorting target in the to-be-sorted scene has changed, and performing the current sorting operation based on the determination result; Repeat the last step until all sorting targets in the to-be-sorted scene are sorted.

7. An intelligent sorting system, characterized in that: include: Intelligent sorting robot; A camera, arranged on the intelligent sorting robot, for collecting two-dimensional images and three-dimensional images of all sorting objects in the scene to be sorted; One or more processors, connected to the intelligent sorting robot and the camera respectively; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the visual recognition-based sorting method as described in any one of claims 1 to 5 or the visual recognition-based sorting method as described in claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual recognition-based sorting method according to any one of claims 1 to 5 or the visual recognition-based sorting method according to claim 6 is implemented.

Images