Object placement method and device, object grasping and placing system, and computer program product

Abstract

The present invention provides an object placement method and apparatus, an object grasping and placing system, and a computer program product. The object placement method includes: determining a first image of a target placement device according to a first point cloud of the target placement device, where the target placement device includes a plurality of accommodation cavities for placing a target object; determining a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image; registering the first sub-point cloud and a second sub-point cloud in a first template point cloud of the target placement device, and determining a target pose corresponding to any one of the accommodation cavities according to the registration result, where the first sub-point cloud and the second sub-point cloud correspond to each other; and placing the target object in the accommodation cavity of the target placement device according to the target pose.

Description

Technical Field

[0001] The present invention relates to the technical field of blanking, and in particular, to a method and device for placing an object, an object grasping and placing system, and a computer program product. Background Art

[0002] The compressor blanking scenario includes two parts: compressor grasping and compressor placing. Among them, when placing the compressor, the compressor needs to be accurately placed at a fixed position according to a specific placement pose. However, in the current compressor blanking method, the process of determining the placement pose of the compressor takes a long time, and the accuracy of the obtained placement pose is poor, thereby reducing the accuracy and efficiency of the compressor placement. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related technologies.

[0004] To this end, a first aspect of the present invention is to propose a method for placing an object.

[0005] A second aspect of the present invention is to propose an object placing device.

[0006] A third aspect of the present invention is to propose another object placing device.

[0007] A fourth aspect of the present invention is to propose an object grasping and placing system.

[0008] A fifth aspect of the present invention is to propose a readable storage medium.

[0009] A sixth aspect of the present invention is to propose a computer program product.

[0010] In view of this, according to one aspect of the present invention, a method for placing an object is proposed. The method includes: determining a first image of a target placement device according to a first point cloud of the target placement device, where the target placement device includes a plurality of accommodation cavities for placing a target object; determining a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image; registering the first sub-point cloud and a second sub-point cloud in a first template point cloud of the target placement device, and determining a target pose corresponding to any one of the accommodation cavities according to the registration result, where the first sub-point cloud and the second sub-point cloud correspond to each other; placing the target object in the accommodation cavity of the target placement device according to the target pose.

[0011] Specifically, in the object placement method provided by the present invention, the target placement device is photographed to obtain the first point cloud of the target placement device in three-dimensional space. Then, two-dimensional mapping is performed on the first point cloud of the target placement device to map the first point cloud in three-dimensional space to two-dimensional space, thereby obtaining the first image of the target placement device in two-dimensional space. Further, the above-mentioned target placement device includes a plurality of accommodation cavities for placing the grabbed target objects.

[0012] On this basis, after obtaining the first image of the target placement device in two-dimensional space, based on this first image, the first point cloud of the target placement device in three-dimensional space is cropped to obtain a plurality of first sub-point clouds from the first point cloud. The plurality of first sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device. Further, the first template point cloud of the target placement device is obtained. The first template point cloud includes a plurality of second sub-point clouds, and the plurality of second sub-point clouds correspond one-to-one to the plurality of first sub-point clouds, that is, the plurality of second sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device.

[0013] On this basis, for any one of the accommodation cavities in the target placement device, that is, for any one of the first sub-point clouds in the first point cloud, the first sub-point cloud is registered with its corresponding second sub-point cloud. Then, based on the registration result, a target pose is determined. The target pose is the placement pose corresponding to the accommodation cavity when the target object is placed in the accommodation cavity corresponding to the first sub-point cloud. On this basis, after the target pose corresponding to each accommodation cavity in the target placement device is determined, the grabbed target object is placed in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity.

[0014] In this way, during the process of placing the target object, the first point cloud of the target placement device in three-dimensional space is projected into two-dimensional space. Combining two-dimensional methods, the positions of the plurality of accommodation cavities in the target placement device are determined. Then, in the first point cloud of the target placement device, a plurality of first sub-point clouds corresponding to the plurality of accommodation cavities are cropped respectively, and the cropped plurality of first sub-point clouds are registered with the plurality of second sub-point clouds in the first template point cloud of the target placement device respectively, thereby obtaining the placement pose corresponding to each accommodation cavity. In this way, by integrating two-dimensional and three-dimensional algorithms and calculating the placement pose of each accommodation cavity in a one-to-one manner, the problems of long time consumption, high dependence on the initial transformation matrix, and poor accuracy of the conventional point cloud registration algorithm are solved, the registration time is reduced, the placement accuracy and placement efficiency are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0015] According to the above object placement method of the present invention, the following additional technical features may also be provided:

[0016] In some technical solutions, optionally, determining a first image of the target placement device based on the first point cloud of the target placement device includes: determining a target area according to the coordinate information of the first point cloud; adjusting the position information of the first point cloud according to the target area to obtain a second point cloud; and determining the first image according to the coordinate information of the second point cloud and the size information of the target area.

[0017] In this technical solution, in the process of two-dimensionally mapping the first point cloud of the target placement device to map the first point cloud in the three-dimensional space to the two-dimensional space to obtain the first image of the target placement device in the two-dimensional space, specifically, the first point cloud is preprocessed to filter out the background information and noise information in the first point cloud, and then, according to the coordinate information of the preprocessed first point cloud, the target area where the first point cloud is located is determined, and the target area is a three-dimensional space. Further, based on the coordinate information of the target area, the position information of the first point cloud is adjusted to adjust the position of the first point cloud to the origin of the coordinate system where it is located to obtain a second point cloud, which is convenient for subsequent calculations of mapping the three-dimensional point cloud into a two-dimensional image.

[0018] On this basis, the coordinate information of the second point cloud obtained after the position adjustment is further obtained, and then, based on the size information of the above target area and the coordinate information of the second point cloud, the first point cloud of the target placement device is two-dimensionally mapped to map each point position of the first point cloud in the three-dimensional space to a pixel point in the two-dimensional space, so as to obtain the first image of the target placement device in the two-dimensional space. In this way, through two-dimensional mapping, the first point cloud of the target placement device in the three-dimensional space is mapped to the first image of the target placement device in the two-dimensional space, and then, by means of the obtained first image through mapping, subsequent registration, pose determination and other work are carried out, which can reduce the registration time consumption, improve the placement accuracy and placement efficiency, and thus enhance the stability of the production line.

[0019] In some technical solutions, optionally, determining the first image according to the coordinate information of the second point cloud and the size information of the target area includes: determining a plurality of pixel abscissas according to the product of the first value and the abscissas of a plurality of point positions in the second point cloud respectively; determining a plurality of pixel ordinates according to the product of the first value and the ordinates of a plurality of point positions in the second point cloud respectively; determining a plurality of pixel values according to the product of the target value and the ratio of the vertical coordinates of a plurality of point positions in the second point cloud to the height of the target area respectively; determining the first image according to the plurality of pixel abscissas, the plurality of pixel ordinates and the plurality of pixel values; wherein, the plurality of pixel abscissas, the plurality of pixel ordinates and the plurality of pixel values correspond to each other one by one.

[0020] In this technical solution, in the process of performing two-dimensional mapping on the first point cloud of the target placement device based on the size information of the above-mentioned target area and the coordinate information of the second point cloud to obtain the first image of the target placement device in the two-dimensional space, specifically, the abscissa value of each point cloud position in the second point cloud is obtained, and each obtained abscissa value is multiplied by the set first value respectively. Then, based on the multiple product values obtained by multiplication, multiple pixel abscissas are determined. Further, the ordinate value of each point cloud position in the second point cloud is obtained, and each obtained ordinate value is multiplied by the above-mentioned set first value respectively. Then, based on the multiple product values obtained by multiplication, multiple pixel ordinates are determined. Further, the vertical coordinate value of each point cloud position in the second point cloud is obtained, and based on the obtained vertical coordinate value and the size information of the target area, the height information of the point cloud position in the target area is determined. Then, the ratio of the height information of the point cloud position to the height value of the target area is determined, and multiple ratios are obtained. Further, the multiple obtained ratios are multiplied by the set target value respectively. Then, based on the multiple product values obtained by multiplication, multiple pixel values are determined.

[0021] On this basis, there is a one-to-one correspondence between the multiple point cloud positions in the first point cloud and the multiple pixel points in the first image. The pixel abscissa, pixel ordinate, and pixel value calculated based on each point cloud position are determined as the pixel information of the pixel point corresponding to the point cloud position. In this way, the multiple calculated pixel ordinates, multiple pixel abscissas, and multiple pixel values are mapped to the two-dimensional space one-to-one, so as to obtain the first image of the target placement device in the two-dimensional space. In this way, through two-dimensional mapping, the first point cloud of the target placement device in the three-dimensional space is mapped into the first image of the target placement device in the two-dimensional space. Then, by means of the obtained first image, subsequent registration, pose determination, etc. are carried out, which can reduce the registration time, improve the placement accuracy and placement efficiency, and thus enhance the stability of the production line.

[0022] In some technical solutions, optionally, determining the first sub-point cloud in the first point cloud corresponding to any one accommodation cavity according to the first image includes: determining the center point information of any one accommodation cavity in the first image according to the first image; registering the first image and the first template image of the first template point cloud to determine the first transformation matrix; determining the first point position in the first point cloud according to the center point information and the first transformation matrix; and determining the first sub-point cloud according to the first point position and the target size information.

[0023] In this technical solution, in the process of cropping the first point cloud of the target placement device in the three-dimensional space based on the above-mentioned first image to obtain the first sub-point cloud corresponding to each accommodation cavity from the first point cloud, specifically, the first image of the target placement device is analyzed and processed to determine the center point information of the accommodation cavity in the target placement device in the first image. Further, a two-dimensional image corresponding to the first template point cloud of the target placement device, that is, the first template image, is obtained, and the first template image is two-dimensionally registered with the first image to obtain a first transformation matrix. Further, according to the first transformation matrix and the center point information, the center point of the accommodation cavity in the first image is mapped back to the first point cloud to determine the first point position corresponding to the center point information in the first point cloud.

[0024] On this basis, the determined first point position is used as the point cloud center of the first sub-point cloud to be cropped, and the first point cloud is cropped according to the target size information of each first sub-point cloud set, so as to obtain the first sub-point cloud corresponding to the accommodation cavity from the first point cloud. In this way, by combining the two-dimensional method, the positions of multiple accommodation cavities in the target placement device in the two-dimensional image are determined, and then the positions are mapped back to the three-dimensional point cloud, and in the first point cloud of the target placement device, multiple first sub-point clouds corresponding to multiple accommodation cavities are cropped out. In this way, by performing registration in the two-dimensional space to determine the position information of the accommodation cavity in the three-dimensional point cloud, and then performing subsequent calculations, the problems of long time consumption, high dependence on the initial transformation matrix, and poor accuracy of the conventional point cloud registration algorithm are solved, the registration time is reduced, the placement accuracy and placement efficiency are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0025] In some technical solutions, optionally, determining the target pose corresponding to any one accommodation cavity according to the registration result includes: determining the second transformation matrix of the first sub-point cloud and the third transformation matrix of the second sub-point cloud according to the registration result; determining the first deflection information of the first sub-point cloud according to the second transformation matrix and the third transformation matrix; and determining the target pose according to the first deflection information of the first sub-point cloud and the template pose information of any one accommodation cavity.

[0026] In this technical solution, in the process of determining the placement pose of the accommodation cavity corresponding to the first sub-point cloud, that is, the above-mentioned target pose, based on the registration result of the second sub-point cloud and the first sub-point cloud, specifically, based on the registration result of the second sub-point cloud and the first sub-point cloud, the third transformation matrix of the second sub-point cloud is determined, and the second transformation matrix of the first sub-point cloud is determined. Further, the third transformation matrix and the second transformation matrix are compared, and according to the deviation information of the third transformation matrix and the second transformation matrix, the first deflection information of the first sub-point cloud relative to the second sub-point cloud is determined.

[0027] On this basis, based on the template pose information of the accommodation cavity corresponding to the second sub-point cloud and the first deflection information of the first sub-point cloud, when determining the actual placement pose of the accommodation cavity corresponding to the target object when the target object is placed in the accommodation cavity corresponding to the first sub-point cloud, that is, the above-mentioned target pose. In this way, through a one-to-one registration method, the placement poses of each accommodation cavity are calculated, solving the problems of long time consumption, high dependence on the initial transformation matrix, and poor accuracy of conventional point cloud registration algorithms, reducing the registration time consumption, improving the placement accuracy and efficiency, enhancing the stability of the production line, and reducing the labor cost.

[0028] In some technical solutions, optionally, placing the target object in the accommodation cavity of the target placement device according to the target pose includes: verifying the registration result of the first sub-point cloud according to the first sub-point cloud and the first template point cloud; and when the registration result of the first sub-point cloud is qualified, placing the target object in the accommodation cavity of the target placement device according to the target pose.

[0029] In this technical solution, in the process of placing the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity, specifically, based on the first template point cloud and the first sub-point cloud, the registration result of the first sub-point cloud is verified, that is, the legality of the placement pose of the accommodation cavity is verified. On this basis, only when the registration results of each first sub-point cloud are all qualified, that is, when the placement poses of any accommodation cavity are all reasonable, the placement operation of the target object is performed, that is, based on the target pose corresponding to the accommodation cavity, the grabbed target object is placed in the accommodation cavity of the target placement device. In this way, before placing the target object, the legality of the placement pose of each accommodation cavity is verified, and only when the placement poses of each accommodation cavity all pass the legality verification, the placement operation is performed on the target object. In this way, the stability of the system operation is improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0030] In some technical solutions, optionally, verifying the registration result of the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: determining the registration score of the first sub-point cloud according to the first sub-point cloud and the first template point cloud; when the registration score is greater than or equal to the first threshold, determining that the registration result of the first sub-point cloud is qualified; and when the registration score is less than the first threshold, determining that the registration result of the first sub-point cloud is unqualified.

[0031] In this technical solution, when verifying the registration result of the first sub-point cloud based on the first template point cloud and the first sub-point cloud, that is, during the process of verifying the legality of the placement pose of the accommodation cavity, specifically, according to the first template point cloud and the first sub-point cloud, the registration score of the first sub-point cloud is determined. This registration score is used to indicate whether the registration result of the first sub-point cloud is reasonable, that is, it is used to indicate the degree of legality of the placement pose of the accommodation cavity. The higher the registration score, the more reasonable the registration result of the first sub-point cloud is, that is, the higher the degree of legality of the placement pose of the accommodation cavity.

[0032] On this basis, the registration score of the first sub-point cloud is compared with the set first threshold. When the first threshold is less than or equal to the registration score of the first sub-point cloud, it indicates that the degree of legality of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively high. At this time, it is determined that the registration result of the first sub-point cloud is qualified. When the set first threshold is greater than the registration score of the first sub-point cloud, it indicates that the degree of legality of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively low. At this time, it is determined that the registration result of the first sub-point cloud is unqualified. In this way, based on the registration score of the first sub-point cloud, the registration result of the first sub-point cloud is verified, that is, the legality of the placement pose of the accommodation cavity is verified, ensuring the accuracy of the verification result, and further ensuring the accuracy of the subsequent placement operation based on the verification result, improving the stability of the system operation, enhancing the stability of the production line, and reducing the labor cost.

[0033] In some technical solutions, optionally, determining the registration score of the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: determining a plurality of second point positions in the first sub-point cloud according to the first sub-point cloud and the first template point cloud, and the distribution rules of the plurality of second point positions are the same as those of some point cloud positions in the first template point cloud; determining the distance information between the plurality of second point positions and the first template point cloud to obtain a plurality of first distance values; screening a plurality of second distance values from the plurality of first distance values according to a preset range; and determining the registration score of the first sub-point cloud according to the plurality of second distance values.

[0034] In this technical solution, during the process of determining the registration score of the first sub-point cloud according to the first template point cloud and the first sub-point cloud, specifically, the first template point cloud and the first sub-point cloud are analyzed to determine a plurality of second point positions in the first sub-point cloud according to the distribution of the point cloud positions in the first template point cloud and the first sub-point cloud. The distribution rules of the plurality of second point positions are the same as those of some point cloud positions in the first template point cloud, that is, the above-mentioned second point positions are the point cloud positions in the first sub-point cloud with the same distribution rules as the point cloud positions of the first template point cloud.

[0035] Further, according to the first template point cloud and the position information of each second point, calculate the distance information between the first template point cloud and each second point to obtain a plurality of first distance values. Further, according to the set preset range, screen the plurality of first distance values to screen out a plurality of second distance values from the plurality of first distance values. On this basis, determine the registration score of the first sub-point cloud according to the screened plurality of second distance values. Specifically, calculate the average value of the plurality of second distance values, and then normalize the average value to obtain a registration score with a value range between 0 and 1. In this way, based on the distribution of the point cloud positions in the first template point cloud and the first sub-point cloud, the registration score of the first sub-point cloud is determined, ensuring the accuracy of the obtained registration score, thus ensuring the accuracy of the subsequent rationality verification of the registration result of the first sub-point cloud based on this registration score, further improving the stability of the system operation, enhancing the stability of the production line, and reducing the labor cost.

[0036] In some technical solutions, optionally, according to the first sub-point cloud and the first template point cloud, verifying the registration result of the first sub-point cloud includes: obtaining the second transformation matrix of the first sub-point cloud, and obtaining the third transformation matrix of the second sub-point cloud corresponding to the first sub-point cloud in the first template point cloud; determining the difference value between the second transformation matrix and the third transformation matrix; when the difference value is less than or equal to the second threshold, determining that the registration result of the first sub-point cloud is qualified; when the difference value is less than the second threshold, determining that the registration result of the first sub-point cloud is unqualified.

[0037] In this technical solution, in the process of verifying the registration result of the first sub-point cloud based on the first template point cloud and the first sub-point cloud, that is, verifying the legality of the placement pose of the accommodation cavity, specifically, obtain the second transformation matrix of the first sub-point cloud during the registration process, and obtain the third transformation matrix of the second sub-point cloud during the registration process. Further, compare the obtained third transformation matrix and the second transformation matrix to determine the difference value between the third transformation matrix and the second transformation matrix. On this basis, compare the obtained difference value with the set second threshold, and verify the registration result of the first sub-point cloud according to the comparison result.

[0038] Specifically, when the set second threshold is greater than or equal to the above difference value, it indicates that the difference between the third transformation matrix and the second transformation matrix is small, that is, the legality degree of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively high. At this time, it is determined that the registration result of the first sub-point cloud is qualified; when the set second threshold is less than the above difference value, it indicates that the difference between the third transformation matrix and the second transformation matrix is large, that is, the legality degree of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively low. At this time, it is determined that the registration result of the first sub-point cloud is unqualified. In this way, based on the difference value between the third transformation matrix of the second sub-point cloud and the second transformation matrix of the first sub-point cloud, the registration result of the first sub-point cloud is verified, that is, the legality of the placement pose of the accommodation cavity is verified, which ensures the accuracy of the verification result, and further ensures the accuracy of the subsequent placement operation based on the verification result, improves the stability of the system operation, enhances the stability of the production line, and reduces the labor cost.

[0039] In some technical solutions, optionally, placing the target object in the accommodation cavity of the target placement device according to the target pose includes: determining the occupancy of the accommodation cavity corresponding to the first sub-point cloud according to the first sub-point cloud and the first template point cloud to obtain a target list of the occupied accommodation cavities; when the target list is empty, placing the target object in the accommodation cavity of the target placement device according to the target pose in the arrangement order of the accommodation cavities in the target placement device; when the target list is not empty and the accommodation cavities in the target list meet the target conditions, placing the target object in the first unoccupied accommodation cavity according to the target pose.

[0040] In this technical solution, during the process of placing the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity, specifically, by analyzing and processing the first template point cloud and the first sub-point cloud, the occupancy of the accommodation cavity corresponding to the first sub-point cloud is determined, that is, it is determined whether a target object has been placed in the accommodation cavity corresponding to the first sub-point cloud. Based on this, by determining the occupancy of each accommodation cavity in the target placement device, a target list of the occupied accommodation cavities in the target placement device is obtained. Further, a logical judgment is made on the target list, and subsequent work is carried out based on the logical judgment result.

[0041] Specifically, when the above-mentioned target list is empty, that is, when there is no occupied accommodation cavity in the target placement device, the captured target objects are sequentially placed in each accommodation cavity of the target placement device according to the arrangement order of each accommodation cavity in the target placement device and the determined target pose. When the above-mentioned target list is not empty and the accommodation cavities in the above-mentioned target list meet the target conditions, the captured target objects are sequentially placed in the unoccupied accommodation cavities starting from the first unoccupied accommodation cavity in the target object placement, according to the determined target pose. In this way, before placing the target objects, the occupancy status of each accommodation cavity is determined, and then subsequent placement operations are carried out based on the occupancy status of each accommodation cavity. In this way, the principle of breakpoint resume during the object placement process is realized, and the stability of the system operation is improved.

[0042] In some technical solutions, optionally, determining the occupancy status of the accommodation cavity corresponding to the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: when the first sub-point cloud is empty, determining that the accommodation cavity corresponding to the first sub-point cloud is occupied; when the proportion value of the number of second point positions in the first sub-point cloud to the total number of point positions in the first template point cloud is less than the third threshold, determining that the accommodation cavity corresponding to the first sub-point cloud is occupied, where the second point position is the point cloud position in the first sub-point cloud with the same distribution law as the point cloud positions in the first template point cloud; when the proportion value is greater than or equal to the third threshold, determining that the accommodation cavity corresponding to the first sub-point cloud is unoccupied.

[0043] In this technical solution, in the process of analyzing and processing the first template point cloud and the first sub-point cloud to determine the occupancy status of the accommodation cavity corresponding to the first sub-point cloud, specifically, when the first sub-point cloud is empty, it is determined that the accommodation cavity corresponding to the first sub-point cloud is occupied, that is, it is determined that a target object has been placed in the accommodation cavity corresponding to the first sub-point cloud. Further, obtain the number of point positions of the second point position in the first sub-point cloud, where the second point position is the point cloud position in the first sub-point cloud with the same distribution law as the point cloud positions in the first template point cloud. Further, calculate the proportion value of the number of point positions of the second point position to the total number of point positions in the first template point cloud, and this proportion value is used to indicate the number of point cloud positions in the first sub-point cloud that match the first template point cloud. On this basis, the above-mentioned proportion value is compared with the set third threshold, and according to the comparison result, the occupancy status of the accommodation cavity corresponding to the first sub-point cloud is determined.

[0044] Specifically, when the set third threshold is greater than the above ratio value, it indicates that the number of point cloud positions in the first sub-point cloud that match the first template point cloud is small. At this time, it is determined that the accommodation cavity corresponding to the first sub-point cloud is occupied, and the second position; when the set third threshold is less than or equal to the above ratio value, it indicates that the number of point cloud positions in the first sub-point cloud that match the first template point cloud is large. At this time, it is determined that the accommodation cavity corresponding to the first sub-point cloud is not occupied. In this way, based on the number of point cloud positions in the first sub-point cloud that match the first template point cloud, the occupancy situation of the accommodation cavity corresponding to the first sub-point cloud is determined, ensuring the accuracy of determining the occupancy situation of the accommodation cavity, and thus ensuring the accuracy of subsequent work based on the occupancy situation of the accommodation cavity.

[0045] According to the second aspect of the present invention, an object placement device is provided. The device includes: a processing unit for determining a first image of the target placement device according to the first point cloud of the target placement device, where the target placement device includes a plurality of accommodation cavities for placing target objects; the processing unit is further configured to determine a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image; the processing unit is further configured to register the first sub-point cloud and a second sub-point cloud in the first template point cloud of the target placement device, and determine the target pose corresponding to any one of the accommodation cavities according to the registration result, where the first sub-point cloud and the second sub-point cloud correspond to each other; a placement unit for placing the target object in the accommodation cavity of the target placement device according to the target pose.

[0046] The object placement device provided by the present invention includes a processing unit and a placement unit. During the process of placing the target object, the target placement device is photographed to obtain the first point cloud of the target placement device in the three-dimensional space. Then, the processing unit performs two-dimensional mapping on the first point cloud of the target placement device to map the first point cloud in the three-dimensional space to the two-dimensional space, thereby obtaining the first image of the target placement device in the two-dimensional space. Further, the above target placement device includes a plurality of accommodation cavities for placing the grabbed target objects.

[0047] On this basis, after obtaining the first image of the target placement device in the two-dimensional space, the processing unit further crops the first point cloud of the target placement device in the three-dimensional space based on this first image to crop a plurality of first sub-point clouds from the first point cloud. The plurality of first sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device. Further, the processing unit retrieves the first template point cloud of the target placement device. The first template point cloud includes a plurality of second sub-point clouds, and the plurality of second sub-point clouds correspond one-to-one to the plurality of first sub-point clouds. That is, the plurality of second sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device.

[0048] On this basis, for any accommodation cavity in the target placement device, that is, for any first sub-point cloud in the first point cloud, the processing unit registers the first sub-point cloud with its corresponding second sub-point cloud, and then determines a target pose based on the registration result. The target pose is the placement pose corresponding to the accommodation cavity when the target object is placed in the accommodation cavity corresponding to the first sub-point cloud. On this basis, after the processing unit determines the target pose corresponding to each accommodation cavity in the target placement device, the placement unit then places the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity.

[0049] In this way, during the process of placing the target object, the first point cloud of the target placement device in the three-dimensional space is projected onto the two-dimensional space, and combined with the two-dimensional method, the positions of multiple accommodation cavities in the target placement device are determined. Then, in the first point cloud of the target placement device, multiple first sub-point clouds corresponding to the multiple accommodation cavities are cropped out, and the cropped multiple first sub-point clouds are respectively registered with multiple second sub-point clouds in the first template point cloud of the target placement device, so as to obtain the placement pose corresponding to each accommodation cavity. In this way, by integrating two-dimensional and three-dimensional algorithms and calculating the placement pose of each accommodation cavity in a one-to-one manner, the problems of long registration time, high dependence on the initial transformation matrix, and poor accuracy of the conventional point cloud registration algorithm are solved, the registration time is reduced, the placement accuracy and placement efficiency are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0050] According to the third aspect of the present invention, another object placement device is proposed, including: a memory storing programs or instructions; a processor, when the processor executes the programs or instructions, implementing the steps of the object placement method in any of the above technical solutions. Therefore, the object placement device proposed in the third aspect of the present invention has all the beneficial effects of the object placement method in any of the technical solutions in the first aspect above, and will not be elaborated here.

[0051] According to the fourth aspect of the present invention, an object grasping and placing system is proposed, including: the object placement device in the technical solution of the third aspect above. The object grasping and placing system proposed in the fourth aspect of the present invention includes the object placement device in the technical solution of the third aspect above. Therefore, the object grasping and placing system proposed in the fourth aspect of the present invention has all the beneficial effects of the object placement device in the technical solution of the third aspect above, and will not be elaborated here.

[0052] According to the fifth aspect of the present invention, a readable storage medium is provided, on which a program or instructions are stored. When the program or instructions are executed by a processor, the object placement method in any of the above technical solutions is implemented. Therefore, the readable storage medium proposed in the fifth aspect of the present invention has all the beneficial effects of the object placement method in any of the technical solutions in the first aspect above, and will not be elaborated here.

[0053] According to the sixth aspect of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the object placement method in any of the above technical solutions is implemented. Therefore, the computer program product proposed in the sixth aspect of the present invention has all the beneficial effects of the object placement method in any of the technical solutions in the first aspect above, and will not be elaborated here.

[0054] The additional aspects and advantages of the present invention will become apparent in the following description section, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0056] Figure 1 FIG. 1 shows one of the schematic flowcharts of the object placement method according to an embodiment of the present invention;

[0057] Figure 2 FIG. 2 shows another schematic flowchart of the object placement method according to an embodiment of the present invention;

[0058] Figure 3 FIG. 3 shows a third schematic flowchart of the object placement method according to an embodiment of the present invention;

[0059] Figure 4 FIG. 4 shows a fourth schematic flowchart of the object placement method according to an embodiment of the present invention;

[0060] Figure 5 FIG. 5 shows a fifth schematic flowchart of the object placement method according to an embodiment of the present invention;

[0061] Figure 6 FIG. 6 shows a sixth schematic flowchart of the object placement method according to an embodiment of the present invention;

[0062] Figure 7 FIG. 7 shows a first structural block diagram of the object placement device according to an embodiment of the present invention;

[0063] Figure 8 FIG. 8 shows a second structural block diagram of the object placement device according to an embodiment of the present invention;

[0064] Figure 9The structural block diagram of the object grasping and placing system according to an embodiment of the present invention is shown. Detailed implementation manners

[0065] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0066] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0067] The following combines Figures 1 to 9 , and through specific embodiments and their application scenarios, the object placement method and device, object grasping and placing system, and computer program product provided by the embodiments of the present application are described in detail.

[0068] In some embodiments of the present invention, as Figure 1 shown, the object placement method may specifically include the following steps 102 to 108:

[0069] Step 102, determining a first image of the target placement device according to the first point cloud of the target placement device;

[0070] Step 104, determining a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image;

[0071] Step 106, registering the first sub-point cloud and a second sub-point cloud in the first template point cloud of the target placement device, and determining a target pose corresponding to any one of the accommodation cavities according to the registration result;

[0072] Step 108, placing the target object in the accommodation cavity of the target placement device according to the target pose;

[0073] Wherein, the target placement device includes a plurality of accommodation cavities for placing the target object, and the second sub-point cloud corresponds to the first sub-point cloud.

[0074] In the object placement method provided by the present invention, during the process of placing the target object into the target placement device, the target placement device is photographed to obtain the first point cloud of the target placement device in the three-dimensional space, and then the first point cloud of the target placement device is two-dimensionally mapped to map the first point cloud in the three-dimensional space to the two-dimensional space, so as to obtain the first image of the target placement device in the two-dimensional space. Further, the above target placement device includes a plurality of accommodation cavities for placing the grabbed target object.

[0075] On this basis, after obtaining the first image of the target placement device in the two-dimensional space, based on this first image, the first point cloud of the target placement device in the three-dimensional space is cropped to obtain a plurality of first sub-point clouds from the first point cloud, and the plurality of first sub-point clouds correspond one-to-one to a plurality of accommodation cavities in the target placement device. Further, a first template point cloud of the target placement device is obtained, and the first template point cloud includes a plurality of second sub-point clouds, and the plurality of second sub-point clouds correspond one-to-one to the plurality of first sub-point clouds, that is, the plurality of second sub-point clouds correspond one-to-one to a plurality of accommodation cavities in the target placement device.

[0076] On this basis, for any one of the accommodation cavities in the target placement device, that is, for any one of the first sub-point clouds in the first point cloud, the first sub-point cloud is registered with its corresponding second sub-point cloud, and then based on the registration result, a target pose is determined, and the target pose is the placement pose corresponding to the accommodation cavity when the target object is placed in the accommodation cavity corresponding to the first sub-point cloud. On this basis, after the target pose corresponding to each accommodation cavity in the target placement device is determined, the grabbed target object is placed in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity.

[0077] In this way, during the process of placing the target object, the first point cloud of the target placement device in the three-dimensional space is projected onto the two-dimensional space, combined with the two-dimensional method, the positions of a plurality of accommodation cavities in the target placement device are determined, and then in the first point cloud of the target placement device, a plurality of first sub-point clouds corresponding to the plurality of accommodation cavities are cropped respectively, and the cropped plurality of first sub-point clouds are respectively registered with a plurality of second sub-point clouds in the first template point cloud of the target placement device, so as to obtain the placement pose corresponding to each accommodation cavity. In this way, by fusing the two-dimensional and three-dimensional algorithms and calculating the placement pose of each accommodation cavity in a one-to-one manner, the problems of long time consumption, high dependence on the initial transformation matrix and poor accuracy of the conventional point cloud registration algorithm are solved, the registration time is reduced, the placement accuracy and placement efficiency are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0078] In some embodiments of the present invention, optionally, as Figure 2 shown, the above step 102 may specifically include the following steps 102a to 102c:

[0079] Step 102a, determining a target area according to the coordinate information of the first point cloud;

[0080] Step 102b, adjusting the position information of the first point cloud according to the target area to obtain a second point cloud;

[0081] Step 102c, determining the first image according to the size information of the target area and the coordinate information of the second point cloud.

[0082] In this embodiment, in the process of two-dimensionally mapping the first point cloud of the target placement device to map the first point cloud in the three-dimensional space to the two-dimensional space to obtain the first image of the target placement device in the two-dimensional space, specifically, the first point cloud is preprocessed to filter out the background information and noise information in the first point cloud, and then, based on the coordinate information of the preprocessed first point cloud, the target area where the first point cloud is located is determined, and this target area is a three-dimensional space. Further, based on the coordinate information of the target area, the position information of the first point cloud is adjusted to adjust the position of the first point cloud to the origin of the coordinate system where it is located, obtaining a second point cloud, so as to facilitate the subsequent calculation of mapping the three-dimensional point cloud into a two-dimensional image.

[0083] On this basis, the coordinate information of the second point cloud obtained after the position adjustment is further acquired, and then, based on the size information of the above-mentioned target area and the coordinate information of the second point cloud, the first point cloud of the target placement device is two-dimensionally mapped to map each point cloud position of the first point cloud in the three-dimensional space to a pixel point in the two-dimensional space, thereby obtaining the first image of the target placement device in the two-dimensional space. In this way, through two-dimensional mapping, the first point cloud of the target placement device in the three-dimensional space is mapped to the first image of the target placement device in the two-dimensional space, and then, by means of the obtained first image through mapping, subsequent work such as registration and pose determination is carried out, which can reduce the registration time consumption, improve the placement accuracy and placement efficiency, and thus enhance the stability of the production line.

[0084] In some embodiments of the present invention, optionally, the above step 102c may specifically include the following steps 102c1 to 102c4:

[0085] Step 102c1, determining a plurality of pixel abscissas according to the products of the abscissas of a plurality of point cloud positions in the second point cloud and the first value respectively;

[0086] Step 102c2, determining a plurality of pixel ordinates according to the products of the ordinates of a plurality of point cloud positions in the second point cloud and the first value respectively;

[0087] Step 102c3, determining a plurality of pixel values according to the products of the ratios of the vertical coordinates of a plurality of point cloud positions in the second point cloud to the height of the target area and the target value respectively;

[0088] Step 102c4, determining the first image according to the plurality of pixel values, the plurality of pixel ordinates, and the plurality of pixel abscissas;

[0089] Among them, the plurality of pixel values, the plurality of pixel ordinates, and the plurality of pixel abscissas correspond to each other one by one.

[0090] In this embodiment, in the process of performing two-dimensional mapping on the first point cloud of the target placement device based on the size information of the above-mentioned target area and the coordinate information of the second point cloud to obtain the first image of the target placement device in the two-dimensional space, specifically, the abscissa value of each point cloud position in the second point cloud is obtained, and each obtained abscissa value is multiplied by a set first value respectively. Then, based on the multiple product values obtained by the multiplication, multiple pixel abscissas are determined. Further, the ordinate value of each point cloud position in the second point cloud is obtained, and each obtained ordinate value is multiplied by the above-mentioned set first value respectively. Then, based on the multiple product values obtained by the multiplication, multiple pixel ordinates are determined. Further, the vertical coordinate value of each point cloud position in the second point cloud is obtained, and based on the obtained vertical coordinate value and the size information of the target area, the height information of the point cloud position in the target area is determined. Then, the ratio of the height information of the point cloud position to the height value of the target area is obtained, and multiple ratios are obtained. Further, the multiple obtained ratios are multiplied by a set target value respectively. Then, based on the multiple product values obtained by the multiplication, multiple pixel values are determined.

[0091] On this basis, there is a one-to-one correspondence between the multiple point cloud positions in the first point cloud and the multiple pixel points in the first image. The pixel abscissa, pixel ordinate, and pixel value calculated based on each point cloud position are determined as the pixel information of the pixel point corresponding to the point cloud position. In this way, the multiple calculated pixel ordinates, multiple pixel abscissas, and multiple pixel values are mapped to the two-dimensional space one by one, so as to obtain the first image of the target placement device in the two-dimensional space. In this way, through two-dimensional mapping, the first point cloud of the target placement device in the three-dimensional space is mapped to the first image of the target placement device in the two-dimensional space. Then, by means of the obtained first image, subsequent registration, pose determination, etc. are carried out, which can reduce the registration time, improve the placement accuracy and placement efficiency, and thus enhance the stability of the production line.

[0092] Among them, the above-mentioned first value is used to indicate the mapping resolution for performing two-dimensional mapping on the first point cloud. The mapping resolution can specifically take values such as 0.001, and no specific limitation is made here. Among them, when the mapping resolution is 0.001, the indicated mapping relationship is: 1 mm of physical size represents one pixel in the two-dimensional image.

[0093] Further, the above-mentioned target value can specifically take values such as 255, and no specific limitation is made here.

[0094] In the actual application process, for the mapping relationship between each point cloud position in the first point cloud and each pixel point in the first image, that is, for the pixel abscissa, pixel ordinate, and pixel value of each pixel point in the first image, they can be specifically determined by the following formulas (1) to (3):

[0095] x = X × Resolution, (1)

[0096] y = Y × Resolution, (2)

[0097]

[0098] Wherein, x represents the abscissa of the pixel, y represents the ordinate of the pixel, z represents the pixel value, Resolution represents the mapping resolution, X represents the abscissa value of the point cloud position, Y represents the ordinate value of the point cloud position, Z represents the vertical coordinate value of the point cloud position, Z max represents the maximum vertical coordinate value of the target area, Z min represents the minimum vertical coordinate value of the target area, Z max -Z min represents the height value of the target area.

[0099] In some embodiments of the present invention, optionally, as Figure 3 shown, the above step 104 may specifically include the following steps 104a to 104d:

[0100] Step 104a, determining the center point information of any accommodation cavity in the first image according to the first image;

[0101] Step 104b, registering the first template image of the first template point cloud and the first image to determine the first transformation matrix;

[0102] Step 104c, determining the first position in the first point cloud according to the first transformation matrix and the center point information;

[0103] Step 104d, determining the first sub-point cloud according to the target size information and the first position.

[0104] In this embodiment, in the process of cropping the first point cloud of the target placement device in the three-dimensional space based on the above first image to crop the first sub-point cloud corresponding to each accommodation cavity from the first point cloud, specifically, the first image of the target placement device is analyzed and processed to determine the center point information of the accommodation cavity in the target placement device in the first image. Further, the two-dimensional image corresponding to the first template point cloud of the target placement device, that is, the first template image, is obtained, and the first template image is two-dimensionally registered with the above first image to obtain the first transformation matrix. Further, according to the above first transformation matrix and the center point information, the center point of the accommodation cavity in the first image is mapped back to the first point cloud to determine the first position in the first point cloud corresponding to the above center point information.

[0105] On this basis, the first point determined above is used as the point cloud center of the first sub-point cloud to be cropped, and the first point cloud is cropped according to the target size information of each first sub-point cloud set, so as to crop the first sub-point cloud corresponding to the accommodating cavity from the first point cloud. In this way, combined with the two-dimensional method, the positions of multiple accommodating cavities in the target placement device in the two-dimensional image are determined, and then this position is mapped back to the three-dimensional point cloud, and in the first point cloud of the target placement device, multiple first sub-point clouds corresponding to the multiple accommodating cavities are cropped respectively. In this way, by performing registration in the two-dimensional space, the position information of the accommodating cavity in the three-dimensional point cloud is determined, and then subsequent calculations are performed, solving the problems of long time consumption, high dependence on the initial transformation matrix, and poor accuracy of the conventional point cloud registration algorithm, reducing the registration time consumption, improving the placement accuracy and placement efficiency, enhancing the stability of the production line, and reducing the labor cost.

[0106] In some embodiments of the present invention, optionally, the step of determining the target pose corresponding to any one accommodating cavity according to the registration result may specifically include the following steps 106a to step 106c:

[0107] Step 106a, according to the registration result, determine the second transformation matrix of the first sub-point cloud and determine the third transformation matrix of the second sub-point cloud;

[0108] Step 106b, according to the third transformation matrix and the second transformation matrix, determine the first deflection information of the first sub-point cloud;

[0109] Step 106c, according to the template pose information of any one accommodating cavity and the first deflection information, determine the target pose.

[0110] In this embodiment, in the process of determining the placement pose of the accommodating cavity corresponding to the first sub-point cloud, that is, the above-mentioned target pose, based on the registration result of the second sub-point cloud and the first sub-point cloud, specifically, based on the registration result of the second sub-point cloud and the first sub-point cloud, determine the third transformation matrix of the second sub-point cloud and determine the second transformation matrix of the first sub-point cloud. Further, compare the third transformation matrix and the second transformation matrix, and determine the first deflection information of the first sub-point cloud relative to the second sub-point cloud according to the deviation information of the third transformation matrix and the second transformation matrix.

[0111] On this basis, based on the template pose information of the accommodation cavity corresponding to the second sub-point cloud and the first deflection information of the first sub-point cloud, when determining the actual placement pose of the accommodation cavity corresponding to the target object when placing the target object in the accommodation cavity corresponding to the first sub-point cloud, that is, the above-mentioned target pose. In this way, through a one-to-one registration method, the placement poses of each accommodation cavity are calculated, solving the problems of long time consumption, high dependence on the initial transformation matrix, and poor accuracy of conventional point cloud registration algorithms, reducing the registration time consumption, improving the placement accuracy and efficiency, enhancing the stability of the production line, and reducing the labor cost.

[0112] In some embodiments of the present invention, optionally, as Figure 4 shown, the above step 108 may specifically include the following step 108a and step 108b:

[0113] Step 108a, check the registration result of the first sub-point cloud according to the first template point cloud and the first sub-point cloud;

[0114] Step 108b, when the registration result of the first sub-point cloud is qualified, place the target object in the accommodation cavity of the target placement device according to the target pose.

[0115] In this embodiment, in the process of placing the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity, specifically, check the registration result of the first sub-point cloud based on the first template point cloud and the first sub-point cloud, that is, verify the legality of the placement pose of the accommodation cavity. On this basis, only when the registration results of each first sub-point cloud are qualified, that is, when the placement poses of any accommodation cavity are reasonable, the placement operation is performed on the target object, that is, place the grabbed target object in the accommodation cavity of the target placement device based on the target pose corresponding to the accommodation cavity. Thus, before placing the target object, the legality of the placement pose of each accommodation cavity is verified, and the placement operation is only performed on the target object when the placement poses of each accommodation cavity all pass the legality verification. In this way, the stability of the system operation is improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0116] In some embodiments of the present invention, optionally, the above step 108a may specifically include the following step 108a1 to step 108a3:

[0117] Step 108a1, determine the registration score of the first sub-point cloud according to the first template point cloud and the first sub-point cloud;

[0118] Step 108a2, when the first threshold is less than or equal to the registration score, determine that the registration result of the first sub-point cloud is qualified;

[0119] Step 108a3: When the first threshold is greater than the registration score, it is determined that the registration result of the first sub-point cloud is unqualified.

[0120] In this embodiment, when verifying the registration result of the first sub-point cloud based on the first template point cloud and the first sub-point cloud, that is, verifying the legality of the placement pose of the accommodation cavity, specifically, according to the first template point cloud and the first sub-point cloud, the registration score of the first sub-point cloud is determined. This registration score is used to indicate whether the registration result of the first sub-point cloud is reasonable, that is, to indicate the degree of legality of the placement pose of the accommodation cavity. The higher the registration score, the more reasonable the registration result of the first sub-point cloud, that is, the higher the degree of legality of the placement pose of the accommodation cavity.

[0121] On this basis, the registration score of the first sub-point cloud is compared with the set first threshold. When the first threshold is less than or equal to the registration score of the first sub-point cloud, it indicates that the degree of legality of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively high. At this time, it is determined that the registration result of the first sub-point cloud is qualified. When the set first threshold is greater than the registration score of the first sub-point cloud, it indicates that the degree of legality of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively low. At this time, it is determined that the registration result of the first sub-point cloud is unqualified. In this way, based on the registration score of the first sub-point cloud, the registration result of the first sub-point cloud is verified, that is, the legality of the placement pose of the accommodation cavity is verified, which ensures the accuracy of the verification result, and further ensures the accuracy of the subsequent placement operation based on the verification result, improves the stability of the system operation, enhances the stability of the production line, and reduces the labor cost.

[0122] Among them, for the specific value of the above first threshold, those skilled in the art can set it according to the actual situation, and no specific limitation is made here.

[0123] In some embodiments of the present invention, optionally, the above step 108a1 may specifically include the following steps 108a11 to 108a14:

[0124] Step 108a11: Determine a plurality of second point positions in the first sub-point cloud according to the first template point cloud and the first sub-point cloud;

[0125] Step 108a12: Determine the distance information between the first template point cloud and the plurality of second point positions respectively to obtain a plurality of first distance values;

[0126] Step 108a13: Screen the plurality of first distance values according to a preset range to obtain a plurality of second distance values;

[0127] Step 108a14: Determine the registration score according to the plurality of second distance values;

[0128] Among them, the distribution rules of multiple second point positions are the same as those of some point cloud positions in the first template point cloud.

[0129] In this embodiment, in the process of determining the registration score of the first sub-point cloud according to the first template point cloud and the first sub-point cloud, specifically, the first template point cloud and the first sub-point cloud are analyzed to determine multiple second point positions in the first sub-point cloud according to the distribution of the point cloud positions in the first template point cloud and the first sub-point cloud. The distribution rules of the multiple second point positions are the same as those of some point cloud positions in the first template point cloud. That is, the above-mentioned second point positions are the point cloud positions in the first sub-point cloud with the same distribution rules as the point cloud positions of the first template point cloud.

[0130] Further, according to the first template point cloud and the position information of each second point position, the distance information between the first template point cloud and each second point position is calculated to obtain multiple first distance values. Further, according to the set preset range, the multiple first distance values are screened to screen out multiple second distance values from the multiple first distance values. On this basis, the registration score of the first sub-point cloud is determined according to the screened multiple second distance values. Specifically, the average value of the multiple second distance values is calculated, and then the average value is normalized to obtain a registration score with a value range between 0 and 1. In this way, based on the distribution of the point cloud positions in the first template point cloud and the first sub-point cloud, the registration score of the first sub-point cloud is determined, which ensures the accuracy of the obtained registration score, thereby ensuring the accuracy of the subsequent rationality verification of the registration result of the first sub-point cloud based on the registration score, further improving the stability of the system operation, enhancing the stability of the production line, and reducing the labor cost.

[0131] Among them, the above preset range can specifically be from 0.9 to 0.98, and those skilled in the art can set the specific value range of the above preset range according to the actual situation, and no specific limitation is made here.

[0132] Further, screening the multiple first distance values through the preset range can remove the point cloud positions in the first sub-point cloud that completely match the first template point cloud, so as to measure the registration effect of the first sub-point cloud through the deviation between the incompletely matched point cloud positions in the first sub-point cloud and the first template point cloud, and further improve the accuracy of the obtained registration score.

[0133] In some embodiments of the present invention, optionally, the above step 108a may specifically include the following steps 108a4 to 108a7:

[0134] Step 108a4, obtaining the second transformation matrix of the first sub-point cloud and obtaining the third transformation matrix of the second sub-point cloud;

[0135] Step 108a5, determine the difference value between the third transformation matrix and the second transformation matrix;

[0136] Step 108a6, when the second threshold is greater than or equal to the difference value, determine that the registration result of the first sub-point cloud is qualified;

[0137] Step 108a7, when the second threshold is less than the difference value, determine that the registration result of the first sub-point cloud is unqualified.

[0138] In this embodiment, when verifying the registration result of the first sub-point cloud based on the first template point cloud and the first sub-point cloud, that is, verifying the legality of the placement pose of the accommodation cavity, specifically, obtain the second transformation matrix of the first sub-point cloud during the registration process, and obtain the third transformation matrix of the second sub-point cloud during the registration process. Further, compare the obtained third transformation matrix and the second transformation matrix to determine the difference value between the third transformation matrix and the second transformation matrix. On this basis, compare the obtained difference value with the set second threshold, and verify the registration result of the first sub-point cloud according to the comparison result.

[0139] Specifically, when the set second threshold is greater than or equal to the above difference value, it indicates that the difference between the third transformation matrix and the second transformation matrix is small, that is, it indicates that the legality degree of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively high. At this time, determine that the registration result of the first sub-point cloud is qualified; while when the set second threshold is less than the above difference value, it indicates that the difference between the third transformation matrix and the second transformation matrix is large, that is, it indicates that the legality degree of the placement pose of the accommodation cavity corresponding to the first sub-point cloud is relatively low. At this time, determine that the registration result of the first sub-point cloud is unqualified. In this way, verify the registration result of the first sub-point cloud based on the difference value between the third transformation matrix of the second sub-point cloud and the second transformation matrix of the first sub-point cloud, that is, verify the legality of the placement pose of the accommodation cavity, ensure the accuracy of the verification result, and then ensure the accuracy of the subsequent placement operation based on the verification result, improve the stability of the system operation, enhance the stability of the production line, and reduce the labor cost.

[0140] Among them, since the rotation angles of the placement poses of multiple accommodation cavities in the same row of the target placement device along the X-axis are the same, therefore, the above third transformation matrix and the second transformation matrix can specifically be the transformation matrix Rz obtained by rotating the point cloud along the Z-axis.

[0141] Further, for the specific value of the above second threshold, those skilled in the art can set it according to the actual situation, and no specific limitation is made here.

[0142] In some embodiments of the present invention, optionally, step 108 may specifically include the following steps 108c to 108e:

[0143] Step 108c: Determine the occupancy situation of the accommodation cavities based on the first template point cloud and the first sub-point cloud, so as to obtain a target list of the occupied accommodation cavities.

[0144] Step 108d: When the target list is empty, place the target object in the accommodation cavity of the target placement device according to the target pose in the arrangement order of the accommodation cavities.

[0145] Step 108e: When the target list is not empty and the accommodation cavities in the target list meet the target conditions, place the target object in the first unoccupied accommodation cavity according to the target pose.

[0146] In this embodiment, in the process of placing the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity, specifically, by analyzing and processing the first template point cloud and the first sub-point cloud, determine the occupancy situation of the accommodation cavity corresponding to the first sub-point cloud, that is, determine whether a target object has been placed in the accommodation cavity corresponding to the first sub-point cloud. Based on this, by determining the occupancy situation of each accommodation cavity in the target placement device, a target list of the occupied accommodation cavities in the target placement device is obtained. Further, perform a logical judgment on the target list and perform subsequent work based on the logical judgment result.

[0147] Specifically, when the above target list is empty, that is, when there is no occupied accommodation cavity in the target placement device, the captured target objects are sequentially placed in each accommodation cavity of the target placement device according to the arrangement order of each accommodation cavity in the target placement device and based on the determined target pose. When the above target list is not empty and the accommodation cavities in the above target list meet the target conditions, then according to the determined target pose, starting from the first unoccupied accommodation cavity in the target object placement, the captured target objects are sequentially placed in the unoccupied accommodation cavities. Among them, the above target conditions are: when numbering each accommodation cavity according to the arrangement order of each accommodation cavity in the target placement device, the numbers of the accommodation cavities in the target list are consecutive, and the minimum value of the numbers of the accommodation cavities in the target list is 1, that is, the target list includes the accommodation cavity located at the first position in the target placement device. Further, when the above target list is not empty and the accommodation cavities in the above target list do not meet the above target conditions, then the three-dimensional point cloud of the target placement device is recollected, and registration, verification, etc. are performed again. When the number of times of collecting the three-dimensional point cloud of the target placement device is relatively large, the placement operation is stopped and an alarm is issued. In this way, before placing the target object, the occupancy situation of each accommodation cavity is determined, and then subsequent placement operations or stop placement operations are performed based on the occupancy situation of each accommodation cavity. In this way, the principle of breakpoint continuation during the object placement process is realized, and the stability of the system operation is improved.

[0148] In some embodiments of the present invention, optionally, step 108c may specifically include the following steps 108c1 to 108c3:

[0149] Step 108c1, when the first sub-point cloud is empty, determine that the accommodation cavity corresponding to the first sub-point cloud is occupied;

[0150] Step 108c2, determine the proportion value of the number of second point positions in the first sub-point cloud to the total number of point positions in the first template point cloud;

[0151] Step 108c3, when the third threshold is greater than the proportion value, determine that the accommodation cavity corresponding to the first sub-point cloud is occupied; when the third threshold is less than or equal to the proportion value, determine that the accommodation cavity corresponding to the first sub-point cloud is not occupied;

[0152] Among them, the second point position is the point cloud position in the first sub-point cloud that has the same distribution law as the point cloud position of the first template point cloud.

[0153] In this embodiment, in the process of analyzing and processing the first template point cloud and the first sub-point cloud to determine the occupancy of the accommodation cavity corresponding to the first sub-point cloud, specifically, when the first sub-point cloud is empty, it is determined that the accommodation cavity corresponding to the first sub-point cloud is occupied, that is, it is determined that the target object has been placed in the accommodation cavity corresponding to the first sub-point cloud. Further, the number of point positions of the second point position in the first sub-point cloud is obtained, and the second point position is the point cloud position in the first sub-point cloud that has the same distribution law as the point cloud positions of the first template point cloud. Further, the ratio of the number of point positions of the second point position to the total number of point positions of the first template point cloud is calculated, and this ratio is used to indicate the number of point cloud positions in the first sub-point cloud that match the first template point cloud. On this basis, the above ratio is compared with the set third threshold, and according to the comparison result, the occupancy of the accommodation cavity corresponding to the first sub-point cloud is determined.

[0154] Specifically, when the set third threshold is greater than the above ratio, it indicates that the number of point cloud positions in the first sub-point cloud that match the first template point cloud is small. At this time, it is determined that the accommodation cavity corresponding to the first sub-point cloud is occupied, and the second point position; while when the set third threshold is less than or equal to the above ratio, it indicates that the number of point cloud positions in the first sub-point cloud that match the first template point cloud is large. At this time, it is determined that the accommodation cavity corresponding to the first sub-point cloud is not occupied. In this way, based on the number of point cloud positions in the first sub-point cloud that match the first template point cloud, the occupancy of the accommodation cavity corresponding to the first sub-point cloud is determined, ensuring the accuracy of determining the occupancy of the accommodation cavity, and thus ensuring the accuracy of subsequent work based on the occupancy of the accommodation cavity.

[0155] To sum up, as Figure 5 shown, taking the above target object as a compressor, the above target placement device as a tray, and the above accommodation cavity as the placement pit in the tray for placing the compressor as an example, the object placement method proposed in the embodiment of the present invention may specifically include the following steps 202 to step 228:

[0156] Step 202, receive a shooting signal and start taking pictures;

[0157] Step 204, determine whether the number of repeated photo shootings exceeds 3 times. If so, execute step 228. If not, execute step 206;

[0158] Step 206, obtain the tray point cloud;

[0159] Step 208, preprocess the tray point cloud;

[0160] Step 210, map the tray point cloud to a two-dimensional space to obtain a tray image;

[0161] Step 212: Register the tray image and the template image of the tray;

[0162] Step 214: Intercept the point clouds of each placement pit in the tray according to the ROI algorithm;

[0163] Step 216: Perform point cloud registration on each placement pit in the tray based on the ICP algorithm;

[0164] Step 218: Determine whether the registration of all placement pits in the tray is completed. If so, execute Step 220; if not, execute Step 214;

[0165] Step 220: Determine whether the registration of each placement pit in the tray is successful and whether the occupancy order of the placement pits in the tray is reasonable. If so, execute Step 222; if not, execute Step 202;

[0166] Step 222: Pose migration;

[0167] Step 224: Obtain the placement poses of each placement pit;

[0168] Step 226: Place the compressor in the placement pit of the tray according to the placement pose;

[0169] Step 228: Stop the production line and give an alarm.

[0170] Specifically, in the embodiment of the present invention, in the scenario of the compressor blanking, after the robotic arm successfully grasps the compressor, the robotic arm sends a shooting signal and collects the pallet point cloud through the camera on the pallet side. Further, by preprocessing the collected pallet point cloud, the pallet point cloud after filtering out the background and noise is obtained. Further, the preprocessed pallet point cloud is mapped to a two-dimensional space to obtain a pallet image corresponding to the pallet point cloud. Further, the pallet image obtained by real-time processing is registered with the template image corresponding to the template point cloud of the pallet to obtain a transformation matrix, and the center points of all the placement pits on the pallet are recorded during the real-time processing of the pallet image for subsequent algorithm calculation. Further, according to the ROI (Region of Interest) algorithm, the point clouds of each placement pit on the pallet are intercepted. Specifically, through coordinate transformation, the center points of all the placement pits in the pallet image are mapped back to the pallet point cloud, and centered on the point cloud positions obtained by the mapping, according to the given physical dimensions, the pit point clouds of multiple placement pits are cropped from the pallet point cloud. Further, the cropped pit point clouds of multiple placement pits and the template pit point clouds are registered to obtain the placement poses and matching scores of each placement pit on the pallet. On this basis, according to parameters such as the matching scores, the legality of the placement poses of each placement pit on the pallet is verified, and at the same time, the rationality of the occupancy order of each placement pit on the pallet is verified. Furthermore, based on the qualified placement poses and the principle of breakpoint continuous placement, the compressor is placed in the placement pit of the pallet.

[0171] Among them, during the process of verifying the legality of the placement pose of the placement pit, specifically, if the ICP (Iterative Closest Point) registration score of any one placement pit fails to reach the set threshold, it indicates that the ICP registration of this placement pit fails. At this time, the serial number of this placement pit is recorded. Further, when placing the compressor, the transformation matrix Rz of the rotation along the Z axis of the placement poses of each row of placement pits on the pallet is consistent. Therefore, the transformation matrix Rz of the pit point cloud of each placement pit on the pallet is compared with the transformation matrix Rz of the corresponding template pit point cloud. If the difference between the two exceeds the set threshold, it indicates that the ICP registration of this placement pit fails, and the serial number of this placement pit is recorded. On this basis, a logical judgment is made on the serial numbers of the placement pits with failed ICP registration recorded above. When the serial number list is not empty, that is, when there is any one placement pit with failed ICP registration, it is determined that the entire pallet matching fails. At this time, the pallet point cloud is collected again by taking a picture through the camera on the pallet side, and the number of times of re-taking pictures by the camera is judged. If the camera takes pictures more than 3 times, the production line alarms and controls the robotic arm to stop working. In this way, after verifying the legality of the placement poses of each placement pit on the pallet, the compressor placement work is carried out, improving the stability of the operation of the blanking system.

[0172] Further, the logic of the breakpoint continuous placement principle is as follows: If the pit point cloud of the intercepted placement pit is empty, it indicates that the compressor has been placed in this placement pit, and the continuous placement should start from the next placement pit. At this time, record the serial number of this placement pit. Further, if the ratio of the number of inlier points (the point cloud positions in the pit point cloud that have the same distribution law as the template point cloud) of the intercepted placement pit to the total number of point cloud positions of the template point cloud is less than the set threshold, it indicates that the compressor has been placed in this placement pit, and the continuous placement should start from the next placement pit. At this time, record the serial number of this placement pit. On this basis, perform business logic judgment on the list of serial numbers of the placement pits that have been occupied by the compressor recorded above. When the list is empty, start placing the compressor from the first placement pit on the tray; when the list is not empty, and the pit serial numbers in the list are continuous and the smallest pit serial number in the list is 1, then start placing the compressor from the first unoccupied placement pit on the tray; while in the case where the pit serial numbers in the list are not continuous or the smallest pit serial number in the list is not 1, it is determined that the entire tray matching fails. At this time, re-take a photo through the camera on the tray side to collect the tray point cloud, and judge the number of times of re-taking photos by the camera. If the camera re-takes photos more than 3 times, the production line alarms and controls the robotic arm to stop working.

[0173] In addition, during the process of placing the compressor based on the qualified placement poses, since the transformation matrix Rz of the placement poses of each row of placement pits on the tray rotating along the Z-axis is consistent, therefore, during the process of placing the compressor, the transformation matrix Rz of the placement pit in the middle position of each row of placement pits can be used as the transformation matrix Rz of the entire row of placement pits to place the compressor, so as to increase the stability of the compressor placement.

[0174] In some embodiments of the present invention, optionally, before the above step 102, the above object placement method may specifically further include the following steps 110 to 116:

[0175] Step 110, determine the first pose according to the target point cloud and the target template point cloud of the target object;

[0176] Step 112, determine the target compensation value according to the target image and the target template image of the target object;

[0177] Step 114, adjust the first pose according to the target compensation value to obtain the second pose;

[0178] Step 116, grasp the target object according to the second pose.

[0179] In this embodiment, by photographing the target object, the target point cloud and the target image of the target object are collected. At the same time, the target template point cloud and the target template image of the target object stored in advance are retrieved. Further, the target point cloud is preprocessed to filter out the background information and noise information in the target point cloud. Then, the obtained target template point cloud and the preprocessed target point cloud are registered, and the initial grasping pose, i.e., the first pose, for grasping the target object is determined according to the registration result. Further, the obtained target template image and the target image are processed and analyzed to determine a target compensation value based on the difference between the target template image and the target image.

[0180] On this basis, through the determined target compensation value, the above-determined initial grasping pose, i.e., the first pose, is compensated and adjusted to obtain a more accurate second pose, and the target object is grasped according to the compensated second pose. In this way, during the process of grasping the target object, after determining the initial grasping pose, i.e., the first pose, for grasping the target object through point cloud registration, the initial grasping pose of the target object is compensated by means of the two-dimensional image of the target object, so as to obtain an accurate second pose. In this way, the workpiece error generated by the target object during the production process can be compensated, thereby improving the accuracy of the grasping pose of the target object, enhancing the accuracy of grasping the target object, thus improving the speed and efficiency of grasping the target object, enhancing the stability of the production line, and reducing the labor cost.

[0181] In an embodiment of the present invention, step 112 may specifically include the following steps 112a to 112c:

[0182] Step 112a, extracting at least two target feature points from the target image;

[0183] Step 112b, determining target feature parameters according to the coordinate information of the at least two target feature points;

[0184] Step 112c, determining the target compensation value according to the template feature parameters and the target feature parameters.

[0185] In this embodiment, in the process of processing and analyzing the target template image and the target image to determine the above target compensation value, specifically, feature extraction is performed on the target image to extract at least two target feature points in the target image. Further, according to the coordinate information of the at least two extracted target feature points, the target feature parameters of the target image are determined. The target feature parameters correspond to the actual features of the target object, and through the target feature parameters, the actual feature information of the target object, such as size feature information, structural feature information, etc., can be indicated. Further, the target template image corresponding to the target template point cloud of the target object is retrieved, and the template feature parameters in the target template image are obtained.

[0186] On this basis, the above target feature parameters and template feature parameters are further compared to determine the difference value between the target feature parameters and the template feature parameters, and then, based on this difference value, the target compensation value for compensating the above first pose is determined. In this way, the target feature parameters of the target image are determined through feature extraction, and then, based on the difference value between the target feature parameters and the template feature parameters, the difference between the target template image and the target image is analyzed, so as to determine the above target compensation value. In this way, the accuracy of determining the target compensation value is ensured, and then the accuracy of determining the second pose is ensured, the accuracy of grasping the target object is improved, and thus the speed and efficiency of grasping the target object are improved.

[0187] In an embodiment of the present invention, step 110 may specifically include the following steps 110a to 110c:

[0188] Step 110a, registering the target template point cloud and the target point cloud;

[0189] Step 110b, determining the registration score of the target point cloud according to the target template point cloud and the target point cloud;

[0190] Step 110c, when the target threshold is less than the registration score of the target point cloud, determining the first pose according to the target template pose and the registration result.

[0191] In this embodiment, in the process of determining the initial grasping pose, that is, the first pose, of the target object according to the target template point cloud and the target point cloud of the target object, specifically, the obtained target template point cloud and the target point cloud are registered, and the registration score for registering the target point cloud is determined according to the registered target template point cloud and the target point cloud. The registration score of the target point cloud is used to indicate whether the registration result of the target point cloud is reasonable, that is, to indicate whether the target point cloud is successfully registered. The higher the registration score of the target point cloud, the more reasonable the registration result of the target point cloud is, that is, the more successful the target point cloud is registered.

[0192] On this basis, the registration score of the target point cloud is compared with a set target threshold. Only when the set target threshold is less than the registration score of the target point cloud, that is, only when the registration result of the target point cloud is reasonable, that is, only when the registration of the target point cloud is successful, the initial grasping pose for grasping the target object, namely the above-mentioned first pose, is determined according to the target template point cloud, the registration result of the target point cloud, and the target template pose corresponding to the target template point cloud. In this way, only when the registration of the target point cloud is successful, the initial grasping pose for the target object, namely the above-mentioned first pose, is obtained, which ensures the usability of the first pose, and further ensures the accuracy of the subsequently determined second pose, improves the accuracy of grasping the target object, and improves the speed and efficiency of grasping the target object.

[0193] Among them, for the specific value of the above target threshold, those skilled in the art can set it according to the actual situation, and no specific limitation is made here.

[0194] In an embodiment of the present invention, the above step 110b may specifically include the following steps 110b1 to 110b4:

[0195] Step 110b1: Determine a plurality of target point positions in the target point cloud according to the target template point cloud and the target point cloud;

[0196] Step 110b2: Determine the distance information between the target template point cloud and each target point position according to the target template point cloud and the position coordinates of the plurality of target point positions, and obtain a plurality of third distance values;

[0197] Step 110b3: Screen the plurality of third distance values according to the target range to obtain a plurality of fourth distance values;

[0198] Step 110b4: Determine the registration score of the target point cloud according to the plurality of fourth distance values;

[0199] Among them, the distribution rules of the plurality of target point positions are the same as those of some point cloud positions in the target template point cloud.

[0200] In this embodiment, in the process of determining the registration score of the target point cloud according to the target template point cloud and the target point cloud, specifically, the target template point cloud and the target point cloud are analyzed to determine a plurality of target point positions in the target point cloud according to the distribution of the point cloud positions in the target template point cloud and the target point cloud. The distribution rules of the plurality of target point positions are the same as those of some point cloud positions in the target template point cloud, that is, the above-mentioned target point positions are the point cloud positions in the target point cloud that have the same distribution rules as the point cloud positions of the target template point cloud.

[0201] Further, based on the target template point cloud and the position information of each target point position, calculate the distance information between the target template point cloud and each target point position to obtain a plurality of third distance values. Further, according to the set target range, screen the plurality of third distance values to screen out a plurality of fourth distance values from the plurality of third distance values. On this basis, determine the registration score of the target point cloud according to the screened plurality of fourth distance values. Specifically, calculate the average value of the plurality of fourth distance values, and then normalize the average value to obtain a registration score with a value range between 0 and 1. In this way, based on the distribution of the point cloud positions in the target template point cloud and the target point cloud, the registration score of the target point cloud is determined, which ensures the accuracy of the obtained registration score, thereby ensuring the accuracy of the subsequent rationality verification of the registration result of the target point cloud based on the registration score, further improving the stability of the system operation, enhancing the stability of the production line, and reducing the labor cost.

[0202] Among them, the above target range may specifically be from 0.9 to 0.98, and those skilled in the art can set the specific value range of the above target range according to the actual situation, and no specific limitation is made here.

[0203] Further, screening the plurality of third distance values through the target range can remove the point cloud positions in the target point cloud that completely match the target template point cloud, so as to measure the registration effect of the target point cloud through the deviation between the point cloud positions in the target point cloud that do not completely match and the target template point cloud, and further improve the accuracy of the obtained registration score.

[0204] In an embodiment of the present invention, step 110c may specifically include the following steps 110c1 to 110c3:

[0205] Step 110c1, determine the target transformation matrix according to the registration result;

[0206] Step 110c2, determine the target deflection information of the target point cloud according to the target transformation matrix;

[0207] Step 110c3, determine the first pose according to the target template pose and the target deflection information.

[0208] In this embodiment, in the process of determining the initial grasping pose for grasping the target object, i.e., the first pose above, based on the registration result of the target template point cloud and the target point cloud, specifically, based on the registration result of the target template point cloud and the target point cloud, the target transformation matrix of the target point cloud is determined, and then, based on this target transformation matrix, the target deflection information of the target point cloud relative to the target template point cloud is determined. On this basis, the target template pose corresponding to the target template point cloud is further obtained, and based on this target template pose and the above target deflection information, the initial grasping pose for grasping the target object, i.e., the first pose above, is determined. In this way, by means of point cloud registration, the initial grasping pose for grasping the target object, i.e., the first pose above, is determined, ensuring the accuracy of the determination of the first pose, and further ensuring the accuracy of the second pose determined subsequently, improving the accuracy of grasping the target object, and enhancing the speed and efficiency of grasping the target object.

[0209] In summary, in the embodiment of the present application, as Figure 6 shown, taking the above target object as a compressor as an example, the object grasping process in the object placement method provided by the embodiment of the present application may specifically include the following steps 302 to step 326:

[0210] Step 302: Determine whether the scanning of the product one-dimensional code is successful. If so, execute step 304; if not, execute step 306.

[0211] Step 304: The camera captures an image.

[0212] Step 306: The production line stops and alarms.

[0213] Step 308: Determine whether the number of repeated photo shootings exceeds 3 times. If so, execute step 306; if not, execute steps 310 and 312.

[0214] Step 310: Obtain the compressor point cloud.

[0215] Step 312: Obtain the compressor image.

[0216] Step 314: Preprocess the compressor point cloud.

[0217] Step 316: Obtain the Rz compensation value.

[0218] Step 318: Perform ICP registration on the compressor point cloud.

[0219] Step 320: Determine whether the registration score of the compressor point cloud is greater than the threshold. If so, execute step 322; if not, execute step 304.

[0220] Step 322: Pose migration.

[0221] Step 324: Obtain the grasping pose.

[0222] Step 326, grasp the compressor according to the grasping pose.

[0223] Specifically, in the embodiment of the present invention, in the scenario of discharging the compressor, after determining the placement pose of the compressor, i.e., the above-mentioned target pose, by the above object placement method, the robotic arm emits a shooting signal to enable the 3D camera and 2D camera on the compressor side to start taking pictures, so as to obtain the compressor image and the compressor point cloud. Further, by preprocessing the compressor point cloud, the compressor point cloud after removing the background and noise is obtained. Further, the ICP registration is performed on the preprocessed compressor point cloud and the template compressor point cloud to obtain the transformation matrix and the initial grasping pose. Further, there is a workpiece error at the bottom of the compressor. The initial grasping pose is compensated by the compressor image to obtain a more accurate grasping pose, and the compensated grasping pose is returned to the robotic arm for the compressor grasping work. In this way, after registering the preprocessed real-time compressor point cloud with the template compressor point cloud to obtain the migrated grasping pose, the migrated grasping pose is further compensated based on the 2D method. In this way, the problem that the compressor cannot be accurately placed in the placement pit of the tray due to the tooling error of the compressor is solved, the grasping accuracy and placement accuracy of the compressor are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0224] Among them, in the process of compensating the initial grasping pose by the compressor image, specifically, by processing the compressor image, the centers of two circular features on the compressor are found in the compressor image. Further, according to the two centers, the included angle corresponding to the straight line segment where the two centers are located is calculated, and this included angle is compared with the included angle corresponding to the two centers in the template compressor image. According to the deviation value of the two included angles, the angular value of the Rz compensation is obtained, and then the compensation value of Rz is obtained, and this compensation value is used to replace the Rz in the initial grasping pose obtained by registration to obtain a more accurate grasping pose.

[0225] In summary, the object placement method proposed in the present invention is aimed at the compressor unloading scene. On the compressor grabbing side, since the compressor has production errors in production, in order to compensate for the errors, in addition to 3D registration of the compressor point cloud, a 2D compensation method is also introduced to compensate for the grabbing posture, thereby improving the compressor grabbing accuracy. On the compressor placement side, the pallet point cloud is projected into the 2D space, and the positions of multiple groups of placement pits are determined by the 2D method, and then multiple groups of pit point clouds are cut out from the 3D pallet point cloud, and the multiple groups of pit point clouds are respectively 3D registered with the template pit points to obtain the placement posture. The above registration method belongs to a one-to-one method, which saves calculation time and improves placement accuracy and efficiency. At the same time, by continuing to place the compressor at the breakpoint, the problem of secondary photography during the compressor placement process is solved, and in the case of three consecutive photography failures or three consecutive registration failures of the compressor, the production line will alarm and stop the line, which improves the stability of the unloading system and reduces the labor cost of the factory. The above-mentioned material cutting solution is highly reusable, and similar projects can be quickly launched without any development, which saves the development cycle and makes the promotion and replication of projects easier and faster.

[0226] In one embodiment of the present invention, an object placement device is also provided. Figure 7 As shown, Figure 7 The structure block diagram of the object placement device 400 according to the embodiment of the present invention is shown. The object placement device 400 may specifically include the following processing unit 402 and placement unit 404:

[0227] A processing unit 402 is used to determine a first image of the target placement device according to a first point cloud of the target placement device;

[0228] The processing unit 402 is further configured to determine, according to the first image, a first sub-point cloud corresponding to any one of the accommodating cavities in the first point cloud;

[0229] The processing unit 402 is further used to register the first sub-point cloud with the second sub-point cloud in the first template point cloud of the target placement device, and determine the target posture corresponding to any one of the accommodating cavities according to the registration result;

[0230] A placement unit 404, used to place the target object in a receiving cavity of the target placement device according to the target posture;

[0231] The target placement device includes a plurality of accommodating cavities, the accommodating cavities are used to place the target objects, and the second sub-point cloud corresponds to the first sub-point cloud.

[0232] The object placement device 400 provided by the embodiment of the present invention includes a processing unit 402 and a placement unit 404. During the process of placing the target object, the target placement device is photographed to obtain the first point cloud of the target placement device in the three-dimensional space. Then, the processing unit 402 performs two-dimensional mapping on the first point cloud of the target placement device to map the first point cloud in the three-dimensional space into the two-dimensional space, thereby obtaining the first image of the target placement device in the two-dimensional space. Further, the above-mentioned target placement device includes a plurality of accommodation cavities, and the accommodation cavities are used to place the grabbed target objects.

[0233] On this basis, after obtaining the first image of the target placement device in the two-dimensional space, the processing unit 402 further crops the first point cloud of the target placement device in the three-dimensional space based on the first image to crop a plurality of first sub-point clouds from the first point cloud, and the plurality of first sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device. Further, the processing unit 402 retrieves the first template point cloud of the target placement device, and the first template point cloud includes a plurality of second sub-point clouds, and the plurality of second sub-point clouds correspond one-to-one to the plurality of first sub-point clouds, that is, the plurality of second sub-point clouds correspond one-to-one to the plurality of accommodation cavities in the target placement device.

[0234] On this basis, for any one of the accommodation cavities in the target placement device, that is, for any one of the first sub-point clouds in the first point cloud, the processing unit 402 registers the first sub-point cloud with its corresponding second sub-point cloud, and then based on the registration result, determines a target pose, and the target pose is the placement pose corresponding to the accommodation cavity when the target object is placed in the accommodation cavity corresponding to the first sub-point cloud. On this basis, after the processing unit 402 has determined the target pose corresponding to each accommodation cavity in the target placement device, the placement unit 404 then places the grabbed target object in the accommodation cavity of the target placement device according to the target pose corresponding to each accommodation cavity.

[0235] In this way, during the process of placing the target object, the first point cloud of the target placement device in the three-dimensional space is projected into the two-dimensional space, combined with the two-dimensional method, to determine the positions of the plurality of accommodation cavities in the target placement device. Then, in the first point cloud of the target placement device, a plurality of first sub-point clouds corresponding to the plurality of accommodation cavities are cropped respectively, and the cropped plurality of first sub-point clouds are respectively registered with the plurality of second sub-point clouds in the first template point cloud of the target placement device, so as to obtain the placement pose corresponding to each accommodation cavity. In this way, by integrating two-dimensional and three-dimensional algorithms and calculating the placement pose of each accommodation cavity in a one-to-one manner, the problems of long time consumption, high dependence on the initial transformation matrix and poor accuracy of the conventional point cloud registration algorithm are solved, the registration time is reduced, the placement accuracy and placement efficiency are improved, the stability of the production line is enhanced, and the labor cost is reduced.

[0236] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine a target area according to the coordinate information of the first point cloud; adjust the position information of the first point cloud according to the target area to obtain a second point cloud; and determine a first image according to the size information of the target area and the coordinate information of the second point cloud.

[0237] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine a plurality of pixel abscissas according to the product of the abscissas of a plurality of point cloud positions in the second point cloud and a first value respectively; determine a plurality of pixel ordinates according to the product of the ordinates of the plurality of point cloud positions in the second point cloud and the first value respectively; determine a plurality of pixel values according to the ratio of the vertical coordinates of the plurality of point cloud positions in the second point cloud to the height of the target area and the product of the ratio and a target value respectively; determine a first image according to the plurality of pixel values, the plurality of pixel ordinates, and the plurality of pixel abscissas; wherein, there is a one-to-one correspondence among the plurality of pixel values, the plurality of pixel ordinates, and the plurality of pixel abscissas.

[0238] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine the center point information of any accommodation cavity in the first image according to the first image; register the first template image of the first template point cloud and the first image to determine a first transformation matrix; determine a first point position in the first point cloud according to the first transformation matrix and the center point information; and determine a first sub-point cloud according to the target size information and the first point position.

[0239] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine a second transformation matrix of the first sub-point cloud and a third transformation matrix of the second sub-point cloud according to the registration result; determine a first deflection information of the first sub-point cloud according to the third transformation matrix and the second transformation matrix; and determine a target pose according to the template pose information of any accommodation cavity and the first deflection information.

[0240] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: verify the registration result of the first sub-point cloud according to the first template point cloud and the first sub-point cloud; the placement unit 404 is specifically configured to: place the target object in the accommodation cavity of the target placement device according to the target pose when the registration result of the first sub-point cloud is qualified.

[0241] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine a registration score of the first sub-point cloud according to the first template point cloud and the first sub-point cloud; determine that the registration result of the first sub-point cloud is qualified when a first threshold is less than or equal to the registration score; and determine that the registration result of the first sub-point cloud is unqualified when the first threshold is greater than the registration score.

[0242] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine a plurality of second point positions in the first sub-point cloud according to the first template point cloud and the first sub-point cloud; determine distance information between the first template point cloud and the plurality of second point positions respectively to obtain a plurality of first distance values; screen the plurality of first distance values according to a preset range to obtain a plurality of second distance values; and determine a registration score according to the plurality of second distance values; wherein, the distribution rules of the plurality of second point positions are the same as those of some point cloud positions in the first template point cloud.

[0243] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: obtain a second transformation matrix of the first sub-point cloud and obtain a third transformation matrix of the second sub-point cloud; determine a difference value between the third transformation matrix and the second transformation matrix; determine that the registration result of the first sub-point cloud is qualified when the second threshold is greater than or equal to the difference value; and determine that the registration result of the first sub-point cloud is unqualified when the second threshold is less than the difference value.

[0244] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: determine the occupancy situation of the accommodation cavities according to the first template point cloud and the first sub-point cloud to obtain a target list of the occupied accommodation cavities; the placement unit 404 is specifically configured to: when the target list is empty, place the target object in the accommodation cavity of the target placement device according to the target pose in the arrangement order of the accommodation cavities; and when the target list is not empty and the accommodation cavities in the target list meet the target conditions, place the target object in the first unoccupied accommodation cavity according to the target pose.

[0245] In some embodiments of the present invention, optionally, the processing unit 402 is specifically configured to: when the first sub-point cloud is empty, determine that the accommodation cavity corresponding to the first sub-point cloud is occupied; determine a proportion value of the number of second point positions in the first sub-point cloud to the total number of point positions in the first template point cloud; determine that the accommodation cavity corresponding to the first sub-point cloud is occupied when the third threshold is greater than the proportion value, and determine that the accommodation cavity corresponding to the first sub-point cloud is not occupied when the third threshold is less than or equal to the proportion value; wherein, the second point positions are the point cloud positions in the first sub-point cloud with the same distribution rules as the point cloud positions of the first template point cloud.

[0246] Further, in the actual application process, the above object placement device 400 can implement the steps of the object placement method in any of the above embodiments, and the above object placement device 400 can implement the steps of the object placement method described above Figures 1 to 6 and can achieve the same technical effects, which will not be elaborated here.

[0247] In an embodiment of the present invention, another object placement device is also proposed. As Figure 8 shown,Figure 8 The structural block diagram of the object placement device 500 provided by the embodiment of the present invention is shown. Among them, the object placement device 500 includes:

[0248] A memory 502, on which programs or instructions are stored;

[0249] A processor 504, when the processor 504 executes the above programs or instructions, implements the steps of the object placement method in any of the above embodiments.

[0250] The object placement device 500 provided in this embodiment includes a memory 502 and a processor 504. When the programs or instructions in the memory 502 are executed by the processor 504, the steps of the object placement method in any of the above embodiments are implemented. Therefore, the object placement device 500 has all the beneficial effects of the object placement method in any of the above embodiments, which will not be elaborated here.

[0251] Specifically, the memory 502 and the processor 504 can be connected through a bus or other means. The processor 504 may include one or more processing units, and the processor 504 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0252] In an embodiment of the present invention, an object grasping and placing system is also proposed. As Figure 9 shown, Figure 9 The structural block diagram of the object grasping and placing system 600 provided by the embodiment of the present invention is shown. Among them, the object grasping and placing system 600 is the object placement device 500 in the above embodiment. Therefore, the object grasping and placing system 600 has all the technical effects of the object placement device 500 in the above embodiment, which will not be elaborated here.

[0253] In the embodiment of the fifth aspect of the present invention, a readable storage medium is proposed. Programs or instructions are stored thereon, and when the programs or instructions are executed by a processor, the steps of the object placement method in any of the above embodiments are implemented.

[0254] For the readable storage medium provided by the embodiment of the present invention, when the stored programs or instructions are executed by a processor, the steps of the object placement method in any of the above embodiments can be implemented. Therefore, the readable storage medium has all the beneficial effects of the object placement method in any of the above embodiments, which will not be elaborated here.

[0255] Specifically, the above-readable storage medium may include any medium capable of storing or transmitting information. Examples of the readable storage medium include electronic circuits, semiconductor memory devices, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), flash memory, erasable ROM (EROM), magnetic tapes, floppy disks, optical discs, hard disks, optical fiber media, radio frequency (RF) links, optical data storage devices, etc. The code segment may be downloaded via a computer network such as the Internet, an intranet, etc.

[0256] An embodiment of the sixth aspect of the present invention provides a computer program product, which includes a computer program that, when executed by a processor, implements the object placement method in any of the above technical solutions. Therefore, the computer program product proposed in the sixth aspect of the present invention has all the beneficial effects of the object placement method in any of the technical solutions in the first aspect above, and will not be described in detail here.

[0257] In the description of this specification, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and defined; the terms "connection", "installation", "fixation", etc. should all be understood in a broad sense. For example, "connection" may be a fixed connection, a detachable connection, or an integral connection; it may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0258] In the description of this specification, the descriptions of the terms "an embodiment", "some embodiments", "specific embodiments", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or instance. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples.

[0259] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0260] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for placing an object, characterized in that, Including: Determining a first image of the target placement device based on a first point cloud of the target placement device, the target placement device including a plurality of accommodation cavities for placing target objects; Determining a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image; Registering the first sub-point cloud and a second sub-point cloud in a first template point cloud of the target placement device, and determining a target pose corresponding to any one of the accommodation cavities according to a registration result, the first sub-point cloud corresponding to the second sub-point cloud; Placing the target object in the accommodation cavity of the target placement device according to the target pose; The determining the first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image includes: Determining center point information of any one of the accommodation cavities in the first image according to the first image; Performing two-dimensional registration on the first image and a first template image of the first template point cloud to determine a first transformation matrix; Determining a first point position in the first point cloud according to the center point information and the first transformation matrix; Determining the first sub-point cloud according to the first point position and target size information.

2. The method for placing an object according to claim 1, characterized in that, The determining the first image of the target placement device based on the first point cloud of the target placement device includes: Determining a target area according to coordinate information of the first point cloud; Adjusting position information of the first point cloud according to the target area to obtain a second point cloud; Determining the first image according to coordinate information of the second point cloud and size information of the target area.

3. The method for placing an object according to claim 2, characterized in that, The determining the first image according to coordinate information of the second point cloud and size information of the target area includes: Determining a plurality of pixel abscissas according to products of a first value and abscissas of a plurality of point cloud positions in the second point cloud respectively; Determining a plurality of pixel ordinates according to products of the first value and ordinates of a plurality of point cloud positions in the second point cloud respectively; Determining a plurality of pixel values according to products of a target value and ratios of vertical coordinates of a plurality of point cloud positions in the second point cloud to the height of the target area respectively; Determining the first image according to the plurality of pixel abscissas, the plurality of pixel ordinates and the plurality of pixel values; Wherein, the plurality of pixel abscissas, the plurality of pixel ordinates and the plurality of pixel values correspond to each other one by one.

4. The method for placing an object according to claim 1, characterized in that, The determining the target pose corresponding to any one of the accommodation cavities according to the registration result includes: Determining a second transformation matrix of the first sub-point cloud and a third transformation matrix of the second sub-point cloud according to the registration result; Determining first deflection information of the first sub-point cloud according to the second transformation matrix and the third transformation matrix; Determining the target pose according to the first deflection information of the first sub-point cloud and template pose information of any one of the accommodation cavities.

5. The method for placing an object according to claim 1, characterized in that, The placing the target object in the accommodation cavity of the target placement device according to the target pose includes: Verifying the registration result of the first sub-point cloud according to the first sub-point cloud and the first template point cloud; When the registration result of the first sub-point cloud is qualified, place the target object in the accommodation cavity of the target placement device according to the target pose.

6. The method for placing an object according to claim 5, characterized in that, The method of verifying the registration result of the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: Determine the registration score of the first sub-point cloud according to the first sub-point cloud and the first template point cloud; When the registration score is greater than or equal to the first threshold, determine that the registration result of the first sub-point cloud is qualified; When the registration score is less than the first threshold, determine that the registration result of the first sub-point cloud is unqualified.

7. The method for placing an object according to claim 6, characterized in that, The method of determining the registration score of the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: Determine a plurality of second point positions in the first sub-point cloud according to the first sub-point cloud and the first template point cloud, and the distribution rules of the plurality of second point positions are the same as those of some point cloud positions in the first template point cloud; Determine the distance information between the plurality of second point positions and the first template point cloud to obtain a plurality of first distance values; Screen a plurality of second distance values from the plurality of first distance values according to a preset range; Determine the registration score of the first sub-point cloud according to the plurality of second distance values.

8. The object placement method according to claim 5, wherein, The method of verifying the registration result of the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: Obtain the second transformation matrix of the first sub-point cloud, and obtain the third transformation matrix of the second sub-point cloud corresponding to the first sub-point cloud in the first template point cloud; Determine the difference value between the second transformation matrix and the third transformation matrix; When the difference value is less than or equal to the second threshold, determine that the registration result of the first sub-point cloud is qualified; When the difference value is greater than the second threshold, determine that the registration result of the first sub-point cloud is unqualified.

9. The object placement method according to any one of claims 1 to 8, wherein, The method of placing the target object in the accommodation cavity of the target placement device according to the target pose includes: Determine the occupancy situation of the accommodation cavity corresponding to the first sub-point cloud according to the first sub-point cloud and the first template point cloud, so as to obtain a target list of occupied accommodation cavities; When the target list is empty, place the target object in the accommodation cavity of the target placement device according to the target pose in the arrangement order of the accommodation cavities in the target placement device; When the target list is not empty and the accommodation cavities in the target list meet the target conditions, place the target object in the first unoccupied accommodation cavity according to the target pose.

10. The object placement method according to claim 9, wherein, The method of determining the occupancy situation of the accommodation cavity corresponding to the first sub-point cloud according to the first sub-point cloud and the first template point cloud includes: When the first sub-point cloud is empty, determine that the accommodation cavity corresponding to the first sub-point cloud is occupied; When the proportion value of the number of second point positions in the first sub-point cloud to the total number of point positions in the first template point cloud is less than the third threshold, determine that the accommodation cavity corresponding to the first sub-point cloud is occupied, and the second point position is the point cloud position in the first sub-point cloud with the same distribution rule as the point cloud position in the first template point cloud. When the ratio value is greater than or equal to the third threshold, it is determined that the accommodation cavity corresponding to the first sub-point cloud is not occupied.

11. An object placement device, wherein, Comprising: A processing unit, configured to determine a first image of the target placement device according to a first point cloud of the target placement device, where the target placement device includes a plurality of accommodation cavities for placing target objects; The processing unit is further configured to determine a first sub-point cloud corresponding to any one of the accommodation cavities in the first point cloud according to the first image; The processing unit is further configured to register the first sub-point cloud and a second sub-point cloud in a first template point cloud of the target placement device, and determine a target pose corresponding to any one of the accommodation cavities according to a registration result, where the first sub-point cloud and the second sub-point cloud correspond to each other; A placement unit, configured to place the target object in the accommodation cavity of the target placement device according to the target pose; Specifically, the processing unit is configured to, according to the first image, determine center point information of any one of the accommodation cavities in the first image; perform two-dimensional registration on the first image and a first template image of the first template point cloud to determine a first transformation matrix; determine a first point position in the first point cloud according to the center point information and the first transformation matrix; and determine the first sub-point cloud according to the first point position and target size information.

12. An object placement device, wherein, Comprising: A memory, storing a program or instruction; A processor, when the processor executes the program or instruction, implementing the steps of the object placement method according to any one of claims 1 to 10.

13. An object grasping and placing system, wherein, Comprising: The object placement device according to claim 12.

14. A readable storage medium, wherein, A program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, implementing the steps of the object placement method according to any one of claims 1 to 10.

15. A computer program product, comprising a computer program, wherein, When the computer program is executed by a processor, implementing the steps of the object placement method according to any one of claims 1 to 10.

Images