Material box pose recognition method and system, storage medium and robot

Abstract

The invention provides a material box pose recognition method and system, a storage medium and a robot, and the method comprises the steps: collecting scene image data and point cloud data, and building a mapping relation; rGB pixels belonging to the feature surface of the workbin in the image data are identified, and a corresponding workbin point cloud is obtained according to the mapping relation; performing plane fitting processing on the point cloud of the workbin to determine a plane normal direction so as to estimate a rotation angle yaw of the workbin; the center P'of the material box is roughly positioned, and plane point clouds in the plane area are screened out based on yaw and the specification of the material box; projecting the plane point cloud into a binary image, screening out the outline of the material box, and carrying out internal area filling treatment; and calculating the center of mass of the internal area of the workbin contour, and estimating the position of the scene space where the center of the workbin is located according to the position and the projection relation of the pixels of the center of mass. Therefore, the posture of the material box is estimated by identifying the shape characteristics of the material box.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision recognition, and in particular to a method and system for recognizing the position and posture of a material box by using machine vision, a storage medium, and a robot. Background Art

[0002] With the rapid development of the manufacturing industry, material picking has become a key procedure in the assembly line operation of manufacturing factories. The manual material picking process is relatively cumbersome, and warehouse personnel shuttle through dense warehouses and engage in heavy physical labor, which is not only inefficient but also unsafe. Warehouse robots free warehouse managers from the picking tasks in dense warehouses, assist warehouse managers in automatically picking materials, and realize the end-to-end picking process from goods to people.

[0003] The key point of warehouse robots picking materials autonomously is to guide the warehouse robots to autonomously identify the cargo boxes and adjust their own state according to the identified posture to complete the cargo picking. In this process, the accuracy of the posture of the cargo boxes obtained through identification is the key to ensure that the warehouse robots pick the correct materials.

[0004] There are many ways to locate the posture of the material box. A typical approach is to post markers on the surface of the material box and estimate the posture of the material box by identifying specific markers, such as QR codes, reflective stickers, etc. However, the disadvantage of this approach is that the markers themselves not only increase the material cost, but also require manual design, management and pasting, which increases labor costs.

[0005] Therefore, if the position of the material box can be directly identified through machine vision solutions, this will greatly save labor and material costs, and further improve the intelligence and efficiency of warehouse management. Summary of the invention

[0006] To this end, the main purpose of the present invention is to provide a method and system for recognizing the posture of a material box, a storage medium, and a robot, so as to estimate the posture of the material box by recognizing the shape characteristics of the material box itself.

[0007] In order to achieve the above object, according to one aspect of the present invention, a method for identifying a container posture is provided, the steps of which include:

[0008] Collect scene image data and point cloud data, and establish mapping relationships;

[0009] Identify the RGB pixels belonging to the characteristic surface of the material box in the image data, and obtain the corresponding material box point cloud according to the mapping relationship;

[0010] Perform plane fitting processing on the material box point cloud to determine the plane normal to estimate the material box rotation angle yaw;

[0011] Roughly locate the center P' of the material box, and filter out the plane point cloud in the plane area based on yaw and material box specifications;

[0012] Project the plane point cloud into a binary image, filter out the outline of the material box and fill the internal area;

[0013] Calculate the centroid of the inner area of ​​the box outline, and estimate the position of the box center in the scene space based on the position and projection relationship of the centroid pixel , .

[0014] In a possible preferred embodiment, the step of identifying RGB pixels belonging to a bin in the image data comprises:

[0015] A pre-trained image recognition network is used to identify the characteristic faces of the bin and obtain the recognition bounding box of the characteristic faces of the bin;

[0016] Extract the RGB pixels within the bounding box.

[0017] In a possible preferred embodiment, the method for identifying the position and posture of a material box further comprises the following steps:

[0018] The material box point cloud is denoised, and the steps include: gridding the material box to identify the bounding box, traversing all grids to vote for the points within the grid according to the preset voting point number threshold, finding the largest continuous interval, and taking the point cloud data in the continuous interval with the largest number of points as the material box point cloud until the traversal is completed.

[0019] In a possible preferred embodiment, the step of estimating the box rotation angle yaw includes:

[0020] Calculate the covariance matrix C of the bin point cloud, and calculate its eigenvectors and eigenvalues ​​based on the covariance matrix C;

[0021] Select the eigenvector with the largest eigenvalue as the normal vector N;

[0022] The normal vector N is normalized to take the angle between its normal and the fitting plane as the box rotation angle yaw.

[0023] In a possible preferred implementation, the step of projecting the plane point cloud into a binary image includes:

[0024] Construct the rotation matrix R of the plane point cloud to the orthographic position according to the rotation angle yaw of the material box;

[0025] The planar point cloud is transformed according to the rotation matrix R and projected into a binary image at a preset scale.

[0026] In a possible preferred implementation, the step of estimating the scene space position of the center of the container includes:

[0027] According to the mapping relationship, the position of the centroid pixel in the scene space is determined, and the size is restored according to the preset scaling ratio to obtain the centroid position , ;

[0028] Calculate the Z-axis average of a plane point cloud ;

[0029] Transformed by the rotation matrix R , , , get the center position of the material box , , .

[0030] In a possible preferred embodiment, the steps of roughly locating the center P' of the material box and screening out the plane point cloud in the plane area based on yaw and the material box specifications include:

[0031] Roughly locate the center P' of the material box according to the extreme boundary of the material box point cloud;

[0032] Based on the center P' of the material box and the rotation angle yaw of the material box, the width W and height H corresponding to the material box specifications are defined to define the boundaries of the plane area:

[0033]

[0034]

[0035]

[0036]

[0037] Based on the boundary of the plane area, the plane point cloud is filtered out.

[0038] In order to achieve the above object, according to another aspect of the present invention, a method for identifying a container posture is provided, the steps of which include:

[0039] Calibrate the viewfinder, transform the viewfinder coordinate system to the robot coordinate system, and construct the transformation matrix T;

[0040] Collect scene image data and point cloud data and establish mapping relationships;

[0041] Identify the RGB pixels belonging to the characteristic surface of the material box in the image data, and obtain the corresponding material box point cloud according to the mapping relationship;

[0042] According to the transformation matrix T, the material box point cloud is transformed into the robot coordinate system, and the material box point cloud is subjected to plane fitting processing to determine the plane normal, so as to preliminarily estimate the material box rotation angle yaw';

[0043] The center P' of the material box is roughly located according to the extreme value boundary of the material box point cloud; the plane area is determined according to the specifications of the material box, and the plane point cloud is screened out based on P' and yaw';

[0044] Perform plane fitting processing based on the plane point cloud to determine the plane normal to estimate the rotation angle yaw of the material box;

[0045] Construct the rotation matrix R of the plane point cloud to the orthogonal position based on yaw;

[0046] According to the rotation matrix R and the preset scaling ratio, the plane point cloud is orthographically projected into a binary image, and the contour of the material box is screened out and the internal area is filled based on the condition of finding the contour with the largest enclosing area;

[0047] Calculate the centroid of the inner area of ​​the box outline and estimate the centroid position based on the position and scale of the centroid pixel , , and calculate the Z-axis average of the plane point cloud , and then transformed by the rotation matrix R to obtain the center position of the material box , , .

[0048] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a material box posture recognition system is also provided, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the material box posture recognition method as described in any one of the above items.

[0049] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed, the steps of any of the above-mentioned methods for identifying the position and posture of the container are implemented.

[0050] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, there is further provided a robot, which comprises: a cargo picking and placing device, a framing device, and the above-mentioned material box posture recognition system.

[0051] In a possible preferred embodiment, the viewing device includes: a combination of a 2D camera and a laser radar, or any one of a 3D camera.

[0052] Through the material box posture recognition method and system provided by the present invention, the storage medium and the robot cleverly use machine vision to recognize the shape characteristics of the material box itself, and estimate the posture of the material box according to the specifications of the material box itself, its inclination and center position, so that there is no need to stick additional markers on the material box, and there is no need to modify the material box, thereby saving a large amount of process of sticking material box labels, as well as manpower and material costs. In addition, this solution can also be compatible with the recognition of material boxes of various different specifications. Therefore, it has a certain degree of versatility and universality, and significantly improves the overall intelligence level and management efficiency of warehouse management. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0054] Figure 1 It is a process schematic diagram of the method for recognizing the position and posture of a material box of the present invention;

[0055] Figure 2 Schematic diagram of the robot coordinate system in the material box posture recognition method of the present invention;

[0056] Figure 3 It is a schematic diagram of 3D camera calibration in the method for recognizing the position and posture of a material box of the present invention;

[0057] Figure 4 A schematic diagram of an image recognition network identifying a characteristic surface of a material box and obtaining a boundary box for identifying the characteristic surface of the material box in the material box posture recognition method of the present invention;

[0058] Figure 5 It is a conceptual schematic diagram of a clustering noise reduction scheme used in the method for identifying the position and posture of a material box of the present invention;

[0059] Figure 6 It is a schematic diagram of the material box projection and processing process in the material box posture recognition method of the present invention;

[0060] Figure 7 It is a schematic diagram of the structure of the material box posture recognition system of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the technical solution of the present invention, the specific technical solution of the present invention will be clearly and completely described in conjunction with the embodiments below to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this case are only embodiments of a part of the present invention, not all of the embodiments. It should be pointed out that for those of ordinary skill in the art, the embodiments in this application and the features in the embodiments can be combined with each other without departing from the concept of the present invention and without conflicting with each other. Based on the embodiments in the present invention, all other embodiments obtained without creative work by those of ordinary skill in the art should belong to the disclosure and protection scope of the present invention.

[0062] In addition, the terms "first", "second", "S100", "S200", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the features used in this way can be interchanged where appropriate, so that the embodiments of the present invention described here can be implemented in an order other than those described here. At the same time, the stages recorded in each step are not mandatory to be implemented in the same step. It should be understood that the implementation order of the contents in each step stage can be adjusted and interchanged without violating the inventive concept, so that the step embodiments of the present invention described here can be implemented in an order other than those described here. In addition, the terms "including" and "having" and any of their variations in the present invention are intended to cover non-exclusive inclusions. Unless otherwise clearly specified and limited, the terms "set", "layout", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in this case can be understood according to specific circumstances and in combination with the prior art.

[0063] In order to estimate the shape characteristics of the box itself by machine vision, such as Figures 1 to 6 As shown, the present invention proposes a method for identifying the position and posture of a container, and its exemplary steps include:

[0064] Step S100 collects scene image data and point cloud data, and establishes a mapping relationship.

[0065] Specifically, in order to realize the collection of scene images and point cloud data, during the implementation process, the robot can be equipped with a combination of a 2D camera and a lidar, or a 3D camera, as a viewfinder to obtain scene RGB images and point cloud data.

[0066] Taking the use of a 3D camera as an example, it can simultaneously output the RGB image and point cloud image of the current camera shooting angle, and according to the conversion relationship provided by the camera, the point cloud coordinates can be converted to the coordinate space of the RGB lens to obtain the mapping relationship.

[0067] In addition, if we take a handling robot equipped with a hopper as a cargo pick-up and placement device as an example, in order to facilitate the output of the hopper's graspable posture when the entire container posture recognition is completed, the viewfinder calibration processing steps can be selected before collecting scene data as follows:

[0068] Step S110 calibrates the viewfinder, transforms the viewfinder coordinate system into the robot coordinate system, and constructs a transformation matrix T.

[0069] Take the calibration of 3D camera as an example, refer to Figure 2 As shown in the example, the origin is the hopper rotation center, the z-axis is perpendicular to the hopper plane and points upward, and the x-axis points from the rotation center to the hopper head. First, the robot coordinate system is established. Then, Figure 3 As shown, the QR code is placed on the hopper, and the height H of the QR code from the hopper base plane is measured. The size of the black border of the QR code, the distance L from the rotation center of the hopper, and the spacing D between the centers of the QR code are the calibration measurement parameters of the 3D camera. After obtaining the parameters, the three-dimensional spatial coordinate point set P of the corner points of the QR code in the robot coordinate system can be calculated, and the pixel coordinates Q of the eight corner points of the QR code can be identified at the same time. R and T are solved according to the PNP problem:

[0070] λ [ u v 1 ] = K [ R t ] [ x y z ]

[0071] For example, the correspondence between K and 8 sets of uv and xyz is known, and R and t are to be found. Finally, R and t form a transformation matrix and then invert it, which is the camera's transformation matrix T. According to this matrix, the coordinates under the 3D camera can be converted to the robot coordinate system.

[0072] Step S200 identifies the RGB pixels belonging to the characteristic surface of the material box in the image data, and obtains the corresponding material box point cloud according to the mapping relationship.

[0073] Specifically, the example steps include:

[0074] Step S210 uses a pre-trained image recognition network to identify the characteristic surface of the container and obtains a boundary box for identifying the characteristic surface of the container.

[0075] For example, collect feature data sets of bin objects in advance, such as collecting several pictures of different brightness, angles, and resolutions, and train a neural network based on YOLO to obtain a weight model for segmentation. Then, obtain the bounding box of the bin object based on inference, such as Figure 4As shown, in order to ensure that the feature surface of the entire bin object is completely framed, the edges of the segmented object may be optionally expanded, such as by adding a few pixels.

[0076] Step S220 extracts RGB pixels within the bounding box.

[0077] For example, based on the relationship between the coordinates of the point cloud and the camera intrinsic parameters, the point cloud falling within the recognition bounding box can be completely segmented.

[0078] [ u v 1 ] = 1 Z [ fx 0 xx 0 fy cy 0 0 1 ] [ X Y Z ]

[0079] Among them, (u, v) refers to the pixel coordinates, (X, Y, Z) refers to the point cloud coordinates, fx, fy, cx, cy refers to the known camera intrinsic parameters, and the point cloud corresponding to the pixels falling in the bounding box is completely screened and retained to obtain the material box point cloud.

[0080] In addition, since the material box point cloud extracted by S220 may have some noise points, the optional examples may also include:

[0081] Step S230 performs denoising on the material box point cloud.

[0082] For example, Figure 5 As shown, a grid clustering algorithm is proposed in this example, and its example steps include: in the camera coordinate system, clustering operation is performed based on the z-axis of the camera. First, the point cloud in the bounding box is divided into grids based on the z-coordinate according to a preset size (such as 5 cm) along the z-axis, and the bounding box is identified by gridding the material box.

[0083] Then, all point clouds are traversed to vote for the grids, and the point cloud index that falls into each different grid is recorded for easy tracing and restoration. At the same time, the largest continuous interval grid is found, and the number of voting points of each grid in the continuous interval needs to be greater than the preset threshold (such as 10), otherwise it will be interrupted and continue to find the next largest continuous interval until all grids are traversed.

[0084] Finally, the point cloud with the largest number of points in the continuous interval is taken out as the denoised material box point cloud. It is worth mentioning that the denoising solution in this example has the advantage of other existing denoising solutions in that it has a faster processing speed under the same scale and number of point clouds. This is mainly because it makes full use of the installation location information. The complexity of processing one-dimensional data is much lower than that of processing three-dimensional data. Therefore, the computing power requirement is lower, and the same effect can be achieved in the end.

[0085] Step S300 performs plane fitting processing on the material box point cloud to determine the plane normal, so as to estimate the material box rotation angle yaw, roughly locate the material box center P', and screen out the plane point cloud in the plane area based on yaw and material box specifications.

[0086] Specifically, in the first implementation, step S300 includes the following steps:

[0087] Step S310 first calculates the covariance matrix C based on the bin point cloud obtained in step S200

[0088] CovarianceMatrix = [ D xx D xy D xz D yx D yy D yz D zx D zy D zz ]

[0089] Then, according to the covariance matrix, the eigenvector and eigenvalue are calculated:

[0090]

[0091] Among them, U and λ are the eigenvector and eigenvalue of the covariance matrix respectively. According to the definition of linear algebra, the matrix is ​​a real symmetric matrix with at most three eigenvectors and eigenvalues. The eigenvector with the largest eigenvalue is selected as the normal vector N that fits the plane, and the normal is normalized. The angle between the normal and the XoZ plane is the required rotation angle yaw of the material box:

[0092] .

[0093] Step S320 uses the minimum value minBoxBounding and the maximum value maxBoxBounding in the point set of the material box point cloud as the extreme value boundaries of the point cloud to roughly locate the box center point P', and the formula is as follows:

[0094]

[0095] Step S330 defines the boundary of the plane area based on the center P' of the material box and the rotation angle yaw of the material box, the width W and the height H corresponding to the material box specifications:

[0096]

[0097]

[0098]

[0099]

[0100] Based on the boundary of the plane area, the plane point cloud is filtered out.

[0101] On the other hand, in the second implementation, step S300 includes the following steps:

[0102] Step S310′ first calculates the covariance matrix C according to the container point cloud obtained in step S200

[0103] CovarianceMatrix = [ D xx D xy D xz D yx D yy D yz D zx D zy D zz ]

[0104] Then, according to the covariance matrix, the eigenvector and eigenvalue are calculated:

[0105]

[0106] Among them, U and λ are the eigenvector and eigenvalue of the covariance matrix respectively. According to the definition of linear algebra, the matrix is ​​a real symmetric matrix with at most three eigenvectors and eigenvalues. The eigenvector with the largest eigenvalue is selected as the normal vector N that fits the plane, and the normal is normalized. The angle between the normal and the XoZ plane is calculated as the preliminary estimate of the box rotation angle yaw':

[0107] .

[0108] Step S320' uses the minimum minBoxBounding and the maximum maxBoxBounding in the point set of the material box point cloud as the extreme value boundaries of the point cloud to roughly locate the box center point P', and the formula is as follows:

[0109]

[0110] Step S330′ defines the boundary of the plane area based on the center P′ of the material box and the initially estimated material box rotation angle yaw′, according to the width W and height H corresponding to the material box specifications:

[0111]

[0112]

[0113]

[0114]

[0115] Based on the boundary of the plane area, the plane point cloud is filtered out.

[0116] In step S340 ′, based on the plane point cloud, step S310 ′ is repeated to calculate the covariance matrix, eigenvalues ​​and eigenvectors to obtain the fitted plane normal, and then estimate the precise rotation angle yaw of the material box.

[0117] In addition, in an optional implementation manner, corresponding to step S110, in the two examples of step S300, the following steps may also be included:

[0118] Step S301 transforms the container point cloud into the robot coordinate system according to the transformation matrix T.

[0119] By converting the material box point cloud into a coordinate system in advance, the material box rotation angle yaw obtained in the subsequent calculation process and the screened plane point cloud will be converted to the robot coordinate system.

[0120] Step S400 projects the plane point cloud into a binary image, screens out the outline of the material box and performs internal area filling processing.

[0121] The specific example steps include:

[0122] Step S410 constructs a rotation matrix R for rotating the plane point cloud to the orthographic position according to the rotation angle yaw of the material box.

[0123] R _ O r t h o g o n a l = [ cos( yaw ) 0 − sin( yaw ) 0 1 0 sin( yaw ) 0 cos( yaw ) ]

[0124] Step S420 transforms the plane point cloud according to the rotation matrix R, and projects it into a binary image at a preset scaling ratio scale.

[0125] Furthermore, in order to accurately estimate the position of the container, this example requires obtaining the centroid position of the container feature plane. Figure 6 As shown in (a), since the point cloud is sparse, when it is projected onto the image, it will cause holes inside, which will interfere with the estimation of the centroid position. For this reason, in this example, the projection is scaled and the holes are filled in to reduce the interference effect, thereby providing a basis for the subsequent accurate acquisition of the centroid position.

[0126] For example, the processing steps for filling holes in this example are as follows:

[0127] Step S430 first searches for all closed boundaries in the image, such as Figure 6 (b) and find the boundary contour with the largest occupied area (for example, traverse all boundaries, calculate the area of ​​each closed package, and then find the largest closed package boundary), use this as the material box outline, and do internal filling processing, the effect is as follows Figure 6 (c) After this treatment, the voids in the outline of the material box can be basically eliminated.

[0128] Step S500 calculates the centroid of the inner area of ​​the material box outline, and estimates the position of the scene space where the center of the material box is located based on the position and projection relationship of the centroid pixel.

[0129] Specifically, when the outline of the filled material box is obtained, the centroid position of the image is calculated according to the moment of the image (that is, the center position of the precisely located material box, such as Figure 6 (d)), the steps include:

[0130] Step S510 calculates the position of the centroid of the image, where Cx, Cy are the positions of the centroid pixels:

[0131] ,

[0132] Where Mij is the moment of the image, which is calculated as:

[0133]

[0134] P represents the value of the pixel coordinate.

[0135] Since the aforementioned plane point cloud has been orthographically projected during projection, it is necessary to restore the real size. The example steps include:

[0136] Step S520 restores the centroid of the image to the scene space position based on the mapping relationship between the image and the preset scaling ratio scale and the image and the point cloud:

[0137] , .

[0138] The calculation of the z-axis position of the center of the material box is based on the average value, as shown in the following example:

[0139] Step S530 calculates the average point cloud thickness of the plane point cloud (i.e., the z-axis average):

[0140]

[0141] Where pz∈ is a plane point cloud point set.

[0142] At this point, the coordinates of the center position of the material box after original scale projection are obtained ( , , ), and then through the matrix transformation of the rotation matrix R, the real position coordinate Center of the material box is obtained:

[0143] C e n t e r = R _ O r t h o g o n a l − 1 [ x world y world z world ] .

[0144] Finally, the estimated rotation angle yaw and center position Center of the material box are obtained as the material box posture. In addition, if the above-mentioned robot coordinate system conversion step is implemented, the accurate posture of the material box in the robot coordinate system can be obtained.

[0145] Thus, through the above-mentioned example of the material box posture recognition method, the overall solution adopts a two-stage posture estimation algorithm from coarse positioning to fine positioning, which can significantly improve the accuracy of posture estimation. In addition, in the process of fine positioning, due to the use of proportional scaling, contour-based search and hole filling, combined with the center of mass calculation, the problem of holes in the image caused by discrete point clouds can be avoided, and the position of the center of the material box can be accurately estimated. In actual experiments, this technical solution was integrated into a warehouse robot. Compared with the existing technology, the recognition cycle and accuracy were significantly improved. The single recognition cycle was shortened to 400ms, and the repeatability recognition accuracy was increased to ±5mm, which significantly improved the recognition speed and accuracy.

[0146] On the other hand, corresponding to the above method example, Figure 7 As shown, the present invention also provides a container posture recognition system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the container posture recognition method described in any of the above examples.

[0147] On the other hand, corresponding to the above method examples, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed, the steps of the container posture recognition method described in any of the above examples are implemented.

[0148] On the other hand, the present invention also provides a robot, which includes: a cargo picking and placing device, a framing device, and the above-mentioned container position recognition system. The framing device includes: a combination of a 2D camera and a laser radar, or any one of a 3D camera.

[0149] To sum up, through the material box posture recognition method and system provided by the present invention, the storage medium and the robot cleverly use machine vision to identify the shape characteristics of the material box itself, and estimate the posture of the material box according to the specifications of the material box itself, its inclination and center position, so that there is no need to stick additional markers on the material box, and there is no need to modify the material box, thereby saving a lot of process of sticking material box labels, as well as manpower and material costs, and this solution can also be compatible with the identification of various different specifications of material boxes, so it has a certain degree of versatility and universality, and significantly improves the overall intelligence level and management efficiency of warehouse management.

[0150] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is only limited by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0151] Those skilled in the art can understand that, in addition to implementing the system, device, unit and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be regarded as structures within the hardware component; the modules for implementing various functions can also be regarded as both software programs for implementing the method and structures within the hardware component.

[0152] In addition, all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program codes.

[0153] In addition, various implementation modes of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed by the embodiments of the present invention.

Claims

1. A method for identifying a material box posture, comprising the steps of: Collect scene image data and point cloud data, and establish mapping relationships; Identify the RGB pixels belonging to the characteristic surface of the material box in the image data, and obtain the corresponding material box point cloud according to the mapping relationship; Perform plane fitting processing on the material box point cloud to determine the plane normal to estimate the material box rotation angle yaw; roughly locate the material box center P', and screen out the plane point cloud in the plane area based on yaw and material box specifications; Project the plane point cloud into a binary image, filter out the outline of the material box and fill the internal area; Calculate the centroid of the inner area of ​​the box outline, and estimate the position of the box center in the scene space based on the position and projection relationship of the centroid pixel.

2. The method for identifying a container pose according to claim 1, wherein the step of identifying RGB pixels belonging to the container in the image data comprises: A pre-trained image recognition network is used to identify the characteristic faces of the bin and obtain the recognition bounding box of the characteristic faces of the bin; Extract the RGB pixels within the bounding box.

3. The method for identifying the position and posture of a container according to claim 2, wherein the steps further include: The material box point cloud is denoised, and the steps include: gridding the material box to identify the bounding box, traversing all grids to vote for the points within the grid according to the preset voting point number threshold, finding the largest continuous interval, and taking the point cloud data in the continuous interval with the largest number of points as the material box point cloud until the traversal is completed.

4. The method for identifying the position and posture of a container according to claim 1, wherein the step of estimating the rotation angle yaw of the container comprises: Calculate the covariance matrix C of the bin point cloud, and calculate its eigenvectors and eigenvalues ​​based on the covariance matrix C; Select the eigenvector with the largest eigenvalue as the normal vector N; The normal vector N is normalized to take the angle between its normal and the fitting plane as the box rotation angle yaw.

5. The method for identifying the position and posture of a container according to claim 1, wherein the step of projecting the plane point cloud into a binary image comprises: Construct the rotation matrix R of the plane point cloud to the orthographic position according to the rotation angle yaw of the material box; The planar point cloud is transformed according to the rotation matrix R and projected into a binary image at a preset scale.

6. The method for identifying the position and posture of a container according to claim 5, wherein the step of estimating the scene space position of the center of the container comprises: According to the mapping relationship, the position of the centroid pixel in the scene space is determined, and the size is restored according to the preset scaling ratio to obtain the centroid position , ; Calculate the Z-axis average of a plane point cloud ; Transformed by the rotation matrix R , , , get the center position of the material box.

7. The method for recognizing the position and posture of a container according to claim 1, wherein the steps of roughly locating the center P' of the container and screening out the plane point cloud in the plane area based on yaw and the container specifications include: Roughly locate the center P' of the material box according to the extreme boundary of the material box point cloud; Based on the center P' of the material box and the rotation angle yaw of the material box, the width W and height H corresponding to the material box specifications are defined to define the boundaries of the plane area: ， ， ， , based on the boundary of the plane area, the plane point cloud is filtered out.

8. A method for identifying a material box posture, comprising the steps of: Calibrate the viewfinder, transform the viewfinder coordinate system to the robot coordinate system, and construct the transformation matrix T; Collect scene image data and point cloud data and establish mapping relationships; Identify the RGB pixels belonging to the characteristic surface of the material box in the image data, and obtain the corresponding material box point cloud according to the mapping relationship; According to the transformation matrix T, the material box point cloud is transformed into the robot coordinate system, and the material box point cloud is subjected to plane fitting processing to determine the plane normal, so as to preliminarily estimate the material box rotation angle yaw'; The center P' of the material box is roughly located according to the extreme value boundary of the material box point cloud; the plane area is determined according to the specifications of the material box, and the plane point cloud is screened out based on P' and yaw'; Perform plane fitting processing based on the plane point cloud to determine the plane normal to estimate the rotation angle yaw of the material box; Construct the rotation matrix R of the plane point cloud to the orthogonal position based on yaw; According to the rotation matrix R and the preset scaling ratio, the plane point cloud is orthographically projected into a binary image, and the contour of the material box is screened out and the internal area is filled based on the condition of finding the contour with the largest enclosing area; Calculate the centroid of the inner area of ​​the box outline and estimate the centroid position based on the position and scale of the centroid pixel , , and calculate the Z-axis average of the plane point cloud , and then transformed by the rotation matrix R to obtain the center position of the material box.

9. A material box posture recognition system, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the method for identifying the position and posture of a container as claimed in any one of claims 1 to 8.

10. A robot comprising: A cargo picking and placing device, a framing device, and a container position recognition system as claimed in claim 9.

11. The robot according to claim 10, wherein the viewing device comprises: A combination of a 2D camera and a LiDAR, or any type of 3D camera.

12. A computer-readable storage medium storing a computer program, wherein when the computer program is executed, the steps of the method for identifying the position and posture of a container as claimed in any one of claims 1 to 8 are implemented.

Images