Package height measurement method and device in disordered grabbing system

Abstract

The present application relates to a method and device for measuring the height of a package in an unordered grasping system, comprising: obtaining a depth map of a working state after a depth camera and a robotic arm are arranged in a specific set position; segmenting the depth map based on a preset rectangular area and retaining the image within the rectangular area; traversing the pixels within the rectangular area and removing invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera; performing contour extraction on the depth map and obtaining a circumscribed rectangle of the maximum contour after performing maximum contour screening; traversing the pixels within the circumscribed rectangle and calculating the package area within the circumscribed rectangle belonging to the package; and calculating the package height based on the calculated package area. The present invention can effectively measure the height of a package in an unordered grasping system with a robotic arm, without the need for additional camera calibration and other steps, making it easy to deploy and meeting the real-time requirements of industrial systems.

Description

Technical Field

[0001] The present application relates to the field of computer machine vision technology, and in particular to a method and device for measuring the height of a package in an unordered grasping system. Background Art

[0002] With the development of computer technology and the internet, online shopping has injected new vitality into society. While this phenomenon has brought convenience to both sellers and buyers, it has also placed enormous pressure on the logistics industry. To cope with the growing volume of parcels and improve parcel sorting efficiency, some logistics transfer stations have begun experimenting with using robotic arms to replace manual labor to sort incoming parcels.

[0003] In current logistics sorting robotic arm systems, manufacturers primarily focus on package detection and position determination, that is, determining "where to grab" by the robotic arm. In contrast, the question of "where to place" the robotic arm seems to be generally ignored or simplified. After the robotic arm completes the grab, most current solutions choose to move the package to a fixed point for delivery, either a point on the conveyor belt or a point on the parcel pallet. For ordinary packages, this delivery method is harmless. However, when the delivered package contains fragile and easily damaged items, methods such as fixed-point delivery can easily break or crush the items in the package, causing unnecessary trouble for buyers, sellers, and logistics companies.

[0004] Existing methods for disordered grasping systems also exist, such as the invention patent document with authorization announcement number CN112070818B, which discloses a machine vision-based robot disordered grasping method and system. By building a grasping system with a Kinect camera and a robot, the Kinect camera is used to collect image data of the target object's surface, and the image data is preprocessed to obtain three-dimensional point cloud data; target detection, target segmentation, target clustering, key point feature extraction, and feature registration are performed on the collected three-dimensional point cloud data to identify the target object and obtain the target object's posture information; based on the calibration results of the hand-eye system, the posture information is converted into coordinates, and control instructions are issued to the robot to achieve grasping of the target object.

[0005] Although the above-mentioned patent document can improve the existing algorithm to adapt to the complex environment in which the robot is located, thereby improving the accuracy of the robot's disordered grasping and the flexibility of the robot, it cannot effectively measure the height of the package in the robotic arm's disordered grasping system, and requires additional camera calibration and other steps, which cannot meet the real-time requirements of the industrial system. Summary of the Invention

[0006] Based on this, it is necessary to provide a method and device for measuring the height of packages in an unordered grasping system of a robotic arm, which can effectively measure the height of packages in an unordered grasping system without the need for additional camera calibration and other steps, is easy to deploy, and can meet the real-time requirements of industrial systems.

[0007] The technical solutions of the present invention are as follows:

[0008] A method for measuring the height of a package in a disordered grasping system, the method comprising:

[0009] Step S100: Acquire a depth map of the working state after the depth camera and the robotic arm are arranged according to a specific set position;

[0010] Step S200: Segmenting the depth map based on a preset rectangular area and retaining the image within the rectangular area;

[0011] Step S300: traversing the pixels within the rectangular area, and removing invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera;

[0012] Step S400: performing contour extraction on the depth map, and obtaining a circumscribed rectangle of the maximum contour after performing maximum contour screening;

[0013] Step S500: traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel;

[0014] Step S600: Calculate the package height according to the calculated package area, and send the calculated package height to the robotic arm, and the robotic arm places the package according to the package height.

[0015] Specifically, step S500: traversing the pixels in the bounding rectangle and calculating the parcel area in the bounding rectangle belonging to the parcel; specifically includes:

[0016] Step S510: Traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area storage stack;

[0017] Step S520: Continuously popping the top pixel of the traversal stack and determining whether to push the top pixel to the stack. If so, the corresponding top pixel is pushed into the traversal stack and the wrapping area storage stack. The determination of whether to push the top pixel to the stack is performed by determining whether the element at the corresponding position of the top pixel in the matrix M2 is 0, so as to skip the neighboring elements that have been traversed.

[0018] Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1;

[0019] Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed;

[0020] The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

[0021] Specifically, step S600: calculating the package height according to the calculated package area, and sending the calculated package height to the robotic arm, and the robotic arm placing the package according to the package height; specifically including:

[0022] Step S610: continuously popping out pixels in the wrapping area storage stack, and restoring the coordinates of the spatial point corresponding to each popped-out pixel in the camera coordinate system based on the depth camera intrinsic parameter;

[0023] Step S620: Calculate the third-order characteristic matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, calculate the three eigenvectors of the third-order characteristic matrix, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication;

[0024] Step S630: Calculate the package height based on the package area obtained by calculating the acquired eigenvector with the highest parallelism to the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, which places the package according to the package height.

[0025] Specifically, step S610: continuously popping out pixels in the wrapping area storage stack, and restoring the coordinates of the spatial point corresponding to each popped-out pixel in the camera coordinate system based on the depth camera intrinsic parameter; specifically includes:

[0026] Step S611: Based on the pixels in the package area storage stack, determine whether the package area storage stack is empty;

[0027] Step S611: If the answer is yes, pop out the pixels in the wrapping area storage stack;

[0028] Step S611: Restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system based on the depth camera intrinsic parameters according to the following formula:

[0029]

[0030] Among them, f x With fy is the focal length of the camera, p x With p y is the coordinate of the camera origin on the imaging plane.

[0031] Specifically, step S620: calculate the third-order characteristic matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each pop-up pixel in the camera coordinate system, calculate the three eigenvectors of the third-order characteristic matrix characteristics, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication screening; specifically including:

[0032] Step S621: According to each popped-up pixel, the relationship between the coordinates of the popped-up pixel and the spatial point corresponding to the popped-up pixel is obtained. The relationship is as follows:

[0033]

[0034] Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system;

[0035] Step S622: Restore and obtain the coordinates (x i ,y i , z i ), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula:

[0036]

[0037] Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack;

[0038] Step S623: Calculate and obtain three eigenvectors of the third-order eigenmatrix characteristics, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

[0039] Specifically, a package height measuring device in a disordered grabbing system includes:

[0040] A depth map acquisition module is used to obtain a depth map of the working state after the depth camera and the robotic arm are arranged in a specific set position;

[0041] a segmentation processing module, configured to segment the depth map based on a preset rectangular area and retain the image within the rectangular area;

[0042] A pixel traversal module, configured to traverse the pixels within the rectangular area and remove invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera;

[0043] An extraction processing module, configured to perform contour extraction on the depth map and obtain a circumscribed rectangle of the maximum contour after performing maximum contour screening;

[0044] a pixel calculation module, configured to traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel;

[0045] The package height calculation module is used to calculate the package height according to the calculated package area and send the calculated package height to the robotic arm, and the robotic arm places the package according to the package height.

[0046] Specifically, the pixel calculation module includes:

[0047] An area saving module is used to traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area saving stack;

[0048] A pixel popping module is used to continuously pop the top pixel of the traversal stack and determine whether to push the top pixel onto the stack. If so, the corresponding top pixel is pushed onto the traversal stack and the wrapping area storage stack. The push determination is performed by determining whether the element at the corresponding position of the top pixel in matrix M2 is 0, thereby skipping the neighboring elements that have been traversed.

[0049] Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1;

[0050] Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed;

[0051] The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

[0052] Specifically, the package height calculation module includes:

[0053] A spatial point acquisition module is used to continuously pop out pixels in the stack saved in the wrapped area and restore the coordinates of the spatial point corresponding to each popped out pixel in the camera coordinate system based on the depth camera intrinsic parameters;

[0054] The feature matrix calculation module is used to calculate the third-order feature matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, and calculate the three eigenvectors of the third-order feature matrix characteristics. Based on vector multiplication, the eigenvector with the highest parallelism to the y-axis of the camera coordinate system is obtained;

[0055] The package placement module is configured to calculate the package height based on the package area obtained by calculating the eigenvector with the highest degree of parallelism with the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, so that the robotic arm places the package according to the package height. The spatial point acquisition module includes:

[0056] a judgment module, configured to judge whether the package area storage stack is empty based on the pixels in the package area storage stack;

[0057] A result generating module, configured to pop out the pixels in the wrapping area storage stack if the judgment is yes;

[0058] The first formula calculation module is used to restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system according to the following formula based on the depth camera intrinsic parameters:

[0059]

[0060] Among them, f x With f y is the focal length of the camera, p x With p y is the coordinate of the camera origin on the imaging plane; wherein the characteristic matrix calculation module includes:

[0061] The second formula calculation module is used to obtain the relationship between the coordinates of each popped-up pixel and the spatial point corresponding to the popped-up pixel according to the following relationship:

[0062]

[0063] Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system;

[0064] The third formula calculation module is used to restore the coordinates (x i ,y i , z i), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula:

[0065]

[0066] Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack;

[0067] The eigenvector calculation module is used to calculate the three eigenvectors of the third-order eigenmatrix characteristics and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

[0068] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the package height measurement method in the disordered grasping system are implemented.

[0069] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for measuring the height of a package in an unordered grasping system.

[0070] The present invention achieves the following technical effects:

[0071] The package height measurement method and device in the above-mentioned disordered grasping system sequentially obtain a depth map of the working state after the depth camera and the robotic arm are arranged according to a specific set position; segment the depth map based on a preset rectangular area and retain the image within the rectangular area; traverse the pixels within the rectangular area and remove invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera; perform contour extraction on the depth map and obtain the circumscribed rectangle of the maximum contour after maximum contour screening; traverse the pixels in the circumscribed rectangle and calculate the package area within the circumscribed rectangle belonging to the package; calculate the package height based on the calculated package area, and send the calculated package height to the robotic arm, which places the package according to the package height, thereby effectively realizing the measurement of package height in the robotic arm disordered grasping system without the need for additional camera calibration and other steps, easy to deploy, and can meet the real-time requirements of industrial systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 1 is a flow chart of a method for measuring the height of a package in an unordered grabbing system according to one embodiment;

[0073] Figure 2 A schematic diagram of the structure of a disordered grasping system in one embodiment in which a depth camera and a robotic arm are arranged according to specific set positions;

[0074] Figure 3 A schematic diagram of the structure of a camera coordinate system of a depth camera in an unordered grasping system in one embodiment;

[0075] Figure 4 A schematic diagram of calculating the height of a package in a camera coordinate system of a depth camera in an unordered grasping system in one embodiment;

[0076] Figure 5 1 is a structural block diagram of a package height measuring device in a disordered grabbing system according to one embodiment;

[0077] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0079] In one embodiment, Figure 1 As shown, a method for measuring the height of a package in an unordered grasping system is provided, the method comprising:

[0080] Step S100: Acquire a depth map of the working state after the depth camera and the robotic arm are arranged according to a specific set position;

[0081] Specifically, if Figure 2 As shown, the depth camera and the robotic arm are arranged according to a specific set position, which means that the depth camera is installed on the frame on the side of the robotic arm, with the shooting direction parallel to the robotic arm mounting plane and pointing towards the robotic arm. The depth map is obtained in the working state by starting the depth camera and keeping it in the form of a video stream so that a frame can be extracted when needed. During the process of the robotic arm performing disorderly grasping, when the front end of the robotic arm tool reaches the vicinity of the specified path point, the system sends a command to the depth camera, and the depth camera returns the current frame in its video stream to the system, which is to obtain the depth map.

[0082] Step S200: Segmenting the depth map based on a preset rectangular area and retaining the image within the rectangular area;

[0083] The present invention utilizes the fixed placement of each hardware to preset a rectangular area to roughly segment the depth map. The pixel values ​​within the area are retained and the pixel values ​​outside the area are discarded to facilitate subsequent processing.

[0084] Step S300: traversing the pixels within the rectangular area, and removing invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera;

[0085] In this step, based on the characteristics of the depth camera used, a matrix M1 of the same size as the original depth map is created, and all values ​​in it are assigned to 0. The pixels within the preset rectangular area are traversed row by row, and pixels with values ​​of 0 (invalid pixels) and 65535 (background pixels) are skipped. The elements of the corresponding positions of the remaining pixels in the matrix M1 are assigned values ​​of 255.

[0086] Step S400: performing contour extraction on the depth map, and obtaining the circumscribed rectangle of the maximum contour after performing maximum contour screening;

[0087] For the matrix M1 assigned in step S300, contour extraction is performed in this step. The extracted contours are all outer contours. The extracted contours are sorted to obtain the largest contour. The circumscribed rectangle R1 of this contour is calculated for the next step.

[0088] Step S500: traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel;

[0089] Specifically, by calculating the parcel area belonging to the parcel in the circumscribed rectangle, the subsequent calculation of the parcel height is facilitated.

[0090] Step S600: Calculate the package height according to the calculated package area, and send the calculated package height to the robotic arm, and the robotic arm places the package according to the package height.

[0091] The present invention sequentially obtains a depth map of a working state after a depth camera and a robotic arm are arranged in a specific set position; segments the depth map based on a preset rectangular area and retains the image within the rectangular area; traverses the pixels within the rectangular area and removes invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera; performs contour extraction on the depth map and obtains a circumscribed rectangle of the maximum contour after maximum contour screening; traverses the pixels in the circumscribed rectangle and calculates the parcel area within the circumscribed rectangle belonging to the parcel; calculates the parcel height based on the calculated parcel area and sends the calculated parcel height to the robotic arm, which places the parcel based on the parcel height, thereby effectively realizing the measurement of the parcel height in the robotic arm's disordered grasping system without the need for additional camera calibration and other steps, facilitating deployment and meeting the real-time requirements of industrial systems.

[0092] In one embodiment, step S500: traversing pixels within the bounding rectangle and calculating the parcel area within the bounding rectangle; specifically includes:

[0093] Step S510: Traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area storage stack;

[0094] In this step, the process is started first, and then a stack is constructed to traverse the depth map to find the pixels belonging to the package.

[0095] Next, we traverse the matrix M1 from bottom to top and from left to right within the bounding rectangle until we find the first element with a value of 255. This element is the first valid pixel. We then push this element into the traversal stack with its row and column numbers so that we can find the corresponding element in the depth map and the created matrices.

[0096] At the same time, a matrix M2 of the same size as the original depth map is constructed, and all elements in it are assigned a value of 0. The element at the corresponding position of the element at the top of the stack in M2 is set to 1.

[0097] Step S520: Continuously popping the top pixel of the traversal stack and determining whether to push the top pixel to the stack. If so, the corresponding top pixel is pushed into the traversal stack and the wrapping area storage stack. The determination of whether to push the top pixel to the stack is performed by determining whether the element at the corresponding position of the top pixel in the matrix M2 is 0, so as to skip the neighboring elements that have been traversed.

[0098] Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1;

[0099] Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed;

[0100] The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

[0101] Since the obtained depth map may show that the robotic arm gripper is stuck to the package, it is necessary to separate the area belonging to the package from the area belonging to the robotic arm in the depth map.

[0102] Furthermore, if all three conditions are positive, the top pixel is determined to be pushed onto the stack. This means that the corresponding top pixel is pushed onto the traversal stack and the wrapping area storage stack for subsequent determination of its neighboring elements. Simultaneously, the neighboring elements that meet the conditions are pushed onto a stack used to record the wrapping area, thereby recording the pixels belonging to the wrapping area for subsequent processing. At the same time, the process returns to step S520, where the top pixel of the traversal stack is continuously popped off.

[0103] In one embodiment, step S520: continuously popping the top pixel of the traversal stack and determining whether to push the top pixel into the stack specifically includes:

[0104] First, start processing the elements in the traversal stack. Then, determine whether the traversal stack is empty. If the traversal stack is not empty, pop the top element of the traversal stack and determine the upper, left, and right neighbors of the element in the depth map. This determination is the above-mentioned stack push determination.

[0105] In one embodiment, step S600: calculating the package height based on the calculated package area, and sending the calculated package height to the robotic arm, and the robotic arm placing the package based on the package height; specifically includes:

[0106] Step S610: continuously popping out pixels in the wrapping area storage stack, and restoring the coordinates of the spatial point corresponding to each popped-out pixel in the camera coordinate system based on the depth camera intrinsic parameter;

[0107] Specifically, by restoring the coordinates of the spatial point corresponding to each pop-up pixel in the camera coordinate system based on the intrinsic parameters of the depth camera, the subsequent calculation of the third-order feature matrix is ​​facilitated, and then the package height is calculated.

[0108] Step S620: Calculate the third-order characteristic matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, calculate the three eigenvectors of the third-order characteristic matrix, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication;

[0109] Step S630: Calculate the package height based on the package area obtained by calculating the acquired eigenvector with the highest parallelism to the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, which places the package according to the package height.

[0110] In one embodiment, step S610: continuously popping out pixels from the wrapping area storage stack and restoring the coordinates of the spatial point corresponding to each popped-out pixel in the camera coordinate system based on the depth camera intrinsic parameter; specifically includes:

[0111] Step S611: Based on the pixels in the package area storage stack, determine whether the package area storage stack is empty;

[0112] Step S611: If the answer is yes, pop out the pixels in the wrapping area storage stack;

[0113] Specifically, if the judgment is yes, that is, if the package area storage stack is determined to be empty, the pixels in the package area storage stack are popped out.

[0114] Step S611: Restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system based on the depth camera intrinsic parameters according to the following formula:

[0115]

[0116] Among them, f x With f y is the focal length of the camera, p x With p y is the coordinate of the camera origin on the imaging plane.

[0117] In one embodiment, step S620: calculating the third-order characteristic matrix of the coordinates of all spatial points in the enclosed area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, calculating three eigenvectors of the third-order characteristic matrix, and obtaining the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication; specifically includes:

[0118] Step S621: According to each popped-up pixel, the relationship between the coordinates of the popped-up pixel and the spatial point corresponding to the popped-up pixel is obtained. The relationship is as follows:

[0119] Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system;

[0120] Specifically, if Figure 3 As shown, in this step, for a pixel C in the image, its corresponding spatial point in the camera coordinate system is T.

[0121] According to the depth camera intrinsic parameter matrix K, the relationship between the coordinates of the pixel C and the coordinates of its corresponding spatial point T can be obtained, which is the above formula.

[0122] Step S622: Restore and obtain the coordinates (x i ,y i , z i ), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula:

[0123] Restore the coordinates (x i ,y i , z i ), and then add it to the feature matrix of the entire package area.

[0124]

[0125] Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack;

[0126] Step S623: Calculate and obtain three eigenvectors of the third-order eigenmatrix characteristics, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

[0127] In one embodiment, step S623: calculating and obtaining three eigenvectors of the third-order eigenmatrix features, and selecting and obtaining the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication, specifically includes:

[0128] First, determine whether the top element of the stack currently popped up is the top leftmost pixel p in the wrapped area of ​​the depth map. top_left or the bottom rightmost pixel p bottom righ t .

[0129] Initially, let ptop_left The row number of the depth map is the bottom row, and the column number is the rightmost column of the depth map; let p bottom riah t p rioh t bottom The row number is the top row of the depth map, and the column number is the leftmost column.

[0130] Each time the top element of the stack is popped, the popped element is compared with p top_left and p bottom nrih t Compare and dynamically modify these two values ​​to obtain the coordinates of the top leftmost pixel and the bottom rightmost pixel in the wrapped area.

[0131] Then, return to step S611.

[0132] If all elements in the wrapping area stack have been popped, start calculating the eigenvectors of the feature matrix M3. The three eigenvectors obtained are respectively compared with the vector (0, 1, 0) representing the y-axis of the camera coordinate system. T Multiply them and select the eigenvector v1 with the largest absolute value of the product. The eigenvector v1 with the largest absolute value of the product is the eigenvector with the highest degree of parallelism to the y-axis of the camera coordinate system. This method is then used to determine the eigenvector that is most parallel to the y-axis of the camera coordinate system.

[0133] In one embodiment, step S630: calculating the package height based on the package area calculated based on the acquired eigenvector having the highest degree of parallelism with the y-axis of the camera coordinate system, and sending the calculated package height to the robotic arm, which places the package based on the package height; specifically includes the following steps:

[0134] First, if Figure 4 As shown, where v a 、v b and v c are the three eigenvectors of M3. For each of these three vectors, perform the following operations:

[0135]

[0136] Where abs is the absolute value calculation, x v 、y v and z v The three elements in the currently used feature vector.

[0137] by Figure 4 For example, the vector with the largest N value is v b , remember v b For v1.

[0138] Then, restore pto by the method described in step S623 p_left With pbottom rih t Corresponding to the coordinates of the spatial points, obtain the vector v2 formed by connecting these two spatial points in the camera coordinate system.

[0139] Furthermore, p top_left correspond Figure 4 p1, p top-left correspond Figure 4 The vector v2 is p2, which is Figure 4 The dotted line connecting p1 and p2.

[0140] Then, according to the laws of spatial geometry, the length of the projection of v2 on v1 is the required package height. Calculate the inner product of v1 and v2 and take the absolute value to get the package height H:

[0141] H=abs(v1·v2)#(3-5)

[0142] The obtained package height H is returned to the robotic arm system to guide the robotic arm to accurately place the package.

[0143] In one embodiment, Figure 5 As shown, a package height measuring device in a disordered grabbing system, the device comprising:

[0144] A depth map acquisition module is used to obtain a depth map of the working state after the depth camera and the robotic arm are arranged in a specific set position;

[0145] a segmentation processing module, configured to segment the depth map based on a preset rectangular area and retain the image within the rectangular area;

[0146] A pixel traversal module, configured to traverse the pixels within the rectangular area and remove invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera;

[0147] An extraction processing module, configured to perform contour extraction on the depth map and obtain a circumscribed rectangle of the maximum contour after performing maximum contour screening;

[0148] a pixel calculation module, configured to traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel;

[0149] The package height calculation module is used to calculate the package height according to the calculated package area and send the calculated package height to the robotic arm, and the robotic arm places the package according to the package height.

[0150] In one embodiment, the pixel calculation module includes:

[0151] An area saving module is used to traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area saving stack;

[0152] A pixel popping module is used to continuously pop the top pixel of the traversal stack and determine whether to push the top pixel onto the stack. If so, the corresponding top pixel is pushed onto the traversal stack and the wrapping area storage stack. The push determination is performed by determining whether the element at the corresponding position of the top pixel in matrix M2 is 0, thereby skipping the neighboring elements that have been traversed.

[0153] Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1;

[0154] Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed;

[0155] The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

[0156] In one embodiment, the package height calculation module includes:

[0157] A spatial point acquisition module is used to continuously pop out pixels in the stack saved in the wrapped area and restore the coordinates of the spatial point corresponding to each popped out pixel in the camera coordinate system based on the depth camera intrinsic parameters;

[0158] The feature matrix calculation module is used to calculate the third-order feature matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, and calculate the three eigenvectors of the third-order feature matrix characteristics. Based on vector multiplication, the eigenvector with the highest parallelism to the y-axis of the camera coordinate system is obtained;

[0159] The package placement module is configured to calculate the package height based on the package area obtained by calculating the eigenvector with the highest degree of parallelism with the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, so that the robotic arm places the package according to the package height. The spatial point acquisition module includes:

[0160] a judgment module, configured to judge whether the package area storage stack is empty based on the pixels in the package area storage stack;

[0161] A result generating module, configured to pop out the pixels in the wrapping area storage stack if the judgment is yes;

[0162] The first formula calculation module is used to restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system according to the following formula based on the depth camera intrinsic parameters:

[0163]

[0164] Wherein, fx and fy are the focal lengths of the camera, and px and py are the coordinates of the camera origin on the imaging plane; wherein the feature matrix calculation module includes:

[0165] The second formula calculation module is used to obtain the relationship between the coordinates of each popped-up pixel and the spatial point corresponding to the popped-up pixel according to the following relationship:

[0166]

[0167] Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system;

[0168] The third formula calculation module is used to restore the coordinates (x i ,y i , z i ), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula:

[0169]

[0170] Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack;

[0171] The eigenvector calculation module is used to calculate the three eigenvectors of the third-order eigenmatrix characteristics and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

[0172] In one embodiment, Figure 3 As shown, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the package height measurement method in the above-mentioned disordered grasping system are implemented.

[0173] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for measuring the height of a package in an unordered grasping system.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for measuring the height of a package in an unordered grabbing system, characterized in that: The method comprises: Step S100: obtaining a depth map of the working state after the depth camera and the robotic arm are arranged according to a specific set position; Step S200: Segmenting the depth map based on a preset rectangular area and retaining the image within the rectangular area; Step S300: traversing the pixels within the rectangular area, and removing invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera; Step S400: performing contour extraction on the depth map, and obtaining the circumscribed rectangle of the maximum contour after performing maximum contour screening; Step S500: traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel; Step S600: Calculating the package height according to the calculated package area, and sending the calculated package height to the robotic arm, so that the robotic arm places the package according to the package height, specifically including: Step S610: continuously popping out pixels in the wrapping area storage stack, and restoring the coordinates of the spatial point corresponding to each popped-out pixel in the camera coordinate system based on the depth camera intrinsic parameter; Step S620: Calculate the third-order characteristic matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, calculate the three eigenvectors of the third-order characteristic matrix, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication; Step S630: Calculate the package height based on the package area obtained by calculating the acquired eigenvector with the highest parallelism to the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, which places the package according to the package height.

2. The method for measuring the height of a package in an unordered grabbing system according to claim 1, characterized in that: Step S500 specifically includes: Step S510: Traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area storage stack; Step S520: Continuously popping the top pixel of the traversal stack and determining whether to push the top pixel to the stack. If so, the corresponding top pixel is pushed into the traversal stack and the wrapping area storage stack. The determination of whether to push the top pixel to the stack is performed by determining whether the element at the corresponding position of the top pixel in the matrix M2 is 0, so as to skip the neighboring elements that have been traversed. Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1; Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed; The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

3. The method for measuring the height of a package in an unordered grabbing system according to claim 1, characterized in that: Step S610 specifically includes: Step S611: Based on the pixels in the package area storage stack, determine whether the package area storage stack is empty; Step S611: If the answer is yes, pop out the pixels in the wrapping area storage stack; Step S611: Restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system based on the depth camera intrinsic parameters according to the following formula: Among them, f x With f y is the focal length of the camera, p x With p y is the coordinate of the camera origin on the imaging plane.

4. The method for measuring the height of a package in an unordered grabbing system according to claim 1, wherein: Step S620 specifically includes: Step S621: According to each popped-up pixel, the relationship between the coordinates of the popped-up pixel and the spatial point corresponding to the popped-up pixel is obtained. The relationship is as follows: Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system; Step S622: Restore and obtain the coordinates (x i ,y i , z i ), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula: Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack; Step S623: Calculate and obtain three eigenvectors of the third-order eigenmatrix characteristics, and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

5. A package height measuring device in a disordered grabbing system, characterized in that: The device comprises: A depth map acquisition module is used to obtain a depth map of the working state after the depth camera and the robotic arm are arranged in a specific set position; a segmentation processing module, configured to segment the depth map based on a preset rectangular area and retain the image within the rectangular area; A pixel traversal module, configured to traverse the pixels within the rectangular area and remove invalid pixels and background pixels within the rectangular area based on the characteristics of the depth camera; An extraction processing module, configured to perform contour extraction on the depth map and obtain a circumscribed rectangle of the maximum contour after performing maximum contour screening; a pixel calculation module, configured to traverse the pixels in the bounding rectangle and calculate the parcel area in the bounding rectangle belonging to the parcel; A package height calculation module, configured to calculate the package height based on the calculated package area and send the calculated package height to the robotic arm, so that the robotic arm places the package based on the package height, includes: A spatial point acquisition module is used to continuously pop out pixels in the stack saved in the wrapped area and restore the coordinates of the spatial point corresponding to each popped out pixel in the camera coordinate system based on the depth camera intrinsic parameters; The feature matrix calculation module is used to calculate the third-order feature matrix of the coordinates of all spatial points in the wrapped area based on the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system, and calculate the three eigenvectors of the third-order feature matrix characteristics. Based on vector multiplication, the eigenvector with the highest parallelism to the y-axis of the camera coordinate system is obtained; The package placement module is used to calculate the package height based on the package area calculated based on the acquired eigenvector with the highest parallelism to the y-axis of the camera coordinate system, and send the calculated package height to the robotic arm, and the robotic arm places the package according to the package height.

6. The package height measuring device in the disordered grabbing system according to claim 5, characterized in that: The pixel calculation module includes: An area saving module is used to traverse the pixels in the bounding rectangle starting from the lower left of the bounding rectangle, and push the first valid pixel traversed into the traversal stack and the wrapping area saving stack; A pixel popping module is used to continuously pop the top pixel of the traversal stack and determine whether to push the top pixel onto the stack. If so, the corresponding top pixel is pushed onto the traversal stack and the wrapping area storage stack. The push determination is performed by determining whether the element at the corresponding position of the top pixel in matrix M2 is 0, thereby skipping the neighboring elements that have been traversed. Determine whether the position of the top pixel of the stack exceeds the range of the circumscribed rectangular area R1, so as to skip the neighboring elements that are not within R1; Determine whether the absolute value of the difference between the pixel value corresponding to the top pixel in the depth map and the pixel value corresponding to the currently popped top element in the depth map is less than a specified threshold, so as to skip neighboring elements that are too far away from the currently traversed element in Euclidean space; if all three are judged to be yes, then determine that the top pixel of the stack can be pushed; The matrix M2 is a matrix of the same size as the depth map, and all elements in the matrix M2 are assigned a value of 0.

7. The package height measuring device in the disordered grabbing system according to claim 5, characterized in that: The spatial point acquisition module includes: a judgment module, configured to judge whether the package area storage stack is empty based on the pixels in the package area storage stack; A result generating module, configured to pop out the pixels in the wrapping area storage stack if the judgment is yes; The first formula calculation module is used to restore the coordinates of the spatial point corresponding to each popped-up pixel in the camera coordinate system according to the following formula based on the depth camera intrinsic parameters: Among them, f x With f y is the focal length of the camera, p x With p y is the coordinate of the camera origin on the imaging plane; wherein the characteristic matrix calculation module includes: The second formula calculation module is used to obtain the relationship between the coordinates of each popped-up pixel and the spatial point corresponding to the popped-up pixel according to the following relationship: Where C is the pixel, and T is the spatial point corresponding to pixel C in the camera coordinate system; The third formula calculation module is used to restore the top element of the currently popped stack i The coordinates of the corresponding space point (x i ,y i , z i ), add it to the feature matrix of the entire package area, and calculate the third-order feature matrix M3 based on the following formula: Where XX is the sum of the squares of the x-axis coordinates of all elements in the stack, YY is the sum of the squares of the y-axis coordinates of all elements in the stack, ZZ is the sum of the squares of the z-axis coordinates of all elements in the stack, XY is the sum of the products of the x-axis coordinates and the y-axis coordinates of all elements in the stack, XZ is the sum of the products of the x-axis coordinates and the z-axis coordinates of all elements in the stack, YZ is the sum of the products of the y-axis coordinates and the z-axis coordinates of all elements in the stack, X is the sum of the x-axis coordinates of all elements in the stack, Y is the sum of the y-axis coordinates of all elements in the stack, Z is the sum of the z-axis coordinates of all elements in the stack, and n is the sum of the number of elements in the stack; The eigenvector calculation module is used to calculate the three eigenvectors of the third-order eigenmatrix characteristics and obtain the eigenvector with the highest parallelism to the y-axis of the camera coordinate system based on vector multiplication.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images