A rapid material-picking method for mash material-picking robots based on machine vision

Abstract

The present invention relates to the technical field of liquor and vinegar brewing, and discloses a machine vision-based method for rapidly retrieving mash using a robot. The method comprises three steps: planning a safe operating area for the robot, characterizing the mash quantity to be retrieved, and implementing a rapid retrieval strategy. The method automates the removal of mash from traditional earthenware vats during solid-state fermentation production, reducing labor intensity and improving retrieval efficiency. The method can be used for mash removal operations in various solid-state fermentation production industries, offering broad applicability and improved efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of brewing liquor and vinegar, and in particular to a method for quickly taking mash out of a vat using a machine vision-based robot, which is suitable for the process of taking mash out of a vat in a traditional earthenware vat solid-state fermentation production process. The method provides a method for the operation of taking mash out of a vat in various industries. Background Art

[0002] Light-fragrance liquor is fermented in ground vats, which are densely distributed in the fermentation workshop. At present, the process of removing mash from the ground vat is generally done by workers using iron shovels to shovel out the mash and then loading it into a traveling crane bucket or a material transport vehicle. This operation method is slow in removing mash from the vat, and the workers have a poor working environment and high labor intensity, which is an important factor limiting the efficiency of liquor production. The material-picking robots for this link at home and abroad are still unable to meet the needs of the industry, and most companies still rely on manual operations. Therefore, designing an automated method for removing mash from the vat has great application prospects and economic value. The applicant has developed a mash-picking robot (CN216328365U, a composite robot for removing mash from the ground vat fermentation process) that can replace manual labor in removing mash from the vat, thereby realizing the intelligent development of the traditional brewing industry. However, during the robot's material-retrieving process, the complex working environment, such as the round and small opening of the ground vat, the fact that it is buried underground during fermentation, the vat opening is level with the ground, and the ground vat wall has low strength, restricts the automation of the mash removal operation. Therefore, it is of great significance to plan a rapid material-retrieving strategy for the mash-retrieving robot to avoid damage to the ground vat and design a rapid material-retrieving method for the mash-retrieving robot based on machine vision. Summary of the Invention

[0003] The problem to be solved by the present invention is to address the deficiencies of the prior art and to provide a method for quickly retrieving mash material using a machine vision-based retrieving robot.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for quickly retrieving mash material using a machine vision-based robot comprises the following steps:

[0006] Step 1: Planning a safe operation area for the mash retrieving robot;

[0007] Step 2, characterizing the amount of mash material taken;

[0008] Step 3: Design the robot's terminal feeding device to preset the feeding point, feeding direction and feeding depth, plan the robot's rapid feeding strategy, and realize the fully automated operation of the mash out of the tank during the traditional earthen tank solid-state fermentation production process.

[0009] Furthermore, the specific implementation of step 1 is as follows:

[0010] Step 1.1: Design a 1:1 digital virtual commissioning platform for the mash retrieving robot to simulate the mash retrieving operation of the robot in the actual industrial physical environment to the greatest extent possible;

[0011] Step 1.2: Analyze the bucket retrieving trajectory of the end-of-line retrieving device of the mash retrieving robot according to the structure of the end-of-line retrieving device of the robot;

[0012] Step 1.3: To prevent the reclaiming device from colliding with the cylinder wall during the reclaiming process, analyze and measure the bucket's dimensional parameters, including bucket length, bucket depth, bucket width, and bucket back length, based on the bucket structure and bucket flipping trajectory.

[0013] Step 1.4, based on the size parameters of the bucket and the ground cylinder, as well as the bucket's material collection trajectory, plan the robot's safe operating area for material collection, while taking material from the edge of the cylinder wall and avoiding collision with the cylinder wall.

[0014] Furthermore, the specific process of step 1.2 is as follows: first, the robot controls the terminal reclaiming device to reach the middle path point above the reclaiming point, then controls the terminal reclaiming device to vertically insert the bucket into the mash to a fixed depth, then controls the electric cylinder to flip the bucket to a horizontal position, completing the mash reclaiming work, then controls the terminal reclaiming device to vertically lift the bucket out of the ground cylinder to the middle path point, and finally runs to the top of the material transport vehicle to flip the bucket to discharge the material, thereby completing the mash reclaiming and discharging process;

[0015] Furthermore, the specific process of step 1.4 is as follows: based on the material picking trajectory and working environment of the material picking robot, the robot's safe operating conditions are analyzed in the digital virtual debugging platform of the material picking robot. According to the safe operating conditions, the internal structure of the ground cylinder is regarded as a frustum, so that a certain safety margin is left between the front edge of the bucket and the cylinder wall. According to the collected depth image of the ground cylinder, the surface of the material is regarded as a plane, that is, the material picking plane. The distance h between the material picking plane in the cylinder and the camera plane is calculated, and the actual radius r of the material picking plane in the ground cylinder at this time is calculated. The calculation method is as follows:

[0016]

[0017] Where, α is the cylinder wall inclination angle; d p is the depth of the ground tank; r m is the actual radius of the ground cylinder mouth; r b is the actual radius of the cylinder bottom; c h is the distance between the camera and the plane of the ground tank mouth;

[0018] Bucket length l b As a benchmark, determine the safe operation area for taking mash, that is, to ensure the safe taking operation, the distance b between the taking point and the center of the mash taking plane at this time is calculated as follows:

[0019] b=rl b / sinα.

[0020] Furthermore, the specific process of step 2 is as follows:

[0021] Step 2.1: Use a structured light depth camera as the visual perception system of the mash retrieval robot to collect RGB images and depth images of the mash jar before and after retrieval.

[0022] Step 2.2: Preprocess the RGB image of the ground jar, then perform edge detection to extract the inner edge contour of the ground jar mouth and approximate fitting circle;

[0023] Step 2.3, solve the actual area surface element s of the ground cylinder image;

[0024] Step 2.4, preprocessing the depth image of the underground vat in step 2.1: based on the neighborhood information of the empty points in the mash area in the vat in the depth image, fill the empty points in the mash area in the vat with the maximum depth value in the neighborhood information, and align the depth image to the color image;

[0025] Step 2.5: Create a mask by fitting a circle based on the inner edge contour of the ground jar mouth obtained in step 2.2, perform image segmentation on the ground jar RGB image and depth image obtained in steps 2.2 and 2.4, and extract the image within the ground jar mouth area;

[0026] Step 2.6, traverse the depth map of the ground jar mouth area obtained by segmentation, reconstruct the point cloud in the ground jar mouth area, and perform color mapping based on the RGB image. The expression is as follows:

[0027]

[0028] Where d is the depth value of each pixel in the depth map within the jar mouth area, (u, v) corresponds to the pixel coordinates; (x, y, z) is the coordinates of each point in the depth map within the jar mouth area in the camera coordinate system; f x 、f y 、c x 、c y is the camera internal parameter; c is the depth map scaling factor in the crater mouth area;

[0029] Step 2.7: To compensate for the robot's control accuracy problem, the point cloud data in the jar mouth area before and after the material removal operation are unified into the same coordinate system through image registration;

[0030] In step 2.8, the changes in the surface distribution point cloud data of the mash in the ground tank before and after the material removal operation are compared to realize the calculation of the material removal amount.

[0031] Furthermore, the specific implementation of step 3 is as follows:

[0032] Step 3.1: Collect the RGB image and depth map of the jar before taking the material in the image acquisition posture, obtain the point cloud in the jar mouth area, establish the empty jar point cloud model in the image acquisition posture, set the empty jar point cloud as the target point cloud, and the point cloud in the jar mouth area as the input point cloud, set the allowed distance threshold between the corresponding points in the two sets of point clouds, calculate the difference between the point clouds, and obtain the mash point cloud;

[0033] Step 3.2: Extract the depth data from the mash point cloud and convert it into millimeters. Perform a histogram analysis. Using the maximum value of the histogram as a benchmark, filter out the mash adhering to the cylinder wall. Statistically analyze the remaining mash data and calculate the average value h. The mash surface is treated as a plane, where h is the distance between the mash extraction plane and the camera plane.

[0034] Step 3.3: Estimate the height of the remaining mash according to the depth of the mash reclaiming plane, and set a mash remaining height threshold based on the bucket length. If the current mash remaining height is lower than the threshold, the mash reclaiming operation is considered to be completed; otherwise, the reclaiming operation continues.

[0035] Step 3.4: Determine the distance b between the material taking point and the center of the material taking plane according to step 1.4, and then perform proportional conversion to obtain the distance b under the pixel coordinates of the ground cylinder in the image. i , the expression for proportional conversion is as follows:

[0036]

[0037] Where, is the inner edge diameter of the ground jar mouth in the image; D is the actual inner edge diameter of the ground jar mouth;

[0038] Step 3.5: Design four preset take-off points with the goal of getting the mash from the edge of the cylinder wall. Based on the cylinder and bucket dimensions, as well as the bucket take-off trajectory, design the bucket take-off direction from the center of the mash take-off plane toward the cylinder wall, with the take-off depth being the length of the bucket back.

[0039] Step 3.6: Based on the bucket reclaiming trajectory of the robot's end-of-line reclaimer, establish a bucket motion coordinate system with the bucket and connecting rod connection point as the coordinate origin, the outward direction of the connecting axis of the bucket and connecting rod as the positive z-axis direction, the vertical downward direction of the connecting rod as the positive y-axis direction, and the direction perpendicular to the connecting rod and away from the bucket as the positive x-axis direction.

[0040] Step 3.7: In the bucket motion coordinate system, construct a function corresponding to the motion trajectory of the midpoint of the bucket front during the bucket flipping and reclaiming process. The reclaiming trajectory function of each point on the bucket front uses the reclaiming trajectory function of the midpoint of the front and converts it into the image coordinate system to obtain the reclaiming trajectory functions in the image coordinate system corresponding to the four preset reclaiming points.

[0041] Step 3.8: Calculate the relative depth of each pixel in the pre-set reclaiming point's reclaiming area relative to the bucket's motion coordinate system xoz plane, and find the corresponding point on the reclaiming trajectory for each point in the pre-set reclaiming point's reclaiming area. Based on the proposed characterization method for mash reclaiming yield, subtract the relative depth of each pixel in the pre-set reclaiming point's reclaiming area from the relative depth of the corresponding point on the reclaiming trajectory to predict the reclaiming yield of each of the four pre-set reclaiming point's reclaiming areas.

[0042] In step 3.9, the preset feeding point with the largest predicted feeding amount is selected as the optimal feeding point. The robot controls the end point of the connecting rod of the feeding device to move to the optimal feeding point to perform the feeding operation, thereby realizing the robot's rapid feeding operation in the traditional earthenware tank solid-state fermentation production process.

[0043] Furthermore, the acquisition process of the RGB image and depth image of the ground vat before and after material taking in step 2.1 is as follows: the structured light depth camera is installed on the end flange of the mash taking robot using the eye-in-hand method, and the camera is calibrated as well as the hand-eye calibration; the mash taking robot moves to the front of the target ground vat to be taken, controls the depth camera to acquire the image of the ground vat, and uses image processing to identify the center of the ground vat mouth; with the goal of the camera being able to acquire a full image of the ground vat mouth, the robot controls the depth camera to move to a fixed height directly above the center of the ground vat mouth, with the lens facing the plane of the ground vat mouth, sets this posture as the image acquisition posture, and acquires the RGB image and depth image of the ground vat before the mash taking operation; the robot performs the mash taking operation, and after the material taking is completed, the robot controls the depth camera to move to the image acquisition posture, and acquires the RGB image and depth image of the ground vat after the mash taking operation.

[0044] Furthermore, the specific process of step 2.2 is as follows:

[0045] Bilateral filtering is used for filtering and noise reduction, and the weighted average method is used to convert the ground cylinder image into a grayscale image. The grayscale expression is as follows:

[0046] Gray=0.299R+0.587G+0.114B

[0047] Where R, G, and B are the three color components in the RGB image of the ground tank; Gray is the grayscale value corresponding to each pixel of the ground tank image after grayscale conversion;

[0048] The Canny edge detection algorithm is used to detect the edge of the grayscale image of the ground jar. The detection results are then processed by morphological closing operation, the contour sequence is traversed and ellipse fitting is performed. Geometric constraints are constructed to extract the inner edge contour of the ground jar mouth and approximate the fitting circle, excluding other contours and eliminating noise interference. The constructed geometric constraint expression is as follows:

[0049]

[0050] Where long_axis and short_axis are the length of the major axis and the length of the minor axis of each contour fitting ellipse, respectively, ratio is the ratio of the major axis length to the minor axis length, R min 、R max are the low constraint threshold and the high constraint threshold of the approximate fitting circle of the inner edge of the ground jar mouth, respectively, which is related to the distance between the RGB-D camera lens plane and the plane where the ground jar mouth is located; r max The constraint threshold for the ratio of the major axis length to the minor axis length of the fitted ellipse.

[0051] Furthermore, the specific process of step 2.3 is as follows: based on the image acquisition posture, a proportional conversion relationship is constructed to calculate the actual area S represented by the acquired ground cylinder image, each small area obtained by dividing the actual area represented by the ground cylinder image by the number of pixels is defined as a surface element s, and then the surface element s of the actual area of ​​the ground cylinder image is calculated;

[0052] The proportional conversion relationship expression is as follows:

[0053]

[0054] Where rows and cols are the width and length of the ground cylinder image respectively; is the inner edge diameter of the ground jar mouth in the image; D is the actual inner edge diameter of the ground jar mouth; width and length are the width and length of the actual area represented by the ground jar image, respectively;

[0055] The calculation method of the actual area bin s of the ground cylinder image is as follows:

[0056] S=width×length

[0057] s=S÷nums=width×length÷nums

[0058] Where nums is the number of pixels in the ground tank image.

[0059] Furthermore, the specific process of step 2.7 is as follows: use scale-invariant feature transformation to extract feature points of the two ground cylinder images before and after material collection, calculate the descriptor of each feature point, use fast approximate nearest neighbor matching algorithm to perform feature matching on the two ground cylinder images, and screen matching pairs based on 4 times the minimum matching distance, use random sampling consistency algorithm to solve the motion relationship between the ground cylinder images before and after material collection, obtain the rotation vector R and translation vector t, and finally realize the unification of the point cloud data coordinate system in the ground cylinder mouth area before and after material collection. The expression based on is as follows:

[0060] q i =R×p i +t

[0061] Where p i and q i are the feature points of the ground cylinder image before and after the material removal operation; R and t are the rotation matrix and displacement vector between the ground cylinder image coordinate systems before and after the material removal operation;

[0062] According to the obtained R and t, the point cloud data in the cylinder mouth area before the material is taken can be converted to the point cloud data coordinate system in the cylinder mouth area after the material is taken. The calculation expression is as follows:

[0063]

[0064] In the formula, [x′ i1 , y′ i1 , z′ i1 ] is the point cloud data in the cylinder mouth area before taking the material, which is converted to the point cloud coordinate system after taking the material; [x i1 ,y i1 , z i1 ] is the point cloud data in the area of ​​the tank mouth before taking the material in the point cloud coordinate system. i1 The depth value d of each point extracted from the depth image in the cylinder mouth area before taking the material i1 Calculated, according to z′ i1 The depth value d′ of each point in the cylinder mouth area before taking the material can be calculated in the point cloud coordinate system after taking the material i1 .

[0065] Furthermore, the specific implementation steps of step 2.8 are as follows:

[0066] 2.81 Based on the collected top views of the vat before and after taking the mash, the mash area in the vat is divided into small rectangular mash blocks with known parameters according to the pixel points.

[0067] 2.82 Compare the changes in the depth values ​​at the same coordinates in the tank mouth area before and after taking the material, and obtain the taking depth h of the mash taking operation at the corresponding surface element. i , calculated as follows:

[0068] h i =d i2 -d′ i1

[0069] Where d′ i1 d i2 They are the depth values ​​of the same pixel point in the tank mouth area before and after taking the material in the same coordinate system;

[0070] 2.83 Combined with the actual area bin s of the ground jar image, the amount of mash material taken at the corresponding bin can be characterized;

[0071] 2.84 According to the principle of integral summation in advanced mathematics, the mash quantity taken at all bins is added together to finally represent the mash quantity M taken for this taking operation. The calculation method is as follows:

[0072]

[0073] Compared with the prior art, the present invention has the following advantages:

[0074] The present invention provides a new automated method for removing mash from a vat, which adopts a mash taking robot to control a terminal taking device to dig the mash from the vat and load it into a material transport vehicle, thereby realizing the automation of the mash taking out operation in the traditional solid-state fermentation production process in the vat, reducing the labor intensity of workers, improving the taking efficiency, and having the advantages of wide applicability and improving benefits.

[0075] The method provided by the present invention can be applied not only to the operation of removing mash from the vat during the traditional solid-state fermentation production process of liquor, but also to the operation of removing mash from the vat in vinegar brewing, as well as the operation of removing mash from the vat in various other solid-state fermentation production industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Develop a flow chart for planning the safe operation area for robotic mash retrieval;

[0077] Figure 2 This is a side view of the robot's end-of-line material handling device structure;

[0078] Figure 3 This is the front view of the robot's end-of-line material removal device structure;

[0079] Figure 4 The bucket flips and reclaims the material track;

[0080] Figure 5 A flow chart for characterizing the amount of mash taken;

[0081] Figure 6 Flowchart for planning a rapid material picking strategy for robots.

[0082] In the figure: 21 is the servo motor, 22 is the servo electric cylinder body, 23 is the servo electric cylinder screw, 24 is the bucket, 25 is the structured light depth camera, 26 is the connecting rod, 27 is the end point of the connecting rod of the material taking device, and 31 is the midpoint of the front edge of the bucket. DETAILED DESCRIPTION

[0083] The present invention is described in detail below with reference to specific embodiments.

[0084] Step 1: Plan the safe operation area for the mash material taking robot. The flowchart of this step is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0085] Step 11: Design a 1:1 digital virtual debugging platform for the mash retrieving robot to simulate the robot mash retrieving operation in the actual industrial physical environment to the greatest extent possible.

[0086] Step 12: According to the structure of the robot end pick-up device, Figure 2 、 Figure 3 As shown, the trajectory of the bucket of the terminal reclaiming device is analyzed: first, the robot controls the terminal reclaiming device to reach the middle path point above the reclaiming point, and then controls the terminal reclaiming device to make the bucket vertically inserted into the mash to a fixed depth. Then, the electric cylinder is controlled to flip the bucket to the horizontal position to complete the mash reclaiming work. Then, the terminal reclaiming device is controlled to lift the bucket vertically out of the ground cylinder to the middle path point, and finally runs to the top of the material transport vehicle to flip the bucket to discharge the material, thereby realizing the mash reclaiming and discharging process.

[0087] The robot end material taking device, such as Figure 2 、 Figure 3 As shown, it includes: a servo motor 21, a servo electric cylinder body 22, a servo electric cylinder screw 23, a bucket 24, a structured light depth camera 25, a connecting rod 26 and an end point 27 of the connecting rod of the material taking device; the servo motor 21 is arranged on the servo electric cylinder body 22, the servo electric cylinder body 22 is fixedly arranged on the connecting rod 26, the bucket 24 is hinged at the lower end of the connecting rod 26, and the servo electric cylinder screw 23 of the servo electric cylinder body 22 is hinged to the bucket 24 and the connecting rod 26 through the connecting rod, and the bucket 24 is driven to move by the servo motor 21.

[0088] Step 13: To avoid collision between the material taking device and the cylinder wall during the material taking process, according to the bucket structure and bucket flipping material taking trajectory, as shown in FIG. Figure 4 As shown, the bucket size parameters are analyzed and measured: bucket length, bucket depth, bucket width and bucket back length.

[0089] Step 14: Based on the bucket and cylinder size parameters, as well as the bucket's material collection trajectory, the robot's safe operating area for material collection is planned, with the premise of collecting material from the cylinder wall edge and avoiding collision with the cylinder wall. The specific implementation steps are as follows:

[0090] Step 141 , performing a mash picking test on the mash picking robot at the mouth of the ground vat in the digital virtual debugging platform of the mash picking robot, so as to test the safe operating conditions of the robot without collision with the ground vat wall.

[0091] Step 142: Based on safe working conditions, the internal structure of the underground cylinder is treated as a frustum, so that a certain safety margin is left between the front edge of the bucket and the cylinder wall. Based on the collected depth image of the underground cylinder, the surface of the mash is treated as a plane, that is, the mash extraction plane. The distance h between the mash extraction plane in the cylinder and the camera plane is calculated, and the actual radius r of the mash extraction plane in the underground cylinder at this time is calculated as follows:

[0092]

[0093] Where, α is the cylinder wall inclination angle; d p is the depth of the ground tank; r m is the actual radius of the ground cylinder mouth; r b is the actual radius of the cylinder bottom; c h is the distance between the camera and the plane of the ground tank mouth.

[0094] Step 143, with bucket length l b As a benchmark, determine the safe operation area for taking mash, that is, the distance b between the taking point and the center of the mash taking plane at this time, and the calculation method is as follows:

[0095] b=rl b / sinα

[0096] Step 2: Characterize the amount of mash taken. The flow chart of this step is as follows: Figure 5 As shown, the specific implementation steps are as follows:

[0097] Step 21: Use the Intel RealSense D455 structured light depth camera as the visual perception system of the mash retrieval robot to collect RGB images and depth images of the vat before and after the mash retrieval operation. The specific implementation method is as follows:

[0098] The present invention adopts the eye-in-hand method, installs the structured light depth camera on the end flange of the robot, and performs camera calibration and hand-eye calibration on the camera; the mash picking robot moves to the target ground vat where the picking operation is to be performed, controls the depth camera to collect the ground vat image, and uses image processing to identify the center of the ground vat mouth; with the camera being able to collect the full image of the ground vat mouth as the goal, the robot controls the depth camera to move to a fixed height directly above the center of the ground vat mouth, with the lens facing the plane of the ground vat mouth, and sets this posture as the image collection posture of the present invention, and collects the RGB image and depth map of the ground vat before the mash picking operation; the robot performs the mash picking operation, and after the picking is completed, the robot controls the depth camera to move to the image collection posture, and collects the RGB image and depth map of the ground vat after the mash picking operation.

[0099] Step 22: Preprocess the RGB image of the ground jar; use bilateral filtering to reduce noise, and use weighted averaging to convert the ground jar image into a grayscale image. The grayscale conversion is based on the following expression:

[0100] Gray=0.299R+0.587G+0.114B

[0101] Where R, G, and B are the three color components in the RGB image of the ground tank; Gray is the grayscale value corresponding to each pixel of the ground tank image after grayscale conversion.

[0102] Step 23: Use the Canny edge detection algorithm to perform edge detection on the grayscale image of the ground jar. Then perform morphological closing operation on the detection results, traverse the contour sequence and perform ellipse fitting. Construct geometric constraints, extract the inner edge contour of the ground jar mouth and approximate the fitting circle, exclude other contours, and eliminate noise interference. The constructed geometric constraint expression is as follows:

[0103]

[0104] Where long_axis and short_axis are the length of the major axis and the length of the minor axis of each contour fitting ellipse, respectively, ratio is the ratio of the major axis length to the minor axis length, R min 、R max are the low constraint threshold and the high constraint threshold of the approximate fitting circle of the inner edge of the ground jar mouth, respectively, which is related to the distance between the RGB-D camera lens plane and the plane where the ground jar mouth is located; r max The constraint threshold for the ratio of the major axis length to the minor axis length of the fitted ellipse.

[0105] Step 24: Based on the image acquisition posture of the present invention, a proportional relationship conversion is constructed to calculate the actual area S represented by the acquired ground cylinder image. Each small area obtained by equally dividing the actual area represented by the ground cylinder image by the number of pixels is defined as a bin s, and then the bin s of the actual area of ​​the ground cylinder image is calculated;

[0106] The proportional conversion relationship expression is as follows:

[0107]

[0108] Where rows and cols are the width and length of the ground cylinder image respectively; is the inner edge diameter of the ground jar mouth in the image; D is the actual inner edge diameter of the ground jar mouth; width and length are the width and length of the actual area represented by the ground jar image, respectively;

[0109] The calculation method of the actual area bin s of the ground cylinder image is as follows:

[0110] S=width×length

[0111] s=S÷nums=width×length÷nums

[0112] Where nums is the number of pixels in the ground tank image.

[0113] Step 25, pre-processing the depth image of the underground vat, filling the void points in the mash area in the vat with the maximum depth value in the neighborhood information based on the neighborhood information of the void points in the mash area in the vat, and aligning the depth image to the color image.

[0114] Step 26, making a mask by approximating the fitting circle of the inner edge of the ground jar mouth obtained in step 23, performing image segmentation on the ground jar RGB image and depth image, and extracting the image within the ground jar mouth area.

[0115] Step 27, traverse the depth map of the ground jar mouth area obtained by segmentation, reconstruct the point cloud in the ground jar mouth area, and perform color mapping according to the RGB map. The expression is as follows:

[0116]

[0117] Where d is the depth value of each pixel in the depth map within the jar mouth area, (u, v) corresponds to the pixel coordinates; (x, y, z) is the coordinates of each point in the depth map within the jar mouth area in the camera coordinate system; f x 、f y 、c x 、c y is the camera internal parameter; c is the depth map scaling factor in the crater mouth area.

[0118] Step 28: Through image registration, the point cloud data in the ground tank mouth area before and after the material collection operation are unified into the same coordinate system to compensate for the robot control accuracy problem; the scale-invariant feature transformation is used to extract the feature points of the two ground tank images before and after the material collection, and the descriptors of each feature point are calculated. The fast approximate nearest neighbor matching algorithm is used to perform feature matching on the two ground tank images, and the matching pairs are selected based on 4 times the minimum matching distance. The random sampling consistency algorithm is used to solve the motion relationship between the ground tank images before and after the material collection, and the rotation vector R and translation vector t are obtained. Finally, the point cloud data coordinate system before and after the material collection operation is unified. The expression based on is as follows:

[0119] q i =R×p i +t

[0120] Where p i and q i are the feature points of the ground cylinder image before and after the material removal operation; R and t are the rotation matrix and displacement vector between the ground cylinder image coordinate systems before and after the material removal operation;

[0121] According to the obtained R and t, the point cloud data in the cylinder mouth area before the material is taken can be converted to the point cloud data coordinate system in the cylinder mouth area after the material is taken. The calculation expression is as follows:

[0122]

[0123] In the formula, [x′ i1 , y′ i1 , z′ i1 ] is the point cloud data in the cylinder mouth area before taking the material, which is converted to the point cloud coordinate system after taking the material; [x i1 ,y i1 , z i1 ] is the point cloud data in the area of ​​the tank mouth before taking the material in the point cloud coordinate system. i1 The depth value d of each point extracted from the depth image in the cylinder mouth area before taking the material i1 Calculated, according to z′ i1 The depth value d′ of each point in the cylinder mouth area before taking the material can be calculated in the point cloud coordinate system after taking the material i1 .

[0124] Step 29, comparing the changes in the surface distribution point cloud data of the mash in the vat before and after the material taking operation, to calculate the amount of mash taken. The specific implementation steps are as follows:

[0125] Step 291 , based on the collected top views of the vat before and after the mash is taken out, the mash area in the vat is divided into small rectangular mash blocks with known parameters according to the pixels.

[0126] Step 292, define z = Z (x, y) as the z coordinate at the (x, y) coordinate in the query point cloud, according to the point cloud conversion result [x′ i1 , y′ i1 , z′ i1 ], query the corresponding coordinates (x′ i1 , y′ i1 ) at the z coordinate z r2 = Z2(x′ r1 , y′ r1 ), the corresponding depth value d of each point in the cylinder mouth area after taking the material can be calculated i2 , compare the changes in the depth value at the same coordinate in the cylinder mouth area before and after taking the material, and obtain the taking depth h of the mash taking operation at the corresponding surface element i , calculated as follows:

[0127] h i =d i2 -d′ i1

[0128] Where d′ i1 d i2 They are the depth values ​​of the same pixel point in the tank mouth area before and after taking the material in the same coordinate system;

[0129] In step 293 , the actual area bin s of the ground jar image obtained in step 24 can be combined to characterize the amount of mash taken at the corresponding bin.

[0130] In step 294, according to the principle of integral summation in advanced mathematics, the mash quantity taken at all bins is added up to finally achieve the mash quantity representation of this taking operation. The calculation method is as follows:

[0131]

[0132] Where M is the calculated value of the mash material quantity; s is the bin of the actual area of ​​the ground jar image; and n is the number of image pixels in the ground jar mouth area.

[0133] Step 3: Plan the robot's rapid material removal strategy to achieve fully automated operation of the mash out of the tank during the traditional solid-state fermentation production process. The flowchart of this step is as follows: Figure 6 As shown, the specific implementation steps are as follows:

[0134] Step 31, collect the RGB image and depth image of the ground vat before taking the material under the image acquisition posture, obtain the point cloud in the ground vat mouth area, establish the empty vat point cloud model under the image acquisition posture, set the empty vat point cloud as the target point cloud, and the point cloud in the ground vat mouth area as the input point cloud, set the allowed distance threshold between the corresponding points in the two groups of point clouds, calculate the difference between the point clouds, and obtain the mash point cloud.

[0135] Step 32: extract and record the depth data in the mash point cloud and convert it into millimeters for histogram statistics. Based on the maximum value of the histogram, filter out the mash adhering to the cylinder wall. Perform statistical analysis on the remaining mash data and calculate the average value h. The mash surface is treated as a plane, and h is the distance between the mash collection plane and the camera plane at this time.

[0136] Step 33, estimate the height of the remaining mash according to the depth of the mash taking plane, set the threshold of the remaining mash height based on the bucket length, and if the current remaining mash height is lower than the threshold, it is considered that the mash taking operation is completed, otherwise, continue the taking operation.

[0137] Step 34: According to step 142 and step 143, determine the distance b between the material taking point and the center of the material taking plane at this time, and then perform proportional conversion to obtain the distance b under the pixel coordinates of the ground cylinder in the image. i , the expression for proportional conversion is as follows:

[0138]

[0139] Where, is the inner edge diameter of the ground cylinder mouth in the image; D is the actual inner edge diameter of the ground cylinder mouth.

[0140] Step 35, with the goal of the bucket taking the mash at the edge of the cylinder wall, design four preset taking point positions; based on the cylinder and bucket size parameters, as well as the bucket taking trajectory, design the bucket taking direction from the center of the mash taking plane to the cylinder wall, and the taking depth is the length of the bucket back.

[0141] Step 36, based on the bucket picking trajectory of the robot's end-of-line picking device, establish a bucket motion coordinate system with the connection point between the bucket and the connecting rod as the coordinate origin, the outward direction of the connection axis between the bucket and the connecting rod as the positive direction of the z-axis, the vertical downward direction of the connecting rod as the positive direction of the y-axis, and the direction perpendicular to the connecting rod and away from the bucket as the positive direction of the x-axis.

[0142] Step 37, in the bucket motion coordinate system, construct the function corresponding to the motion trajectory of the midpoint of the bucket front during the bucket flipping and material taking process. The material taking trajectory function of each point on the bucket front uses the material taking trajectory function of the midpoint of the front, and converts it into the image coordinate system to obtain the material taking trajectory functions in the image coordinate system corresponding to the four preset material taking points.

[0143] Step 38, calculate the relative depth value of each pixel point in the material taking area of ​​the preset material taking point relative to the xoz plane of the bucket motion coordinate system, and find the corresponding point of each point in the material taking area of ​​the preset material taking point on the corresponding material taking trajectory; according to the proposed method for characterizing the material taking amount of mash, the relative depth value of each pixel point in the material taking area of ​​each preset material taking point is subtracted from the relative depth value of the corresponding point on the corresponding material taking trajectory, and the material taking amount of the four preset material taking point material taking areas is predicted respectively.

[0144] Step 39, select the preset feeding point with the largest predicted feeding amount as the optimal feeding point, and the robot controls the end point of the connecting rod of the feeding device to move to the optimal feeding point to perform the feeding operation, thereby realizing the robot's rapid feeding operation in the traditional earthenware jar solid-state fermentation production process.

[0145] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for quickly taking mash material by a machine vision-based robot, characterized in that: The following steps are involved: Step 1: Planning a safe operation area for the mash retrieving robot; Step 2, characterizing the amount of mash material taken; Step 3: Design the robot's end-of-line material handling device to preset the material handling point, material handling direction, and material handling depth, and plan the robot's rapid material handling strategy to achieve fully automated operation of the mash unloading process in the traditional ground tank solid-state fermentation production process; The specific implementation of step 1 is as follows: Step 1.1: Design a 1:1 digital virtual commissioning platform for the mash retrieving robot to simulate the mash retrieving operation of the robot in the actual industrial physical environment to the greatest extent possible; Step 1.2, analyzing the bucket retrieving trajectory of the end-of-line retrieving device of the mash retrieving robot according to the structure of the end-of-line retrieving device of the robot; Step 1.3: To prevent the reclaiming device from colliding with the cylinder wall during the reclaiming process, analyze and measure the bucket's dimensional parameters, including bucket length, bucket depth, bucket width, and bucket back length, based on the bucket structure and bucket reclaiming trajectory. Step 1.4: Based on the bucket and vat dimensions, as well as the bucket's retrieving trajectory, plan a safe operating area for the robot to retrieve mash while ensuring it can retrieve mash from the edge of the vat wall and avoid collisions with the vat wall. The specific process of step 1.2 is as follows: first, the robot controls the terminal reclaiming device to reach the middle path point above the reclaiming point, then controls the terminal reclaiming device to vertically insert the bucket into the mash to a fixed depth, then controls the electric cylinder to flip the bucket to a horizontal position, completing the mash reclaiming work, then controls the terminal reclaiming device to vertically lift the bucket out of the ground cylinder to the middle path point, and finally runs to the top of the material transport vehicle to flip the bucket to discharge the material, thereby completing the mash reclaiming and discharging process; The specific process of step 1.4 is as follows: based on the material taking trajectory and working environment of the material taking robot, the safe operating conditions of the robot are analyzed in the digital virtual debugging platform of the material taking robot. According to the safe operating conditions, the internal structure of the ground cylinder is regarded as a frustum, so that a certain safety margin is left between the front edge of the bucket and the cylinder wall. According to the collected depth image of the ground cylinder, the surface of the material is regarded as a plane, that is, the material taking plane. The distance h between the material taking plane in the cylinder and the camera plane is calculated, and the actual radius r of the material taking plane in the ground cylinder at this time is calculated. The calculation method is as follows: Where, α is the cylinder wall inclination angle; d p is the depth of the ground tank; r m is the actual radius of the ground cylinder mouth; r b is the actual radius of the cylinder bottom; c h is the distance between the camera and the plane of the ground tank mouth; Bucket length l b As a benchmark, determine the safe operation area for taking mash, that is, to ensure the safe taking operation, the distance b between the taking point and the center of the mash taking plane at this time is calculated as follows: b=rl b / sinα。 2. The method for quickly retrieving mash material by a machine vision-based robot according to claim 1, characterized in that: The specific process of step 2 is: Step 2.1: Use a structured light depth camera as the visual perception system of the mash retrieval robot to collect RGB images and depth images of the mash jar before and after retrieval. Step 2.2: Pre-process the RGB images of the ground jar before and after taking the material, then perform edge detection to extract the approximate fitting circle of the edge contour inside the ground jar mouth; Step 2.3, solve the actual area surface element s of the RGB image and depth image of the ground tank before and after material collection; Step 2.4, pre-processing the depth map of the vat before and after taking the material in step 2.1: based on the neighborhood information of the empty points in the material area of ​​the vat in the depth map, fill the empty points in the material area of ​​the vat with the maximum depth value in the neighborhood information, and align the depth map to the color image; Step 2.5: Create a mask by fitting a circle based on the inner edge contour of the ground jar mouth obtained in step 2.2, perform image segmentation on the ground jar RGB image and depth image obtained in steps 2.2 and 2.4, and extract the image within the ground jar mouth area; Step 2.6, traverse the depth map of the ground jar mouth area obtained by segmentation, reconstruct the point cloud in the ground jar mouth area, and perform color mapping based on the RGB image. The expression is as follows: Where d is the depth value of each pixel in the depth map within the jar mouth area, (u, v) corresponds to the pixel coordinates; (x, y, z) is the coordinates of each pixel in the depth map within the jar mouth area in the camera coordinate system; f x 、f y 、c x 、c y is the camera internal parameter; c is the depth map scaling factor in the crater mouth area; Step 2.7: To compensate for the robot's control accuracy problem, the point cloud data in the jar mouth area before and after the material removal operation are unified into the same coordinate system through image registration; In step 2.8, the changes in the surface distribution point cloud data of the mash in the ground tank before and after the material removal operation are compared to realize the calculation of the material removal amount.

3. The method for quickly retrieving mash material by a machine vision-based robot according to claim 1, characterized in that: The specific implementation of step 3 is: Step 3.1: Collect the RGB image and depth map of the jar before taking the material in the image acquisition posture, obtain the point cloud in the jar mouth area, establish the empty jar point cloud model in the image acquisition posture, set the empty jar point cloud as the target point cloud, and the point cloud in the jar mouth area as the input point cloud, set the allowed distance threshold between the corresponding points in the two sets of point clouds, calculate the difference between the point clouds, and obtain the mash point cloud; Step 3.2: Extract the depth data from the mash point cloud and convert it into millimeters. Perform a histogram analysis. Using the maximum value of the histogram as a benchmark, filter out the mash adhering to the cylinder wall. Statistically analyze the remaining mash data and calculate the average value h. The mash surface is treated as a plane, where h is the distance between the mash extraction plane and the camera plane. Step 3.3: Estimate the height of the remaining mash according to the depth of the mash reclaiming plane, and set a mash remaining height threshold based on the bucket length. If the current mash remaining height is lower than the threshold, the mash reclaiming operation is considered to be completed; otherwise, the reclaiming operation continues. Step 3.4: Determine the distance b between the material taking point and the center of the material taking plane at this time according to step 1.4, and then perform proportional conversion to obtain the distance b in the pixel coordinates of the ground cylinder in the RGB image and depth image before taking the material. i , the expression for proportional conversion is as follows: Where, is the inner edge diameter of the ground jar mouth in the image; D is the actual inner edge diameter of the ground jar mouth; Step 3.5: Design four preset take-off points with the goal of getting the mash from the edge of the cylinder wall. Based on the cylinder and bucket dimensions, as well as the bucket take-off trajectory, design the bucket take-off direction from the center of the mash take-off plane toward the cylinder wall, with the take-off depth being the length of the bucket back. Step 3.6: Based on the bucket reclaiming trajectory of the robot's end-of-line reclaimer, establish a bucket motion coordinate system with the bucket and connecting rod connection point as the coordinate origin, the outward direction of the connecting axis of the bucket and connecting rod as the positive z-axis direction, the vertical downward direction of the connecting rod as the positive y-axis direction, and the direction perpendicular to the connecting rod and away from the bucket as the positive x-axis direction. Step 3.7: In the bucket motion coordinate system, construct a function corresponding to the motion trajectory of the midpoint of the bucket front during the bucket flipping and reclaiming process. The reclaiming trajectory function of each point on the bucket front uses the reclaiming trajectory function of the midpoint of the front and converts it into the image coordinate system to obtain the reclaiming trajectory functions in the image coordinate system corresponding to the four preset reclaiming points. Step 3.8: Calculate the relative depth of each pixel in the designated reclaiming area relative to the xoz plane of the bucket's motion coordinate system, and find the corresponding point on the corresponding reclaiming trajectory for each pixel in the designated reclaiming area. Based on the proposed characterization method for mash reclaiming yield, subtract the relative depth of each pixel in the designated reclaiming area from the relative depth of the corresponding point on the corresponding reclaiming trajectory to predict the reclaiming yield of each of the four designated reclaiming areas. In step 3.9, the preset feeding point with the largest predicted feeding amount is selected as the optimal feeding point. The robot controls the end point of the connecting rod of the feeding device to move to the optimal feeding point to perform the feeding operation, thereby realizing the robot's rapid feeding operation in the traditional earthenware tank solid-state fermentation production process.

4. The method for quickly retrieving mash material by a machine vision-based robot according to claim 2, characterized in that: The process of collecting the RGB images and depth images of the ground vat before and after material taking in the step 2.1 is as follows: the structured light depth camera is installed on the end flange of the mash taking robot using the eye-in-hand method, and the camera is calibrated and the hand-eye calibrated; the mash taking robot moves to the front of the target ground vat where the material taking operation is to be performed, controls the depth camera to collect the ground vat image, and uses image processing to identify the center of the ground vat mouth; with the goal of the camera being able to collect the full image of the ground vat mouth, the robot controls the depth camera to move to a fixed height directly above the center of the ground vat mouth, with the lens facing the plane of the ground vat mouth, sets this posture as the image collection posture, and collects the RGB image and depth map of the ground vat before the material taking operation; the robot performs the material taking operation, and after the material taking is completed, the robot controls the depth camera to move to the image collection posture, and collects the RGB image and depth map of the ground vat after the material taking operation.

5. The method for quickly retrieving mash material by a machine vision-based robot according to claim 2, characterized in that: The specific process of step 2.2 is as follows: Bilateral filtering is used for filtering and noise reduction, and the weighted average method is used to convert the ground cylinder image into a grayscale image. The grayscale expression is as follows: Gray=0.299R+0.587G+0.114B Where R, G, and B are the three color components in the RGB image of the ground tank; Gray is the grayscale value corresponding to each pixel of the ground tank image after grayscale conversion; The Canny edge detection algorithm is used to detect the edge of the grayscale image of the ground jar. The detection results are then processed by morphological closing operation, the contour sequence is traversed and ellipse fitting is performed. Geometric constraints are constructed to extract the inner edge contour of the ground jar mouth and fit the ellipse. Other contours are excluded and noise interference is eliminated. The constructed geometric constraint expression is as follows: Where long_axis and short_axis are the length of the major axis and the length of the minor axis of each contour fitting ellipse, respectively, ratio is the ratio of the major axis length to the minor axis length, R min 、R max are the low constraint threshold and high constraint threshold of the ellipse fitted to the inner edge of the jar mouth, respectively, which are related to the distance between the structured light depth camera lens plane and the jar mouth plane; r max The constraint threshold for the ratio of the major axis length to the minor axis length of the fitted ellipse.

6. The method for quickly retrieving mash material by a machine vision-based robot according to claim 2, characterized in that: The specific process of step 2.3 is as follows: based on the image acquisition posture, a proportional conversion relationship is constructed to calculate the actual area S represented by the acquired ground cylinder image, each small area obtained by equally dividing the actual area represented by the ground cylinder image by the number of pixels is defined as a surface element s, and then the surface element s of the actual area of ​​the ground cylinder image is calculated; The proportional conversion relationship expression is as follows: Where rows and cols are the width and length of the ground cylinder image respectively; is the inner edge diameter of the ground jar mouth in the image; D is the actual inner edge diameter of the ground jar mouth; width and length are the width and length of the actual area represented by the ground jar image, respectively; The calculation method of the actual area bin s of the ground cylinder image is as follows: S=width×length s=S÷nums=width×length÷nums Where nums is the number of pixels in the ground tank image.

7. The method for quickly retrieving mash material by a machine vision-based robot according to claim 2, characterized in that: The specific process of step 2.7 is as follows: using scale-invariant feature transformation to extract feature points of the two floor cylinder images before and after material collection, and calculating the descriptor of each feature point, using fast approximate nearest neighbor matching algorithm to perform feature matching on the two floor cylinder images, and screening matching pairs based on 4 times the minimum matching distance, using random sampling consistency algorithm to solve the motion relationship between the floor cylinder images before and after material collection, and obtain the rotation matrix R between the coordinate systems of the floor cylinder images before and after material collection and the displacement vector t between the coordinate systems of the floor cylinder images before and after material collection, and finally realize the unification of the point cloud data coordinate system in the floor cylinder mouth area before and after material collection, based on the following expression: q i =R×p i +t Where p i and q i are the feature points of the ground cylinder image before and after the material removal operation; R and t are the rotation matrix and displacement vector between the ground cylinder image coordinate systems before and after the material removal operation; According to the obtained R and t, the point cloud data in the cylinder mouth area before the material is taken can be converted to the point cloud data coordinate system in the cylinder mouth area after the material is taken. The calculation expression is as follows: In the formula, [x′ i1 ,y′ i1 ,z′ i1 ] is the point cloud data in the area of ​​the jar mouth before taking the material, which is converted to the point cloud data coordinate system in the area of ​​the jar mouth after taking the material; [x i1 ,y i1 , z i1 ] is the point cloud data coordinate system in the area of ​​the cylinder mouth before taking the material, i1 The depth value d of each pixel point extracted from the depth image in the cylinder mouth area before taking the material i1 Calculated, according to z′ i1 The depth value d′ of each pixel point in the area of ​​the jar mouth before taking the material can be calculated in the point cloud data coordinate system of the jar mouth after taking the material. i1 .

8. The method for quickly retrieving mash material by a machine vision-based robot according to claim 2, characterized in that: The specific implementation steps of step 2.8 are as follows: 2.81 Based on the collected top views of the vat before and after taking the mash, the mash area in the vat is divided into small rectangular mash blocks with known parameters according to the pixel points. 2.82 Compare the changes in the depth values ​​at the same coordinates in the tank mouth area before and after taking the material, and obtain the taking depth h of the mash taking operation at the corresponding surface element. i , calculated as follows: h i =d i2 -d′ i1 Where d′ i1 d i2 They are the depth values ​​of the same pixel point in the tank mouth area before and after taking the material in the same coordinate system; 2.83 Combined with the actual area bin s of the ground jar image, the amount of mash taken at the corresponding bin can be characterized; 2.84 According to the principle of integral summation in advanced mathematics, the mash quantity taken at all bins is added together to finally represent the mash quantity M taken for this taking operation. The calculation method is as follows:

Images