A complex environment-oriented lees robot visual perception system and method

Abstract

The present application belongs to the technical field of liquor brewing, and discloses a liquor extraction robot visual perception system and method for complex environment. The method adds the number of cylinder edge of the preliminarily obtained cylinder area range as an iteration condition by using a cylinder positioning detection algorithm, obtains a real cylinder area, splits the obtained cylinder area mask into a plurality of single cylinder area masks, and performs cylinder opening recognition; uses a fermented grain material surface segmentation algorithm to segment the fermented grain material surface; uses a material surface and fermented grain extraction environment three-dimensional reconstruction algorithm to perform three-dimensional curved surface reconstruction to obtain a material surface cylinder model, converts the cylinder model to the material surface point cloud through coordinate transformation, performs preliminary registration, and aligns the cylinder model with the original cylinder point cloud. The present application realizes the area detection and positioning of the cylinder opening, the identification of the material surface, and provides information for the digging action of the fermented grain extraction robot, thereby realizing efficient automatic digging of fermented grains.

Description

Technical Field

[0001] This invention belongs to the field of liquor brewing technology, and in particular relates to a visual perception system and method for a fermentation robot oriented towards complex environments. Background Technology

[0002] Currently, the baijiu (Chinese liquor) brewing industry largely relies on manual operation for production. This is due to two main reasons: firstly, the reliability of current automated baijiu production equipment is poor, making it difficult to apply in actual production scenarios; secondly, the baijiu production process is complex and diverse, making it impossible for equipment to be used in all situations. Regarding the fermentation of baijiu mash, most baijiu companies use fermentation pits or similar facilities, which facilitate automation. However, companies like Laobaigan and Fenjiu use earthen vats for fermentation, resulting in a more complex environment for mash removal. Achieving automation in these vat-based systems requires attention not only to the design of the extraction process but also to the implementation of advanced sensing methods. Therefore, an efficient earthen vat environment sensing system can significantly improve equipment utilization and facilitate the upgrading and automation of the entire brewing process.

[0003] Based on the above analysis, the problems and defects of the existing technology are as follows: the existing technology cannot realize the regional detection and positioning of the vat opening and the identification of the material surface in the complex baijiu brewing environment. At the same time, it has not established a three-dimensional model of the mash material surface and the mash discharge environment under the mash discharge operation, and cannot provide information for the digging action of the mash discharge robot, thus failing to realize the efficient and automated digging operation of mash. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the present invention discloses a visual perception system and method for a fermentation robot in complex environments.

[0005] The technical solution is as follows: a visual perception method for a fermentation robot in complex environments, comprising the following steps:

[0006] S1. Using the cylinder positioning detection algorithm, the number of cylinder edges of the initially obtained cylinder area is added as an iterative condition to obtain the real cylinder area. The obtained cylinder area mask is then divided into several single cylinder area masks for cylinder mouth recognition.

[0007] S2, based on the identification of the vat opening, the mash surface is segmented using a mash surface segmentation algorithm;

[0008] S3, based on the segmented mash surface, uses a 3D reconstruction algorithm of the mash surface and the mash discharge environment to reconstruct the 3D surface of the fermentation vat model, transforms the vat model into the mash surface point cloud through coordinate transformation, performs preliminary registration, and aligns the vat model with the original vat point cloud.

[0009] In step S1, the cylinder positioning detection algorithm consists of the initial cylinder positioning algorithm and the cylinder port accurate identification algorithm;

[0010] The initial positioning algorithm for the ground cylinder is a combination of a depth data analysis algorithm and an improved Hough circle detection algorithm. The depth data analysis algorithm includes the following steps:

[0011] Step a: Obtain ground depth data features: Convert the depth images of the ground and the top of the tank into a depth histogram;

[0012] Step b: Create a depth histogram with the following parameters: the horizontal axis represents the depth value, and the vertical axis represents the frequency, which is the number of pixels within the same depth group.

[0013] Step c: Use the maximum frequency two-sided classification to determine the range of ground and ground cylinder data. Compare the current frequency with the frequencies of its left and right groups. Determine whether it is ground by whether the frequency difference is within 40% of the current frequency.

[0014] Step d: After completing the maximum frequency two-sided class search, a binary map is built. The resolution of the binary map is the same as that of the depth map. The pixels corresponding to the ground and top of the tank are represented by 0 in the binary map, and the rest are represented by 1. The part with a value of 1 in the binary map is the range of the tank area.

[0015] The improved Hough circle detection algorithm includes: adding the number of cylinder edges as an iteration condition on the basis of Hough circle detection, and completing the upper threshold self-updating of Canny edge detection before Hough circle detection; the iteration and self-updating strategy is: the upper threshold is initially set to 40. If the number of cylinder edges detected cannot meet the requirements, the upper threshold is reduced and detection continues. If the upper threshold and the lower threshold are equal and the conditions are still met, the detection fails.

[0016] The cylinder port accurate identification algorithm uses an improved Canny edge detection algorithm to achieve accurate cylinder port identification.

[0017] Furthermore, the initial positioning algorithm for the ground cylinder includes:

[0018] The contour is extracted from the binary image obtained by depth data analysis, and the bounding rectangle of the approximate area of ​​the underground reservoir is drawn. Let the length of the obtained bounding rectangle be x. r y r The geometric center is (x i y i The expression for the virtual radius is:

[0019]

[0020] In the formula, r r For virtual radius;

[0021] The following conditions must be met for the specified constant D:

[0022]

[0023] In the formula, r c The radius of the circle obtained by Hough circle detection, (x c y c () represents the center of the circle obtained by Hough circle detection;

[0024] If the conditions are met, the cylinder area detected by the Hough circle is the real cylinder area. If the conditions are not met, the cylinder area obtained by the current Hough circle detection is discarded, and the calculation of whether other areas meet the requirements continues until all are detected. After the detection is completed, the obtained cylinder area mask is split into several single cylinder area masks for accurate identification by the cylinder mouth recognition algorithm.

[0025] Furthermore, the Canny edge detection algorithm has been improved, including:

[0026] Step 1: Perform an AND operation between the color image and the mask of the single-cylinder area to obtain the color image of the single-cylinder area;

[0027] Step II: Convert the obtained color image of the single-cylinder tank to grayscale. The grayscale conversion formula is as follows:

[0028] Gray=0.299×R+0.587×G+0.114×B

[0029] In the formula, Gray is the gray value of the point, R is the pixel value of the red part in the color image of the point, G is the pixel value of the green part in the color image of the point, and B is the pixel value of the blue part in the color image of the point.

[0030] The obtained grayscale image is subjected to Gaussian filtering to reduce noise interference with the inspection. The specific formula is shown below:

[0031]

[0032] In the formula, the kernel function is a (5, 5) Gaussian kernel, the kernel function f(x, y) is the formula for calculating the value of each point in the matrix formed by Gaussian filtering, δ is the standard deviation of the normal distribution, exp(·) is an exponential function with the constant e as the base, and x and y are the coordinates of the pixel points in the image.

[0033] Step III: Use the Sobel operator to calculate the first derivatives in the horizontal and vertical directions to obtain the magnitude and direction of the gradient. The specific calculation formula is shown below:

[0034]

[0035] In the formula, Edge_Grandient(G) is the magnitude of the edge gradient, and G xG represents the gradient magnitude in the x-direction of the image. y Angle(θ) is the gradient magnitude in the y-direction of the image, and Angle(θ) is the angle between the edge gradient direction and the x-direction of the image.

[0036] Step IV: Based on the magnitude of Angle(θ), the gradient directions are divided into four categories: vertical, horizontal, and two diagonals; each pixel is checked to determine whether the gradient of the point is the largest in the same gradient direction around it. If so, the point is kept; otherwise, the point is removed; after non-maximum suppression, for several edge points in the same direction, basically only one is kept.

[0037] Step V: Using a dual threshold to determine the edges, the extracted edges are optimized using an adaptive iterative method; specifically including:

[0038] Step V-1: Perform several rounds of manual detection on the detection image, and count the high and low threshold data when the Canny detection meets the requirements. Set the initial high threshold to 1.5 times the larger value among the statistical high thresholds, and set the initial low threshold to the larger value among the statistical low thresholds.

[0039] Step V-2: For single-cylinder images, the initial threshold is used for detection first. If an inner contour is present and meets the size requirements, the edge of the cylinder is considered to have been detected.

[0040] Step V-3: The threshold is changed using an adaptive iterative method. Two logical loops are used to iteratively update the threshold: the inner loop updates the high threshold, and the outer loop updates the low threshold. If no closed edge data is detected, the high threshold is lowered first, and edge detection is performed again. If no closed edge is detected, the high threshold is lowered further until the difference between the high and low thresholds is within 100. If no closed edge information is obtained at this point, the low threshold is lowered, the high threshold is restored to its initial value, and detection continues until a closed contour is obtained. If the difference between the low threshold and its initial value reaches 100, the detection is considered a failure, and the detection is performed in a different area.

[0041] Step VI: Draw the smallest bounding rectangle for the contour, obtain the centroid coordinates, and use the camera intrinsic parameters to transform the 2D centroid coordinates to the 3D coordinate system as shown in the formula:

[0042]

[0043] In the formula, (x, y, z) are the coordinates of the point in the camera coordinate system, u is the pixel value of the point in the x-axis direction, v is the pixel value of the point in the y-axis direction, and c x Let c be the initial value of the camera center in the x-axis direction. y Let x be the initial value of the camera center along the y-axis, and depth be the depth value of that point. w y w, z w (T) represents the coordinates of a point in the world coordinate system. x T y T z Let be the translations in the x, y, and z directions, and R be the rotation matrix.

[0044] In step S2, the fermented mash surface segmentation algorithm uses the YOLOv7 image segmentation algorithm to label the surface area in the color image captured by the camera, create a dataset for network training, set the network training parameters and train it, and then test the obtained model and adjust some parameters and training samples to achieve the segmentation of the fermented mash surface.

[0045] In step S3, the three-dimensional reconstruction algorithm of the material surface and the fermentation environment includes: synthesis of the material surface point cloud, synthesis of the original point cloud of the fermentation tank, establishment of the fermentation tank model, and matching algorithm between the fermentation tank model and the material surface point cloud; the fermentation tank model is obtained by modeling based on the actual production data of the fermentation tank.

[0046] The specific steps for synthesizing the material surface point cloud are as follows: The material surface mask obtained from image segmentation is ANDed with the color image and depth image respectively to obtain the material surface color image and material surface depth image; the material surface point cloud synthesis process involves using camera intrinsic parameters to convert the material surface color image and material surface depth image into a 3D point cloud; the color and depth images inside the vat obtained from accurate vat opening recognition are synthesized into a point cloud; based on this, the material surface point cloud is removed to obtain the original vat point cloud; outliers are filtered out from the material surface point cloud.

[0047] The matching algorithm between the cylinder model and the material surface point cloud includes: converting the cylinder edge information obtained by the cylinder positioning and detection algorithm into real coordinates.

[0048]

[0049] The circle was fitted using the least squares method, and the center coordinates were obtained as (x0, y0), with z0 being the average value of the depth of the cylinder edge.

[0050] The RANSAC method is used to fit a plane to the ground point cloud, obtaining the equation of the plane containing the ground. The normal vector of the ground is denoted as... In the ground cylinder model, the normal vector at the center of the ground cylinder is The top center coordinates are (0, 0, 0). A transformation matrix is ​​established using the normal vector and the center coordinates to achieve the initial registration of the cylinder model and the point cloud. The formula is as follows:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] P p =P p +T·P g

[0057] in, Let be the axis of rotation, and c be the cosine of the rotation angle, [v] x for The oblique symmetric cross product matrix, where I is the three-dimensional identity matrix, R is the rotation matrix, and T1 is (x0, y0, z0). T T is the transformation matrix, P p For material surface clouding, P g Point cloud for the earthen cylinder model;

[0058] Based on the initial alignment, the ICP registration algorithm in Open3D is used to register the ground cylinder model with the original point cloud of the ground cylinder to achieve accurate alignment. The surface reconstruction is then performed using the a-shape algorithm in Open3D to obtain the material surface ground cylinder model.

[0059] Furthermore, the ICP registration algorithm is as follows: In the target point cloud P and the source point cloud Q to be matched, the nearest point is found according to the constraints, and the optimal matching parameters R and t are calculated to minimize the error function F(R, t). The error function is as follows:

[0060]

[0061] In the formula, p i For the point in the target point cloud P, i.e., p i ∈P, q i For a point in the source point cloud Q, i.e., q i ∈Q, where n is the number of points in the point cloud;

[0062] The ICP registration algorithm includes:

[0063] Step 1: Extract a portion of the point cloud from the target point cloud P as a point set p and find its corresponding point set q in the source point cloud Q;

[0064] Step 2: Calculate the rotation matrix R and translation matrix T that transforms the target point cloud point set p to the corresponding point set q of the source point cloud, and minimize the value of the error function F(R, t);

[0065] Step 3: Perform coordinate transformation on point set p using the rotation matrix R and translation matrix T described above to obtain a new point set denoted as p′; calculate the average distance d between points in point set p′ and corresponding points in point set q, using the following formula:

[0066]

[0067] In the formula, p i ′ represents a point within the point set p′;

[0068] Step 4: If d is less than the given value d0 or the number of iterations is greater than the maximum number of iterations, then stop the calculation. At this time, the rotation matrix R and the translation matrix T are the transformation matrices required for registration. If the conditions are not met, then select a new point set q in the source point cloud Q and continue with step 2.

[0069] Furthermore, the a-shape algorithm uses a ball of radius a to roll randomly on the point cloud. If the ball intersects with three points in the point cloud, the triangle formed by these three points is taken as the boundary surface of the point set.

[0070] The steps of the a-shape are as follows:

[0071] The first step is to randomly select a point p from the point cloud. i The points in the point cloud that are less than 2α away from the given point form a point set S. i From S i Choose any pair of points p1p2 and calculate the value passing through point p. i A sphere with p1p2 and radius a, and centers C1 and C2;

[0072] The second step is to traverse the point set S. i Calculate the distances from other points to the center C1 and C2 of the sphere, denoted as sets D1 and D2 respectively. If either set D1 or set D2 contains a set where all distance values ​​are greater than or equal to the radius α, then the p-th point is considered to be a point on the sphere. i The triangle formed by p1p2 is not an edge contour, so stop traversing.

[0073] The third step is to use point set S. i Select another set of points p1′p2′ and continue to the second step;

[0074] Step 4: Select the point cloud that is not in S i p that is not in the middle i Another point is to continue repeating the first, second, and third steps until all points in the point cloud have been determined. All the obtained contours can then form the three-dimensional reconstructed surface of the point cloud.

[0075] Another objective of this invention is to provide a visual perception system for a fermentation robot oriented towards complex environments. This system is achieved through the aforementioned visual perception method for a fermentation robot oriented towards complex environments. The system includes: a ground vat positioning and detection algorithm module, a fermentation mash surface segmentation algorithm module, and a three-dimensional reconstruction algorithm module for the mash surface and the fermentation environment.

[0076] The cylinder positioning and detection algorithm module is used to add the number of cylinder edges of the cylinder to the initially obtained cylinder area range as an iteration condition to obtain the real cylinder area, and then split the obtained cylinder area mask into several single cylinder area masks for cylinder mouth recognition.

[0077] The fermented mash surface segmentation algorithm module is used to segment the fermented mash surface based on the identification of the vat opening using the fermented mash surface segmentation algorithm;

[0078] The three-dimensional reconstruction algorithm module for the material surface and the fermentation environment is used to reconstruct the three-dimensional surface of the fermented mash based on the segmented material surface and the three-dimensional reconstruction algorithm for the material surface and the fermentation environment to obtain the material surface vat model. The vat model is then transformed into the material surface point cloud through coordinate transformation for preliminary registration. The vat model is then aligned with the original vat point cloud.

[0079] Furthermore, the aforementioned cylinder positioning and detection algorithm module includes a cylinder initial positioning algorithm module and a cylinder mouth accurate identification algorithm;

[0080] The initial positioning algorithm module for the ground cylinder uses a combination of depth data analysis algorithm and improved Hough circle detection to achieve the initial positioning of the ground cylinder.

[0081] The cylinder bore precision recognition algorithm module achieves precise cylinder bore recognition by improving the Canny edge detection algorithm.

[0082] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows: The visual perception system for fermentation robots in complex environments provided by this invention can realize the regional detection and positioning of the vat opening and the identification of the material surface in complex liquor brewing environments. At the same time, it can establish a three-dimensional model of the fermentation material surface and the fermentation environment during fermentation operations, providing information for the digging action of the fermentation robot, thereby realizing efficient and automated digging of the fermentation material.

[0083] The cylinder positioning and detection algorithm of the present invention consists of two parts: a cylinder initial positioning algorithm and a cylinder mouth precise identification algorithm. First, the cylinder initial positioning algorithm is used to find the approximate area of ​​the cylinder, and then the cylinder mouth precise identification algorithm is used to achieve precise identification. This algorithm reduces the area to be detected for precise identification, and on the other hand, improves the data purity of precise identification and reduces the interference of external irrelevant factors and other cylinders in the same picture on the detection results.

[0084] The initial localization algorithm for the ground cylinder in this invention combines a depth data analysis algorithm with an improved Hough circle detection algorithm. The improved Hough circle detection algorithm adds a constraint on the number of cylinder edges to the traditional Hough circle detection algorithm and uses this constraint to modify the upper threshold parameter of Canny edge detection in Hough circle detection, thereby achieving automatic updates of the Hough circle detection parameters for better recognition results. Compared to traditional visual detection methods, the application of the depth information analysis algorithm introduces depth information as a key factor into traditional color image detection and analysis, making the detection results more reliable.

[0085] The cylinder edge accurate recognition algorithm of this invention, based on the traditional Canny edge detection algorithm, selects parameters specifically for the actual cylinder edge detection environment and adds an adaptive threshold iteration algorithm, achieving accurate detection of cylinder edges. Furthermore, to improve detection efficiency, a selection mechanism is designed for the initial detection threshold values ​​of other cylinders in the same image, reducing the threshold iteration time.

[0086] The three-dimensional reconstruction algorithm for the material surface and the fermentation environment of this invention does not reconstruct the entire scene, but only synthesizes the point cloud of the material surface. It uses a standard earthen vat model to register with the obtained point cloud of the material surface, which solves the problem of incomplete reconstruction caused by missing earthen vat image information. At the same time, it reduces the computational load of point cloud synthesis and subsequent processing. Moreover, the earthen vat model is modeled using real data from earthen vat production, which is more accurate than the earthen vat point cloud synthesized from point cloud.

[0087] The matching algorithm between the cylinder model and the material surface point cloud of this invention prioritizes the use of the least squares method and the RANSAC fitting plane algorithm to match the positions of the cylinder model and the material surface point cloud. In response to the possibility of matching deviations, the ICP registration algorithm is used to register the cylinder model with the original cylinder point cloud based on the above matching, thus ensuring the registration accuracy of the two. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0089] Figure 1 This is a schematic diagram of a visual perception system for a fermentation robot designed for complex environments, provided in an embodiment of the present invention.

[0090] Figure 2 This is a flowchart of a visual perception method for a fermentation robot in a complex environment, provided by an embodiment of the present invention.

[0091] Figure 3 This is a flowchart of the initial positioning algorithm for the ground cylinder provided in an embodiment of the present invention;

[0092] Figure 4 This is a flowchart of the cylinder port accurate identification algorithm provided in an embodiment of the present invention;

[0093] Figure 5 This is a flowchart of the fermentation mash surface segmentation algorithm provided in an embodiment of the present invention;

[0094] Figure 6 This is a flowchart of the matching algorithm between the ground cylinder model and the material surface point cloud provided in an embodiment of the present invention;

[0095] In the diagram: 1. Ground vat positioning and detection algorithm module; 101. Ground vat initial positioning algorithm module; 102. Vat mouth precise identification algorithm module; 2. Fermentation mash surface segmentation algorithm module; 3. Fermentation mash surface and fermentation environment three-dimensional reconstruction algorithm module. Detailed Implementation

[0096] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0097] The innovation of the visual perception system and method for fermentation robots in complex environments provided in this invention is as follows: This invention proposes a visual perception system that combines a ground vat positioning algorithm with a material surface and fermentation environment reconstruction algorithm, which solves the visual perception problem of fermentation equipment in the complex environment of ground vat brewing during the fermentation process, and effectively promotes the development of automated equipment for fermentation in ground vats.

[0098] Example 1, such as Figure 1 As shown, the visual perception system for a fermentation robot oriented towards complex environments provided in this embodiment of the invention includes a ground vat positioning and detection algorithm module 1, a fermentation mash surface segmentation algorithm module 2, and a three-dimensional reconstruction algorithm module for the mash surface and the fermentation environment 3.

[0099] The aforementioned cylinder positioning and detection algorithm module 1 includes a cylinder initial positioning algorithm module 101 and a cylinder mouth precise identification algorithm 102. The cylinder initial positioning algorithm 101 reduces the area to be detected for precise identification and improves the data purity for precise identification, reducing interference from external irrelevant factors and other cylinders in the same image.

[0100] The initial positioning algorithm module 101 for the ground cylinder is implemented by combining a depth data analysis algorithm and an improved Hough circle detection. Both the depth data and the color image are acquired using a depth camera; the depth data is stored as a depth map, and the color image is stored as a color image.

[0101] The depth data analysis algorithm primarily utilizes the characteristics of depth values ​​in the ground and the top of the vat region to determine whether each pixel in the depth image belongs to the vat region. The algorithm's detection principle is as follows: In the depth image obtained by the depth camera under unobstructed conditions, the number of pixels corresponding to the ground and the top of the vat accounts for the vast majority of the depth information, and there is a significant difference compared to the number of pixels corresponding to the vat. Converting this to the depth histogram, the ground and the top of the vat should correspond to the group with the highest frequency and the group with the smallest frequency difference among their surrounding groups. Therefore, by selecting an appropriate threshold, the ground and the top of the vat region can be filtered out from the depth image.

[0102] The depth histogram settings are as follows: the depth value of each point in the depth map is set as the x-axis, and the number of points corresponding to each depth value is set as the y-axis. Since the equipment's working range is 1000mm-3000mm, and the distance from the camera installation position to the ground of the tank is approximately 1600mm, to reduce the computational workload during the statistical process, the x-axis range is set to 1500mm-2000mm, with a group interval of 100mm. By analyzing multiple depth histograms, the maximum value of the y-axis is found to be approximately 100,000; therefore, the y-axis range is set to 0-120,000, with an interval of 20,000. The algorithm implementation steps are as follows: First, perform a two-sided search for the maximum frequency in the depth histogram. Extract the horizontal coordinate corresponding to the maximum frequency in the depth data histogram, which is the depth value. If the depth value is within the range of 1500mm-2000mm, compare the current frequency with the frequency of the group to its left. If the difference is within 40% of the current frequency, the data in the group to the left is considered to be ground, and the search continues to the left until the difference is no longer less than 40% of the current frequency or the group containing 1500mm is reached. The same method is used for the data on the right side, comparing the frequency of the current group with the frequency on its right. If the difference is within 40% of the current frequency, the data in the group to the right is considered to be ground, and the search continues to the right until the difference is no longer less than 40% of the current frequency or the group containing 2000mm is reached. After completing the two-sided search for the maximum frequency, the ground and the top of the tank data are represented by 0 in the binary image, and the rest by 1. The resolution of the binary image is the same as the depth image. At this point, the part with a value of 1 in the binary image represents the area of ​​the tank.

[0103] The traditional Hough circle detection principle is as follows: the color image obtained by the depth camera is converted into a grayscale image, and by traversing each potential circle with non-zero pixels, the weight of the curve intersection point in Hough space is used to determine whether the part is the target circle, i.e., the edge of the cylinder.

[0104] This invention improves the Hough circle detection algorithm by adding the number of cylinder edges as an iterative condition to the traditional Hough circle detection algorithm. This enables self-updating of the upper threshold for Canny edge detection before Hough circle detection, thus optimizing the detection effect. The iteration and self-updating strategy is as follows: The upper threshold is initially set to 40 based on the traditional Hough circle detection algorithm. If the number of cylinder edges detected does not meet the requirements, the upper threshold is lowered, and detection continues. If the condition is still not met when the upper threshold equals the lower threshold, the detection fails. The cylinder regions obtained by the depth data analysis algorithm and the improved Hough circle detection algorithm are analyzed, and the nearly overlapping parts are retained as the cylinder regions, while the rest are discarded.

[0105] The cylinder rim accurate recognition algorithm module 102 is mainly implemented through an improved Canny edge detection algorithm. This improved Canny edge detection algorithm first converts the color image of a single cylinder area to grayscale, then performs Gaussian filtering to denoise the image. The processed image is slightly blurry compared to the original image, but the interference of noise on image information is reduced. Next, the first derivatives in the horizontal and vertical directions are calculated using the Sobel operator to obtain the magnitude and direction of the gradient. Then, non-maximum suppression is performed, that is, each pixel is checked to determine whether its gradient is the largest in the same gradient direction among its surrounding pixels. If so, the pixel is retained; otherwise, it is removed. After non-maximum suppression, for several edge points in the same direction, basically only one is retained, achieving refined edge information. After Gaussian filtering, gradient calculation, and non-maximum suppression, the obtained edge information is quite messy and contains a lot of noise. Dual threshold filtering is used to optimize the extracted edges to obtain the desired edge information. However, due to the influence of factors such as lighting, the high and low thresholds of different images are not fixed, and the selection of high and low thresholds of different regions in the same image is also slightly different. Therefore, this invention adopts an adaptive iterative method for threshold selection.

[0106] In selecting the threshold, we first statistically analyzed the high and low threshold data when Canny detection met the requirements. We then set the initial high threshold to 1.5 times the larger value among the statistical high thresholds and the initial low threshold to the larger value among the statistical low thresholds.

[0107] In this embodiment of the invention, Canny edge detection is an edge detection algorithm that utilizes image gradients to obtain image edges. The algorithm principle is as follows: the grayscale values ​​of object edges in a grayscale image undergo significant changes, i.e., large gradient values. By calculating the gradient values ​​of each point in the image and analyzing them in a certain way, the edge contour information of the object can be obtained. Traditional Canny edge detection mainly consists of Gaussian filtering, gradient calculation, non-maximum suppression, and double threshold filtering. Gaussian filtering is a linear smoothing filter that uses a normal distribution to calculate the transformation of each pixel in the image. Compared to median filtering and mean filtering, in Gaussian filtering, the value of each pixel is obtained by a weighted average of its own value and the values ​​of other pixels in its neighborhood. The weights decrease as the distance between the pixel and other pixels in the neighborhood increases, making it closer to the real image and reducing the destruction of edge information while achieving image smoothing. Gradient calculation uses the Sobel operator to calculate the gradient in the x and y directions. Then, the arithmetic mean of the gradients in the x and y directions is taken as the magnitude of the gradient at that point, and the arctangent of the gradients in the x and y directions is taken as the direction of the gradient at that point. Non-maximum suppression is an algorithm for refining edge information. Its principle is to convert the gradient direction into eight directions: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°, based on the principle of small differences. It then iterates through the gradient values ​​of each point in the image and compares the gradient value of each point with the gradient value of the same direction in the neighborhood. If it is the maximum value, it is retained; otherwise, the gradient value of that point is set to 0. After non-maximum suppression (NMS), strong edges are already within the acquired edge image. However, some false edges may also be present. Therefore, a double threshold filter is needed to process the NMS result. The double threshold filter sets two thresholds: a high threshold and a low threshold. If the gradient magnitude of the current edge pixel is greater than or equal to the high threshold, the current edge pixel is marked as a strong edge, meaning it needs to be retained. If the gradient magnitude of the current edge pixel is between the high and low thresholds, the current edge pixel is marked as a false edge, which also needs to be retained. If the gradient magnitude of the current edge pixel is less than or equal to the low threshold, the current edge pixel is suppressed, meaning the edge at that point is discarded. Further judgment is needed for the obtained false edges. If the false edge is connected to a strong edge, it is considered a strong edge and its edge information is retained. If the false edge is not connected to a strong edge, it is considered a weak edge and discarded.

[0108] The high and low thresholds that meet the requirements refer to the high and low thresholds for Canny edge detection when the Canny detection result has an inner contour and the inner contour meets the size requirements. Whether an inner contour exists can be obtained by OpenCV's built-in contour acquisition algorithm, and the inner contour size is obtained by manually calculating the inner contour size before detection.

[0109] Then, for single-cylinder images, the initial threshold is used for detection. If there is an inner contour and the inner contour meets the size requirements, it is considered that the edge of the cylinder has been detected. If there is no inner contour or the inner contour does not meet the size requirements, the Canny threshold is changed and Canny detection is performed again.

[0110] The adaptive iterative thresholding method employs two logical loops: the inner loop updates the high threshold, and the outer loop updates the low threshold. If no closed edge data is detected, the high threshold is lowered first, and edge detection is performed again. If no closed edge is detected, the high threshold is lowered further until the difference between the high and low thresholds is within 100. If no closed edge information is obtained at this point, the low threshold is lowered, and the high threshold is restored to its initial value, and detection continues until a closed contour is obtained. If the difference between the low threshold and the initial low threshold reaches 100, the detection is considered a failure, and the detection is repeated in a different tank area. Since the Canny detection parameters for different tanks in the same image are not significantly different, to reduce computational load, the average value of the detected Canny thresholds is used to assign values ​​to the initial high and low thresholds. The initial high threshold is 1.2 times the average high threshold, and the initial low threshold is 1.2 times the average low threshold. After obtaining the tank edge, the position of the tank center can be obtained by drawing the minimum bounding rectangle of the edge and using coordinate transformation.

[0111] Due to poor lighting conditions and inconsistent color within the material handling tank, traditional visual inspection methods struggle to achieve satisfactory material surface segmentation results.

[0112] The fermentation mash surface segmentation algorithm module 2 is used to identify and segment the fermentation mash surface using the YOLOv7 network. First, the surface areas in the color images captured by the depth camera are labeled to create a dataset for network training. Then, the relevant network training parameters are set and the network is trained. After that, the trained model is tested and some parameters and training samples are adjusted to achieve high-speed and accurate segmentation of the fermentation mash surface.

[0113] The 3D reconstruction algorithm module 3 for the material surface and the fermentation environment is used to reconstruct the 3D curved surface to obtain the material surface cylinder model, and to transform the cylinder model into the material surface point cloud through coordinate transformation for preliminary registration; and to accurately align the cylinder model with the original cylinder point cloud.

[0114] The 3D reconstruction algorithm module 3 for the material surface and the fermentation environment includes several parts: synthesis of material surface point clouds, synthesis of original point clouds of the fermentation tank, establishment of the fermentation tank model, and matching algorithm between the fermentation tank model and the material surface point clouds. Since the fermentation tank absorbs some of the infrared light from visual detection, complete point cloud data of the fermentation tank cannot be obtained in the actual detection process. Therefore, the fermentation tank model is obtained by modeling using actual production data of the fermentation tank.

[0115] The material surface point cloud is obtained by synthesizing the material surface color image and the material surface depth image obtained from image segmentation. After synthesizing the cylinder interior color and depth images obtained from accurate cylinder inlet recognition and removing the material surface point cloud, the original point cloud of the cylinder is obtained. After obtaining the material surface point cloud, outliers are filtered and downsampled to reduce the density of the point cloud and reduce the point cloud processing time while preserving the approximate shape of the material surface.

[0116] The matching algorithm between the cylinder model and the material surface point cloud is as follows: First, the least squares method is used to fit the point cloud information of the cylinder edge to obtain the coordinates of the center of the circle where the cylinder is located, and then the coordinates are transformed to the world coordinate system. Second, the RANSAC method is used to fit the plane of the ground point cloud to obtain the equation of the plane where the ground is located, and the normal vector of the ground is obtained. The cylinder model is transformed into the material surface point cloud through coordinate transformation to achieve preliminary registration. The ICP registration algorithm in Open3D is used to register the cylinder model with the original cylinder point cloud to achieve accurate alignment. The material surface cylinder model can be obtained by reconstructing the surface using the a-shape algorithm in Open3D.

[0117] In this embodiment of the invention, the principle of the ICP registration algorithm is as follows: In the target point cloud P and the source point cloud Q to be matched, the nearest neighbor point is found according to certain constraints. Then, the optimal matching parameters R and t are calculated to minimize the error function F(R, t). The error function is as follows:

[0118]

[0119] In the formula, p i For the point in the target point cloud P, i.e., p i ∈P, q i For a point in the source point cloud Q, i.e., q i ∈Q, where n is the number of points in the point cloud;

[0120] The ICP registration algorithm steps are as follows:

[0121] Step 1: Extract a portion of the point cloud from the target point cloud P as a point set p and find its corresponding point set q in the source point cloud Q;

[0122] Step 2: Calculate the rotation matrix R and translation matrix T that transforms the target point cloud point set p to the corresponding point set q of the source point cloud, and minimize the value of the error function F(R, t);

[0123] Step 3: Perform coordinate transformation on point set p using the rotation matrix R and translation matrix T described above to obtain a new point set denoted as p′; calculate the average distance d between points in point set p′ and corresponding points in point set q, using the following formula:

[0124]

[0125] In the formula, p i ′ represents a point within the point set p′;

[0126] Step 4: If d is less than the given value d0 or the number of iterations is greater than the maximum number of iterations, then stop the calculation. At this time, the rotation matrix R and the translation matrix T are the transformation matrices required for registration. If the conditions are not met, then select a new point set q in the source point cloud Q and continue with step 2.

[0127] In this embodiment of the invention, the a-shape algorithm is also known as the rolling ball method. Its principle is to use a ball with a radius of α to roll randomly on the point cloud. If the ball intersects with three points in the point cloud, the triangle formed by the three points is taken as the boundary surface of the point set.

[0128] The steps of a-shape are as follows:

[0129] The first step is to randomly select a point p from the point cloud. i The points in the point cloud that are less than 2α away from the given point form a point set S. i From S i Choose any pair of points p1p2 and calculate the value passing through point p. i A sphere with p1p2 and radius a, and centers C1 and C2;

[0130] The second step is to traverse the point set S. i Calculate the distances from other points to the center C1 and C2 of the sphere, denoted as sets D1 and D2 respectively. If either set D1 or set D2 contains a set where all distance values ​​are greater than or equal to the radius α, then the p-th point is considered to be a point on the sphere. i The triangle formed by p1p2 is not an edge contour, so stop traversing.

[0131] The third step is to use point set S. i Select another set of points p1′p2′ and continue to the second step;

[0132] Step 4: Select the point cloud that is not in S i p that is not in the middle i Another point is to continue repeating the first, second, and third steps until all points in the point cloud have been determined. All the obtained contours can then form the three-dimensional reconstructed surface of the point cloud.

[0133] As demonstrated by the above embodiments, this invention solves the problem of automating the fermentation process of earthen vats in baijiu brewing. It proposes a novel visual perception system that addresses the perception problem of the earthen vat fermentation robot, thus promoting the automation of brewing processes in earthen vat fermentation enterprises such as Fenjiu and Hengshui Laobaigan. This invention fills the gap in effective perception methods for the earthen vat fermentation process, proposing a visual perception method for fermentation robots in complex environments, greatly accelerating the automation of fermentation equipment. This invention solves the problem of automating the fermentation process of earthen vats in baijiu brewing, proposing a novel visual perception system that addresses the perception problem of the earthen vat fermentation robot.

[0134] Example 2, as another embodiment of the present invention, such as Figure 2 As shown, the visual perception method for fermentation robots in complex environments provided in this embodiment of the invention includes a vat positioning and detection algorithm and a fermentation mash surface segmentation algorithm (such as...). Figure 5 The algorithm consists of several parts, including the three-dimensional reconstruction algorithm of the material surface and the fermentation environment.

[0135] Specifically, it includes:

[0136] S1. Using the cylinder positioning detection algorithm, the number of cylinder edges of the initially obtained cylinder area is added as an iterative condition to obtain the real cylinder area. The obtained cylinder area mask is then divided into several single cylinder area masks for cylinder mouth recognition.

[0137] S2, based on the identification of the vat opening, the mash surface is segmented using a mash surface segmentation algorithm;

[0138] S3, based on the segmented mash surface, uses a 3D reconstruction algorithm of the mash surface and the mash discharge environment to reconstruct the 3D surface of the fermentation vat model, transforms the vat model into the mash surface point cloud through coordinate transformation, performs preliminary registration, and aligns the vat model with the original vat point cloud.

[0139] The aforementioned cylinder positioning and detection algorithm consists of a cylinder initial positioning algorithm (such as...) Figure 3 ) and cylinder port precision identification algorithms (such as Figure 4 It consists of two parts. The initial positioning algorithm for the ground cylinder is implemented by combining a depth data analysis algorithm and an improved Hough circle detection algorithm.

[0140] The depth data analysis algorithm works as follows: The depth map can be divided into two parts: the ground and the top of the tank area, and the tank area itself. In the absence of obstructions, the depth data obtained by the depth camera shows certain characteristics in the ground and tank top areas. By identifying these characteristics, the approximate areas of the ground and tank top can be obtained, thus indirectly revealing the tank area. The image data includes depth and color images, acquired by the depth camera.

[0141] The deep data analysis algorithm mainly includes the following steps: acquiring data features from the ground and the top of the tank, building a depth histogram, performing maximum frequency two-sided class search, and obtaining the range of the tank area. The specific steps are as follows:

[0142] Step a: Obtain ground depth data features. Through multiple preliminary experiments, the characteristics of the ground and the top of the tank depth data are as follows: in the depth image under unobstructed conditions, the number of pixel values ​​corresponding to the ground and the top of the tank accounts for the vast majority of the depth information, and there is a significant difference compared to the number of pixels in the tank. When converted to a depth histogram, this means that the group with the highest frequency corresponding to the ground and the top of the tank should be the group with the smallest difference in frequency among the surrounding groups.

[0143] Step b: The parameters for the established depth histogram are set as follows: the horizontal axis represents the depth value, and the vertical axis, i.e., the frequency, represents the number of pixels within the same depth group; the horizontal axis range is set to 1500-2000mm, with a group interval of 100mm. By statistically analyzing multiple depth histograms, the maximum value of the vertical axis is found to be approximately 100,000; therefore, the frequency range is set to 0-120,000, and the interval is set to 20,000.

[0144] Step c: Maximum frequency two-sided class search is used to determine the range of ground and ground tank data. The algorithm is as follows: If the group corresponding to the maximum frequency in the depth histogram is located in the range of 1500mm-2000mm, compare the current frequency with the frequency of the group to its left. If the difference is within 40% of the current frequency, the data in the group to the left is considered to be ground, and the search continues to the left until the difference is no longer less than 40% of the current frequency or the group at 1500mm is reached. The same method is used for the data on the right. Compare the frequency of the current group with the frequency on its right. If the difference is within 40% of the current frequency, the data in the group on the right is considered to be ground, and the search continues to the right until the difference is no longer less than 40% of the current frequency or the group at 2000mm is reached.

[0145] Step d: After completing the maximum frequency two-sided class search, a binary map is built. The resolution of the binary map is the same as that of the depth map. The pixels corresponding to the ground and top of the tank are represented by 0 in the binary map, and the rest are represented by 1. The part with a value of 1 in the binary map is the range of the tank area.

[0146] The improved Hough circle detection algorithm adds the number of cylinder edges as an iterative condition to the traditional Hough circle detection algorithm. This enables self-updating of the upper threshold for Canny edge detection before Hough circle detection, thus optimizing the detection effect. The Hough circle detection algorithm is an algorithm for obtaining circle detection results on grayscale images. It determines the target circle by traversing every non-zero pixel potential circle and determining the weight of the curve intersection points in Hough space. Points in the Cartesian coordinate system that lie on the same circle should also lie at the same point in the ABR coordinate system. Therefore, in the ABR coordinate system, this point should have curves intersecting with the total number of pixels of the circle. By judging the number of intersections at each point in the ABR coordinate system, if it exceeds a certain threshold, the point is considered a circle. Hough circle detection is implemented using built-in OpenCV functions. The iteration and self-updating strategy is as follows: the upper threshold is initially set to 40. If the number of cylinder edges detected does not meet the requirement, the upper threshold is lowered, and detection continues. If the upper threshold equals the lower threshold and the condition is still not met, the detection fails.

[0147] In this embodiment of the invention, the region of the underground water tank obtained by the depth data analysis algorithm and the improved Hough circle detection algorithm is optimized and analyzed. The contour is extracted from the binary image obtained by the depth data analysis, and the circumscribed rectangle of the approximate region of the underground water tank is drawn. Let the length of the obtained circumscribed rectangle be x. r y r The geometric center is (x i y i The expression for the virtual radius is:

[0148]

[0149] In the formula, r r For virtual radius;

[0150] The following conditions must be met for the specified constant D:

[0151]

[0152] In the formula, r c The radius of the circle obtained by Hough circle detection, (x c y c () represents the center of the circle obtained by Hough circle detection;

[0153] If the conditions are met, the cylinder area detected by the Hough circle is the real cylinder area. If the conditions are not met, the cylinder area obtained by the current Hough circle detection is discarded, and the calculation of whether other areas meet the requirements continues until all are detected. After the detection is completed, the obtained cylinder area mask is split into several single cylinder area masks for accurate identification by the cylinder mouth recognition algorithm.

[0154] In this embodiment of the invention, the cylinder bore accurate identification algorithm is implemented by an improved Canny edge detection algorithm. The specific steps of the algorithm are as follows:

[0155] Step 1: Perform an AND operation between the color image and the mask of the single-cylinder area to obtain the color image of the single-cylinder area.

[0156] Step II: Convert the obtained color image of a single cylinder to grayscale. The grayscale conversion formula is shown below:

[0157] Gray=0.299×R+0.587×G+0.114×B

[0158] In the formula, Gray is the gray value of the point, R is the pixel value of the red part in the color image of the point, G is the pixel value of the green part in the color image of the point, and B is the pixel value of the blue part in the color image of the point.

[0159] Gaussian filtering is applied to the obtained grayscale image to reduce noise interference with the inspection. The specific formula is shown below, where the kernel function is a (5, 5) Gaussian kernel.

[0160]

[0161] In this embodiment of the invention, the kernel function f(x, y) is the formula for calculating the value of each point within the matrix formed by Gaussian filtering. For example, the (5, 5) Gaussian kernel mentioned above is a 5×5 matrix, and the value of each point within the matrix is ​​the function value corresponding to f(x, y). For example, the value corresponding to the first row and first column of the matrix is ​​f(0, 0). The value of each point within the Gaussian filtering matrix conforms to a normal distribution, that is, the kernel function f(x, y) is the probability density function of the normal distribution. The letter δ represents the standard deviation of the normal distribution, which is used to adjust the proportional relationship between the value at the middle position and other values ​​in the matrix during Gaussian filtering. The magnitude of this value is determined by the size of the Gaussian kernel (i.e., (5, 5) mentioned above).

[0162] Step III: Use the Sobel operator to calculate the first derivatives in the horizontal and vertical directions to obtain the magnitude and direction of the gradient. The specific calculation formula is shown below:

[0163]

[0164] In the formula, Edqe_Grandient(G) is the magnitude of the edge gradient, and G x G represents the gradient magnitude in the x-direction of the image. y Angle(θ) is the gradient magnitude in the y-direction of the image, and Angle(θ) is the angle between the edge gradient direction and the x-direction of the image.

[0165] Step IV: Based on the magnitude of Angle(θ), the gradient directions are divided into four categories: vertical, horizontal, and the two diagonals. Then, each pixel is checked to determine whether the gradient of the point is the largest in the same gradient direction around it. If it is, the point is retained; otherwise, the point is removed. This process is also called non-maximum suppression. After non-maximum suppression, for several edge points in the same direction, basically only one is retained, thus achieving the refinement of edge information.

[0166] Step V: Optimize the extracted edges using a dual-threshold method. The two thresholds are a high threshold and a low threshold. If the gradient magnitude of the current edge pixel is greater than or equal to the high threshold, the current edge pixel is marked as a strong edge, meaning it needs to be retained. If the gradient magnitude of the current edge pixel is between the high and low thresholds, the current edge pixel is marked as a virtual edge, which also needs to be retained. If the gradient magnitude of the current edge pixel is less than or equal to the low threshold, the current edge pixel is suppressed, meaning the edge at that location is discarded. Further judgment is needed for the obtained virtual edges. If the virtual edge is connected to a strong edge, it is considered a strong edge and its edge information is retained. If the virtual edge is not connected to a strong edge, it is considered a weak edge and discarded.

[0167] Since the high and low thresholds are not fixed for different images, and even within the same image, the high and low thresholds may vary slightly depending on factors such as lighting, this invention employs an adaptive iterative method for threshold selection to achieve better detection results. The specific steps are as follows:

[0168] Step V-1: First, perform several rounds of manual detection on the image and count the high and low threshold data when the Canny detection meets the requirements. Set the initial high threshold to 1.5 times the larger value among the high thresholds and the initial low threshold to the larger value among the low thresholds.

[0169] Step V-2: For single-tank images, the initial threshold is used for detection. If an inner contour exists and meets the size requirements, the tank edge is considered detected. The contour detection algorithm uses OpenCV's built-in algorithm. If there is no inner contour or the inner contour does not meet the size requirements, the Canny threshold is changed and Canny detection is performed again.

[0170] Step V-3: The threshold is changed using an adaptive iterative method. This algorithm uses two logical loops to iteratively update the threshold. The inner loop updates the high threshold, and the outer loop updates the low threshold. If no closed edge data is detected, the high threshold is lowered first, and then edge detection is performed again. If no closed edge is detected, the high threshold is lowered again until the difference between the high and low thresholds is within 100. If no closed edge information is obtained at this point, the low threshold is lowered, the high threshold is restored to its initial value, and detection continues until a closed contour is obtained. If the difference between the low threshold and the initial low threshold reaches 100, the detection is considered to have failed, and the detection is continued in a different canny area. Since the detection parameters of canny areas in the same image are not significantly different, in order to reduce the computational load, the average value of the detected canny thresholds is used to assign values ​​to the initial high and initial low thresholds. The initial high threshold is 1.2 times the average high threshold, and the initial low threshold is 1.2 times the average low threshold.

[0171] Step VI: Draw the smallest bounding rectangle for the contour, obtain the centroid coordinates, and use the camera intrinsic parameters to transform the two-dimensional centroid coordinates to the three-dimensional coordinate system. The specific formula is shown in (8):

[0172]

[0173] In the formula, (x, y, z) are the coordinates of the point in the camera coordinate system, u is the pixel value of the point in the x-axis direction, v is the pixel value of the point in the y-axis direction, and c x Let c be the initial value of the camera center in the x-axis direction. y Let x be the initial value of the camera center along the y-axis, and depth be the depth value of that point. w y w , z w (T) represents the coordinates of a point in the world coordinate system. x T y T z Let be the translations in the x, y, and z directions, and R be the rotation matrix.

[0174] In this embodiment of the invention, the fermentation mash surface segmentation algorithm is implemented using the YOLOv7 image segmentation algorithm. The mash surface regions within the color images captured by the camera are labeled, a dataset for network training is created, the network training parameters are set and trained, and then the obtained model is tested and some parameters and training samples are adjusted to achieve high-speed and accurate segmentation of the fermentation mash surface.

[0175] The 3D reconstruction algorithm for the material surface and the fermentation environment includes the synthesis of material surface point clouds, the synthesis of original point clouds of the fermentation tank, the establishment of the fermentation tank model, and a matching algorithm between the fermentation tank model and the material surface point clouds (e.g., Figure 6 ).

[0176] The model of the earthen vat is modeled based on the actual production data of the earthen vat. The specific steps for synthesizing the material surface point cloud are as follows: the material surface mask obtained by image segmentation is ANDed with the color image and the depth image to obtain the material surface color image and the material surface depth image; the material surface point cloud synthesis process is to use the camera intrinsic parameters to convert the material surface color image and the material surface depth image into a three-dimensional point cloud, as shown in formula (8); the color and depth images inside the vat obtained by accurate vat mouth recognition are synthesized into a point cloud, as shown in formula (8), and the original point cloud of the earthen vat is obtained by removing the material surface point cloud. Outliers are filtered out from the material surface point cloud and downsampling is performed. While retaining the approximate shape of the material surface, the density of the point cloud is reduced, and the time of point cloud computation is reduced.

[0177] For example, the matching algorithm between the ground vat model and the point cloud of the mash surface realizes the matching of the position of the ground vat model and the mash surface. First, the vat edge information obtained by the ground vat positioning and detection algorithm is converted into real coordinates, as shown in formula (8) and formula (9);

[0178]

[0179] The circle was fitted using the least squares method, and the center coordinates were obtained as (x0, y0), with z0 being the average value of the depth of the cylinder edge.

[0180] The RANSAC method is used to fit a plane to the ground point cloud, obtaining the equation of the plane containing the ground. The normal vector of the ground is denoted as... In the ground cylinder model, the normal vector at the center of the ground cylinder is The top center coordinates are (0, 0, 0). A transformation matrix is ​​established using the normal vector and the center coordinates to achieve the initial registration of the cylinder model and the point cloud. The formula is as follows:

[0181]

[0182]

[0183]

[0184]

[0185]

[0186] P p =P p +T•P g (15)

[0187] in, Let be the axis of rotation, and c be the cosine of the rotation angle, [v] x for The oblique symmetric cross product matrix; I is the three-dimensional identity matrix, R is the rotation matrix, and T1 is (x0, y0, z0).T T is the transformation matrix, P p For material surface clouding, P g Point cloud for the earthen cylinder model;

[0188] For example, the RANSAC method is an iterative approach to compute mathematical model parameters from a series of data containing outliers, and it is also frequently used for plane fitting in point clouds. The specific algorithm flow is as follows:

[0189] (1) Randomly select three points from the point cloud.

[0190] (2) These three points form a plane.

[0191] (3) Calculate the distance of all other points to the plane. If the distance is less than the threshold T, they are considered to be points on the same plane.

[0192] (4) If there are more than n points in the same plane, save the plane and mark all the points in the plane as matched.

[0193] (5) The termination condition is that after N iterations, the plane found has less than n points, or no three unmarked points can be found.

[0194] Based on the initial alignment, the ICP registration algorithm in Open3D is used to register the ground cylinder model with the original point cloud of the ground cylinder to achieve accurate alignment. The surface reconstruction is then performed using the α-shape algorithm in Open3D to obtain the ground cylinder model with material surface.

[0195] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0196] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0197] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0198] This invention also provides a computer device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0199] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps described in the various method embodiments above.

[0200] This invention also provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, or switches.

[0201] This invention also provides a server that, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments.

[0202] This invention provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0204] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A visual perception method for a fermentation robot in complex environments, characterized in that, The method includes the following steps: S1. Using the cylinder positioning detection algorithm, the number of cylinder edges of the initially obtained cylinder area is added as an iterative condition to obtain the real cylinder area. The obtained cylinder area mask is then divided into several single cylinder area masks for cylinder mouth recognition. S2, based on the identification of the vat opening, the mash surface is segmented using a mash surface segmentation algorithm; S3, based on the segmented mash surface, uses a 3D reconstruction algorithm of the mash surface and the mash discharge environment to reconstruct the 3D surface of the fermentation vat model, transforms the vat model into the mash surface point cloud through coordinate transformation, performs preliminary registration, and aligns the vat model with the original vat point cloud; In step S3, the three-dimensional reconstruction algorithm of the material surface and the fermentation environment includes: synthesis of the material surface point cloud, synthesis of the original point cloud of the fermentation tank, establishment of the fermentation tank model, and matching algorithm between the fermentation tank model and the material surface point cloud; the fermentation tank model is obtained by modeling based on the actual production data of the fermentation tank. The specific steps for synthesizing the material surface point cloud are as follows: The material surface mask obtained from image segmentation is ANDed with the color image and depth image respectively to obtain the material surface color image and material surface depth image; the material surface point cloud synthesis process involves using camera intrinsic parameters to convert the material surface color image and material surface depth image into a 3D point cloud; the color and depth images inside the vat obtained from accurate vat opening recognition are synthesized into a point cloud; based on this, the material surface point cloud is removed to obtain the original vat point cloud; outliers are filtered out from the material surface point cloud. The matching algorithm between the cylinder model and the material surface point cloud includes: converting the cylinder edge information obtained by the cylinder positioning and detection algorithm into real coordinates. The circle was fitted using the least squares method, and the coordinates of the center were obtained as follows: , The average depth of the cylinder rim is used; where, Let these be the coordinates of the point in the world coordinate system. for Translation in the direction, It is a rotation matrix; The RANSAC method is used to fit a plane to the ground point cloud, obtaining the equation of the plane containing the ground. The normal vector of the ground is denoted as... In the ground cylinder model, the normal vector at the center of the ground cylinder is... The coordinates of the top center are A transformation matrix is ​​established using the normal vector and the center coordinates to achieve preliminary registration between the ground cylinder model and the point cloud. The formula is as follows: in, For the axis of rotation, The value of the cosine of the rotation angle. for The oblique symmetric cross product matrix, It is a three-dimensional identity matrix. for , Let be the transformation matrix. Add clouds to the surface of the material. Point cloud for the earthen cylinder model; Based on the initial alignment, the ICP registration algorithm within Open3D is used to register the ground tank model with the original point cloud of the ground tank, achieving precise alignment. The algorithm reconstructs the surface to obtain the material surface cylinder model.

2. The visual perception method for a fermentation robot in complex environments according to claim 1, characterized in that, In step S1, the cylinder positioning detection algorithm consists of the initial cylinder positioning algorithm and the cylinder port accurate identification algorithm; The initial positioning algorithm for the ground cylinder is a combination of a depth data analysis algorithm and an improved Hough circle detection algorithm. The depth data analysis algorithm includes the following steps: Step a: Obtain ground depth data features: Convert the depth images of the ground and the top of the tank into a depth histogram; Step b: Create a depth histogram with the following parameters: the horizontal axis represents the depth value, and the vertical axis represents the frequency, which is the number of pixels within the same depth group. Step c: Use the maximum frequency two-sided classification to determine the range of ground and ground cylinder data. Compare the current frequency with the frequencies of its left and right groups. Determine whether it is ground by whether the frequency difference is within 40% of the current frequency. Step d: After completing the maximum frequency two-sided class search, a binary map is built. The resolution of the binary map is the same as that of the depth map. The pixels corresponding to the ground and top of the tank are represented by 0 in the binary map, and the rest are represented by 1. The part with a value of 1 in the binary map is the range of the tank area. The improved Hough circle detection algorithm includes: adding the number of cylinder edges as an iteration condition on the basis of Hough circle detection, and completing the upper threshold self-updating of Canny edge detection before Hough circle detection; the iteration and self-updating strategy is: the upper threshold is initially set to 40. If the number of cylinder edges detected cannot meet the requirements, the upper threshold is reduced and detection continues. If the upper threshold and the lower threshold are equal and the conditions are still met, the detection fails. The cylinder port accurate identification algorithm uses an improved Canny edge detection algorithm to achieve accurate cylinder port identification.

3. The visual perception method for a fermentation robot in complex environments according to claim 2, characterized in that, The initial positioning algorithm for the ground cylinder includes: The contour is extracted from the binary image obtained by depth data analysis, and the circumscribed rectangle of the approximate area of ​​the underground reservoir is drawn. Let the length of the obtained circumscribed rectangle be... The geometric center is The expression for the virtual radius is: In the formula, For virtual radius; Satisfy the specified constant The following conditions: In the formula, The radius of the circle obtained by Hough circle detection. The center of the circle obtained by Hough circle detection; If the conditions are met, the cylinder area detected by the Hough circle is the real cylinder area. If the conditions are not met, the cylinder area obtained by the current Hough circle detection is discarded, and the calculation of whether other areas meet the requirements continues until all are detected. After the detection is completed, the obtained cylinder area mask is split into several single cylinder area masks for accurate identification by the cylinder mouth recognition algorithm.

4. The visual perception method for a fermentation robot in complex environments according to claim 2, characterized in that, Improvements to the Canny edge detection algorithm include: Step 1: Perform an AND operation between the color image and the mask of the single-cylinder area to obtain the color image of the single-cylinder area; Step II: Convert the obtained color image of the single-cylinder tank to grayscale. The grayscale conversion formula is as follows: In the formula, This represents the grayscale value of that point. This represents the pixel value of the red portion in the color image at that point. This represents the pixel value of the green portion in the color image at that point. This represents the pixel value of the blue portion in the color image at that point; The obtained grayscale image is subjected to Gaussian filtering to reduce noise interference with the inspection. The specific formula is shown below: In the formula, the kernel function is: Gaussian kernel, kernel function Here is the formula for calculating the values ​​of points within the matrix formed by Gaussian filtering. The standard deviation of the normal distribution. As a constant An exponential function with base 0. These are the coordinates of a pixel in the image; Step III: Use the Sobel operator to calculate the first derivatives in the horizontal and vertical directions to obtain the magnitude and direction of the gradient. The specific calculation formula is shown below: In the formula, The magnitude of the edge gradient. For image The magnitude of the gradient in the direction, For image The magnitude of the gradient in the direction, Edge gradient direction and image The angle between directions; Step IV: According to The size of the gradient direction is divided into four categories: vertical, horizontal and two diagonals. Each pixel is checked to determine whether the gradient of the point is the largest gradient value in the same gradient direction around it. If it is, the point is kept; otherwise, the point is removed. After non-maximum suppression, for several edge points in the same direction, basically only one is kept. Step V: Using a dual threshold to determine the edges, the extracted edges are optimized using an adaptive iterative method; specifically including: Step V-1: Perform several rounds of manual detection on the detection image, and count the high and low threshold data when the Canny detection meets the requirements. Set the initial high threshold to 1.5 times the larger value among the statistical high thresholds, and set the initial low threshold to the larger value among the statistical low thresholds. Step V-2: For single-cylinder images, the initial threshold is used for detection first. If an inner contour is present and meets the size requirements, the edge of the cylinder is considered to have been detected. Step V-3: The threshold is changed using an adaptive iterative method. Two logical loops are used to iteratively update the threshold: the inner loop updates the high threshold, and the outer loop updates the low threshold. If no closed edge data is detected, the high threshold is lowered first, and edge detection is performed again. If no closed edge is detected, the high threshold is lowered further until the difference between the high and low thresholds is within 100. If no closed edge information is obtained at this point, the low threshold is lowered, the high threshold is restored to its initial value, and detection continues until a closed contour is obtained. If the difference between the low threshold and its initial value reaches 100, the detection is considered a failure, and the detection is performed in a different area. Step VI: Draw the smallest bounding rectangle for the contour, obtain the centroid coordinates, and use the camera intrinsic parameters to transform the 2D centroid coordinates to the 3D coordinate system as shown in the formula: In the formula, Let these be the coordinates of the point in the camera coordinate system. For this point at Pixel values ​​along the axis, For this point at Pixel values ​​along the axis, For the camera center at Initial values ​​in the axial direction, For the camera center at Initial values ​​in the axial direction, This is the depth value at that point. Let these be the coordinates of the point in the world coordinate system. for Translation in the direction, It is a rotation matrix.

5. The visual perception method for a fermentation robot in complex environments according to claim 1, characterized in that, In step S2, the fermented mash surface segmentation algorithm uses the YOLOv7 image segmentation algorithm to label the surface area in the color image captured by the camera, create a dataset for network training, set the network training parameters and train it, and then test the obtained model and adjust some parameters and training samples to achieve the segmentation of the fermented mash surface.

6. The visual perception method for a fermentation robot in complex environments according to claim 1, characterized in that, The ICP registration algorithm is as follows: in the target point cloud to be matched... Heyuan Point Cloud In the process, the nearest point is found according to the constraints, and the optimal matching parameters are calculated. and This makes the error function Minimum error function is as follows: In the formula, For target point cloud The point in, that is , Source Point Cloud The point in, that is , The number of points in the point cloud; The ICP registration algorithm includes: Step 1, Extract the target point cloud Part of the point cloud is used as a point set And find its location in the source cloud. The corresponding point set ; Step 2, calculate the target point cloud point set. Transformed to the corresponding point set of the source point cloud rotation matrix Translation matrix and make the error function The value is the smallest; Step 3, for the point set Using the rotation matrix described above Translation matrix The coordinate transformation yields a new set of points, denoted as . ; Compute point set Interior points and point sets Average distance between corresponding points The formula is as follows: In the formula, For point set points within; Step 4, if Less than the given value If the number of iterations exceeds the maximum number of iterations, the calculation stops, and the rotation matrix is ​​then... Translation matrix This is the transformation matrix required for registration; if the condition is not met, then in the source point cloud... Select a new set of points Proceed to step 2.

7. The visual perception method for a fermentation robot in complex environments according to claim 1, characterized in that, The algorithm uses a radius of A small ball rolls randomly on a point cloud. If the ball intersects with three points in the point cloud, the triangle formed by these three points is taken as the boundary surface of the point set. The The steps are as follows: The first step is to randomly select a point from the point cloud. The points in the point cloud that are less than 2α away from the given point form a point set. ,from Choose any set of points Calculate the points And the radius is Let the center of the ball be... ; The second step is to traverse the point set. Calculate the distance from other points to the center of the sphere. The distances between them are denoted by sets. and set If set or set There is a set of sets where all distance values ​​are greater than or equal to the radius. Then the If the triangle formed is not an edge contour, stop traversing; The third step is to use the point set. Choose another set of points Proceed to the second step; Step 4: Select the point cloud that is not present. And not for Another point is to continue repeating the first, second, and third steps until all points in the point cloud have been determined. All the obtained contours can then form the three-dimensional reconstructed surface of the point cloud.

8. A visual perception system for a fermentation robot in complex environments, characterized in that, The system is implemented by the visual perception method of the fermentation robot for complex environment as described in any one of claims 1-7. The system includes: a ground cylinder positioning and detection algorithm module (1), a fermentation material surface segmentation algorithm module (2), and a three-dimensional reconstruction algorithm module of material surface and fermentation environment (3). The cylinder positioning and detection algorithm module (1) is used to add the number of cylinder edges of the cylinder to the initially obtained cylinder area range as an iteration condition to obtain the real cylinder area, and to split the obtained cylinder area mask into several single cylinder area masks for cylinder mouth recognition. The fermented mash surface segmentation algorithm module (2) is used to segment the fermented mash surface based on the identification of the vat opening using the fermented mash surface segmentation algorithm; The three-dimensional reconstruction algorithm module (3) of the material surface and the fermentation environment is used to reconstruct the three-dimensional surface of the fermented mash based on the segmented fermented mash material surface, and to convert the fermentation model into the material surface point cloud through coordinate transformation for preliminary registration; and to align the fermentation model with the original point cloud of the fermentation.

9. The visual perception system for a fermentation robot in complex environments according to claim 8, characterized in that, The cylinder positioning detection algorithm module (1) includes a cylinder initial positioning algorithm module (101) and a cylinder mouth accurate identification algorithm module (102). The initial positioning algorithm module (101) of the ground cylinder adopts a combination of depth data analysis algorithm and improved Hough circle detection to realize the initial positioning of the ground cylinder; The cylinder port precision recognition algorithm module (102) achieves precise cylinder port recognition by improving the Canny edge detection algorithm.

Images