Visual identification correction method and device for steaming pot and electronic equipment

Abstract

The invention provides a retort pot visual identification deviation correction method and device and electronic equipment, and belongs to the technical field of robotics.The method comprises the steps that a depth camera is used for obtaining a depth image and an RGB image of a retort pot, the depth image and the RGB image are registered, and a registered grey-scale image is generated; cutting the grey-scale map to obtain a cut grey-scale map, and processing the cut grey-scale map to obtain a processed grey-scale map; based on the processed grey-scale map, calculating the deviation of a pixel coordinate system of the depth camera and the height of the material in the retort pot; and calculating the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculating the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera. The correction accuracy of the thermal imager coordinate system and the robot coordinate system is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a method, device and electronic equipment for visual recognition and correction of a steamer pot. Background Art

[0002] In the process of automated steaming of liquor, robots play a vital role, as they need to spread the mash evenly in the steamer. However, the steamer needs to be rotated or flipped after being covered with mash. This process will cause a certain deviation when the steamer returns to its original position, which will cause deviations in the coordinate systems of the depth camera, thermal imager and robot, resulting in the robot being unable to spread the material accurately, increasing the risk of safety accidents. In the process of automated steaming of liquor, how to correct the different coordinate systems is a key issue that needs to be solved urgently. In the process of correcting the coordinate systems of the thermal imager and the robot, the existing technology only considers a single factor, and the accuracy of the correction is low. Summary of the invention

[0003] The present invention provides a method, device and electronic equipment for visual recognition and correction of a steamer pot, which are used to solve the defect that in the prior art, only a single factor is considered during the correction of the coordinate system of a thermal imager and a robot, resulting in low correction accuracy.

[0004] The present invention provides a method for visually identifying and correcting a steamer pot, comprising: A method for visual recognition and correction of a steamer pot is applied to an automatic steaming system for liquor, wherein the automatic steaming system for liquor comprises a steamer pot, a depth camera, a thermal imager and a robot. The method comprises: Acquire a depth map and an RGB map of the steamer pot using the depth camera, align the depth map and the RGB map, and generate an aligned grayscale map; Cutting the grayscale image to obtain a cut grayscale image, and processing the cut grayscale image to obtain a processed grayscale image; Based on the processed grayscale image, calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and the deviation of the coordinate system of the robot is calculated according to the deviation of the pixel coordinate system of the depth camera.

[0005] In some embodiments, the calculating, based on the processed grayscale image, the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot comprises: Based on the processed grayscale image, determining the central pixel coordinates and range of the steamer pot; According to the range of the steamer pot, the inclination angle of the steamer pot compared to the ground is calculated, and according to the central pixel coordinates of the steamer pot and the original calibrated central pixel coordinates, the deviation of the pixel coordinate system of the depth camera is calculated; Based on the processed grayscale image, the material surface range of the steamer is determined, the average material surface height is calculated according to the material surface range and the inclination angle, a new material surface range is calculated according to the average material surface height, and a new material surface average height is calculated according to the new material surface range, and the iteration is performed successively until the difference between the average material surface height calculated for the nth time and the average material surface height calculated for the n-1th time is less than or equal to a preset threshold value, and the iteration is terminated, and the average material surface height calculated for the nth time is determined to be the material height of the steamer, where n is a natural number greater than 1.

[0006] In some embodiments, determining the material surface range of the steamer pot based on the processed grayscale image, and calculating the average material surface height according to the material surface range and the inclination angle, comprises: Based on the processed grayscale image, determining the material surface range of the steamer pot, selecting a plurality of target points within the material surface range, and determining the coordinates of the plurality of target points and the distances between the plurality of target points and the depth camera; The average height of the material surface is calculated according to the coordinates of the multiple target points and the inclination angle, and the distances between the multiple target points and the depth camera.

[0007] In some embodiments, calculating the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material includes: Determine the first upper and lower bottom pixel lengths of the steamer pot according to the RGB image; Acquire a thermal image of the steamer pot using the thermal imager, and determine the second upper and lower bottom pixel lengths of the steamer pot according to the thermal imager image; Calculating the pixel ratio of the RGB image to the thermal imager image according to the height of the steamer pot, the height of the material, the first upper and lower bottom pixel lengths, and the second upper and lower bottom pixel lengths; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the pixel ratio.

[0008] In some embodiments, calculating the deviation of the robot's coordinate system according to the deviation of the pixel coordinate system of the depth camera includes: Calculating a deviation of a camera coordinate system of the depth camera according to a deviation of a pixel coordinate system of the depth camera; Determine the angle between the coordinate system of the robot and the camera coordinate system of the depth camera; The deviation of the coordinate system of the robot is determined according to the deviation of the camera coordinate system of the depth camera and the angle.

[0009] In some embodiments, registering the depth map and the RGB map includes: The depth value information of each pixel in the depth map is aligned with the pixel coordinates of each pixel in the RGB map to generate an aligned grayscale map, wherein the pixel value of the grayscale map is the depth information.

[0010] In some embodiments, the step of cutting the grayscale image to obtain a cut grayscale image includes: Determine the central pixel coordinates and range of the steamer pot according to the RGB image, and perform a primary cut on the grayscale image according to the central pixel coordinates and range of the steamer pot to obtain a grayscale image after the primary cut; According to the depth map, the distance from the edge of the steamer pot to the depth camera is determined, and according to the distance from the edge of the steamer pot to the depth camera, the grayscale image after the initial cutting is cut again to obtain the cut grayscale image.

[0011] In some embodiments, the processing of the cut grayscale image to obtain a processed grayscale image includes: Performing binarization and dilation corrosion preprocessing on the cut grayscale image to obtain a preprocessed grayscale image; Edge extraction and contour detection are performed on the preprocessed grayscale image to obtain the processed grayscale image.

[0012] The present invention also provides a visual identification and correction device for a steamer pot, comprising: An acquisition unit, used for acquiring a depth map and an RGB map of the steamer pot using a depth camera, registering the depth map and the RGB map, and generating a registered grayscale map; A processing unit, used for cutting the grayscale image to obtain a cut grayscale image, and processing the cut grayscale image to obtain a processed grayscale image; A first calculation unit, configured to calculate a deviation of a pixel coordinate system of the depth camera and a height of the material in the steamer based on the processed grayscale image; The second calculation unit is used to calculate the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculate the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the visual recognition and correction method for a steamer pot as described above is implemented.

[0014] The steamer pot visual recognition correction method, device and electronic device provided by the present invention use a depth camera to obtain a depth map and an RGB map of the steamer pot, align the depth map and the RGB map to generate a registered grayscale map, cut the grayscale map to obtain a cut grayscale map, process the cut grayscale map to obtain a processed grayscale map, calculate the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot based on the processed grayscale map, calculate the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, calculate the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera, thereby improving the accuracy of the correction of the thermal imager coordinate system and the robot coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 1 is a flow chart of a method for visually identifying and correcting a steamer pot provided by an embodiment of the present invention; Figure 2 is a material surface depth diagram of a steamer pot provided in an embodiment of the present invention; Figure 3 1 is a schematic structural diagram of a visual recognition and deviation correction device for a steamer pot provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0019] In an embodiment of the present invention, a system for automatically steaming liquor is provided, which includes a steamer pot, a depth camera, a thermal imager and a robot. The depth camera and the thermal imager are used to capture images of the steamer pot. The robot is used to evenly spread the mash in the steamer pot according to the captured images. After the steamer pot is covered with the mash, it is rotated or flipped and then returned to its original position. The depth camera includes a depth camera and an RGB camera.

[0020] Among them, the depth camera is a camera that can obtain the physical distance between objects in the scene and the camera. The distance between each point in the image and the depth camera is obtained by the depth camera, and the three-dimensional coordinates of each point in the image are obtained by adding the two-dimensional coordinates of the point in the 2D image.

[0021] Among them, a thermal imager is a device that can convert the invisible infrared energy emitted by an object into a visible thermal image. By viewing the thermal image, the overall temperature distribution of the target can be observed, the heating conditions of the target can be studied, and the next step of work can be judged.

[0022] Optionally, the steamer pot is barrel-shaped or truncated cone-shaped.

[0023] Figure 1 The following is a flow chart of a method for visually identifying and correcting a steamer pot according to an embodiment of the present invention. Figure 1 As shown, a method for visual identification and correction of a steamer pot is provided, which is applied to the above-mentioned automatic steaming system for liquor, and the method comprises the following steps: step 110, step 120, step 130 and step 140. The method flow steps are only used as a possible implementation mode of the present invention.

[0024] Step 110: Use a depth camera to obtain a depth map and an RGB map of the steamer pot, align the depth map and the RGB map, and generate an aligned grayscale map.

[0025] Optionally, in a scenario where the robot spreads the fermented grains into a steamer pot, a depth camera is used to obtain a depth map and an RGB map of the steamer pot, and a thermal imager is used to obtain a thermal imager image of the steamer pot.

[0026] Optionally, a depth map of the steamer pot is acquired using a depth camera of the depth camera, and an RGB map of the steamer pot is acquired using an RGB camera of the depth camera.

[0027] Among them, the depth map includes at least the depth information of each pixel point on the upper edge of the steamer pot and the upper surface of the material in the steamer pot, that is, the distance between each pixel point and the depth camera; the RGB map refers to the color image of the steamer pot, including at least the pixel coordinate information of each pixel point on the upper edge of the steamer pot and the upper surface of the material in the steamer pot.

[0028] In some embodiments, registering the depth map and the RGB map includes: The depth value information of each pixel in the depth map is aligned with the pixel coordinates of each pixel in the RGB map to generate a aligned grayscale image, and the pixel value of the grayscale image is the depth information.

[0029] Optionally, for each pixel in the depth map, a three-dimensional vector p_ir = (x, y, z) is constructed according to its depth value (z) and pixel coordinates (x, y) to obtain the spatial coordinates of each pixel in the depth camera coordinate system.

[0030] Optionally, the spatial coordinates of each pixel point in the depth camera coordinate system are transformed to obtain the pixel coordinates of the point on the RGB image, the color value at the pixel coordinates is extracted from the RGB image, and assigned to the corresponding pixel point in the depth image.

[0031] Step 120: cut the grayscale image to obtain a cut grayscale image, and process the cut grayscale image to obtain a processed grayscale image.

[0032] It can be understood that by cutting the grayscale image to obtain a cut grayscale image, and processing the cut grayscale image to obtain a processed grayscale image, noise interference can be removed and the region of interest can be extracted.

[0033] In some embodiments, cutting the grayscale image to obtain the cut grayscale image includes: According to the RGB image, the central pixel coordinates and range of the steamer pot are determined, and according to the central pixel coordinates and range of the steamer pot, the grayscale image is initially cut to obtain the grayscale image after the initial cutting; According to the depth map, the distance from the edge of the steamer pot to the depth camera is determined. According to the distance from the edge of the steamer pot to the depth camera, the grayscale image after the initial cutting is cut again to obtain the cut grayscale image.

[0034] Optionally, the grayscale image is cut horizontally and vertically according to the central pixel coordinates and range of the steamer pot to eliminate interference from objects other than the steamer pot, thereby obtaining a grayscale image after initial cutting.

[0035] Optionally, according to the distance from the rim surface of the steamer pot to the depth camera, the grayscale image after the initial cutting is cut again to retain the rim data of the steamer pot.

[0036] In some embodiments, processing the cut grayscale image to obtain a processed grayscale image includes: The cut grayscale image is subjected to binarization and expansion corrosion preprocessing to obtain a preprocessed grayscale image; Perform edge extraction and contour detection on the preprocessed grayscale image to obtain a processed grayscale image.

[0037] Optionally, based on a preset grayscale threshold, the cut grayscale image is binarized, and the pixels in the cut grayscale image whose grayscale values ​​are greater than or equal to the grayscale threshold are set to 1 (or white), and the pixels whose grayscale values ​​are less than the grayscale threshold are set to 0 (or black), so as to obtain a binarized grayscale image, which can simplify the image and improve the efficiency of subsequent data processing.

[0038] Optionally, the binarized grayscale image is subjected to dilation and corrosion processing to fill the holes in the image and enhance the boundaries of the steamer pot in the image to make it easier to identify and detect, eliminate small objects and noise in the image, and make the steamer pot area smoother; corrosion is performed after dilation to remove additional boundaries that may be introduced by the dilation operation and further refine the outline of the steamer pot.

[0039] Optionally, edge extraction and contour detection are performed on the preprocessed grayscale image to extract the complete contour of the steamer pot, and shape matching is performed to calculate the bounding box of the contour to determine the coordinates and range of the center point of the steamer pot.

[0040] Step 130: Calculate the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer based on the processed grayscale image.

[0041] It should be noted that as the height of the material increases, the pixel ratio of the RGB image and the thermal imager image changes linearly. The pixel ratio of the RGB image and the thermal imager image can be calculated according to the height of the material, and then the deviation of the thermal imager coordinate system can be calculated according to the deviation of the pixel coordinate system of the depth camera and the pixel ratio; according to the deviation of the pixel coordinate system of the depth camera, coordinate transformation is performed to calculate the deviation of the robot's coordinate system.

[0042] In some embodiments, in step 130, based on the processed grayscale image, calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot includes the following steps: Step 131, based on the processed grayscale image, determine the central pixel coordinates and range of the steamer pot; Step 132: Calculate the inclination angle of the steamer pot relative to the ground according to the range of the steamer pot, and calculate the deviation of the pixel coordinate system of the depth camera according to the central pixel coordinate of the steamer pot and the original calibrated central pixel coordinate; Step 133, based on the processed grayscale image, determine the material surface range of the steamer, calculate the average material surface height according to the material surface range and the inclination angle, calculate a new material surface range according to the average material surface height, calculate a new average material surface height according to the new material surface range, and iterate in sequence until the difference between the average material surface height calculated for the nth time and the average material surface height calculated for the n-1th time is less than or equal to a preset threshold, the iteration terminates, and the average material surface height calculated for the nth time is determined to be the material height of the steamer, where n is a natural number greater than 1.

[0043] Optionally, the deviation values ​​of the pixel coordinate system of the depth camera in the X and Y axis directions are calculated according to the central pixel coordinates of the steamer pot and the original calibrated central pixel coordinates. and .

[0044] Optionally, the range of the steamer pot is an ellipse, and the inclination angle of the steamer pot compared to the ground is calculated based on the major axis and minor axis of the ellipse.

[0045] Optionally, when the steamer pot is completely parallel to the ground, the identified upper edge of the steamer pot is a circle with a radius of R. When the inclination angle between the steamer pot and the ground is When the upper edge of the steamer is an ellipse, the calculation formula of the inclination angle is: ; in, is the major axis of the ellipse, with a length of R, is the minor axis of the ellipse.

[0046] Optionally, the preset threshold is determined based on user input or empirical data, and the preset threshold may be 0.5 mm, 1 mm, 1.5 mm, etc.

[0047] Optionally, the range of the material surface is calculated based on the range of the upper bottom and the lower bottom of the steamer pot, the height of the steamer pot, and the average height of the material surface.

[0048] It is understandable that by calculating the average height of the material surface through multiple iterations, the calculation error of the average height of the material surface can be reduced, thereby improving the accuracy of the correction of the thermal imager coordinate system.

[0049] In some embodiments, based on the processed grayscale image, determining the material surface range of the steamer pot, and calculating the average material surface height according to the material surface range and the inclination angle, comprises: Based on the processed grayscale image, determine the material surface range of the steamer pot, select multiple target points within the material surface range, determine the coordinates of the multiple target points and the distances between the multiple target points and the depth camera; The average height of the material surface is calculated based on the coordinates and inclination angles of multiple target points and the distances between the multiple target points and the depth camera.

[0050] It should be noted that since the steamer has a certain inclination angle compared to the ground, the shape of the material surface (composed of different depth values) obtained by the depth camera is an inclined surface compared to the ground. After being flattened according to the height, it will be parallel to the ground, but in the steamer it will form a phenomenon of one side being high and the other side being low.

[0051] Figure 2 The material surface depth diagram of the steamer pot provided by the embodiment of the present invention. Figure 2 As shown, the average height of the material surface is , The corresponding plane is , , These are the concave points below the average height. If you want to spread it along the plane of the steamer, you need to find the concave points below the plane of the steamer. The concave point has a three-dimensional coordinate of (x, y, z). When the height of the concave point is greater than the average height, such as , ,exist Point, depth camera to The distance is , greater than the depth camera to Distance , the concave point does not need to be filled. Point, depth camera to The distance is , which is smaller than the depth camera Distance , the concave point needs to be filled; when the height of the concave point is less than the average height, such as , ,exist Point, depth camera to The distance is , greater than the depth camera to Distance , the concave point does not need to be filled. Point, depth camera to The distance is , which is smaller than the depth camera Distance , the concave point needs to be filled.

[0052] Optionally, The calculation formula is as follows: ; Among them, when for When (x, y) is The pixel coordinates of the point (x1, y1); when for When (x, y) is The pixel coordinates of the point (x2, y2); when for When (x, y) is The pixel coordinates of the point (x3, y3); when for When (x, y) is The pixel coordinates of the point (x4, y4).

[0053] Optionally, according to Calculate the average height of the material surface as .

[0054] Step 140: Calculate the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculate the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera.

[0055] It should be noted that the resolution and coordinate system directions of the depth camera and the thermal imager are different (the X-axis of the thermal imager is the Y-axis of the depth camera, and the Y-axis of the thermal imager is the X-axis of the depth camera). To obtain the deviation of the thermal imager coordinate system, the deviation of the pixel coordinate system of the depth camera needs to be converted.

[0056] In some embodiments, calculating the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material includes: According to the RGB image, determine the first upper and lower bottom pixel lengths of the steamer pot; Using a thermal imager to obtain a thermal image of the steamer pot, and determining the second upper and lower bottom pixel lengths of the steamer pot according to the thermal image; Calculate the pixel ratio of the RGB image to the thermal imager image according to the height of the steamer pot, the height of the material, the first upper and lower bottom pixel lengths, and the second upper and lower bottom pixel lengths; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the pixel ratio.

[0057] It should be noted that the thermal image and the RGB image are taken based on the same scene.

[0058] Optionally, the calculation formula of the deviation of the coordinate system of the thermal imager is as follows: ; ; in, is the deviation of the thermal imager's coordinate system in the X-axis direction, is the deviation of the thermal imager's coordinate system in the Y-axis direction, is the deviation of the pixel coordinate system of the depth camera in the X-axis direction, is the deviation of the pixel coordinate system of the depth camera in the Y-axis direction, is the pixel ratio of the RGB image to the thermal image, is the height of the material, is the height of the steamer pot, is the pixel length of the upper bottom of the steamer in the RGB image, is the pixel length of the bottom of the steamer in the RGB image, is the pixel length of the upper bottom of the steamer pot in the thermal image, is the pixel length of the bottom of the steamer pot in the thermal image.

[0059] In some embodiments, calculating the deviation of the robot's coordinate system according to the deviation of the pixel coordinate system of the depth camera includes: Calculate the deviation of the camera coordinate system of the depth camera according to the deviation of the pixel coordinate system of the depth camera; Determine the angle between the robot's coordinate system and the camera coordinate system of the depth camera; The deviation of the robot's coordinate system is determined based on the deviation and angle of the camera coordinate system of the depth camera.

[0060] Optionally, let the pixel coordinates of the identified steamer center be , the pixel coordinates of the calibrated center of the steamer are , the coordinates after converting them into the camera coordinate system are and , the height corresponding to the pixel coordinate is H, then and The calculation formula is: ; in, is the pixel scale factor of the depth camera in the horizontal direction, It is the pixel scale factor in the vertical direction of the depth camera, which is used to convert the coordinates on the normalized plane into pixel coordinates, indicating the actual size of each pixel of the depth camera in the horizontal and vertical directions; and It is the optical center coordinate of the depth camera, which represents the coordinate of the center pixel of the depth camera image plane and is used to correspond the pixel coordinate system to the camera coordinate system.

[0061] Optionally, the calculation formula of the deviation of the robot's coordinate system is: ; in, Indicates the deviation of the robot's coordinate system in the X-axis direction. Indicates the deviation of the robot's coordinate system in the Y-axis direction. Represents the angle between the robot's coordinate system and the depth camera's camera coordinate system.

[0062] In an embodiment of the present invention, a depth camera is used to obtain a depth map and an RGB map of the steamer pot, the depth map and the RGB map are aligned to generate an aligned grayscale map, the grayscale map is cut to obtain a cut grayscale map, the cut grayscale map is processed to obtain a processed grayscale map, based on the processed grayscale map, the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot are calculated, the deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and the deviation of the coordinate system of the robot is calculated according to the deviation of the pixel coordinate system of the depth camera, thereby improving the accuracy of the deviation correction of the thermal imager coordinate system and the robot coordinate system.

[0063] The following is a description of a visual recognition and correction device for a steamer pot provided in an embodiment of the present invention. The visual recognition and correction device for a steamer pot described below and the visual recognition and correction method for a steamer pot described above can be referred to in correspondence with each other.

[0064] Figure 3 A schematic diagram of the structure of a visual recognition and correction device for a steamer pot provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the steamer pot visual recognition and correction device 300 includes: An acquisition unit 310 is used to acquire a depth map and an RGB map of the steamer pot using a depth camera, align the depth map and the RGB map, and generate an aligned grayscale map; The processing unit 320 is used to cut the grayscale image to obtain a cut grayscale image, and process the cut grayscale image to obtain a processed grayscale image; A first calculation unit 330 is used to calculate the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot based on the processed grayscale image; The second calculation unit 340 is used to calculate the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculate the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera.

[0065] Optionally, based on the processed grayscale image, calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot includes: Based on the processed grayscale image, determine the central pixel coordinates and range of the steamer pot; According to the range of the steamer, the inclination angle of the steamer relative to the ground is calculated, and according to the central pixel coordinates of the steamer and the original calibrated central pixel coordinates, the deviation of the pixel coordinate system of the depth camera is calculated; Based on the processed grayscale image, the material surface range of the steamer is determined, the average material surface height is calculated according to the material surface range and the inclination angle, a new material surface range is calculated according to the average material surface height, and a new average material surface height is calculated according to the new material surface range, and the iteration is performed successively until the difference between the average material surface height calculated for the nth time and the average material surface height calculated for the n-1th time is less than or equal to a preset threshold value, and the iteration is terminated, and the average material surface height calculated for the nth time is determined to be the material height of the steamer, where n is a natural number greater than 1.

[0066] Optionally, based on the processed grayscale image, determining the material surface range of the steamer pot, and calculating the average material surface height according to the material surface range and the inclination angle, including: Based on the processed grayscale image, determine the material surface range of the steamer pot, select multiple target points within the material surface range, determine the coordinates of the multiple target points and the distances between the multiple target points and the depth camera; The average height of the material surface is calculated based on the coordinates and inclination angles of multiple target points and the distances between the multiple target points and the depth camera.

[0067] Optionally, the deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the height of the material, including: According to the RGB image, determine the first upper and lower bottom pixel lengths of the steamer pot; Using a thermal imager to obtain a thermal image of the steamer pot, and determining the second upper and lower bottom pixel lengths of the steamer pot according to the thermal image; Calculate the pixel ratio of the RGB image to the thermal imager image according to the height of the steamer pot, the height of the material, the first upper and lower bottom pixel lengths, and the second upper and lower bottom pixel lengths; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the pixel ratio.

[0068] Optionally, calculating the deviation of the robot's coordinate system according to the deviation of the pixel coordinate system of the depth camera includes: Calculate the deviation of the camera coordinate system of the depth camera according to the deviation of the pixel coordinate system of the depth camera; Determine the angle between the robot's coordinate system and the camera coordinate system of the depth camera; The deviation of the robot's coordinate system is determined based on the deviation and angle of the camera coordinate system of the depth camera.

[0069] Optionally, the depth map and the RGB map are registered, including: The depth value information of each pixel in the depth map is aligned with the pixel coordinates of each pixel in the RGB map to generate a aligned grayscale image, and the pixel value of the grayscale image is the depth information.

[0070] Optionally, the grayscale image is cut to obtain a cut grayscale image, including: According to the RGB image, the central pixel coordinates and range of the steamer pot are determined, and according to the central pixel coordinates and range of the steamer pot, the grayscale image is initially cut to obtain the grayscale image after the initial cutting; According to the depth map, the distance from the edge of the steamer pot to the depth camera is determined. According to the distance from the edge of the steamer pot to the depth camera, the grayscale image after the initial cutting is cut again to obtain the cut grayscale image.

[0071] Optionally, the cut grayscale image is processed to obtain a processed grayscale image, including: The cut grayscale image is subjected to binarization and expansion corrosion preprocessing to obtain a preprocessed grayscale image; Perform edge extraction and contour detection on the preprocessed grayscale image to obtain a processed grayscale image.

[0072] It should be noted here that the steamer pot visual recognition and correction device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned steamer pot visual recognition and correction method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0073] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the method for visual identification and correction of the steamer pot, the method comprising: using a depth camera to obtain a depth map and an RGB map of the steamer pot, aligning the depth map and the RGB map, and generating an aligned grayscale map; cutting the grayscale map to obtain a cut grayscale map, processing the cut grayscale map to obtain a processed grayscale map; based on the processed grayscale map, calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer pot; calculating the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculating the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera.

[0074] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visual identification and correction of a steamer pot, characterized in that: Applied to an automatic steaming system for liquor, the automatic steaming system for liquor comprises a steamer pot, a depth camera, a thermal imager and a robot, and the method comprises: Acquire a depth map and an RGB map of the steamer pot using the depth camera, align the depth map and the RGB map, and generate an aligned grayscale map; Cutting the grayscale image to obtain a cut grayscale image, and processing the cut grayscale image to obtain a processed grayscale image; Based on the processed grayscale image, calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and the deviation of the coordinate system of the robot is calculated according to the deviation of the pixel coordinate system of the depth camera.

2. The method for visually identifying and correcting a steamer pot according to claim 1, characterized in that: The step of calculating the deviation of the pixel coordinate system of the depth camera and the height of the material in the steamer based on the processed grayscale image comprises: Based on the processed grayscale image, determining the central pixel coordinates and range of the steamer pot; According to the range of the steamer pot, the inclination angle of the steamer pot compared to the ground is calculated, and according to the central pixel coordinates of the steamer pot and the original calibrated central pixel coordinates, the deviation of the pixel coordinate system of the depth camera is calculated; Based on the processed grayscale image, the material surface range of the steamer is determined, the average material surface height is calculated according to the material surface range and the inclination angle, a new material surface range is calculated according to the average material surface height, and a new material surface average height is calculated according to the new material surface range, and the iteration is performed successively until the difference between the average material surface height calculated for the nth time and the average material surface height calculated for the n-1th time is less than or equal to a preset threshold value, and the iteration is terminated, and the average material surface height calculated for the nth time is determined to be the material height of the steamer, where n is a natural number greater than 1.

3. The method for visually identifying and correcting a steamer pot according to claim 2, characterized in that: The step of determining the material surface range of the steamer pot based on the processed grayscale image, and calculating the average material surface height according to the material surface range and the inclination angle comprises: Based on the processed grayscale image, determining the material surface range of the steamer pot, selecting a plurality of target points within the material surface range, and determining the coordinates of the plurality of target points and the distances between the plurality of target points and the depth camera; The average height of the material surface is calculated according to the coordinates of the multiple target points and the inclination angle, and the distances between the multiple target points and the depth camera.

4. The method for correcting deviation of a steamer pot by visual recognition according to claim 1, characterized in that: The calculating the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material includes: Determine the first upper and lower bottom pixel lengths of the steamer pot according to the RGB image; Acquire a thermal image of the steamer pot using the thermal imager, and determine the second upper and lower bottom pixel lengths of the steamer pot according to the thermal imager image; Calculating the pixel ratio of the RGB image to the thermal imager image according to the height of the steamer pot, the height of the material, the first upper and lower bottom pixel lengths, and the second upper and lower bottom pixel lengths; The deviation of the coordinate system of the thermal imager is calculated according to the deviation of the pixel coordinate system of the depth camera and the pixel ratio.

5. The method for visual identification and correction of a steamer pot according to claim 1, characterized in that: The calculating the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera comprises: Calculating a deviation of a camera coordinate system of the depth camera according to a deviation of a pixel coordinate system of the depth camera; Determine the angle between the coordinate system of the robot and the camera coordinate system of the depth camera; The deviation of the coordinate system of the robot is determined according to the deviation of the camera coordinate system of the depth camera and the angle.

6. The method for visually identifying and correcting a steamer pot according to claim 1, characterized in that: The registering the depth map and the RGB map comprises: The depth value information of each pixel in the depth map is aligned with the pixel coordinates of each pixel in the RGB map to generate an aligned grayscale map, wherein the pixel value of the grayscale map is the depth information.

7. The method for visually identifying and correcting a steamer pot according to claim 1, characterized in that: The step of cutting the grayscale image to obtain a cut grayscale image includes: Determine the central pixel coordinates and range of the steamer pot according to the RGB image, and perform a primary cut on the grayscale image according to the central pixel coordinates and range of the steamer pot to obtain a grayscale image after the primary cut; According to the depth map, the distance from the edge of the steamer pot to the depth camera is determined, and according to the distance from the edge of the steamer pot to the depth camera, the grayscale image after the initial cutting is cut again to obtain the cut grayscale image.

8. The method for visually identifying and correcting a steamer pot according to any one of claims 2 to 7, characterized in that: The step of processing the cut grayscale image to obtain a processed grayscale image includes: Performing binarization and dilation corrosion preprocessing on the cut grayscale image to obtain a preprocessed grayscale image; Edge extraction and contour detection are performed on the preprocessed grayscale image to obtain the processed grayscale image.

9. A visual identification and correction device for a steamer pot, characterized in that: include: An acquisition unit, used for acquiring a depth map and an RGB map of the steamer pot using a depth camera, registering the depth map and the RGB map, and generating a registered grayscale map; A processing unit, used for cutting the grayscale image to obtain a cut grayscale image, and processing the cut grayscale image to obtain a processed grayscale image; A first calculation unit, configured to calculate a deviation of a pixel coordinate system of the depth camera and a height of the material in the steamer based on the processed grayscale image; The second calculation unit is used to calculate the deviation of the coordinate system of the thermal imager according to the deviation of the pixel coordinate system of the depth camera and the height of the material, and calculate the deviation of the coordinate system of the robot according to the deviation of the pixel coordinate system of the depth camera.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steamer visual recognition and correction method as described in any one of claims 1 to 8 is implemented.

Images