Estimation of smart dishwashing poses and grasping strategy system

Abstract

The application discloses a kind of estimation and grasping strategy systems for intelligent dishwashing pose, the method includes: the initialization setting of depth camera is completed, establishes camera coordinate system, establishes image coordinate system and completes initial image acquisition of depth camera;After completing the initial image acquisition of depth camera, the rapid identification and two-dimensional coordinate extraction of all dishware in image are completed by convolution network object recognition algorithm;The data information obtained is accepted by pose recognition algorithm, and the final 6D pose is selected with the help of CAD model and Transform pose iteration etc.;Each 6D pose is placed in task pool, and the task pool is reordered by the help of pose recognition algorithm and redistributed according to the spatial position of robot;After task pool generation is completed, data file is generated according to preset format, sent to mechanical arm, and the issuing of instruction is realized.The method fuses convolution network object recognition algorithm and pose recognition algorithm, and realizes the integrated system process of identification and pose estimation.

Description

Technical Field

[0001] This invention belongs to the field of intelligent dishwashing technology, and more specifically, relates to a system for estimating and grasping the pose of intelligent dishwashing devices. Background Technology

[0002] In the development of the modern catering service industry, the continuous rise in labor costs has become an important driving factor for industry transformation and upgrading. The application of automated equipment has become a core development direction. As an important supporting process in catering operations, the automation transformation of the dishwashing process has received widespread attention. The traditional manual dishwashing mode has many problems such as low efficiency, high labor intensity, and inconsistent cleaning standards. The intervention of robotics technology provides an effective way to solve these pain points. Among them, robot vision technology, with its advantages of high precision and strong versatility, has been maturely applied in dynamic operation scenarios such as industrial sorting, logistics handling, and defect detection, becoming the core sensing unit of automated equipment. However, when applied to the tableware processing in the dishwashing process, the bowls are often piled up in a messy state when they are to be washed, and their postures are highly complex, which faces many unique technical challenges.

[0003] Existing technologies have attempted to solve the problem of tableware identification and processing in the dishwashing process. For example, photoelectric sensors are used to identify the orientation of bowls and basins to perform different subsequent processing, which is beneficial for stacking different types of bowls. Some existing technologies also identify the type of bowl by recognizing the graphic code on the back of the bowl, realizing intelligent identification of the bowl. At the same time, the application of machine vision technology in the field of tableware processing is gradually advancing. Traditional machine vision technology mainly relies on two-dimensional images to carry out recognition work, attempting to provide basic information for robotic arms to grasp bowls. Various recognition algorithms and sensor technologies are also constantly trying to integrate into the automated process of intelligent dishwashing.

[0004] The existing technologies applied to the field of intelligent dishwashing have the following obvious drawbacks: (1) The relevant technologies can only identify the front and back of the bowls or distinguish the types of bowls. They cannot obtain the complete pose information of the bowls in three-dimensional space, and it is difficult to deal with the complex situation of multiple overlapping, tilted placement, and partial occlusion of bowls in actual applications; (2) Traditional machine vision technology can only obtain the coordinate information of objects on a plane. It cannot perceive the depth and angle in three-dimensional space, cannot accurately judge the actual stacking level and tilt state of the bowls, and lacks an effective estimation method for complex poses; (3) The existing technologies cannot plan a reasonable grasping strategy for the robotic arm according to the actual state of the bowls. At the same time, it is difficult to scientifically allocate and schedule the grasping tasks of multiple robotic arms. This can easily lead to low accuracy and efficiency of robotic arm grasping operations, and even collision risks, which seriously restricts the promotion and application of automation technology in the dishwashing process of catering services. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a system for estimating and grasping the pose of a smart dishwashing appliance. Through convolutional network object recognition, optimized FP algorithm pose recognition, and collaborative robotic arm task pool generation, and with the assistance of the ROS system for communication, it can accurately identify various poses of the bowl and provide them to the robotic arm for subsequent grasping.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for estimating and grasping the pose of an intelligent dishwasher is provided, comprising: S1: Before the depth camera performs its first operation, the initial setup of the depth camera must be completed; define the x-axis along the horizontal right direction of the image, the y-axis along the vertical downward direction of the image, and the z-axis perpendicular to the image plane and pointing into the scene to establish the camera coordinate system; with the upper left corner of the acquired image as the origin, define the x-axis along the horizontal right direction of the image and the y-axis along the vertical downward direction of the image to establish the image coordinate system; after the depth camera starts, acquire the complete image of the bowl to be processed within the current field of view to complete the initial image acquisition; S2: After the initial image acquisition by the depth camera is completed, the color image is communicated to the convolutional network object recognition algorithm via ROS to complete the rapid recognition and two-dimensional coordinate extraction of all bowls in the image; the backbone network performs multi-scale downsampling on the input image to extract features at different levels; the neck network performs top-down and bottom-up fusion of the multi-scale features; the features of the backbone network and the neck network are integrated to obtain the input features of the detection head, and the data information of the rotated rectangle is obtained for each spatial position of the input features; S3: The pose recognition algorithm receives the obtained data information and generates an initial pose hypothesis set through the CAD model; it obtains the region of interest of the object using the RGB-D image and the mask information of the object recognition by the convolutional network; it refines the pose iteratively using the Transformer to obtain the refined candidate poses; it filters out suitable candidate poses through the positive and negative pose estimation of the object recognition by the CAD rendering and the convolutional network and eliminates the negative solutions; it selects the effective pose with the highest matching score as the final 6D pose, realizing the integrated output of "recognition-pose estimation"; S4: The 6D poses of each object are generated and placed into the task pool. The 6D poses are reordered in the task pool using a pose recognition algorithm to avoid errors in the identification of the top and bottom order of the bowls. The task pool is arranged in descending order. Based on the deep and shallow allocation, the task pool is reassigned according to the robot's spatial position. S5: After the task pool is generated, a data file is generated according to the preset format and sent to the robotic arm through the ROS node to realize the issuance of instructions.

[0007] Furthermore, the multi-scale sampling includes: in, The hierarchy of output features in the backbone network; The number of feature layers (e.g., 3 layers, corresponding to downsampling times of 8 / 16 / 32). To preserve details such as object edges and textures; For global semantic information of the object; As the first in the backbone network The output features of the layer; I The input is an RGB image; I∈ ( H high, W (Wide, 3 channels) ∈ ( (Number of channels); Backbone network for target detection The feature extraction mapping function of the layer, where I Given the original RGB input image, this function performs multi-level downsampling and feature extraction on the input image, and outputs the first... Feature map of layer .

[0008] Furthermore, the top-down processing includes: in, The neck network is fused from top to bottom. The output features of the layer; Conv is a two-dimensional convolution operation function used to perform channel transformation, feature encoding and information fusion on the feature map, eliminate the aliasing effect after feature addition, and output an enhanced feature map with dimension matching; Upsample is an upsampling operation function used to enlarge the size of the deep low-resolution feature map so that it matches the spatial size of the shallow feature map, so as to realize the transmission of high-level semantic information from top to bottom to the shallow layer.

[0009] Furthermore, the bottom-up processing includes: in, The output features of the 1st layer of the Neck network are obtained by bottom-up fusion; Downsample is a downsampling operation function used to reduce the size of the shallow high-resolution feature map so that it matches the spatial size of the deep feature map, thereby realizing the transfer of shallow details from the bottom up to the deep layers.

[0010] Furthermore, the center coordinates of the rotating rectangle include: in, Normalize the network output offset to the grid. The coordinates; Normalize the network output offset to the grid. The coordinates; s is the current feature map step size (e.g., 8 / 16 / 32); The x-coordinate of the top-left corner of the feature map grid; The coordinate of the top-left corner of the feature map grid is in the y-direction.

[0011] Furthermore, the region of interest of the object includes: Where Ω is the ROI pixel region based on the mask; M is the target mask output by the convolutional network; u is the horizontal coordinate of the image pixel, corresponding to the width dimension of the image; and v is the vertical coordinate of the image pixel, corresponding to the height dimension of the image.

[0012] Furthermore, the 3D scene point cloud includes: in, 3D scene point cloud within the ROI region; For the pixel coordinates in the input RGB-D image The depth value at that location, in mm, represents the vertical distance from the scene point corresponding to that pixel to the camera's optical center; For the back projection transformation from pixel coordinates to normalized 3D coordinates in the camera coordinate system, where, K This is the camera intrinsic parameter matrix, describing the camera's intrinsic parameters such as focal length and principal point. It is the inverse of the camera intrinsic parameter matrix. pixel coordinates The homogeneous form of .

[0013] Furthermore, the convolutional network's positive and negative pose matching score includes: in, is the feature extraction function of the convolutional network; ⊙ represents element-wise multiplication, retaining only the features of the target region labeled by the mask; For pose matching degree, A higher score indicates a higher pose matching degree, and vice versa. Well below the threshold τ; The input is an RGB image.

[0014] According to a second aspect of the present invention, a system for estimating and grasping poses for intelligent dishwashing is provided, which is implemented using a method for estimating and grasping poses for intelligent dishwashing, comprising: The depth camera image acquisition module provides data for subsequent bowl recognition and pose estimation. It uses a RealSense depth camera as the vision unit to simultaneously acquire color and depth images. Before the depth camera operation, a camera coordinate system and an image coordinate system consistent with the corresponding recognition algorithm need to be established. To establish the camera coordinate system, the x-axis should be defined horizontally to the right along the image, the y-axis vertically downward along the image, and the z-axis perpendicular to the image plane and pointing into the scene. To establish the image coordinate system, the upper left corner of the acquired image should be used as the origin, and the x-axis should be defined horizontally to the right along the image, and the y-axis vertically downward along the image. The rotating rectangle acquisition module is used for the rapid identification and 2D coordinate extraction of all bowls in the image. It adopts a convolutional network object recognition algorithm and transmits color image data through ROS communication. In the convolutional network object recognition algorithm, the backbone network samples the input image at multiple scales and extracts features at different levels. The neck network performs top-down and bottom-up fusion processing on the multi-scale features extracted by the backbone network. The input features of the detection head obtained by the fusion processing are directly regressed to the rotating rectangle parameters to obtain the final rotating rectangle of the input color image. The 6D pose acquisition module aggregates RGB-D images from a depth camera, the CAD model of an existing object, the bounding boxes from a convolutional network object recognition system, and information on both front and back poses to form the final 6D pose of the bowl. The FP algorithm receives the data from the final rotated bounding box and generates a hypothetical initial pose of the object using the CAD model. It then uses the RGB-D image and mask information from the convolutional network object recognition system to obtain the region of interest (ROI). The pose is iteratively refined using a Transformer and suitable candidate poses are selected through CAD rendering and front and back pose estimation from the convolutional network object recognition system, while excluding negative solutions. Finally, the effective pose with the highest matching score is selected as the final 6D pose, achieving an integrated output of "recognition-pose estimation". The collaborative robotic arm task pool generation module is used to reorder the task pool based on the 6D pose data. The task pool is arranged in descending order. Based on the deep and shallow allocation, the nearest bowl is allocated according to the robot's spatial position, thereby improving the high efficiency of the collaborative robotic arm. The robotic arm task allocation and reallocation module is used to quickly read and process data from the robotic arm's control system. The data in the task pool is encapsulated in JSON format and sent to the robotic arm through ROS nodes to issue commands. According to a third aspect of the present invention, a computer-readable storage medium is also provided, having a computer program stored thereon, the computer program being executed by a processor of any one of the methods for estimating and grasping intelligent dishwashing poses.

[0015] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The method for estimating and grasping the pose of a smart dishwashing device proposed in this embodiment of the invention uses convolutional network object recognition, optimized FP algorithm pose recognition, and collaborative robotic arm task pool generation. With the help of ROS system communication, it can accurately identify various poses of the bowl and provide them to the robotic arm for subsequent grasping.

[0016] 2. The core technical feature of the method for estimating and grasping the pose of a smart dishwashing machine proposed in this embodiment of the invention lies in its ability to obtain a rotated rectangle with more accurate position coordinates of the target based on a convolutional network object recognition algorithm. This significantly reduces the proportion of irrelevant background and transmits the 2D position of the bowl to the pose estimation algorithm, which can greatly reduce the recognition time of the pose estimation algorithm and improve the recognition accuracy. This method integrates the convolutional network object recognition algorithm and the FP algorithm to realize an integrated system process for recognition and pose estimation. By using the coarse estimation of the 2D position and the forward and reverse recognition of the object through convolutional network object recognition, the recognition accuracy of the FP algorithm is improved, eliminating the fatal problem of reverse solution, and making the pose recognition more accurate.

[0017] 3. The core technical feature of the method for estimating and grasping the pose of intelligent dishwashing proposed in this embodiment of the invention lies in the efficient communication between the camera, object recognition, pose recognition, and robot task allocation by leveraging the ROS system and efficient communication protocols, thereby achieving orderly and safe operation of tasks. Furthermore, through the principle of depth-first and the principle of collaborative robotic arm allocation, the task allocation and grasping strategy of the collaborative robotic arm are optimized, thus improving the overall system efficiency. Attached Figure Description

[0018] Figure 1 This invention describes the control workflow of a robot recognition system used in an intelligent dishwashing pose estimation and grasping strategy system. Figure 2 This is a schematic diagram illustrating the working principle of the convolutional network object recognition algorithm used in the intelligent dishwashing pose estimation and grasping strategy system according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating the pose recognition algorithm of the intelligent dishwashing pose estimation and grasping strategy system according to an embodiment of the present invention. Figure 4 The diagram shows the experimental results of pose recognition and task pool of the intelligent dishwashing pose estimation and grasping strategy system according to an embodiment of the present invention. Figure 5 This is an example diagram of the JSON file format used in the intelligent dishwashing pose estimation and capture strategy system according to an embodiment of the present invention; Figure 6This is a schematic diagram of the system modules of the intelligent dishwashing pose estimation and grasping strategy system according to an embodiment of the present invention. Detailed Implementation

[0019] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0020] The method provided by this invention can be implemented in a terminal environment that may include one or more of the following components: a processor, a storage medium, and a display screen. The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0021] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in the storage medium, and by calling data stored in the storage medium.

[0022] Storage media can include random access memory (RAM) or read-only memory (ROM). Storage media can be used to store instructions, programs, code, code sets, or instructions.

[0023] The display screen is used to show the interactive sections of various applications.

[0024] In the formula of this invention, all subscripts are only used to distinguish parameters and have no actual meaning.

[0025] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.

[0026] Example 1 like Figure 1 As shown, this embodiment 1 provides a method for estimating and capturing the pose of an intelligent dishwasher, including: S1: Before the depth camera performs its first operation, the initial setup of the depth camera must be completed; define the x-axis along the horizontal right direction of the image, the y-axis along the vertical downward direction of the image, and the z-axis perpendicular to the image plane and pointing into the scene to establish the camera coordinate system; with the upper left corner of the acquired image as the origin, define the x-axis along the horizontal right direction of the image and the y-axis along the vertical downward direction of the image to establish the image coordinate system; after the depth camera starts, acquire the complete image of the bowl to be processed within the current field of view to complete the initial image acquisition; S2: After the initial image acquisition by the depth camera is completed, the color image is communicated to the convolutional network object recognition algorithm via ROS to complete the rapid recognition and two-dimensional coordinate extraction of all bowls in the image; the backbone network performs multi-scale downsampling on the input image to extract features at different levels; the neck network performs top-down and bottom-up fusion of the multi-scale features; the features of the backbone network and the neck network are integrated to obtain the input features of the detection head, and the data information of the rotated rectangle is obtained for each spatial position of the input features; S3: The pose recognition algorithm receives the obtained data information and generates an initial pose hypothesis set through the CAD model; it obtains the region of interest of the object by using the RGB-D image and the mask information of the object recognition of the convolutional network; it refines the pose iteratively with Transformer to obtain the refined candidate pose; it selects suitable candidate poses by using CAD rendering and positive and negative pose estimation of the object recognition of the convolutional network, and excludes negative solutions; it selects the effective pose with the highest matching score as the final 6D pose, realizing the integrated output of "recognition-pose estimation".

[0027] S4: The 6D poses of each object are generated and placed into the task pool. The 6D poses are reordered in the task pool using a pose recognition algorithm to avoid errors in the identification of the top and bottom order of the bowls. The task pool is arranged in descending order. Based on the deep and shallow allocation, the task pool is reassigned according to the robot's spatial position. S5: After the task pool is generated, a data file is generated according to the preset format and sent to the robotic arm through the ROS node to realize the issuance of instructions.

[0028] This embodiment 1 further elaborates on the above steps.

[0029] (1) Depth camera image acquisition In the modern catering service industry, with the continuous rise in labor costs, the application of automated equipment has become a core direction for industry transformation and upgrading. As an important supporting process in catering operations, the automation transformation of the dishwashing process has always attracted much attention. In this process, a high-precision RealSense depth camera is first used as the visual perception unit. This camera can simultaneously acquire color and depth images of the scene, retaining the ability of traditional color cameras to recognize the appearance features of objects, while also obtaining the positional information of objects in three-dimensional space through depth images, providing a data foundation for subsequent depth calculation and pose estimation. Secondly, it is necessary to complete the acquisition of complete images (including color and depth images) of the bowls to be processed within the current field of view using the RealSense depth camera. Before the first operation of the RealSense depth camera, the initialization settings of the RealSense depth camera must be completed.

[0030] Based on this, the core steps of the RealSense depth camera during the initial setup process are the establishment of the camera coordinate system and the image coordinate system and the initial image acquisition. During this process, the setting of the image coordinate system follows the principle of being consistent with the coordinate system of the object convolutional network recognition algorithm, and the setting of the camera coordinate system follows the principle of being consistent with the pose estimation recognition coordinate system. Through the unified design of the coordinate system and the corresponding algorithm, errors generated in the subsequent coordinate transformation process can be effectively avoided, which facilitates the subsequent joint calculation.

[0031] To establish a camera coordinate system, the x-axis should be defined as horizontally to the right along the image, the y-axis as vertically downward along the image, and the z-axis as perpendicular to the image plane and pointing into the scene.

[0032] To establish an image coordinate system, the top left corner of the acquired image should be the origin. The x-axis should be defined along the horizontal direction to the right of the image, and the y-axis should be defined along the vertical direction to the bottom of the image.

[0033] (2) Convolutional network object recognition and 2D localization After acquiring images from the depth camera, the color images are communicated to the convolutional network object recognition algorithm via ROS (Robot Operating System) to achieve rapid recognition and extraction of the two-dimensional coordinates of all bowls in the image.

[0034] ROS, as an open-source operating system in the robotics field, provides rich communication mechanisms and tool libraries, enabling efficient data transmission between different modules, and is especially suitable for real-time robot control scenarios.

[0035] Based on this, compared with traditional recognition algorithms, the rotating rectangle of the convolutional network object recognition algorithm can maximize the wrapping of the bowl target and significantly reduce the proportion of irrelevant background areas. For tilted bowls, traditional axis-aligned rectangles will contain a large number of background pixels, while rotating rectangles can accurately fit the edge of the bowl. This not only improves the calculation accuracy of subsequent pose estimation algorithms (by reducing the interference of background noise), but also reduces the amount of computation, providing support for real-time operations.

[0036] like Figure 2 As shown, in the convolutional network object recognition algorithm, firstly, the backbone network needs to sample the input image at multiple scales in order to extract features at different levels. In the backbone network... The output feature formula for the layer is: in, The hierarchy of output features in the backbone network; The number of feature layers (e.g., 3 layers, corresponding to downsampling times of 8 / 16 / 32). To preserve details such as object edges and textures; For global semantic information of the object; As the first in the backbone network The output features of the layer; I The input is an RGB image; I∈ ( H high, W (Wide, 3 channels) ∈ ( (Number of channels); Backbone network for target detection The feature extraction mapping function of the layer, where I Given the original RGB input image, this function performs multi-level downsampling and feature extraction on the input image, and outputs the first... Feature map of layer .

[0037] Furthermore, the neck network performs top-down and bottom-up fusion processing on the multi-scale features extracted by the backbone network. The top-down processing formula is as follows: in, The neck network is fused from top to bottom. The output features of the layer; Conv is a two-dimensional convolution operation function used to perform channel transformation, feature encoding and information fusion on the feature map, eliminate the aliasing effect after feature addition, and output an enhanced feature map with dimension matching; Upsample is an upsampling operation function used to enlarge the size of the deep low-resolution feature map so that it matches the spatial size of the shallow feature map, so as to realize the transmission of high-level semantic information from top to bottom to the shallow layer.

[0038] The bottom-up processing formula is as follows: in, The output features of the 1st layer of the Neck network are obtained by bottom-up fusion; Downsample is a downsampling operation function used to reduce the size of the shallow high-resolution feature map so that it matches the spatial size of the deep feature map, thereby realizing the transfer of shallow details from the bottom up to the deep layers.

[0039] Finally, after enhancing the output of the neck network, the following multi-scale feature set is obtained: in, This is the enhanced multi-scale feature set output by the Neck network.

[0040] Subsequently, the features of the backbone network and the neck network are integrated to obtain the following formula for the input features of the detection head: in, The input features for the detection head (the final detection features after multi-scale feature fusion); This is a multi-scale feature fusion mapping function, whose input includes the original multi-scale feature set output by the backbone network. Enhanced multi-scale feature set output by the neck network It is used to integrate features from different levels and semantic information, and output the final fused feature map for use by the detection head. .

[0041] Finally, the input features of the detection head are... Each spatial location is directly regressed to the parameters of the rotated rectangle. During the processing of the rotated rectangle, the center coordinates of the final rotated rectangle are first determined using the following formula: in, Normalize the network output offset to the grid. The coordinates; Normalize the network output offset to the grid. The coordinates;s The current feature map step size (e.g., 8 / 16 / 32); The x-coordinate of the top-left corner of the feature map grid; The coordinate of the top-left corner of the feature map grid is in the y-direction.

[0042] The formulas for the width and height coordinates of the final rotated rectangle are then set as follows: in, w The coordinates of the width of the rotated rectangle; h These are the height coordinates of the rotated rectangle; The width of the preset anchor frame; The height of the preset anchor frame; To output Map to the (0, 2) interval to avoid extreme width values; To output Map to the (0, 2) interval to avoid highly extreme values.

[0043] Next, the formula for the final rotation angle coordinates of the rotating rectangle is as follows: Where θ is the rotation angle of the rotating rectangle, defined as the amount of rotation relative to the positive x-axis; To output the network Mapped to [−π / 2,π / 2].

[0044] Therefore, defining the rotation angle θ as the positive x-axis direction with counterclockwise as positive, the formula for the total loss (composed of rotation loss, angle loss, classification loss, and confidence loss) is as follows: in, This represents the total loss of the rotating frame; For rotation frame loss; For angle loss; For classification loss; This represents the confidence loss.

[0045] The formula for the rotation frame loss is as follows: Where RIoU is the intersection-over-union ratio of the rotated bounding boxes, used to measure the overlap and positional consistency between the predicted and ground truth bounding boxes; B is the rotated bounding box predicted by the detection network, taken from the rotation detection algorithm documentation, and its complete data consists of a set of 5 parameters: ,in, and The pixel coordinates of the center point of the rotating rectangle on the input image; This is a true rotating frame.

[0046] The formula for angle loss is as follows: Wherein, SmoothL1 is the smooth L1 loss function, which is a robust loss function in regression tasks; This is a true rotating frame.

[0047] The formula for classification loss is as follows: Where classes is the complete set of target categories for the detection task. The summation symbol in the formula indicates that the classification loss is calculated and summed for each of the target categories to be detected, which is suitable for the training requirements of multi-class object detection. Let be the true label of the c-th category, and be a binary value. If the true category of the target corresponding to the current predicted anchor box / grid is the c-th category, then... =1, if the target corresponding to the current predicted anchor box / grid does not belong to class c. =0, The core function of the Sigmoid activation function is to map the unbounded original output of the network to the interval (0,1), and transform it into a classification confidence score that conforms to the probability definition. The original predicted value output by the detection head network for the c-th category is the unactivated original output obtained from the forward propagation of the network. It has no fixed range of values ​​and only represents the model's original predicted response to "the target belongs to the c-th category". This represents the classification prediction probability after Sigmoid activation, i.e., the confidence level of the model predicting that the target within the current anchor box / grid belongs to the c-th class. Its value is strictly constrained within the (0,1) interval and is used to compare with the true label. Calculate the classification error.

[0048] The formula for confidence loss is as follows: in, The true label for the existence of the target is a binary variable. It takes the value 1 when the target to be detected exists in the corresponding grid cell, and takes the value 0 when the corresponding grid cell is the background and there is no target. There is a confidence level for the target predicted by the network, where To detect the raw output value of the head network, The sigmoid activation function maps the output value to the interval [0,1], representing the probability of a target existing within that grid cell; grid is the set of grid cells in the feature map, i.e., the fused feature map input to the detection head. The spatial grid. The feature map is divided into... Each grid cell corresponds to a fixed receptive field region of the input image, serving as the basic unit for the network to predict targets.

[0049] The final formula for the rotation rectangle parameters of each target output by the detection head is as follows: Where conf_thresh is the confidence threshold (e.g., 0.6), used to filter low-confidence prediction boxes; Output is the rotation rectangle parameter for each target output by the detection head.

[0050] Based on the above convolutional network object recognition algorithm, the final rotated bounding box of the input color image can be obtained.

[0051] (3) Optimize the pose recognition algorithm for pose recognition After optimization, the FoundationPose pose recognition algorithm can aggregate RGB-D images from a depth camera, CAD models of existing objects, bounding boxes of objects identified by convolutional networks, and information on their front and back poses via ROS communication.

[0052] Among them, the FoundationPose algorithm, as a pose recognition algorithm based on the Tranformer architecture, is widely popular for its high speed and accuracy. However, the FoundationPose algorithm cannot recognize the 2D position information of objects on its own, and there are cases where the inverse solution exists for objects with strong orthosymmetry, such as square bowls. This may lead to fatal system errors. Therefore, convolutional network object recognition algorithms are combined with the FP algorithm (FoundationPose pose recognition algorithm) to achieve integrated recognition and pose estimation.

[0053] like Figure 3 As shown, after the FP algorithm receives the final rotated rectangle data, it first generates the initial pose hypothesis set of the object based on the object's CAD model, as follows: in, This is a mapping function for the FP algorithm to generate initial pose assumptions based on the CAD model; This is a set of initial pose assumptions used to address the problem of large initial value deviations in traditional pose estimation. It is a 3D point set of the object's CAD model.

[0054] Next, the region of interest (ROI) is obtained using the RGB-D image and the mask information from the convolutional network object recognition. The following is the definition of the ROI pixel region based on the mask: Where Ω represents the mask-based ROI pixel region; M represents the target mask output by the convolutional network; u The horizontal coordinates of the image pixels correspond to the width dimension of the image; v These are the vertical coordinates of the image pixels, corresponding to the height dimension of the image.

[0055] The 3D scene point cloud (depth map back projection) in the ROI region is as follows: in, 3D scene point cloud within the ROI region; For the pixel coordinates in the input RGB-D image The depth value at that location, in mm, represents the vertical distance from the scene point corresponding to that pixel to the camera's optical center; For the back projection transformation from pixel coordinates to normalized 3D coordinates in the camera coordinate system, where, K This is the camera intrinsic parameter matrix, describing the camera's intrinsic parameters such as focal length and principal point. It is the inverse of the camera intrinsic parameter matrix. pixel coordinates The homogeneous form of .

[0056] The above formula can be used to convert 2D pixel coordinates on the image plane into 3D normalized coordinates in the camera coordinate system, and then multiply by the depth value. This allows us to obtain the 3D spatial coordinates of the pixel in the camera coordinate system, ultimately forming a 3D point cloud of the target area. .

[0057] exist The calculation formula retains only the 3D points of the target area marked by the mask to eliminate background interference and narrow the calculation range of the FP algorithm.

[0058] Furthermore, by using Transformer pose iteration and refinement, the refined candidate pose formula is obtained as follows: in, Let be a rotation matrix, belonging to the special orthogonal group so(3), satisfying det(R) = 1, which describes the rotational transformation of the object's CAD model in the camera coordinate system and is the rotational component of the 6D pose. It is a translation vector used to describe the translation transformation of the object's CAD model in the camera coordinate system, and is the translation component of the 6D pose. Let be the function value of the FP energy optimization function at the i-th candidate pose in the k-th iteration; This is the Transformer pose iteration refinement function.

[0059] During the refined candidate pose iteration process, the termination condition for the iteration is as follows: Where ϵ is the precision threshold.

[0060] The refined candidate pose set is as follows: in, This is the set of candidate poses obtained after iterative refinement using the Transformer pose.

[0061] Subsequently, suitable candidate poses were selected through CAD rendering and convolutional network object recognition for both forward and reverse pose estimation, while inverse solutions were excluded. The CAD rendering images of the candidate poses were generated as follows: in, Render the image of the CAD model corresponding to the i-th candidate pose and compare it with the target region features of the real RGB-D image; Functions for rendering CAD models.

[0062] The convolutional network pose matching score is as follows: in, is the feature extraction function of the convolutional network; ⊙ represents element-wise multiplication, retaining only the features of the target region labeled by the mask; For pose matching degree, A higher score indicates a higher pose matching degree, and vice versa. Well below the threshold τ; The input is an RGB image.

[0063] In this process, valid poses need to be selected to exclude inverse solutions. The selection formula is as follows: in, This is the set of effective poses; τ The pose selection threshold (the minimum score for a valid pose set manually).

[0064] After the above step-by-step filtering, the final 6D pose can be obtained as follows: in, The optimal rotation matrix is ​​the rotation component corresponding to the effective pose with the highest matching score after pose filtering, which belongs to so (3). The optimal translation vector is the final output of the algorithm, which is the translation component corresponding to the optimal pose mentioned above; argmax is the operator for maximizing the independent variable, which means maximizing the subsequent objective function (here, pose matching score). The independent variable that achieves the maximum value (here, the 6D pose) That is, to select the effective pose with the highest score; The final 6D pose of the object output by the algorithm is determined by the optimal rotation matrix. and optimal translation vector Together, they form a complete description of the spatial pose of the object's 3D CAD model in the camera coordinate system, and are the final output of this algorithm.

[0065] The effective pose with the highest matching score is selected as the final 6D pose, realizing the integrated output of "recognition-pose estimation".

[0066] (4) Generation of task pool for collaborative robotic arm After generating the 6D poses of each object, the 6D pose data is placed into the task pool, and then the task pool is reordered according to the 6D poses.

[0067] During this process, the average depth of the stacked bowls may be used in conjunction with the depth of other bowls, which may lead to errors in the identification of the bowls' vertical order and potentially cause collision risks during operation. Therefore, the 6D poses obtained by the pose recognition algorithm are reordered, and the task pool is redistributed based on the spatial position characteristics of each object and the spatial position of the robotic arm.

[0068] Specifically, the task pools are arranged in descending order of depth, and based on the deep and shallow allocation, the nearest bowl is assigned to the robot according to its spatial position, thereby improving the high efficiency of the collaborative robotic arm.

[0069] like Figure 4 The image shows the experimental results of bowl pose recognition and task pool.

[0070] (5) Task allocation and reallocation of the robotic arm After the task pool is generated, a data file needs to be generated according to a preset format and sent to the robotic arm through the ROS node to enable the issuance of commands.

[0071] The system communication protocol of the robotic arm includes: bowl serial number (assigned by the task pool algorithm), bowl type (identified by the classification branch of the convolutional network object recognition algorithm, supporting the differentiation of common bowls such as rice bowls and plates), homogeneous transformation matrix (pose information), and robotic arm serial number (used for multi-robotic arm collaboration scenarios).

[0072] Furthermore, the data in the above task pool is encapsulated in JSON format. JSON format is lightweight and easy to parse, and can be quickly read and processed by the control system of the robotic arm.

[0073] Specifically, such as Figure 5 As shown in the figure, this is a typical example of JSON data format. Its main function is to enable communication between the robotic arm and this method after the robotic arm completes a task, so as to redistribute the task. Through the orderly extraction and redistribution of the task pool, it is used to realize the parallel operation of the collaborative robotic arm, thereby improving safety performance and efficiency.

[0074] The method for estimating and grasping the pose of a smart dishwashing device proposed in this invention uses convolutional network object recognition, optimized FP algorithm pose recognition, and collaborative robotic arm task pool generation. With the help of ROS system communication, it can accurately identify various poses of the bowl and provide them to the robotic arm for subsequent grasping.

[0075] The core technical feature of the intelligent dishwashing pose estimation and grasping strategy method proposed in this invention lies in its convolutional network object recognition algorithm, which can obtain a rotated rectangle with more accurate target position coordinates, significantly reduce the proportion of irrelevant background, and transmit the 2D position of the identified bowl to the pose estimation algorithm, which can significantly reduce the recognition time of the pose estimation algorithm and improve the recognition accuracy. This method integrates the convolutional network object recognition algorithm and the FP algorithm to realize an integrated system process of recognition and pose estimation. By using the coarse estimation of the 2D position of the convolutional network object recognition and the identification of front and back, the recognition accuracy of the FP algorithm is improved, the fatal problem of inverse solution is eliminated, and the pose recognition is more accurate.

[0076] The core technical feature of the method for estimating and grasping the pose of intelligent dishwashing proposed in this invention is that it uses the ROS system and efficient communication protocols to achieve efficient communication between the camera, object recognition, pose recognition, and robot task allocation, so as to achieve orderly and safe and stable operation of the task. Furthermore, through the principle of depth priority and the principle of collaborative robotic arm allocation, the task allocation and grasping strategy of the collaborative robotic arm are optimized, thereby improving the overall system efficiency.

[0077] Example 2 like Figure 6 As shown, this embodiment 2 provides a system for estimating and capturing the pose of an intelligent dishwasher, including: The depth camera image acquisition module provides data for subsequent bowl recognition and pose estimation. It uses a RealSense depth camera as the vision unit to simultaneously acquire color and depth images. Before the depth camera operation, a camera coordinate system and an image coordinate system that are consistent with the corresponding recognition algorithm need to be established. To establish the camera coordinate system, the x-axis should be defined as horizontally to the right along the image, the y-axis as vertically downward along the image, and the z-axis as perpendicular to the image plane and pointing into the scene. To establish the image coordinate system, the upper left corner of the acquired image should be used as the origin, and the x-axis should be defined as horizontally to the right along the image, and the y-axis as vertically downward along the image.

[0078] The rotating bounding box acquisition module is used for the rapid identification and 2D coordinate extraction of all bowls in the image. It adopts a convolutional network object recognition algorithm and transmits color image data through ROS communication. In the convolutional network object recognition algorithm, the backbone network samples the input image at multiple scales and extracts features at different levels. The neck network performs top-down and bottom-up fusion processing on the multi-scale features extracted by the backbone network. The input features of the detection head obtained by the fusion processing are directly regressed to the rotating bounding box parameters to obtain the final rotating bounding box of the input color image.

[0079] The 6D pose acquisition module aggregates RGB-D images from a depth camera, the CAD model of an existing object, the bounding boxes from a convolutional network object recognition system, and information on both front and back poses to form the final 6D pose of the bowl. The FP algorithm receives the data from the final rotated bounding box and generates a hypothetical initial pose of the object using the CAD model. It then uses the RGB-D image and mask information from the convolutional network object recognition system to obtain the region of interest (ROI). The Transformer pose iterative refinement is used, and suitable candidate poses are selected through CAD rendering and front and back pose estimation from the convolutional network object recognition system, while excluding negative solutions. Finally, the effective pose with the highest matching score is selected as the final 6D pose, achieving an integrated "recognition-pose estimation" output.

[0080] The collaborative robotic arm task pool generation module is used to reorder the task pool based on the 6D pose data. The task pool is arranged in descending order of importance. Based on the deep and shallow allocation, the robot is assigned the nearest bowl according to its spatial position, thereby improving the high efficiency of the collaborative robotic arm.

[0081] The robotic arm task allocation and reallocation module is used to quickly read and process data from the robotic arm's control system. The data in the task pool is encapsulated in JSON format and sent to the robotic arm through ROS nodes to issue commands.

[0082] Example 3 In embodiment 3 of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it can implement any step of a method for estimating and grasping the pose of an intelligent dishwashing machine.

[0083] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0085] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0086] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for estimating and grasping the pose of an intelligent dishwasher, characterized in that, include: S1: Before the depth camera performs its first operation, the initial setup of the depth camera must be completed; define the x-axis along the horizontal right direction of the image, the y-axis along the vertical downward direction of the image, and the z-axis perpendicular to the image plane and pointing into the scene to establish the camera coordinate system; with the upper left corner of the acquired image as the origin, define the x-axis along the horizontal right direction of the image and the y-axis along the vertical downward direction of the image to establish the image coordinate system; after the depth camera starts, acquire the complete image of the bowl to be processed within the current field of view to complete the initial image acquisition; S2: After the initial image acquisition by the depth camera is completed, the color image is communicated to the convolutional network object recognition algorithm via ROS to complete the rapid recognition and two-dimensional coordinate extraction of all bowls in the image; the backbone network performs multi-scale downsampling on the input image to extract features at different levels; the neck network performs top-down and bottom-up fusion of the multi-scale features; the features of the backbone network and the neck network are integrated to obtain the input features of the detection head, and the data information of the rotated rectangle is obtained for each spatial position of the input features; S3: The pose recognition algorithm receives the obtained data information and generates an initial pose hypothesis set through the CAD model; it obtains the region of interest of the object by using the RGB-D image and the mask information of the object recognition of the convolutional network; it refines the pose iteratively with Transformer to obtain the refined candidate pose; it selects suitable candidate poses by using CAD rendering and the positive and negative pose estimation of the object recognition of the convolutional network, and excludes the negative solutions. The effective pose with the highest matching score is selected as the final 6D pose, realizing the integrated "recognition-pose estimation" output; S4: The 6D poses of each object are generated and placed into the task pool. The 6D poses are reordered in the task pool using a pose recognition algorithm to avoid errors in the identification of the top and bottom order of the bowls. The task pool is arranged in descending order. Based on the deep and shallow allocation, the task pool is reassigned according to the robot's spatial position. S5: After the task pool is generated, a data file is generated according to the preset format and sent to the robotic arm through the ROS node to realize the issuance of instructions.

2. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 1, characterized in that, The multi-scale sampling includes: in, The hierarchy of output features in the backbone network; The number of feature layers (e.g., 3 layers, corresponding to downsampling times of 8 / 16 / 32). To preserve details such as object edges and textures; For global semantic information of the object; As the first in the backbone network The output features of the layer; I The input is an RGB image; I∈ ( H high, W (Wide, 3 channels) ∈ ( , , (Number of channels); Backbone network for target detection The feature extraction mapping function of the layer, where I Given the original RGB input image, this function performs multi-level downsampling and feature extraction on the input image, and outputs the first... Feature map of layer .

3. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 2, characterized in that, The top-down processing includes: in, The neck network is fused from top to bottom. The output features of the layer; Conv is a two-dimensional convolution operation function used to perform channel transformation, feature encoding and information fusion on the feature map, eliminate the aliasing effect after feature addition, and output an enhanced feature map with dimension matching; Upsample is an upsampling operation function used to enlarge the size of the deep low-resolution feature map so that it matches the spatial size of the shallow feature map, so as to realize the transmission of high-level semantic information from top to bottom to the shallow layer.

4. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 3, characterized in that, The bottom-up processing includes: in, The output features of the 1st layer of the Neck network are obtained by bottom-up fusion; Downsample is a downsampling operation function used to reduce the size of the shallow high-resolution feature map so that it matches the spatial size of the deep feature map, thereby realizing the transfer of shallow details from the bottom up to the deep layers.

5. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 4, characterized in that, The center coordinates of the rotated rectangle include: Where, is the coordinate of the network output offset normalized to the grid; is the coordinate of the network output offset normalized to the grid; s is the current feature map step size (e.g., 8 / 16 / 32); is the x-coordinate of the upper left corner of the feature map grid; is the y-coordinate of the upper left corner of the feature map grid.

6. A method for estimating and grasping the pose of an intelligent dishwasher according to any one of claims 1-5, characterized in that, The region of interest of the object includes: Where Ω is the ROI pixel region based on the mask; M is the target mask output by the convolutional network; u is the horizontal coordinate of the image pixel, corresponding to the width dimension of the image; and v is the vertical coordinate of the image pixel, corresponding to the height dimension of the image.

7. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 6, characterized in that, The 3D scene point cloud includes: in, 3D scene point cloud within the ROI region; For the pixel coordinates in the input RGB-D image The depth value at that location, in mm, represents the vertical distance from the scene point corresponding to that pixel to the camera's optical center; For the back projection transformation from pixel coordinates to normalized 3D coordinates in the camera coordinate system, where, K This is the camera intrinsic parameter matrix, describing the camera's intrinsic parameters such as focal length and principal point. It is the inverse of the camera intrinsic parameter matrix. pixel coordinates The homogeneous form of .

8. The method for estimating and grasping the pose of an intelligent dishwasher according to claim 7, characterized in that, The convolutional network's pose matching score includes: in, This is the feature extraction function for convolutional networks; To perform element-wise multiplication, only the target region features marked by the mask are retained; For pose matching degree, A higher score indicates a higher pose matching degree, and vice versa. Well below the threshold τ; The input is an RGB image.

9. A system for estimating and grasping the pose of an intelligent dishwasher, characterized in that, include: The depth camera image acquisition module provides data for subsequent bowl recognition and pose estimation. It uses a RealSense depth camera as the vision unit to simultaneously acquire color and depth images. Before the depth camera operation, a camera coordinate system and an image coordinate system consistent with the corresponding recognition algorithm need to be established. To establish the camera coordinate system, the x-axis should be defined horizontally to the right along the image, the y-axis vertically downward along the image, and the z-axis perpendicular to the image plane and pointing into the scene. To establish the image coordinate system, the upper left corner of the acquired image should be used as the origin, and the x-axis should be defined horizontally to the right along the image, and the y-axis vertically downward along the image. The rotating rectangle acquisition module is used for the rapid identification and 2D coordinate extraction of all bowls in the image. It adopts a convolutional network object recognition algorithm and transmits color image data through ROS communication. In the convolutional network object recognition algorithm, the backbone network samples the input image at multiple scales and extracts features at different levels. The neck network performs top-down and bottom-up fusion processing on the multi-scale features extracted by the backbone network. The input features of the detection head obtained by the fusion processing are directly regressed to the rotating rectangle parameters to obtain the final rotating rectangle of the input color image. The 6D pose acquisition module is used to aggregate the RGB-D image from the depth camera, the CAD model of the existing object, the bounding box of the object recognition by the convolutional network, and the front and back pose information to form the final 6D pose of the bowl; the FP algorithm receives the data information of the final rotated rectangle, generates the assumed initial pose of the object through the CAD model of the object; and obtains the region of interest of the object with the help of the RGB-D image and the mask information of the object recognition by the convolutional network. By using Transformer pose iteration to refine and then filtering out suitable candidate poses through CAD rendering and convolutional network object recognition for both positive and negative pose estimation, the negative solutions are excluded. The effective pose with the highest matching score is selected as the final 6D pose, realizing the integrated output of "recognition-pose estimation"; The collaborative robotic arm task pool generation module is used to reorder the task pool based on the 6D pose data. The task pool is arranged in descending order. Based on the deep and shallow allocation, the nearest bowl is allocated according to the robot's spatial position, thereby improving the high efficiency of the collaborative robotic arm. The robotic arm task allocation and reallocation module is used to quickly read and process data from the robotic arm's control system. The data in the task pool is encapsulated in JSON format and sent to the robotic arm through ROS nodes to issue commands.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8, which is a method for estimating and grasping the pose of an intelligent dishwasher.

Images