Handheld transparent object pose estimation method and robot grasping control method

Abstract

This paper proposes a method for estimating the pose of a handheld transparent object and a robot grasping control method. The method comprises: acquiring RGB and depth images; using an RGB feature encoder (Enco R) to extract RGB image features, Deco R1 to decode conventional features, and Deco R2 to decode geometric information and a hand segmentation map to assist in pose estimation; using pixel multiplication techniques to filter transparent objects and background interference from the depth map, and extracting hand depth features using Enco D and Deco D; fusing and stacking the RGB and depth features; and finally, using a pose decoding module to decode the fused features to obtain precise 6D pose information for the transparent object. This method significantly improves pose estimation accuracy by deeply utilizing hand information, which is of great significance for precise robot grasping and safe handover, especially effectively improving the safety and accuracy of robot handover of transparent objects in a home environment.

Description

Technical Field

[0001] The present invention relates to object posture detection and robot control technology, and in particular to a handheld transparent object posture estimation method and a robot grasping control method. Background Art

[0002] Transparent objects are widely present in homes and industrial environments, posing significant challenges for various visual tasks. For one thing, the non-Lambertian surfaces of transparent objects prevent effective diffuse reflection, making it impossible to acquire depth images of them using depth cameras. Furthermore, the appearance of transparent objects is significantly affected by the background and lighting. In extreme environments, such as those with high-frequency background changes, chaotic lighting, and underwater, transparent objects are difficult to effectively detect visually. These challenges make it extremely difficult for home service robots to interact with transparent objects in the home. In the home, one of the most common and important interactions involving transparent objects is handing over a glass of water. Failure to properly handle such a situation can result in not only shattering of the glass but also more serious consequences, such as burns. Therefore, a more comprehensive solution is needed to address the issue of handing over transparent objects.

[0003] Generally speaking, robotic grasping technology is required for a robot to transfer objects held in its hand. To accurately locate the appropriate grasping point, it is necessary to obtain the 6D pose information of the handheld transparent object (the 6D pose consists of position and attitude, each with three degrees of freedom, for a total of six degrees of freedom, hence the term "6D"). However, due to hand occlusion, detecting the 6D pose of a handheld transparent object is more difficult than for a stationary transparent object.

[0004] With the development of home service robots, safety considerations are gradually increasing, and the handover of handheld transparent objects is one of the issues that need special attention. Handheld transparent objects generally have the following characteristics:

[0005] Fragile: Transparent objects are usually glass products. Improper robot handover may cause glass products to break and cause losses.

[0006] Contains contents: Often, transparent objects such as glasses contain contents (such as beverages or hot water). Failure in handover can cause the contents to spill, making it difficult to clean up and potentially causing burns.

[0007] Therefore, it is urgent to find a solution that can adapt to the handover of handheld transparent objects, that is, to estimate the pose of the transparent object in the hand, so as to realize robot grasping.

[0008] Currently, there are several technologies for estimating the pose of handheld opaque objects. The main difference between handheld opaque objects and transparent objects is that handheld transparent objects are affected by the hand pattern behind the transparent object (which can be considered as the transparent object's background), lighting, and the liquid inside the transparent object, making them more difficult to identify and locate than opaque objects. Therefore, the technology of identifying and locating opaque objects through surface texture information is no longer applicable to transparent objects.

[0009] There are still several technologies for pose estimation of transparent objects. However, they do not consider the influence of hands. The occlusion of hands makes it difficult for such algorithms to extract enough information to estimate the pose of transparent objects. At the same time, the hand posture also partially reflects the posture of transparent objects. Therefore, completely ignoring the hand posture is also a waste of available information. More importantly, most current solutions take into account that depth cameras cannot obtain the depth of transparent objects, so depth information is no longer considered in pose estimation. However, depth cameras can accurately detect the depth of the hand, and the depth of the hand can help locate the position of transparent objects inside the hand.

[0010] The paper Learning to Estimate Pose and Shape of Hand-Held Objects from RGBImages proposes a pose estimation method for handheld transparent objects (referred to as Method 1), such as Figure 1 shown.

[0011] In this algorithm, the input images are hands and held objects. During the algorithm, the hand posture information and object posture information are extracted respectively, and the hand posture information is used as a supplement to estimate the object posture.

[0012] Disadvantages of method 1:

[0013] When processing handheld objects, the algorithm directly extracts the appearance features of the handheld object without any additional processing, which is inappropriate for transparent objects. The influence of changing backgrounds, lighting, and internal solutions on positioning and recognition of transparent objects makes direct appearance feature extraction ineffective for transparent objects. Therefore, although the algorithm proposes to use the additional information provided by hand posture to assist in 6D pose estimation of handheld objects, it is not well adapted to the case of handheld transparent objects.

[0014] The paper Tgf-net: Sim2real transparent object 6d pose estimation based on geometric fusion proposes a 6D pose estimation method for transparent objects based on geometric fusion (referred to as Method 2).

[0015] like Figure 2 As shown in the figure, the algorithm takes an RGB image as input. After the encoder completes feature extraction, it splits into two independent branches. The decoder in the upper part, shown in yellow, is the conventional stream. The conventional stream extracts features other than geometric features, thereby providing a more comprehensive description of transparent objects. The decoder in the lower part, shown in pink, is the geometric stream. It extracts geometric information about the object, such as surface fragments, edge features, and object masks. This geometric information is less susceptible to object color interference and effectively mitigates the effects of background and illumination variations on the 6D pose estimation of transparent objects. After feature extraction in the geometric and conventional streams, the extracted features are concatenated and extracted using the corresponding feature extraction network. This results in a dense 3D coordinate map and confidence maps for both channels. The dense 3D coordinate map is then overlaid with the 2D dense coordinate map to obtain 2D-3D correspondences. In the subsequent steps, the point-to-point (PnP) algorithm (shown in yellow in the figure) is used to solve the 6D pose.

[0016] Disadvantages of method 2:

[0017] The features extracted by this algorithm are all features at the visible position of transparent objects. However, when holding transparent objects, the visible part of the transparent object is drastically reduced due to the occlusion of the hand. This results in the algorithm extracting very limited feature information of the visible position of the transparent object. Too little feature information makes it difficult to produce sufficiently accurate pose estimation. Secondly, the input of the algorithm is an RGB image. This is because the depth of transparent objects cannot be effectively measured by the depth camera. If only the transparent object exists, adding the depth image will not provide additional supplementary information. However, the depth camera can reliably measure the depth information of the hand. Adding the depth information of the hand can assist in the pose estimation of the object in the hand (for example, when we know that the depth of a point on the hand is 0.5m, that is, the distance between the point on the hand and the camera is 0.5m, we can infer that the position between the object in the hand and the camera is also around 0.5m. In this way, the information of the transparent object in the hand is inferred with the help of the hand information). Finally, in addition to the hand depth information that can provide additional information to assist in estimating the pose of objects within the hand, the hand information in the RGB image can also provide additional information to assist in estimating the pose of objects within the hand. For example, we can roughly judge the pose of the object within the hand based on the hand posture, which is also not considered in this algorithm.

[0018] The shortcomings of Methods 1 and 2 above are summarized as follows: Method 1: The influence of changing backgrounds, lighting, and internal solutions on the positioning and recognition of transparent objects makes the direct extraction of appearance features ineffective for transparent objects. Method 2: When holding transparent objects, the algorithm extracts very limited feature information about the visible position of the transparent object due to occlusion by the hand. This insufficient feature information makes it difficult to produce sufficiently accurate pose estimation. Hand information (including depth and RGB information) is not fully utilized to assist in pose estimation of transparent objects in the hand.

[0019] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0020] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a handheld transparent object posture estimation method and a robot grasping control method.

[0021] To achieve the above object, the present invention adopts the following technical solutions:

[0022] A method for estimating the pose of a handheld transparent object comprises the following steps:

[0023] S1. Acquire RGB image and depth image of handheld transparent object;

[0024] S2. RGB processing module: uses the RGB feature encoder Enco R to extract features from the RGB image; decodes the image using the conventional feature decoder Deco R1 and the geometric feature decoder Deco R2, where the conventional feature decoder Deco R1 extracts RGB image features and the geometric feature decoder Deco R2 extracts the object's geometric information and hand segmentation map to assist in pose estimation using hand posture;

[0025] S3. Depth Processing Module: This module performs pixel-wise multiplication on the extracted hand segmentation image and the original depth image to filter out interference from transparent object depth and background depth. The hand depth information in the depth image is retained and fed into the hand depth feature encoder Enco D for feature encoding. The hand depth feature information is then extracted and decoded by the hand depth feature decoder Deco D to obtain the decoded hand depth features.

[0026] S4. Feature fusion: Feature stacking of the output of the RGB processing module and the output of the depth processing module;

[0027] S5. Posture decoding module: decodes the stacked features to obtain the pose information of the transparent object.

[0028] Further:

[0029] The RGB feature encoder Enco R, the conventional feature decoder Deco R1 and the geometric feature decoder Deco R2 are constructed based on the Swin-Transformer structure.

[0030] The geometric feature decoder Deco R2 extracts geometric information of an object including surface fragments, edge features and object segmentation maps.

[0031] In step S3, performing pixel multiplication on the extracted hand segmentation image and the original depth image includes: taking the pixel value of the hand position in the hand segmentation image as 1 and the pixel value of the non-hand position as 0, and performing pixel multiplication on the depth image.

[0032] The hand depth feature encoder Enco D and the hand depth feature decoder Deco D are constructed based on the Swin-Transformer structure.

[0033] The obtained pose information of the transparent object is 6D pose information including three degrees of freedom of position and attitude.

[0034] The hand includes a human hand and a mechanical gripper.

[0035] A robot grasping control method uses the handheld transparent object posture estimation method to estimate the posture information of the transparent object, and guides the robot to grasp and hand over the transparent object according to the posture information.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the handheld transparent object pose estimation method or the robot grasping control method.

[0037] A computer program product includes a computer program, which, when executed by a processor, implements the handheld transparent object pose estimation method or the robot grasping control method.

[0038] The present invention has the following beneficial effects:

[0039] This paper proposes a method for estimating the pose of a handheld transparent object by integrating hand information. This method effectively utilizes the additional information provided by hand posture to assist in 6D pose estimation of a handheld transparent object, thereby obtaining more accurate pose information for the handheld transparent object and minimizing the impact of changing background, lighting, and internal solutions on positioning and identification. Compared with previous methods, this method has the following advantages:

[0040] The present invention integrates hand posture information and depth information into the transparent object pose estimation algorithm. The supplementation of additional information makes the transparent object pose estimation more accurate and can cope with the situation where the hand severely occludes the object in the hand.

[0041] The extracted hand pose and depth information can also serve as a basis for robot grasping, meaning the robot needs to avoid the position of the human hand and grasp the transparent object in the appropriate position. Therefore, the solution proposed by this invention not only estimates the object pose, but also avoids collision between the robot and the human hand (the algorithm also obtains the hand segmentation map and avoids grasping the area where the hand segmentation map is located during grasping), thus achieving two functions with one algorithm.

[0042] The geometric information of the visible part of the transparent object is effectively utilized, which can effectively avoid the influence of background and lighting changes and the liquid inside the container on the 6D pose estimation of the transparent object.

[0043] In summary, the method of the present invention makes full use of the assistance of hand information, effectively improves the accuracy of pose estimation of handheld transparent objects, and thus effectively improves the accuracy and safety of robot grasping and handover.

[0044] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the framework diagram of the handheld object pose estimation process of existing method one.

[0046] Figure 2 This is the framework diagram of the handheld object pose estimation process of the existing method 2.

[0047] Figure 3 This is a flow chart of a method for estimating the pose of a handheld transparent object by integrating hand information according to an embodiment of the present invention.

[0048] Figure 4 This is a flow chart of the RGB processing module according to an embodiment of the present invention.

[0049] Figure 5 Schematic diagram of different parts of a handheld transparent object.

[0050] Figure 6 This is a flow chart of the depth processing module of an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0052] See Figure 3, an embodiment of the present invention provides a method for estimating the pose of a handheld transparent object, comprising the following steps:

[0053] S1. Acquire RGB image and depth image of handheld transparent object;

[0054] S2. RGB processing module: uses the RGB feature encoder Enco R to extract features from the RGB image; decodes the image using the conventional feature decoder Deco R1 and the geometric feature decoder Deco R2, where the conventional feature decoder Deco R1 extracts RGB image features and the geometric feature decoder Deco R2 extracts the object's geometric information and hand segmentation map to assist in pose estimation using hand posture;

[0055] S3. Depth Processing Module: This module performs pixel-wise multiplication on the extracted hand segmentation image and the original depth image to filter out interference from transparent object depth and background depth. The hand depth information in the depth image is retained and fed into the hand depth feature encoder Enco D for feature encoding. The hand depth feature information is then extracted and decoded by the hand depth feature decoder Deco D to obtain the decoded hand depth features.

[0056] S4. Feature fusion: Feature stacking of the output of the RGB processing module and the output of the depth processing module;

[0057] S5. Posture decoding module: Decodes the stacked features to obtain the pose information of the transparent object.

[0058] It should be understood that in the present invention, the "hand" in the scene of holding a transparent object includes not only a human hand, but also various robotic manipulators or grippers for picking up and grasping objects, as well as bionic hands, various clamps for performing object grasping and carrying tasks, etc.

[0059] like Figure 4 As shown, in some embodiments, in the RGB processing module, the RGB feature encoder Enco R, the conventional feature decoder Deco R1 and the geometric feature decoder Deco R2 are constructed based on a Swin-Transformer structure.

[0060] In some embodiments, the geometric feature decoder Deco R2 extracts geometric information of an object including surface fragments, edge features, and object segmentation maps.

[0061] In some embodiments, in step S3, performing pixel multiplication on the extracted hand segmentation image and the original depth image includes: taking the pixel value of the hand position in the hand segmentation image as 1 and the pixel value of the non-hand position as 0, and performing pixel multiplication on the depth image.

[0062] like Figure 6 As shown, in some embodiments, in the depth processing module, the hand depth feature encoder Enco D and the hand depth feature decoder Deco D are constructed based on the Swin-Transformer structure.

[0063] An embodiment of the present invention also provides a robot grasping control method, which uses the handheld transparent object pose estimation method of any of the aforementioned embodiments to estimate the pose information of the transparent object, and guides the robot to grasp and hand over the transparent object based on the pose information.

[0064] The hand-integrated hand information-based pose estimation method for handheld transparent objects in this embodiment of the present invention effectively utilizes the additional information provided by hand posture to assist in 6D pose estimation of handheld transparent objects, thereby obtaining more accurate pose information for handheld transparent objects and minimizing the impact of changing backgrounds, lighting, and internal solutions on positioning and identification. Compared to previous methods, the method of this invention has the following advantages:

[0065] The present invention integrates hand posture information and depth information into the transparent object pose estimation algorithm. The supplementation of additional information makes the transparent object pose estimation more accurate and can cope with the situation where the hand severely occludes the object in the hand.

[0066] The extracted hand pose and depth information can also serve as a basis for robot grasping, meaning the robot needs to avoid the position of the human hand and grasp the transparent object in the appropriate position. Therefore, the solution proposed by this invention not only estimates the object pose, but also avoids collision between the robot and the human hand (the algorithm also obtains the hand segmentation map and avoids grasping the area where the hand segmentation map is located during grasping), thus achieving two functions with one algorithm.

[0067] The geometric information of the visible part of the transparent object is effectively utilized, which can effectively avoid the influence of background and lighting changes and the liquid inside the container on the 6D pose estimation of the transparent object.

[0068] In summary, the method of the present invention makes full use of the assistance of hand information, effectively improves the accuracy of pose estimation of handheld transparent objects, and thus effectively improves the accuracy and safety of robot grasping and handover.

[0069] Specific embodiments of the present invention are further described below.

[0070] The embodiment of the present invention proposes a method for estimating the pose of a handheld transparent object by integrating hand information. Figure 3As shown. The present invention proposes a transparent object RGB processing module, in which the geometric information of the transparent object, such as surface fragments, edge features and object segmentation map, is fully considered. The rich geometric information can effectively avoid the influence of background and illumination changes on the 6D pose estimation of the transparent object. At the same time, the segmentation of the hand is added, and the 6D pose estimation of the transparent object is assisted by the hand posture. The present invention proposes a transparent object depth processing module, which eliminates the influence of irrelevant depth by pixel multiplication of the hand segmentation map and the original depth map, and integrates the hand depth information into the algorithm to assist in the 6D pose estimation of the transparent object.

[0071] In the method of this embodiment of the present invention, the input images are an RGB image of a handheld object and a depth image of the handheld object. In the depth image, the depth of the transparent object is lost due to the special optical properties of the transparent object. However, the depth image can accurately capture the depth information of the hand position. The algorithm of this embodiment of the present invention can be generally divided into three modules: an RGB processing module, a depth processing module, and a pose decoding module. The implementation of these three modules is described in detail below.

[0072] RGB processing module

[0073] The RGB processing module takes as input an RGB image of a handheld transparent object captured by the camera. This RGB image is processed by the encoder Enco R (short for Encoder RGB) for feature extraction, followed by decoding of the extracted features by Deco R1 and Deco R2 (short for Decoder RGB). Deco R1 is a conventional feature decoder that extracts deep image features in addition to geometric features, thereby providing a more comprehensive description of the transparent object. Deco R2 is a geometric feature decoder that extracts geometric information about the object, such as surface fragmentation, edge features, and an object segmentation map. This geometric information is less susceptible to color interference and effectively mitigates the effects of background and illumination variations on the 6D pose estimation of the transparent object. (The definitions of surface fragmentation, edge features, and object segmentation map are the same as in the paper Tgf-net: Sim2real transparent object 6D pose estimation based on geometric fusion.) Considering the unique characteristics of handheld objects, Deco R2 not only extracts surface fragmentation, edge features, and an object segmentation map, but also extracts a segmentation map of the hand. In RGB images, hand images are obvious and easy to segment. At the same time, the hand posture reflected in the hand image plays an auxiliary role in predicting the object posture. For example, when the hand is in a gripping state, the approximate state of the transparent object in the hand can be judged.

[0074] In one embodiment, at the network structure level, Enco R, Deco R1 and Deco R2 can all adopt the Swin-Transformer structure (see the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021). The overall structure of the RGB processing module is as follows: Figure 4 As shown in the figure, these include modules such as Skip Connection, PatchPartition, Linear Embedding, Swin Transformer block, Patch Merging, and Linear Projection. Together, these components form the Swin-Transformer-based network structure, enabling feature extraction and decoding.

[0075] Deep processing module

[0076] In the depth processing module, the input part is the depth image of the handheld transparent object collected by the depth camera, in which the depth of the transparent object is missing. We consider dividing the depth of the handheld transparent object depth image into three parts, namely the hand depth, the transparent object depth, and the background depth, such as Figure 5 shown.

[0077] exist Figure 5 In the image, due to the poor visualization effect of the depth map, the depth of the hand, the depth of the transparent object, and the depth of the background are difficult to distinguish, so the RGB image is used as an example (the depth image corresponds to the RGB image one-to-one). For the object part in the depth image, since the depth camera cannot effectively measure the depth of the transparent object, this part appears as depth loss and depth holes, so the depth of this part cannot provide valuable information. For the background part in the depth image, since the handheld object is in the air, that is, the depth of the background behind it is completely irrelevant to the posture state of the object in the hand, the depth of this part cannot provide valuable information. As for the hand part in the depth image, since the object is held in the hand, the hand depth can provide a lot of useful information about the object posture, and the depth of this part can be fully utilized as supplementary information for estimating the posture of the object in the hand.

[0078] In order to make full use of the hand depth while filtering out useless object depth and background depth, we multiply the hand segmentation map obtained by Deco R2 decoding in the RGB processing module with the original depth map, that is, Figure 3 in Since the pixel value of the hand position in the hand segmentation map is 1, and the pixel value of the non-hand position is 0, pixel multiplication can be used to filter out the interference of transparent object depth and background depth while retaining the hand depth.

[0079] After filtering the depth of transparent objects and background in the depth map, the filtered hand depth will enter Enco D and Deco D for feature encoding and decoding. In one embodiment, Enco D and Deco D can also be based on the Swin-Transformer structure, and the structures of Deco D and Deco R can be completely consistent (except for the final output). Figure 6 shown.

[0080] Pose decoding module

[0081] As described above, the final output of the RGB processing module is the RGB image features output by Deco R1, the boundary features, surface fragments, object segmentation map, and hand segmentation map output by Deco R2, and the final output of the depth processing module is the depth image features. Before performing pose decoding, all of the above features are stacked. After stacking, the stacked features are input into the pose decoding module. In one embodiment, the pose decoding module can adopt the pose decoding module in the paper Tgf-net: Sim2realtransparent object 6d pose estimation based on geometric fusion (IEEE Robotics and Automation Communications, 2023). After pose decoding, the pose information of the transparent object in the hand can be obtained. In addition, those skilled in the art will understand that the present invention can also use other alternatives to complete the decoding process from stacked features to the 6D pose of the transparent object, such as using a multi-task learning model based on deep learning to extract the pose and shape information of the object; and using other machine learning methods or pose estimation methods based on physical models.

[0082] As described above, embodiments of the present invention propose a method for estimating the pose of a handheld transparent object that incorporates hand information. This method integrates hand pose information and depth information into the transparent object pose estimation algorithm. This additional information makes transparent object pose estimation more accurate. The handheld transparent object RGB processing module encodes and decodes the transparent object's RGB image using an encoder, EncoR, a conventional feature decoder, Deco R1, and a geometric feature decoder, Deco R2. The geometric feature decoder, Deco R2, extracts the object's geometric information, such as surface fragments, edge features, and an object segmentation map. This rich geometric information effectively mitigates the effects of background and illumination changes on the transparent object's 6D pose estimation. To account for the influence of the hand, the geometric feature decoder, Deco R2, also extracts a hand segmentation map, thereby aiding in 6D pose estimation of the transparent object using hand pose. The handheld transparent object depth processing module, in order to eliminate interfering transparent object depth and background depth, performs pixel-wise multiplication of the hand segmentation map obtained by Deco R2 decoding in the RGB processing module with the original depth map. The filtered hand depth is sent to Enco D and Deco D for feature encoding and decoding, and finally the depth image features are obtained, thereby incorporating the hand depth into the algorithm to assist in the 6D pose estimation of transparent objects.

[0083] In summary, this paper proposes an innovative method for estimating the pose of a handheld transparent object. By integrating hand information, this method improves the accuracy and security of pose estimation. This method is particularly applicable to (but not limited to) household service robots interacting with transparent objects in a domestic environment, such as handing over fragile or filled glasses.

[0084] The present invention uses an RGB processing module, an encoder (Enco R) and decoders (Deco R1 and Deco R2) to extract deep features from RGB images. Deco R1 extracts deep image features, while Deco R2 focuses on geometric features, such as surface fragmentation and edge features of objects, as well as segmentation maps of hands and objects. The depth processing module uses pixel multiplication technology to filter out interference information in the depth image, retaining only hand depth information. Feature encoding and decoding are then performed using the encoder (Enco D) and decoder (Deco D). Through feature fusion, the outputs of the RGB and depth processing modules are stacked and input into the pose decoding module to ultimately obtain the pose information of the transparent object.

[0085] By fusing hand posture and depth information, the present invention significantly improves the accuracy of pose estimation, especially when the hand severely obscures the object. The algorithm simultaneously obtains a hand segmentation map, which is not only used for pose estimation, but also guides the robot to avoid the position of the human hand, achieve safe grasping, and avoid collisions. By utilizing the geometric information of the visible part of the transparent object, the pose estimation is effectively resisted by background and lighting changes, as well as the influence of the liquid inside the container. Compared with the prior art, the present invention makes up for the shortcomings of the prior art in processing handheld transparent objects, and provides a more accurate and safe pose estimation method, which is particularly suitable for scenarios where home service robots interact with humans.

[0086] This invention has broad application prospects and high practical value. For example, in a home environment, a robot needs to safely hand over a transparent object, such as a glass filled with a drink or hot water. This invention accurately estimates the object's position and guides the robot for precise grasping, preventing object breakage or spillage due to failed handovers, thereby ensuring the safety and reliability of the home service robot.

[0087] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0088] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0089] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0090] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0091] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0092] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0094] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0095] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0096] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0097] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0098] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0099] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for estimating the pose of a handheld transparent object, characterized in that: The following steps are involved: S1. Acquire RGB image and depth image of handheld transparent object; S2.RGB processing module: Use RGB feature encoder Enco R to extract features from the RGB image; The conventional feature decoder Deco R1 and the geometric feature decoder Deco R2 perform decoding, wherein the conventional feature decoder Deco R1 extracts RGB image features, and the geometric feature decoder Deco R2 extracts the geometric information of the object and the hand segmentation map to assist in pose estimation using hand posture; S3. Depth Processing Module: This module performs pixel-wise multiplication on the extracted hand segmentation image and the original depth image to filter out interference from transparent object depth and background depth. The hand depth information in the depth image is retained and fed into the hand depth feature encoder Enco D for feature encoding. The hand depth feature information is then extracted and decoded by the hand depth feature decoder Deco D to obtain the decoded hand depth features. S4. Feature fusion: Feature stacking of the output of the RGB processing module and the output of the depth processing module; S5. Posture decoding module: Decodes the stacked features to obtain the pose information of the transparent object.

2. The handheld transparent object pose estimation method according to claim 1, wherein: The RGB feature encoder Enco R, the conventional feature decoder Deco R1 and the geometric feature decoder Deco R2 are constructed based on the Swin-Transformer structure.

3. The handheld transparent object pose estimation method according to claim 1 or 2, wherein: The geometric feature decoder Deco R2 extracts geometric information of an object including surface fragments, edge features and object segmentation maps.

4. The method for estimating the position and posture of a handheld transparent object according to any one of claims 1 to 3, wherein: In step S3, performing pixel multiplication on the extracted hand segmentation image and the original depth image includes: taking the pixel value of the hand position in the hand segmentation image as 1 and the pixel value of the non-hand position as 0, and performing pixel multiplication on the depth image.

5. The method for estimating the position and posture of a handheld transparent object according to any one of claims 1 to 4, wherein: The hand depth feature encoder Enco D and the hand depth feature decoder Deco D are constructed based on the Swin-Transformer structure.

6. The method for estimating the position and posture of a handheld transparent object according to any one of claims 1 to 5, wherein: The obtained pose information of the transparent object is 6D pose information including three degrees of freedom of position and attitude.

7. The method for estimating the position and posture of a handheld transparent object according to any one of claims 1 to 6, wherein: The hand includes a human hand and a mechanical gripper.

8. A robot grasping control method, characterized in that: The handheld transparent object pose estimation method according to any one of claims 1 to 7 is used to estimate the pose information of the transparent object, and the robot is guided to grasp and hand over the transparent object according to the pose information.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for estimating the pose of a handheld transparent object according to any one of claims 1 to 7 or the robot grasping control method according to claim 8 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for estimating the pose of a handheld transparent object according to any one of claims 1 to 7 or the robot grasping control method according to claim 8 is implemented.

Images