A robot grasping control method, device, equipment and storage medium

Abstract

The application provides a robot grasping control method, device and equipment and a storage medium. The method first acquires an image containing daily necessities as a target image; then performs feature extraction on the target image by using a feature extraction network to obtain a corresponding feature map; performs grasping pose detection on the feature map to acquire a direction angle and bounding box regression correction information of a candidate grasping box; performs semantic segmentation on the feature map to acquire grasping enclosure information and grasping region semantic information; performs information fusion on the grasping enclosure information, the grasping region semantic information, the direction angle and the bounding box regression correction information of the candidate grasping box to obtain a grasping dataset; and finally controls a robot to grasp an object according to the grasping dataset. The application fully considers grasping quality on the premise of ensuring grasping accuracy and rapidity by integrating grasping enclosure into a deep semantic segmentation neural network grasping pose detection model, so that better grasping effect is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a robot grasping control method, device, equipment, and storage medium. Background Technology

[0002] With economic and social development, the aging population is becoming increasingly prominent, and this trend inevitably has negative impacts on economic development and social stability. To address this challenge, we should vigorously promote the high-quality development of elderly care services in the new era, and build a universal, standardized, professional, and diversified elderly care service system. Against this backdrop, the investment and development of robotics technology in elderly care support plays a crucial role in improving all aspects of elderly care services. Furthermore, in today's fast-paced world, most young and middle-aged people not only need to balance work and life but also need to care for elderly family members, inevitably leading to situations where care is inadequate, significantly impacting their own and the elderly's daily lives. Robots can provide better support for the elderly, such as retrieving daily necessities, cleaning, monitoring physiological indicators, and issuing alarms, thus greatly changing this situation.

[0003] For life support robots, object grasping is a crucial function. However, whether a robot can perform stable and reliable grasping in a given environment remains a challenging problem that has attracted widespread attention and research. Robotic grasping can be mainly divided into two parts: grasping pose generation and grasping path planning. Grasping pose generation refers to obtaining the object region that the robot's end-effector can stably grasp. In this process, the external environment is not considered; only the geometry and material properties of the object and the end-effector are taken into account. Grasping path planning, on the other hand, involves planning a complete grasping path while considering the external environment. Objects in the external environment may be obstructed, stacked, or leaning against each other. Therefore, it is necessary to plan the grasping order of objects and select appropriate grasping poses to avoid objects falling or being damaged.

[0004] Currently, robotic grasping technology still faces several challenges. For example, existing grasping techniques do not adequately consider certain grasping details and require further improvement: ① Since not all areas of an object are suitable for grasping, the graspable area needs to be considered during the grasping process, but current grasping technologies rarely take this into account; ② Current grasping technologies typically focus on the pick-and-place operation of a two-finger gripper. The requirement for a successful grasping operation is generally simply the ability to grasp the object and complete the corresponding task, without fully considering the enclosed nature of the grasp. Therefore, how to effectively control robot grasping is a problem that urgently needs to be solved. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, this application provides a robot grasping control method, apparatus, device and storage medium to solve the above technical problems.

[0006] This application provides a robot grasping control method, including the following steps:

[0007] Acquire images containing daily necessities, either pre-captured or captured in real-time, and record them as target images;

[0008] The target image is used to extract features using a feature extraction network to obtain the corresponding feature map;

[0009] The feature map is subjected to grasping posture detection to obtain the orientation angle and bounding box regression correction information of the candidate grasping box; and the feature map is subjected to semantic segmentation to obtain grasping closure information and grasping region semantic information.

[0010] The crawling closure information, the semantic information of the crawling region, the orientation angle of the candidate crawling box, and the bounding box regression correction information are fused to obtain the crawling dataset.

[0011] The robot is controlled to grasp objects based on the grasping dataset.

[0012] In one embodiment of this application, if the feature extraction network includes a residual network and a feature pyramid network, then when using the feature extraction network to extract features from the target image, the process of obtaining the corresponding feature map includes:

[0013] The residual network is used to extract features from the target image to obtain a first feature image;

[0014] The feature pyramid network is used to extract features from the target image to obtain a second feature image;

[0015] The first feature image and the second feature image are fused to obtain the feature map.

[0016] In one embodiment of this application, the process of performing grasping pose detection on the feature map and obtaining the orientation angle and bounding box regression correction information of the candidate grasping box includes:

[0017] A region proposal network is used to find candidate capture boxes from the feature map;

[0018] The candidate capture box is calibrated by performing region of interest calibration to obtain the calibrated candidate capture box;

[0019] The calibrated candidate grab box is rotated angularly in the first direction to obtain an axis-aligned rectangle.

[0020] The axis-aligned rectangle is classified as a target, and bounding box regression is performed according to the target classification result. A rotation opposite to the first direction but at the same angle is also performed to obtain the direction angle and bounding box regression correction information of the candidate grabbing box. The bounding box regression correction information includes: the center point coordinates of the candidate grabbing box in the image coordinate system, the width of the candidate grabbing box in the image coordinate system, and the height of the candidate grabbing box in the image coordinate system.

[0021] In one embodiment of this application, the process of classifying the axis-aligned rectangle to obtain the target classification result includes:

[0022] The angle range corresponding to the calibrated candidate capture box when it is rotated in the first direction is denoted as the first angle range;

[0023] The first angle interval is superimposed to generate the second angle interval;

[0024] The second angle interval is uniformly quantized, dividing the second angle interval into multiple sub-intervals, and each sub-interval is treated as a target category, denoted as the first target category; wherein, each sub-interval is spaced at the same angle.

[0025] Obtain the blank grasping region in the feature map, and treat the blank grasping region as a separate target category, denoted as the second target category;

[0026] The first target category and the second target category are superimposed in terms of quantity, and the result of the superimposed target categories is used as the target classification result of the axis-aligned rectangle.

[0027] In one embodiment of this application, the process of performing semantic segmentation on the feature map to obtain grasping closure information and grasping region semantic information includes:

[0028] Obtain the robot's gripper and establish a grasping coordinate system with the center point of the gripper as the origin;

[0029] When the object to be grasped is translated within the first interval on the x-axis in the grasping coordinate system, and at least one finger of the gripper contacts the object to be grasped, the distance between the two grippers in the robot is recorded, as well as the angle between each gripper and the object to be grasped is recorded.

[0030] The minimum value of the region is selected from the distance values ​​between the two grippers of the robot, and the grasping closure information is generated based on the selected minimum value of the region and the angle between each gripper and the object to be grasped; and

[0031] The semantic information of the crawled region is obtained by semantic segmentation of the crawled closed information using the watershed algorithm.

[0032] This application also provides a robot grasping control device, the device comprising:

[0033] The image acquisition module is used to acquire images containing daily necessities, either pre-captured or captured in real time, and these images are denoted as target images.

[0034] The feature extraction module is used to extract features from the target image using a feature extraction network to obtain the corresponding feature map;

[0035] The grasping posture detection module is used to perform grasping posture detection on the feature map and obtain the orientation angle and bounding box regression correction information of the candidate grasping box;

[0036] The semantic segmentation module is used to perform semantic segmentation on the feature map to obtain grasping closure information and grasping region semantic information;

[0037] The information fusion module is used to fuse the captured closure information, the semantic information of the captured region, the orientation angle of the candidate captured box, and the bounding box regression correction information to obtain the captured dataset.

[0038] The grasping control module is used to control the robot to grasp objects based on the grasping dataset.

[0039] In one embodiment of this application, if the feature extraction network includes a residual network and a feature pyramid network, then when the feature extraction module uses the feature extraction network to extract features from the target image, the process of obtaining the corresponding feature map includes:

[0040] The residual network is used to extract features from the target image to obtain a first feature image;

[0041] The feature pyramid network is used to extract features from the target image to obtain a second feature image;

[0042] The first feature image and the second feature image are fused to obtain the feature map.

[0043] In one embodiment of this application, the process by which the grasping posture detection module performs grasping posture detection on the feature map and obtains the orientation angle and bounding box regression correction information of the candidate grasping box includes:

[0044] A region proposal network is used to find candidate capture boxes from the feature map;

[0045] The candidate capture box is calibrated by performing region of interest calibration to obtain the calibrated candidate capture box;

[0046] The calibrated candidate grab box is rotated angularly in the first direction to obtain an axis-aligned rectangle.

[0047] The axis-aligned rectangle is classified as a target, and bounding box regression is performed according to the target classification result. A rotation opposite to the first direction but at the same angle is also performed to obtain the direction angle and bounding box regression correction information of the candidate grabbing box. The bounding box regression correction information includes: the center point coordinates of the candidate grabbing box in the image coordinate system, the width of the candidate grabbing box in the image coordinate system, and the height of the candidate grabbing box in the image coordinate system.

[0048] This application also provides a robot grasping control device, the device comprising:

[0049] One or more processors;

[0050] A storage device for storing one or more programs that, when executed by one or more processors, cause the device to implement the robot grasping control method as described above.

[0051] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the robot grasping control method as described in any of the above-described applications.

[0052] As described above, this application provides a robot grasping control method, apparatus, device, and storage medium, which have the following beneficial effects:

[0053] This application first acquires images containing daily necessities, captured in advance or in real-time, denoted as target images; then, it uses a feature extraction network to extract features from the target images, obtaining corresponding feature maps; next, it performs grasping pose detection on the feature maps to obtain the orientation angle of candidate grasping boxes and bounding box regression correction information; and then, it performs semantic segmentation on the feature maps to obtain grasping closure information and grasping region semantic information; next, it fuses the grasping closure information, the grasping region semantic information, the orientation angle of the candidate grasping boxes, and the bounding box regression correction information to obtain a grasping dataset; finally, it controls the robot to grasp objects based on the grasping dataset. Therefore, this application, by incorporating grasping closure into a deep semantic segmentation neural network grasping pose detection model, fully considers grasping quality while ensuring grasping accuracy and speed, achieving better grasping results. This provides a more accurate grasping pose detection method for daily necessities in life support scenarios. Furthermore, this application proposes to integrate semantic information describing the graspable region of an object into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. Regarding the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure of the grasp and output grasping detection results that meet the required performance. This approach improves both grasping detection accuracy and grasping quality, thereby promoting the development and application of grasping technology for life support robots and better providing life support for elderly people in need of care.

[0054] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0056] Figure 1 This is a schematic diagram illustrating an exemplary system architecture that applies the technical solutions in one or more embodiments of this application;

[0057] Figure 2 This is a flowchart illustrating a robot grasping control method provided in one embodiment of this application;

[0058] Figure 3 This is a schematic diagram illustrating the principle of grasping pose detection provided in one embodiment of this application;

[0059] Figure 4 This is a schematic diagram illustrating the principle of forming a closed grasp according to an embodiment of this application;

[0060] Figure 5 This is a schematic diagram illustrating the principle of a robot grasping control method provided in one embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the hardware structure of a robot grasping control device provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the hardware structure of a robot grasping control device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0063] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0064] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0065] In this application, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0066] The term "multiple" in this application refers to two or more.

[0067] In the description of this application, the terms "first," "second," etc., are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0068] Additionally, in the embodiments of this application, the term "exemplary" is used to indicate that it is an example, illustration, or description. Any embodiment or implementation described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or implementations. Rather, the use of the term "exemplary" is intended to present the concept in a specific manner.

[0069] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0070] Figure 1 A schematic diagram of an exemplary system architecture that can apply the technical solutions of one or more embodiments of this application is shown. Figure 1 As shown, the system architecture 100 may include terminal device 110, network 120, and server 130. Terminal device 110 may include various electronic devices such as smartphones, tablets, laptops, and desktop computers. Server 130 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.

[0071] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided in this application embodiment can be applied to terminal device 110, or to server 130, or can be implemented jointly by terminal device 110 and server 130. This application does not impose any special limitations on this.

[0072] In one embodiment of this application, the terminal device 110 or server 130 can acquire images containing daily necessities captured in advance or in real time, denoted as target images; then, a feature extraction network is used to extract features from the target images to obtain corresponding feature maps; then, grasping posture detection is performed on the feature maps to obtain the orientation angle of candidate grasping boxes and border regression correction information; and, semantic segmentation is performed on the feature maps to obtain grasping closure information and grasping region semantic information; then, the grasping closure information, the grasping region semantic information, the orientation angle of the candidate grasping boxes, and the border regression correction information are fused to obtain a grasping dataset; finally, the robot is controlled to grasp objects based on the grasping dataset. By using the terminal device 110 or server 130 to execute the robot grasping control method, grasping closure can be incorporated into the deep semantic segmentation neural network grasping posture detection model. While ensuring grasping accuracy and speed, grasping quality is fully considered, resulting in better grasping effects. This forms a more accurate grasping posture detection method for daily necessities in life support scenarios. Meanwhile, for the graspable region of an object, the proposed approach integrates semantic information describing the graspable region into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. Regarding the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure properties of the grasp and output grasping detection results that meet these performance requirements. This approach improves both grasping detection accuracy and grasping quality, thereby promoting the development and application of grasping technology in life support robots and better providing life support for elderly people in need of care.

[0073] The above section introduced an exemplary system architecture that applies the technical solution of this application. Next, we will continue to introduce the robot grasping control method of this application.

[0074] Figure 2 A schematic flowchart of a robot grasping control method according to an embodiment of this application is shown. Specifically, in an exemplary embodiment, as follows... Figure 2 As shown, this embodiment provides a robot grasping control method, which includes the following steps:

[0075] S210: Acquire images containing daily necessities, either pre-captured or captured in real-time, and record them as target images;

[0076] S220, the feature extraction network is used to extract features from the target image to obtain the corresponding feature map;

[0077] S230, perform grasping posture detection on the feature map to obtain the orientation angle and bounding box regression correction information of the candidate grasping box; and perform semantic segmentation on the feature map to obtain grasping closure information and grasping region semantic information.

[0078] S240, information fusion is performed on the captured closure information, the captured region semantic information, the orientation angle of the candidate captured box and the bounding box regression correction information to obtain the captured dataset;

[0079] S250, control the robot to grasp objects according to the grasping dataset.

[0080] Therefore, this embodiment, by incorporating grasping closure into the deep semantic segmentation neural network grasping pose detection model, fully considers grasping quality while ensuring grasping accuracy and speed, achieving better grasping results. This results in a more accurate grasping pose detection method for everyday items in life support scenarios. Furthermore, this embodiment proposes integrating semantic information describing the graspable region of an object into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. Regarding grasping closure, a shape closure algorithm is combined with the grasping pose detection network, allowing the network to fully learn the grasping closure and output grasping detection results that meet the required performance. This approach improves both grasping detection accuracy and grasping quality, thereby promoting the development and application of grasping technology in life support robots and better providing life support for elderly people in need of care.

[0081] In an exemplary embodiment, if the feature extraction network includes a residual network and a feature pyramid network, then the process of obtaining a corresponding feature map when using the feature extraction network to extract features from the target image includes: extracting features from the target image using the residual network to obtain a first feature image; extracting features from the target image using the feature pyramid network to obtain a second feature image; and fusing the first feature image and the second feature image to obtain the feature map. As an example, in this embodiment, a residual network (ResNet) combined with a feature pyramid network (FPN) can be used for feature extraction, and then multi-scale feature fusion can be performed to obtain a feature map with more detailed information. In this embodiment, the residual network can be ResNet-101.

[0082] In an exemplary embodiment, the process of detecting the grasping posture of the feature map and obtaining the orientation angle and bounding box regression correction information of the candidate grasping box includes: using a region proposal network to find candidate grasping boxes from the feature map; performing region of interest calibration on the candidate grasping boxes to obtain calibrated candidate grasping boxes; rotating the calibrated candidate grasping boxes by an angle in a first direction to obtain an axis-aligned rectangle; classifying the axis-aligned rectangles as objects, performing bounding box regression according to the object classification results, and performing a rotation opposite to the first direction but with the same angle to obtain the orientation angle and bounding box regression correction information of the candidate grasping boxes; wherein, the bounding box regression correction information includes: the center point coordinates of the candidate grasping box in the image coordinate system, the width of the candidate grasping box in the image coordinate system, and the height of the candidate grasping box in the image coordinate system. In this embodiment, the process of classifying the axis-aligned rectangle to obtain the target classification result includes: obtaining the angle interval corresponding to the calibrated candidate grasping box when rotating in the first direction, denoted as the first angle interval; superimposing the first angle interval to generate a second angle interval; uniformly quantizing the second angle interval to divide it into multiple sub-intervals, and taking each sub-interval as a target category, denoted as the first target category; wherein, the angle interval between each sub-interval is the same; obtaining the blank grasping area in the feature map, and taking the blank grasping area as a separate target category, denoted as the second target category; superimposing the first target category and the second target category, and taking the target category superposition result as the target classification result of the axis-aligned rectangle. Specifically, in the target classification stage of this embodiment, regarding the direction angle problem of the grasping box, if the angle interval for rotating in the first direction is , the angle interval is first processed into a new angle interval by adding, and then uniform quantization is used to divide the new angle interval into 18 sub-intervals, each sub-interval spaced 10, corresponding to an angle category, i.e., 0-17. In addition, for cases where there is no grabbable region in the image, an ungrabable class is added, resulting in a total of 19 target categories. These 19 target categories are then used as the target classification results for axis-aligned rectangles.

[0083] In an exemplary embodiment, the process of semantically segmenting the feature map to obtain grasping closure information and grasping region semantic information includes: acquiring the robot's claws and establishing a grasping coordinate system with the center point of the claws as the origin; when the object to be grasped translates within a first interval on the x-axis of the grasping coordinate system, and at least one finger of the claws contacts the object to be grasped, recording the distance value between the two claws of the robot, and recording the angle between each claw and the object to be grasped; filtering out the minimum value of the region from the distance values ​​between the two claws of the robot, and generating the grasping closure information based on the filtered minimum value of the region and the angle between each claw and the object to be grasped; and performing semantic segmentation on the grasping closure information using a watershed algorithm to obtain the grasping region semantic information. Regarding the grasping closure property, the overall technical concept of this embodiment is to obtain the grasping positions on the object that satisfy the closure property through the object's shape contour features. Therefore, the shape closure theory is used to generate grasping positions that satisfy the grasping closure property, i.e., the annotation information of the grasping rectangle, by using the semantic information of the graspable area of ​​the object in the two-dimensional plane through the watershed algorithm. Specifically, it can be described as follows: Figure 4 As shown, the entire grasping coordinate system has its origin at the center point of the hand. Assuming the object translates along the x-axis within this coordinate system, when at least one finger on each side of the hand contacts the object, the distance between the fingers and the angle between the hand and the object are recorded. After the entire process is completed, the local minimum value needs to be calculated, thus obtaining the unique, closed grasping position on the object that satisfies the shape closure condition. This generates a grasping dataset containing semantic information of the object's graspable region and 5D parameter information of the grasping rectangle.

[0084] According to the above description, in another exemplary embodiment of this application, this embodiment provides a robot grasping control method, including the following steps:

[0085] First, based on the performance requirements of grasping pose detection, Faster R-CNN (Faster Region-based Convolutional Neural Network or Faster Regions With Convolutional Neural Network Features, abbreviated as Faster R-CNN) was selected as the model foundation. After a series of improvements, a grasping pose detection model based on Faster R-CNN was proposed. The principle structure diagram of this grasping pose detection model is shown below. Figure 3As shown. This grasping pose detection model represents the grasping pose using 5-dimensional parameters of a 2D grasping rectangle. It treats the orientation angle representing the grasping posture as a target classification prediction problem, and the coordinates of the rectangle's center point, width, and height representing the grasping position as a regression prediction problem. Specific improvements are as follows: ① In the feature extraction network, a ResNet-101 (Residual Network, ResNet for short) combined with a Feature Pyramid Network (FPN, FPN) is used for feature extraction. Through multi-scale feature fusion, more detailed information is obtained. ② In the region proposal network and the target classification and bounding box regression correction stages, considering that in the Faster R-CNN target detection model, the Region Proposal Network (RPN) and the final target classification and bounding box regression process use axis-aligned rectangles (i.e., these rectangles have no orientation angle), while grasping pose detection uses oriented rectangles to represent the grasping pose. To address this, the grasping rectangle is rotated to obtain an axis-aligned rectangle, and then the orientation angle of the grasping rectangle is treated as a target classification problem. Therefore, the axis-aligned rectangle is still used for region extraction and bounding box regression. Finally, the candidate boxes output by the network are rotated in the opposite direction to obtain the final oriented candidate grasping boxes. ③ In the pooling stage, RoI Align is used instead of RoI Pooling to reduce quantization errors in the candidate box size transformation operation. ④ In the target classification stage, regarding the orientation angle of the grasping box, the orientation angle range is...

[0086] First, the angle intervals are processed into new angle intervals by adding them. Then, a uniform quantization operation is used to divide the new angle intervals into 18 sub-intervals, each with a 10-degree interval, corresponding to an angle category, i.e., 0-17. In addition, for cases where there are no graspable regions in the image, an ungraspable category is added, resulting in a total of 19 target categories.

[0087] Secondly, based on the research scenario of life support robots, a grasping dataset containing various daily necessities is constructed within a local life support scenario. This dataset considers both the semantic information describing the shape contour of the graspable area of ​​the object and the closure of the grasping mechanism. For semantic information acquisition, the graspable areas of the object are manually annotated semantically based on human grasping experience. Regarding grasping closure, this embodiment aims to obtain the grasping positions on the object that satisfy the closure property through the object's shape contour features. Therefore, shape closure theory is used to generate grasping positions that satisfy the closure property in a two-dimensional plane using the watershed algorithm, i.e., the annotation information of the grasping rectangle. For example... Figure 4 As shown, the specific process can be described as follows: A grasping coordinate system is established with the center point of the hand as the origin. Assuming the object translates within this coordinate system along the x-axis, when at least one finger on each side of the hand contacts the object, the distance between the fingers on both sides and the angle between the hand and the object are recorded. After the entire process is completed, the local minimum value among all distance values ​​is calculated. This yields the unique, closed grasping position on the object that satisfies the shape closure condition, thus generating a grasping dataset containing semantic information about the graspable region of the object and 5-dimensional parameter information of the grasping rectangle.

[0088] Finally, considering the poor grasping pose detection results of the Faster R-CNN-based model for small objects and object details, this embodiment designs an improved grasping pose detection network that integrates semantic information, such as... Figure 5 As shown, this network adds a parallel semantic segmentation module to the grasping pose detection model based on Faster R-CNN. This module is used to segment and predict the graspable region of the object. Then, the detection results of the two branches are fused and refined by an information fusion and refinement module based on a multilayer perceptron (MLP). This combines the semantic information of the graspable region of the object with the grasping closure property. By training on the generated grasping dataset, the network outputs grasping detection results with higher detection accuracy and while satisfying the grasping closure property.

[0089] Therefore, the grasping pose detection module in this embodiment is based on the classic Faster R-CNN algorithm, with a series of improvements. The feature extraction network uses ResNet-101 combined with the Feature Pyramid Network (FPN) for feature extraction, which yields more detailed information, crucial for grasping everyday items, especially small objects. ROI Align is used for pooling to reduce quantization errors, further ensuring accurate and reliable grasping operations. Treating the grasping angle as a target classification problem instead of using directional anchor boxes reduces network computation. Secondly, a localized grasping dataset containing various everyday items is designed for the grasping task of the life support robot. This dataset considers both the semantic information describing the graspable area's outline and the closure of the grasping action, providing a large number of data samples for the network. After training, the network learns more accurate grasping poses that satisfy the closure requirement. Finally, the improved grasping pose detection network, which integrates semantic information, incorporates pixel-level semantic information about the graspable area, better considering small objects and object details, achieving better grasping detection results.

[0090] In summary, this application provides a robot grasping control method. First, it acquires an image containing daily necessities, captured in advance or in real-time, denoted as the target image. Then, it uses a feature extraction network to extract features from the target image, obtaining a corresponding feature map. Next, it performs grasping pose detection on the feature map, obtaining the orientation angle of candidate grasping boxes and bounding box regression correction information. Then, it performs semantic segmentation on the feature map, obtaining grasping closure information and grasping region semantic information. Finally, it fuses the grasping closure information, the grasping region semantic information, the orientation angle of the candidate grasping boxes, and the bounding box regression correction information to obtain a grasping dataset. Finally, it controls the robot to grasp objects based on the grasping dataset. Therefore, this method, by incorporating grasping closure into a deep semantic segmentation neural network grasping pose detection model, fully considers grasping quality while ensuring grasping accuracy and speed, achieving better grasping results. This provides a more accurate grasping pose detection method for daily necessities in life support scenarios. Meanwhile, this method proposes to integrate semantic information describing the graspable region of an object into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. For the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure of the grasp and output grasping detection results that meet the required performance. This achieves both improved grasping detection accuracy and full consideration of grasping quality, thereby promoting the development and application of grasping technology for life support robots and better providing life support for elderly people in need of care. Essentially, this method addresses existing technical problems by researching grasping pose detection technology for life support robots and designs a robot grasping pose detection model based on the fusion of deep semantic segmentation and object detection. For the graspable region of an object, this paper proposes to integrate the semantic information describing the graspable region into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. For the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure of the grasp and output grasping detection results that meet this performance requirement. This achieves both improved grasping detection accuracy and full consideration of grasping quality, thereby promoting the development and application of grasping technology in life support robots. Furthermore, this method integrates grasping closure into the grasping pose detection algorithm, taking into account both grasping quality and accuracy. Moreover, based on shape closure theory, a watershed algorithm is used to generate a grasping position annotation method that satisfies grasping closure for the semantic information of the graspable region in the two-dimensional plane. Simultaneously, this method generates a grasping dataset for life support scenarios containing semantic information of the graspable region and grasping closure annotations. By training on the generated grasping dataset, higher detection accuracy and grasping closure satisfaction can be achieved.

[0091] like Figure 6 As shown, this application also provides a robot grasping control device, the device comprising:

[0092] Image acquisition module 610 is used to acquire images containing daily necessities, either pre-captured or captured in real time, and is denoted as target image;

[0093] Feature extraction module 620 is used to extract features from the target image using a feature extraction network to obtain a corresponding feature map;

[0094] The grasping posture detection module 630 is used to perform grasping posture detection on the feature map and obtain the orientation angle and bounding box regression correction information of the candidate grasping box.

[0095] The semantic segmentation module 640 is used to perform semantic segmentation on the feature map to obtain grasping closure information and grasping region semantic information;

[0096] The information fusion module 650 is used to fuse the crawling closure information, the semantic information of the crawling region, the orientation angle of the candidate crawling box, and the bounding box regression correction information to obtain the crawling dataset.

[0097] The grasping control module 660 is used to control the robot to grasp objects based on the grasping dataset.

[0098] Therefore, this embodiment, by incorporating grasping closure into the deep semantic segmentation neural network grasping pose detection model, fully considers grasping quality while ensuring grasping accuracy and speed, achieving better grasping results. This results in a more accurate grasping pose detection method for everyday items in life support scenarios. Furthermore, this embodiment proposes integrating semantic information describing the graspable region of an object into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. Regarding grasping closure, a shape closure algorithm is combined with the grasping pose detection network, allowing the network to fully learn the grasping closure and output grasping detection results that meet the required performance. This approach improves both grasping detection accuracy and grasping quality, thereby promoting the development and application of grasping technology in life support robots and better providing life support for elderly people in need of care.

[0099] In an exemplary embodiment, if the feature extraction network includes a residual network and a feature pyramid network, then when the feature extraction module uses the feature extraction network to extract features from the target image, the process of obtaining the corresponding feature map includes: using the residual network to extract features from the target image to obtain a first feature image; using the feature pyramid network to extract features from the target image to obtain a second feature image; and fusing the first feature image and the second feature image to obtain the feature map. As an example, in this embodiment, the feature extraction network can use a residual network (ResNet) combined with a feature pyramid network (FPN) for feature extraction, and then obtain a feature map with more detailed information through multi-scale feature fusion. In this embodiment, the residual network can be ResNet-101.

[0100] In an exemplary embodiment, the process by which the grasping posture detection module detects the grasping posture of the feature map and obtains the orientation angle and bounding box regression correction information of candidate grasping boxes includes: using a region proposal network to find candidate grasping boxes from the feature map; performing region of interest calibration on the candidate grasping boxes to obtain calibrated candidate grasping boxes; rotating the calibrated candidate grasping boxes by an angle in a first direction to obtain axis-aligned rectangles; classifying the axis-aligned rectangles as targets, performing bounding box regression according to the target classification results, and performing a rotation opposite to the first direction but with the same angle to obtain the orientation angle and bounding box regression correction information of the candidate grasping boxes; wherein, the bounding box regression correction information includes: the center point coordinates of the candidate grasping box in the image coordinate system, the width of the candidate grasping box in the image coordinate system, and the height of the candidate grasping box in the image coordinate system. In this embodiment, the process of classifying the axis-aligned rectangle to obtain the target classification result includes: obtaining the angle interval corresponding to the calibrated candidate grasping box when rotating in the first direction, denoted as the first angle interval; superimposing the first angle interval to generate a second angle interval; uniformly quantizing the second angle interval to divide it into multiple sub-intervals, and taking each sub-interval as a target category, denoted as the first target category; wherein, the angle interval between each sub-interval is the same; obtaining the blank grasping area in the feature map, and taking the blank grasping area as a separate target category, denoted as the second target category; superimposing the first target category and the second target category, and taking the target category superposition result as the target classification result of the axis-aligned rectangle. Specifically, in the target classification stage of this embodiment, regarding the direction angle problem of the grasping box, if the angle interval for rotating in the first direction is , the angle interval is first processed into a new angle interval by adding, and then uniform quantization is used to divide the new angle interval into 18 sub-intervals, each sub-interval spaced 10, corresponding to an angle category, i.e., 0-17. In addition, for cases where there is no grabbable region in the image, an ungrabable class is added, resulting in a total of 19 target categories. These 19 target categories are then used as the target classification results for axis-aligned rectangles.

[0101] In an exemplary embodiment, the process of semantically segmenting the feature map to obtain grasping closure information and grasping region semantic information includes: acquiring the robot's claws and establishing a grasping coordinate system with the center point of the claws as the origin; when the object to be grasped translates within a first interval on the x-axis of the grasping coordinate system, and at least one finger of the claws contacts the object to be grasped, recording the distance value between the two claws of the robot, and recording the angle between each claw and the object to be grasped; filtering out the minimum value of the region from the distance values ​​between the two claws of the robot, and generating the grasping closure information based on the filtered minimum value of the region and the angle between each claw and the object to be grasped; and performing semantic segmentation on the grasping closure information using a watershed algorithm to obtain the grasping region semantic information. Regarding the grasping closure property, the overall technical concept of this embodiment is to obtain the grasping positions on the object that satisfy the closure property through the object's shape contour features. Therefore, the shape closure theory is used to generate grasping positions that satisfy the grasping closure property, i.e., the annotation information of the grasping rectangle, by using the semantic information of the graspable area of ​​the object in the two-dimensional plane through the watershed algorithm. Specifically, it can be described as follows: Figure 4 As shown, the entire grasping coordinate system has its origin at the center point of the hand. Assuming the object translates along the x-axis within this coordinate system, when at least one finger on each side of the hand contacts the object, the distance between the fingers and the angle between the hand and the object are recorded. After the entire process is completed, the local minimum value needs to be calculated, thus obtaining the unique, closed grasping position on the object that satisfies the shape closure condition. This generates a grasping dataset containing semantic information of the object's graspable region and 5D parameter information of the grasping rectangle.

[0102] According to the above description, in another exemplary embodiment of this application, this embodiment provides a robot grasping control device for performing the following steps:

[0103] First, based on the performance requirements of grasping pose detection, Faster R-CNN (Faster Region-based Convolutional Neural Network or Faster Regions With Convolutional Neural Network Features, abbreviated as Faster R-CNN) was selected as the model foundation. After a series of improvements, a grasping pose detection model based on Faster R-CNN was proposed. The principle structure diagram of this grasping pose detection model is shown below. Figure 3As shown. This grasping pose detection model represents the grasping pose using 5-dimensional parameters of a 2D grasping rectangle. It treats the orientation angle representing the grasping posture as a target classification prediction problem, and the coordinates of the rectangle's center point, width, and height representing the grasping position as a regression prediction problem. Specific improvements are as follows: ① In the feature extraction network, a ResNet-101 (Residual Network, ResNet for short) combined with a Feature Pyramid Network (FPN, FPN) is used for feature extraction. Through multi-scale feature fusion, more detailed information is obtained. ② In the region proposal network and the target classification and bounding box regression correction stages, considering that in the Faster R-CNN target detection model, the Region Proposal Network (RPN) and the final target classification and bounding box regression process use axis-aligned rectangles (i.e., these rectangles have no orientation angle), while grasping pose detection uses oriented rectangles to represent the grasping pose. To address this, the grasping rectangle is rotated to obtain an axis-aligned rectangle, and then the orientation angle of the grasping rectangle is treated as a target classification problem. Therefore, the axis-aligned rectangle is still used for region extraction and bounding box regression. Finally, the candidate boxes output by the network are rotated in the opposite direction to obtain the final oriented candidate grasping boxes. ③ In the pooling stage, RoI Align is used instead of RoI Pooling to reduce quantization errors in the candidate box size transformation operation. ④ In the target classification stage, regarding the orientation angle of the grasping box, the orientation angle range is...

[0104] First, the angle intervals are processed into new angle intervals by adding them. Then, a uniform quantization operation is used to divide the new angle intervals into 18 sub-intervals, each with a 10-degree interval, corresponding to an angle category, i.e., 0-17. In addition, for cases where there are no graspable regions in the image, an ungraspable category is added, resulting in a total of 19 target categories.

[0105] Secondly, based on the research scenario of life support robots, a grasping dataset containing various daily necessities is constructed within a local life support scenario. This dataset considers both the semantic information describing the shape contour of the graspable area of ​​the object and the closure of the grasping mechanism. For semantic information acquisition, the graspable areas of the object are manually annotated semantically based on human grasping experience. Regarding grasping closure, this embodiment aims to obtain the grasping positions on the object that satisfy the closure property through the object's shape contour features. Therefore, shape closure theory is used to generate grasping positions that satisfy the closure property in a two-dimensional plane using the watershed algorithm, i.e., the annotation information of the grasping rectangle. For example... Figure 4 As shown, the specific process can be described as follows: A grasping coordinate system is established with the center point of the hand as the origin. Assuming the object translates within this coordinate system along the x-axis, when at least one finger on each side of the hand contacts the object, the distance between the fingers on both sides and the angle between the hand and the object are recorded. After the entire process is completed, the local minimum value among all distance values ​​is calculated. This yields the unique, closed grasping position on the object that satisfies the shape closure condition, thus generating a grasping dataset containing semantic information about the graspable region of the object and 5-dimensional parameter information of the grasping rectangle.

[0106] Finally, considering the poor grasping pose detection results of the Faster R-CNN-based model for small objects and object details, this embodiment designs an improved grasping pose detection network that integrates semantic information, such as... Figure 5 As shown, this network adds a parallel semantic segmentation module to the grasping pose detection model based on Faster R-CNN. This module is used to segment and predict the graspable region of the object. Then, the detection results of the two branches are fused and refined by an information fusion and refinement module based on a multilayer perceptron (MLP). This combines the semantic information of the graspable region of the object with the grasping closure property. By training on the generated grasping dataset, the network outputs grasping detection results with higher detection accuracy and while satisfying the grasping closure property.

[0107] Therefore, the grasping pose detection module in this embodiment is based on the classic Faster R-CNN algorithm, with a series of improvements. The feature extraction network uses ResNet-101 combined with the Feature Pyramid Network (FPN) for feature extraction, which yields more detailed information, crucial for grasping everyday items, especially small objects. ROI Align is used for pooling to reduce quantization errors, further ensuring accurate and reliable grasping operations. Treating the grasping angle as a target classification problem instead of using directional anchor boxes reduces network computation. Secondly, a localized grasping dataset containing various everyday items is designed for the grasping task of the life support robot. This dataset considers both the semantic information describing the graspable area's outline and the closure of the grasping action, providing a large number of data samples for the network. After training, the network learns more accurate grasping poses that satisfy the closure requirement. Finally, the improved grasping pose detection network, which integrates semantic information, incorporates pixel-level semantic information about the graspable area, better considering small objects and object details, achieving better grasping detection results.

[0108] In summary, this application provides a robot grasping control device. First, it acquires an image containing daily necessities, captured in advance or in real-time, designated as the target image. Then, it uses a feature extraction network to extract features from the target image, obtaining a corresponding feature map. Next, it performs grasping posture detection on the feature map, obtaining the orientation angle of candidate grasping boxes and bounding box regression correction information. Then, it performs semantic segmentation on the feature map, obtaining grasping closure information and grasping region semantic information. Finally, it fuses the grasping closure information, the grasping region semantic information, the orientation angle of the candidate grasping boxes, and the bounding box regression correction information to obtain a grasping dataset. Finally, it controls the robot to grasp objects based on the grasping dataset. Therefore, this device, by incorporating grasping closure into a deep semantic segmentation neural network grasping posture detection model, fully considers grasping quality while ensuring grasping accuracy and speed, achieving better grasping results. This provides a more accurate grasping posture detection method for daily necessities in life support scenarios. Meanwhile, this device proposes to integrate semantic information describing the graspable region of an object into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. Regarding the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure of the grasp and output grasping detection results that meet the required performance. This achieves both improved grasping detection accuracy and full consideration of grasping quality, thereby promoting the development and application of grasping technology for life support robots and better providing life support for elderly people in need of care. Essentially, this device addresses existing technical problems by researching grasping pose detection technology for life support robots and designing a robot grasping pose detection model based on the fusion of deep semantic segmentation and object detection. For the graspable region of an object, this paper proposes to integrate the semantic information describing the graspable region into the grasping pose detection network to achieve better grasping detection results and generate more accurate grasping poses. For the closure of the grasp, a shape closure algorithm is combined with the grasping pose detection network, enabling the network to fully learn the closure of the grasp and output grasping detection results that meet this performance requirement. This achieves both improved grasping detection accuracy and full consideration of grasping quality, thereby promoting the development and application of grasping technology in life support robots. Furthermore, this device integrates grasping closure into the grasping pose detection algorithm, taking into account both grasping quality and accuracy. Moreover, based on shape closure theory, a watershed algorithm is used to generate a grasping position annotation method that satisfies grasping closure in the two-dimensional plane by analyzing the semantic information of the graspable region. Simultaneously, this device generates a grasping dataset for life support scenarios containing semantic information of the graspable region and grasping closure annotations. By training on the generated grasping dataset, higher detection accuracy and grasping closure satisfaction can be achieved in the output of grasping detection results.

[0109] It should be noted that the robot grasping control device provided in the above embodiments and the robot grasping control method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the robot grasping control device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0110] Embodiments of this application also provide a robot grasping control device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the robot grasping control device to implement the robot grasping control methods provided in the above embodiments.

[0111] Figure 7 A schematic diagram of a computer device suitable for implementing the robot grasping control device of the embodiments of this application is shown. It should be noted that... Figure 7 The computer system 1000 for robot grasping control shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 7 As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage portion 1008 into Random Access Memory (RAM) 1003, such as performing the methods described in the above embodiments. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004.

[0113] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0114] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs the various functions defined in the apparatus of this application.

[0115] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0118] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the robot grasping control method as described above. This computer-readable storage medium may be included in the robot grasping control device described in the above embodiments, or it may exist independently and not incorporated into the robot grasping control device.

[0119] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the robot grasping control method provided in the various embodiments described above.

[0120] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A robot grasping control method, characterized in that, The method includes the following steps: Acquire images containing daily necessities, either pre-captured or captured in real-time, and record them as target images; The target image is used to extract features using a feature extraction network to obtain the corresponding feature map; The feature map is subjected to grasping pose detection to obtain the orientation angle and bounding box regression correction information of candidate grasping boxes; and semantic segmentation is performed on the feature map to obtain grasping closure information and grasping region semantic information; wherein, the process of grasping pose detection on the feature map to obtain the orientation angle and bounding box regression correction information of candidate grasping boxes includes: using a region proposal network to find candidate grasping boxes from the feature map; performing region of interest calibration on the candidate grasping boxes to obtain calibrated candidate grasping boxes; rotating the calibrated candidate grasping boxes by an angle in a first direction to obtain axis-aligned rectangles; classifying the axis-aligned rectangles for targets, and performing bounding box regression according to the target classification results, and performing a rotation opposite to the first direction but with the same angle to obtain the orientation angle and bounding box regression correction information of the candidate grasping boxes; wherein, the bounding box regression correction information includes: the center point coordinates of the candidate grasping box in the image coordinate system, the width of the candidate grasping box in the image coordinate system, and the height of the candidate grasping box in the image coordinate system; The crawling closure information, the semantic information of the crawling region, the orientation angle of the candidate crawling box, and the bounding box regression correction information are fused to obtain the crawling dataset. The robot is controlled to grasp objects based on the grasping dataset.

2. The robot grasping control method according to claim 1, characterized in that, If the feature extraction network includes a residual network and a feature pyramid network, then when using the feature extraction network to extract features from the target image, the process of obtaining the corresponding feature map includes: The residual network is used to extract features from the target image to obtain a first feature image; The feature pyramid network is used to extract features from the target image to obtain a second feature image; The first feature image and the second feature image are fused to obtain the feature map.

3. The robot grasping control method according to claim 1 or 2, characterized in that, The process of classifying the target using the axis-aligned rectangle to obtain the target classification result includes: The angle range corresponding to the calibrated candidate capture box when it is rotated in the first direction is denoted as the first angle range; The first angle interval is superimposed to generate the second angle interval; The second angle interval is uniformly quantized, dividing the second angle interval into multiple sub-intervals, and each sub-interval is treated as a target category, denoted as the first target category; wherein, each sub-interval is spaced at the same angle. Obtain the blank grasping region in the feature map, and treat the blank grasping region as a separate target category, denoted as the second target category; The first target category and the second target category are superimposed in terms of quantity, and the result of the superimposed target categories is used as the target classification result of the axis-aligned rectangle.

4. The robot grasping control method according to claim 1, characterized in that, The process of semantic segmentation of the feature map to obtain grasping closure information and grasping region semantic information includes: Obtain the robot's gripper and establish a grasping coordinate system with the center point of the gripper as the origin; When the object to be grasped is translated within the first interval on the x-axis in the grasping coordinate system, and at least one finger of the gripper contacts the object to be grasped, the distance between the two grippers in the robot is recorded, as well as the angle between each gripper and the object to be grasped is recorded. The minimum value of the region is selected from the distance values ​​between the two grippers of the robot, and the grasping closure information is generated based on the selected minimum value of the region and the angle between each gripper and the object to be grasped; and The semantic information of the crawled region is obtained by semantic segmentation of the crawled closed information using the watershed algorithm.

5. A robot grasping control device, characterized in that, The device includes: The image acquisition module is used to acquire images containing daily necessities, either pre-captured or captured in real time, and these images are denoted as target images. The feature extraction module is used to extract features from the target image using a feature extraction network to obtain the corresponding feature map; A grasping posture detection module is used to detect the grasping posture of the feature map and obtain the orientation angle and bounding box regression correction information of candidate grasping boxes. This includes: finding candidate grasping boxes from the feature map using a region proposal network; calibrating the candidate grasping boxes using a region of interest to obtain calibrated candidate grasping boxes; rotating the calibrated candidate grasping boxes by an angle in a first direction to obtain axis-aligned rectangles; classifying the axis-aligned rectangles as objects and performing bounding box regression according to the object classification results; and performing a rotation opposite to the first direction but with the same angle to obtain the orientation angle and bounding box regression correction information of the candidate grasping boxes. The bounding box regression correction information includes: the center point coordinates of the candidate grasping box in the image coordinate system, the width of the candidate grasping box in the image coordinate system, and the height of the candidate grasping box in the image coordinate system. The semantic segmentation module is used to perform semantic segmentation on the feature map to obtain grasping closure information and grasping region semantic information; The information fusion module is used to fuse the captured closure information, the semantic information of the captured region, the orientation angle of the candidate captured box, and the bounding box regression correction information to obtain the captured dataset. The grasping control module is used to control the robot to grasp objects based on the grasping dataset.

6. The robot grasping control device according to claim 5, characterized in that, If the feature extraction network includes a residual network and a feature pyramid network, then when the feature extraction module uses the feature extraction network to extract features from the target image, the process of obtaining the corresponding feature map includes: The residual network is used to extract features from the target image to obtain a first feature image; The feature pyramid network is used to extract features from the target image to obtain a second feature image; The first feature image and the second feature image are fused to obtain the feature map.

7. A robot grasping control device, characterized in that, The device includes: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the device to implement the robot grasping control method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the robot grasping control method as described in any one of claims 1 to 4.

Images