Mechanical arm grabbing method and equipment based on few-sample semantic segmentation model and medium

Abstract

The invention discloses a mechanical arm grabbing method and equipment based on a few-sample semantic segmentation model and a medium, and relates to the technical field of computer systems based on specific calculation models. The method comprises the following steps: constructing a support set according to a scene image containing a target object and an annotation image containing multiple types of target objects; performing object segmentation on the input scene image to obtain a candidate mask image set corresponding to a plurality of segmented objects; in response to the grabbing instruction, determining a target mask corresponding to the specified segmentation object according to the candidate mask image set and the support set based on the few-sample segmentation device, and generating a mask thermodynamic diagram corresponding to the target mask; inputting the mask thermodynamic diagram and a depth map corresponding to the input scene image into a UNet network to determine a grabbing point and a grabbing attitude parameter corresponding to the specified segmented object; and according to the grabbing points and the grabbing posture parameters, the mechanical arm is controlled to execute the corresponding grabbing action, and grabbing of the designated segmented object is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer systems based on specific computing models, in particular to a mechanical arm grabbing method based on a few-shot semantic segmentation model, equipment and a medium. BACKGROUND

[0002] Robot grabbing action is one of the most basic and important capabilities of robots. Traditional robot grabbing technology usually relies on geometric information, depth images, point cloud data, etc. to generate a grabbing pose. However, such methods are difficult to achieve identification and grabbing of specific target objects in a scene where objects are randomly stacked. Although a semantic segmentation model can be used to segment images to assist object identification, the semantic segmentation model needs a large amount of labeled data for training and has poor generalization, making it difficult to handle specific target objects that have not been seen before.

[0003] Existing few-shot semantic segmentation models (such as ASGNet and GSMR-CNN) alleviate the dependence on labeled data to some extent, but their segmentation performance still decreases significantly when the test data and the training data are very different. Although a fine-tuning strategy can be used to improve the success rate of grabbing in a specific scene, this approach often weakens the generalization ability of the semantic segmentation model, and the quality of the segmentation mask is unstable, affecting the accuracy of subsequent grabbing operations. SUMMARY

[0004] To solve the above problems, the present application provides a mechanical arm grabbing method based on a few-shot semantic segmentation model, comprising: According to a scene image containing a target object and a labeled image containing multiple types of target objects, a support set is constructed; wherein the support set includes Is masks corresponding to each target object category; An input scene image to be identified is input into a SAM model, and the SAM model is used to segment the input scene image to obtain a candidate mask image set corresponding to a plurality of segmented objects; A grabbing instruction for a specified segmented object is received, and in response to the grabbing instruction, a few-shot segmenter is used to determine a target mask corresponding to the specified segmented object based on the candidate mask image set and the support set, and generate a mask heat map corresponding to the target mask; The mask heat map and the depth map corresponding to the input scene image are input into a UNet network to determine the grabbing point and the grabbing pose parameters corresponding to the specified segmented object; According to the grabbing point and the grabbing pose parameters, the mechanical arm is controlled to perform a corresponding grabbing action to achieve grabbing of the specified segmented object.

[0005] In one implementation of the present invention, determining the target mask corresponding to the specified segmented object based on the candidate mask image set and the support set specifically includes: Extract the candidate mask feature vector corresponding to each candidate mask image in the candidate mask image set, and determine the pseudo-class set corresponding to each segmented object based on the candidate mask feature vector; Extract the feature vector corresponding to each image in the support set, and determine the average feature set composed of the average feature vectors corresponding to various target objects based on the feature vectors; For each segmented object, the candidate mask feature vector corresponding to the segmented object in the pseudo-category set is matched with the average feature set to determine the target object category corresponding to the segmented object; The target mask corresponding to the specified segmented object is determined based on the mapping relationship between the target object category and the candidate mask feature vector.

[0006] In one implementation of the present invention, the candidate mask feature vectors corresponding to the segmented objects in the pseudo-category set are matched with the average feature set to determine the target object category corresponding to the segmented object, specifically including: The candidate mask feature vectors corresponding to the segmented objects in the pseudo-category set are matched with the average feature set to calculate the cosine similarity between the candidate mask feature vectors and the average feature vectors corresponding to each type of target object in the average feature set. From the average feature set, the target object corresponding to the average feature vector with the highest cosine similarity is selected as the target object category corresponding to the segmented object.

[0007] In one implementation of the present invention, the mask heatmap and the depth map corresponding to the input scene image are input into a UNet network to determine the grasping point and grasping posture parameters corresponding to the specified segmented object, specifically including: The mask heatmap and the depth map corresponding to the input scene image are input into the UNet network. Based on different branches of the UNet network, a quality heatmap, an angle heatmap, and a width heatmap are output. The grab point corresponding to the specified segmented object is determined using the mass heat map. The angle and width values ​​corresponding to the grab point are extracted from the angle heatmap and the width heatmap, respectively, and the angle and width values ​​are used as the grab posture parameters corresponding to the specified segmented object.

[0008] In one implementation of the present invention, determining the gripping point corresponding to the specified segmented object using the mass heatmap specifically includes: Based on the pixel values ​​corresponding to each pixel in the mass heatmap, a target pixel value is determined; wherein, the target pixel value is the product of the maximum pixel value among the pixel values ​​and a preset ratio; From the mass heatmap, select a target region where the pixel value of each pixel is greater than the target pixel value, and use the center point of the target region as the grab point corresponding to the specified segmented object.

[0009] In one implementation of the present invention, the robotic arm is controlled to perform a corresponding grasping action based on the grasping point and the grasping posture parameters, specifically including: The coordinates of the gripping points are transformed, and the rotation angles of each servo motor in the robotic arm are determined based on the transformed global coordinates and the gripping posture parameters. A corresponding pulse signal is generated based on the rotation angle, and the servo motor is controlled to move by the pulse signal until it reaches the gripping point. Then, the robotic arm is controlled to perform the corresponding gripping action according to the gripping posture parameters.

[0010] In one implementation of the present invention, the input scene image is segmented using the SAM model to obtain a set of candidate mask images corresponding to several segmented objects, specifically including: The SAM model includes an image encoder, a cue encoder, and a mask encoder; The input scene image is encoded using the image encoder to obtain an image code, and a prompt position code corresponding to the input scene image is constructed using the prompt encoder; wherein, the prompt position code is used to instruct the SAM model to perform object segmentation; The image encoding and the prompt location encoding are fused to obtain fused features; The mask encoder outputs candidate masks corresponding to each fusion feature, and obtains a set of candidate mask images corresponding to several segmented objects.

[0011] In one implementation of the present invention, the prompt encoder constructs a prompt position code corresponding to the input scene image, specifically including: The input scene image is divided into multiple uniformly distributed grid points using the prompt encoder. For each grid point, the coordinates of the grid point are converted into the corresponding prompt location code.

[0012] This invention provides a robotic arm grasping device based on a few-shot semantic segmentation model, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the robotic arm grasping method based on the few-shot semantic segmentation model as described in any of the preceding claims.

[0013] This invention provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows: The robotic arm grasping method based on a few-shot semantic segmentation model, as described in any of the preceding items.

[0014] The robotic arm grasping method based on a few-shot semantic segmentation model proposed in this invention can bring the following beneficial effects: By automatically generating candidate mask images using the SAM model, the reliance on target object training data is avoided, enabling the perception of unknown objects without training and achieving preliminary segmentation of any object. Combining a few-shot classifier with a support set, target objects can be accurately identified with only a small number of labeled samples, avoiding the decline in generalization ability caused by model fine-tuning and significantly improving the adaptive recognition capability for unknown segmented objects. Furthermore, fusing mask heatmaps and depth maps, and using the UNet network to predict grasping pose parameters, effectively improves grasping accuracy. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart illustrating the robotic arm grasping method based on a few-shot semantic segmentation model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a robotic arm grasping device based on a few-sample semantic segmentation model provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0017] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0018] like Figure 1As shown, the robotic arm grasping method based on a few-shot semantic segmentation model provided in this embodiment of the invention includes: S101: Construct a support set based on a scene image containing target objects and an labeled image containing multiple types of target objects; wherein, the support set includes the Is mask corresponding to each target object category.

[0019] In cluttered backgrounds, target identification can be challenging. To accurately identify the target, a support set is constructed using a scene image containing the target and labeled images containing multiple target categories. This support set serves as a dataset for few-shot learning. The support set includes the Is mask for each target category. The Is mask is a binary image used to identify pixel regions in the image belonging to that category. The support set provides the semantic segmentation model with visual prior information about various target categories, enabling it to accurately identify and segment similar objects in the query image with only a very small number of samples. Specifically, the support set consists of an RGB-D scene image containing the target and n labeled images containing the target, represented as S={(Is,Ms,s)i,i=1,…k}, s=1,…,n, where Is represents the image of category s, Ms represents the Is mask of the target of category s, s is the target category, and i represents k samples for each target category. Setting multiple samples for each category enhances the semantic model's understanding of that target category. For example, if the target objects include two categories, screwdrivers and wrenches, the support set will contain several labeled screwdriver images and their masks, as well as several wrench images and their masks.

[0020] The process of constructing a support set is actually a kind of visual knowledge injection, which enables the semantic segmentation model to identify the most matching target object by comparing the visual features of candidate objects in the input scene image with the samples of each category in the support set during subsequent analysis. Only a very small number of labeled samples are needed to achieve the recognition and segmentation of new objects, which significantly reduces the dependence on a large amount of labeled data.

[0021] S102: Input the input scene image to be identified into the SAM model. Through the SAM model, object segmentation is performed on the input scene image to obtain a set of candidate mask images corresponding to several segmented objects.

[0022] The Segment Anything Model (SAM) is a general-purpose image segmentation model based on deep learning. It can segment various objects in an image with high quality without requiring training on specific objects, possessing zero-shot generalization capability and handling untrained scenes and blurred segmentation objects. Therefore, in identifying and grasping targets, this invention requires inputting the input scene image to the SAM model. The SAM model extracts features from the input scene image to segment objects in the image, ultimately obtaining a set of candidate mask images corresponding to several segmented objects. The candidate mask image set consists of multiple candidate mask images, each a binary image that identifies the pixel region containing a segmented object in the image. The candidate masks cover almost all visually separable objects in the input scene image, regardless of whether their category appeared during training, enabling preliminary segmentation of unknown objects.

[0023] Specifically, object segmentation of the input scene image is accomplished collaboratively with three parts of the SAM model: the image encoder, the cue encoder, and the mask decoder. The image encoder, as the backbone network for feature extraction, encodes the input scene image, mapping it from pixel space to a high-dimensional feature space to form an image code containing semantic information. Through the image encoder, the global context and local details of the input scene image can be captured, providing basic feature representations for subsequent object segmentation. The cue encoder is responsible for constructing cue location codes. First, the input scene image is divided into multiple uniformly distributed grid points, each grid point representing a possible segmentation starting position. Then, the cue encoder converts the coordinate information of each grid point into a high-dimensional vector form of a cue location code, which serves as input cue to the SAM model, instructing the model to start segmenting objects from this position. Through cue location codes, the SAM model can automatically initiate multiple segmentation attempts without requiring manual specification of the segmentation target.

[0024] Furthermore, after obtaining the image encoding and the cue location encoding, the image encoding and the cue location encoding are fused using cross-attention and other feature fusion mechanisms to generate fused features with spatial awareness and semantic information. Feature fusion ensures that the SAM model can simultaneously consider image content and location cues during decoding, achieving accurate object localization and boundary recognition.

[0025] Finally, based on the aforementioned fusion features, the mask decoder progressively decodes and generates corresponding candidate mask images Mqi (i=0,…,m), where m is the number of candidate objects to be segmented. A candidate mask image is output for each cue location, ultimately forming a candidate mask image set {Mq0,…,Mqm}. Regardless of whether the segmented object belongs to a known category, it can cover all visually separable object regions in the input scene image.

[0026] S103: Receive a grabbing instruction for a specified segmented object, and in response to the grabbing instruction, determine the target mask corresponding to the specified segmented object based on the few-shot segmenter, according to the candidate mask image set and the support set, and generate a mask heatmap corresponding to the target mask.

[0027] After object segmentation, a grasping action is required. When the control system receives a grasping command for a specific segmented object, it initiates the few-shot segmentation and recognition process. Based on the few-shot segmenter, according to the category of the target object to be grasped, the system accurately identifies the candidate mask image most similar to the support set through the constructed candidate mask image set and support set. This determines the target mask corresponding to the specified segmented object and generates a mask heatmap. The generated mask heatmap not only provides a stable and accurate input for subsequent robotic arm grasping posture estimation but is also a key prerequisite for achieving a high success rate of robotic arm grasping.

[0028] In one embodiment, few-shot learning essentially involves accurately identifying the mask corresponding to the target object specified by the operator from a set of candidate mask images. This requires comparing the similarity between unknown segmented objects and known category prototypes in the support set, thereby achieving accurate target category identification and target selection.

[0029] First, a pre-trained feature extraction network, such as the ViT encoder, is used to extract features from each binary candidate mask image in the candidate mask image set to obtain the corresponding candidate mask feature vector fi, which has a dimension of D. The candidate mask feature vectors corresponding to each candidate mask image collectively constitute a pseudo-class set, which represents the feature set of all segmented unknown objects in the current scene.

[0030] Then, the support set contains pre-labeled images of a small number of known categories and their Is masks. Extract each image within the support set. corresponding feature vector The feature vector has dimension D, s represents the target object category, and j is the j-th image. Then, using the formula... The average feature vectors of all samples within the same category are averaged to generate the average feature vectors for each category of target objects. The average feature vectors of all target object categories together constitute the average feature set, which can serve as a reference standard for classification decisions of unknown targets.

[0031] At this point, the category of the segmented objects in the input scene image is unknown. To achieve accurate object capture, it is necessary to match them with the support set to determine the target object category corresponding to each segmented object. Therefore, for each segmented object, i.e., the target object with an unknown category, the candidate mask feature vector corresponding to the segmented object in the pseudo-category set is matched with the average feature set to determine the target object category corresponding to the segmented object.

[0032] Specifically, matching each element in the pseudo-class set with the average feature set involves calculating the cosine similarity between the candidate mask feature vector corresponding to each segmented object and the average feature vector corresponding to each category of the target object in the average feature set. Cosine similarity determines the semantic similarity by calculating the directional consistency between two feature vectors. Therefore, the target object corresponding to the average feature vector with the highest cosine similarity is selected from the average feature set to represent the category; this category is the target object category to which the segmented object belongs.

[0033] For each segmented object, the above matching process is performed to establish a mapping relationship between the candidate mask feature vector and its corresponding target object category. Based on this mapping relationship, upon receiving a grasping instruction for a specific segmented object, the system can directly find the candidate mask feature vector in the pseudo-category set according to the target object category corresponding to the specified segmented object, and use its corresponding candidate mask image as the target mask for the specified segmented object. For example, when the operator specifies that a cup needs to be grasped, the control system can find the candidate mask feature vector classified as a cup in the pseudo-category set, and determine its corresponding original binary candidate mask image as the final target mask.

[0034] By comparing vector similarity, unknown objects can be classified, avoiding the complex model retraining process. It can quickly identify target objects of a specified category from a large number of unknown objects, greatly improving the autonomy and adaptability of the robot system when facing new objects and new environments.

[0035] S104: Input the mask heatmap and the depth map corresponding to the input scene image into the UNet network to determine the grab point and grab pose parameters corresponding to the specified segmented object.

[0036] The mask heatmap is a binary image derived from the target mask. Highlighted areas indicate the pixel-level location of the target object in the image, ensuring the uniqueness of the object being grasped. The depth map provides three-dimensional spatial information of the scene, recording the distance between each pixel and the camera, and is crucial for calculating the real-world grasping coordinates. To ultimately achieve precise grasping of the specified segmented object by the robotic arm, the control system needs to convert the visual recognition results into executable grasping actions. Therefore, the mask heatmap and depth map are used as inputs to the UNet network. The UNet network then determines the grasping points and grasping posture parameters corresponding to the specified segmented object.

[0037] In this embodiment of the invention, the UNet network has a three-branch structure, with each branch used to predict the grasping confidence Q, the grasping angle A, and the grasping width W, respectively. Specifically, the Q branch is used to infer the center point position of the grasp, the A branch is used to infer the rotation direction of the robotic arm's end effector, and the W branch is used to infer the opening and closing size of the robotic arm's end effector. Therefore, the mask heatmap and the depth map corresponding to the input scene image are input into the UNet network, and based on the different branches of the UNet network, a quality heatmap, an angle heatmap, and a width heatmap are output.

[0038] Each pixel value on the quality heatmap represents the confidence level for performing a grab at that point, and is constrained to the range [0,1] by the softmax function. The quality heatmap allows us to determine the grab point corresponding to a specific segmented object. Essentially, it selects a location from the probability distribution of the UNet network that not only has the highest confidence level but is also the most stable and reliable in physical space, as the executable grab coordinate.

[0039] Specifically, the maximum pixel value is selected from the quality heatmap, and then the target pixel value is determined by multiplying the maximum pixel value by a preset ratio. The preset ratio is relatively high, such as 95%, and can also be adaptively adjusted according to different scenarios. The target pixel value is used to filter pixels with higher reliability. By comparing the pixel values ​​of each pixel with the target pixel value, region filtering can be performed on the quality heatmap to determine target regions where all pixel values ​​are greater than the target pixel value. Pixels in the target region need to be connected. The target region is a high-confidence region; generally, these regions are located on the surface of the target object and have a stable shape. After determining the target region, the coordinates (x, y) of the geometric center point of the target region are determined as the final grab point.

[0040] In determining the gripping point, choosing a high-confidence region rather than a single point greatly enhances the system's robustness. This effectively offsets the inherent minor errors in camera perception, neural network prediction, and robotic arm control. As long as the robotic arm lands within the target area, the gripping success rate is extremely high. Secondly, selecting the center point of the target area as the gripping point, rather than an edge point, ensures the mechanical balance of the gripping action and avoids potential tilting or slippage during gripping.

[0041] After determining the gripping point (x, y), the pixel value at the coordinates of the gripping point (x, y) is directly extracted from the angle heatmap; this value is the gripping angle. The angle defines the angle by which the robotic arm's end effector (such as a gripper) needs to rotate around the Z-axis when approaching the object, ensuring that the gripper's direction is aligned with the object's optimal gripping axis. Simultaneously, the pixel value at the same gripping point (x, y) coordinates is extracted from the width heatmap to obtain the gripping width. This width directly determines the width at which the robotic arm's gripper needs to open to accommodate the specific dimensions of the target object, achieving a grip that is both secure and does not damage it. Ultimately, the gripping angle and gripping width values ​​together constitute the gripping posture parameters that can be used for execution.

[0042] S105: Based on the gripping point and gripping posture parameters, control the robotic arm to perform the corresponding gripping action to achieve the gripping of the specified segmented object.

[0043] Based on the calculated gripping point and gripping posture parameters, the robotic arm is controlled to perform gripping actions. This requires transforming abstract parameters into specific mechanical movements in the real physical world, ultimately achieving successful gripping of the specified segmented object.

[0044] First, the grasping point needs to be transformed into coordinates. The grasping point (x, y) is obtained based on the image pixel coordinate system, and it must be transformed into the camera coordinate system using the camera calibration parameters to obtain the camera coordinates (xc, yc) of the target point. Then, using the transformation relationship between the camera coordinate system and the global coordinate system, the camera coordinate system (xc, yc) is transformed into the global coordinate system (xa, ya), thus obtaining the global coordinates in the real world that the robotic arm can understand.

[0045] Then, based on the transformed global coordinates and grasping posture parameters, the rotation angle of each servo motor in the robotic arm is determined using an inverse kinematics algorithm. The calculated rotation angle of each servo motor is converted into a corresponding pulse signal. The working principle of the servo motor is to precisely control the angular position of its axis of rotation by recognizing the width of the pulse signal. The control system sends these pulse signal sequences to the servos driving each joint, thereby controlling the coordinated movement of each joint of the robotic arm until its end effector accurately moves to the grasping point and orients itself in the posture specified by the angle value. At this point, a control signal is generated based on the width value. The control signal drives the end effector of the robotic arm to first open to the width W, and then perform a closing action to achieve precise grasping of the specified segmented object.

[0046] The above are embodiments of the method proposed in this invention. Based on the same idea, some embodiments of this invention also provide devices and non-volatile computer storage media corresponding to the above methods.

[0047] Figure 2 This is a schematic diagram of the structure of a robotic arm grasping device based on a few-shot semantic segmentation model, provided as an embodiment of the present invention. Figure 2 As shown, it includes: At least one processor; and, At least one processor-communication-connected memory; wherein, The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the robotic arm grasping method based on the few-shot semantic segmentation model as described in any of the preceding claims.

[0048] This invention provides a non-volatile computer storage medium storing computer-executable instructions, which are configured as follows: a robotic arm grasping method based on a few-sample semantic segmentation model as described in any of the preceding claims.

[0049] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0050] The devices, media, and methods provided in the embodiments of the present invention are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0051] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0055] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0056] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0057] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0058] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A robotic arm grasping method based on a few-shot semantic segmentation model, characterized in that, The method includes: A support set is constructed based on a scene image containing target objects and an labeled image containing multiple types of target objects; wherein, the support set includes the Is mask corresponding to each target object category; The input scene image to be identified is input into the SAM model. The SAM model is used to segment the input scene image into objects to obtain a set of candidate mask images corresponding to several segmented objects. Receive a grabbing instruction for a specified segmented object, and in response to the grabbing instruction, determine the target mask corresponding to the specified segmented object based on the few-shot segmenter, according to the candidate mask image set and the support set, and generate a mask heatmap corresponding to the target mask; The mask heatmap and the depth map corresponding to the input scene image are input into the UNet network to determine the grasping point and grasping posture parameters corresponding to the specified segmented object; Based on the gripping point and the gripping posture parameters, the robotic arm is controlled to perform the corresponding gripping action to grasp the specified segmented object.

2. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 1, characterized in that, Determining the target mask corresponding to the specified segmented object based on the candidate mask image set and the support set specifically includes: Extract the candidate mask feature vector corresponding to each candidate mask image in the candidate mask image set, and determine the pseudo-class set corresponding to each segmented object based on the candidate mask feature vector; Extract the feature vector corresponding to each image in the support set, and determine the average feature set composed of the average feature vectors corresponding to various target objects based on the feature vectors; For each segmented object, the candidate mask feature vector corresponding to the segmented object in the pseudo-category set is matched with the average feature set to determine the target object category corresponding to the segmented object; The target mask corresponding to the specified segmented object is determined based on the mapping relationship between the target object category and the candidate mask feature vector.

3. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 2, characterized in that, Matching the candidate mask feature vectors corresponding to the segmented objects in the pseudo-category set with the average feature set to determine the target object category corresponding to the segmented object, specifically including: The candidate mask feature vectors corresponding to the segmented objects in the pseudo-category set are matched with the average feature set to calculate the cosine similarity between the candidate mask feature vectors and the average feature vectors corresponding to each type of target object in the average feature set. From the average feature set, the target object corresponding to the average feature vector with the highest cosine similarity is selected as the target object category corresponding to the segmented object.

4. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 1, characterized in that, The mask heatmap and the depth map corresponding to the input scene image are input into the UNet network to determine the grasping points and grasping pose parameters corresponding to the specified segmented object, specifically including: The mask heatmap and the depth map corresponding to the input scene image are input into the UNet network. Based on different branches of the UNet network, a quality heatmap, an angle heatmap, and a width heatmap are output. The grab point corresponding to the specified segmented object is determined using the mass heat map. The angle and width values ​​corresponding to the grab point are extracted from the angle heatmap and the width heatmap, respectively, and the angle and width values ​​are used as the grab posture parameters corresponding to the specified segmented object.

5. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 4, characterized in that, The mass heatmap is used to determine the grab points corresponding to the specified segmented object, specifically including: Based on the pixel values ​​corresponding to each pixel in the mass heatmap, a target pixel value is determined; wherein, the target pixel value is the product of the maximum pixel value among the pixel values ​​and a preset ratio; From the mass heatmap, select a target region where the pixel value of each pixel is greater than the target pixel value, and use the center point of the target region as the grab point corresponding to the specified segmented object.

6. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 1, characterized in that, Based on the grasping point and the grasping posture parameters, the robotic arm is controlled to perform a corresponding grasping action, specifically including: The coordinates of the gripping points are transformed, and the rotation angles of each servo motor in the robotic arm are determined based on the transformed global coordinates and the gripping posture parameters. A corresponding pulse signal is generated based on the rotation angle, and the servo motor is controlled to move by the pulse signal until it reaches the gripping point. Then, the robotic arm is controlled to perform the corresponding gripping action according to the gripping posture parameters.

7. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 1, characterized in that, The SAM model is used to segment objects in the input scene image to obtain a set of candidate mask images corresponding to the segmented objects, specifically including: The SAM model includes an image encoder, a cue encoder, and a mask encoder; The input scene image is encoded using the image encoder to obtain an image code, and a prompt position code corresponding to the input scene image is constructed using the prompt encoder; wherein, the prompt position code is used to instruct the SAM model to perform object segmentation; The image encoding and the prompt location encoding are fused to obtain fused features; The mask encoder outputs candidate masks corresponding to each fusion feature, and obtains a set of candidate mask images corresponding to several segmented objects.

8. The robotic arm grasping method based on a few-shot semantic segmentation model according to claim 7, characterized in that, The prompt encoder is used to construct the prompt location code corresponding to the input scene image, specifically including: The input scene image is divided into multiple uniformly distributed grid points using the prompt encoder. For each grid point, the coordinates of the grid point are converted into the corresponding prompt location code.

9. A robotic arm grasping device based on a few-shot semantic segmentation model, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the robotic arm grasping method based on a few-sample semantic segmentation model as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: The robotic arm grasping method based on a few-sample semantic segmentation model as described in any one of claims 1-8.

Images