Methods, apparatus, equipment and storage media for controlling robotic arms

Abstract

According to embodiments of this disclosure, a method, apparatus, device, and storage medium for controlling a robotic arm are provided. The method includes: receiving control commands for the robotic arm; determining descriptive information associated with an object to be grasped by the robotic arm by parsing the control commands; determining evaluation information of multiple candidate regions in image content based on text features corresponding to the descriptive information; determining a target object from the multiple candidate objects based on the evaluation information, and controlling the robotic arm to grasp the target object. In this manner, embodiments of this disclosure can improve the accuracy of the robotic arm grasping objects.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and storage media for controlling robotic arms. Background Technology

[0002] With the development of technology, robotic arms are widely used in various industries. To meet people's needs for grasping objects, robotic arms can be controlled by commands to grasp target objects. However, traditional methods usually train and deploy models on a known fixed set of categories. Therefore, when encountering new categories not seen in the training data, or when the input commands are complex, the system performance will drop significantly. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for controlling a robotic arm is provided. The method includes: receiving control commands for the robotic arm, the control commands corresponding to natural language representations; determining, by parsing the control commands, descriptive information associated with an object to be grasped by the robotic arm, the descriptive information including first information, second information, and third information, the first information describing the type of the object, the second information describing the grasping components in the object, and the third information describing objects or components to be avoided from grasping during the operation of the robotic arm; determining evaluation information for multiple candidate regions in image content based on text features corresponding to the descriptive information, the multiple candidate regions corresponding to multiple candidate objects, the image content being captured by a camera mounted on the robotic arm; determining a target object from the multiple candidate objects based on the evaluation information; and controlling the robotic arm to grasp the target object.

[0004] In a second aspect of this disclosure, an apparatus for controlling a robotic arm is provided. The apparatus includes: a receiving module configured to receive control commands for the robotic arm, the control commands corresponding to natural language representations; a parsing module configured to determine descriptive information associated with an object to be grasped by the robotic arm by parsing the control commands, the descriptive information including first information, second information, and third information, the first information describing the type of the object, the second information describing the grasping components in the object, and the third information describing objects or components to be avoided from being grasped during the operation of the robotic arm; an evaluation module configured to determine evaluation information of multiple candidate regions in image content based on text features corresponding to the descriptive information, the multiple candidate regions corresponding to multiple candidate objects, the image content being captured by a camera mounted on the robotic arm; a determining module configured to determine a target object from the multiple candidate objects based on the evaluation information; and a control module configured to control the robotic arm to grasp the target object.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] The embodiments of this disclosure can accurately identify and control a robotic arm to grasp target objects without relying on training with sample images by parsing control commands, thereby improving the control efficiency of the robotic arm.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 A first flowchart illustrating an example process for controlling a robotic arm according to some embodiments of the present disclosure is shown;

[0012] Figure 3A and Figure 3B A second flowchart illustrates an example process for controlling a robotic arm according to some embodiments of the present disclosure;

[0013] Figure 4 A schematic structural block diagram of an example device for controlling a robotic arm according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0019] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0020] As mentioned above, with the development of technology, robotic arms are widely used in various industries. To meet people's needs for grasping objects, robotic arms can be controlled by commands to grasp target objects. However, traditional methods typically train and deploy models on a known, fixed set of categories. This approach has poor generalization ability, high maintenance costs, and is prone to failure in open-world scenarios. Therefore, when encountering new categories not seen in the training data, or when the input commands are complex, the system performance will drop significantly.

[0021] Embodiments of this disclosure propose a scheme for controlling a robotic arm. The scheme includes: receiving control commands for the robotic arm, the control commands corresponding to natural language representations; parsing the control commands to determine descriptive information associated with an object to be grasped by the robotic arm, the descriptive information including first information, second information, and third information, wherein the first information describes the type of the object, the second information describes the grasping components within the object, and the third information describes objects or components that the robotic arm should avoid grasping during operation; determining evaluation information for multiple candidate regions in image content based on text features corresponding to the descriptive information, the multiple candidate regions corresponding to multiple candidate objects, the image content being captured by a camera mounted on the robotic arm; determining a target object from the multiple candidate objects based on the evaluation information; and controlling the robotic arm to grasp the target object.

[0022] In this way, the embodiments of this disclosure can parse control commands into multi-layered descriptive information, thereby obtaining not only information related to the object type in the control commands, but also information related to the part to be grasped, and information related to objects or parts that need to be avoided from grasping, thus improving the understanding of control commands. Furthermore, based on the multi-layered descriptive information, evaluation information corresponding to multiple candidate regions can be generated to evaluate the graspability of multiple candidate objects, thereby determining a suitable target object from among the multiple candidate objects, and finally controlling the robotic arm to grasp the target object to complete the grasping task. In this way, target objects can be accurately identified without relying on training with sample images. It also allows the robotic arm to grasp target objects without requiring specially supervised training for new categories, thereby improving the overall efficiency of the method.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example environment:

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1As shown, the example environment 100 may include a control device 110, a model 120, a robotic arm 130, and an object 140. The control device 110 can implement the above-described scheme of controlling the robotic arm through the model 120. The model 120 can be deployed on the control device 110, or it can be deployed on other devices, which will not be elaborated here.

[0026] In some embodiments, the robotic arm may consist of multiple joints and links. The robotic arm may have multiple degrees of freedom, enabling complex movements in three-dimensional space. Alternatively, the robotic arm may also include an end effector. Such an end effector can support the configuration of different tools to achieve different functions. Exemplarily, the end effector of the robotic arm may be implemented as a gripper, wrench, surgical instrument, spraying equipment, etc.

[0027] In some embodiments, a camera may be mounted on the robotic arm to assist it in performing object grasping tasks. For example, the camera may be mounted on the end effector of the robotic arm. As an example, object 140 may correspond to the object to be grasped. Such an object may be, for example, a cup, a tool, or a part.

[0028] The control device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the control device 110 can also support any type of interface for the target user (such as "wearable" circuitry).

[0029] The control device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. The control device 110 may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in a cloud environment, etc.

[0030] For example, control commands can be input to control device 110. After receiving the control commands, control device 110 parses the control commands through model 120 to obtain descriptive information. Then, based on the text features corresponding to the descriptive information, evaluation information for multiple candidate regions is determined. Next, based on the evaluation information, target object 140 is determined from multiple candidate objects, and finally, control device 110 controls robotic arm 130 to grasp target object 140.

[0031] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0032] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0033] Example process:

[0034] Figure 2 A flowchart illustrating an example process for controlling a robotic arm according to some embodiments of the present disclosure is shown. Process 200 can be implemented at control device 110.

[0035] like Figure 2 As shown, in block 210, control device 110 can receive control commands for the robotic arm.

[0036] In some embodiments, reference Figure 3A In step 302, the control device 110 can receive control commands. The control commands correspond to natural language representations. For example, the control command for controlling the robotic arm can be "Take a container that can be used to pour water, and grasp the handle, avoiding pressing the rim of the cup, not the lid, and not a disposable paper cup." Applying this control command will control the robotic arm to grasp the handle of the cup.

[0037] In some embodiments, to assist a robotic arm in performing object grasping tasks, a robotic arm equipped with a camera can be used. For example, a color-depth camera can be used, which is a camera that not only takes color photographs but also measures the distance from pixels to the camera. And refer to... Figure 3A Before execution, step 301, namely camera calibration and color depth alignment, needs to be completed. Camera calibration is a calibration process that corrects distortion and obtains the camera's internal parameters. Color depth alignment involves precisely matching each pixel in the depth map acquired by the color-depth camera with the corresponding pixel in the color map, so that each color pixel in the generated aligned image has an accurate depth value.

[0038] In some embodiments, if the camera is mounted on the end effector of a robotic arm, hand-eye calibration is required. Hand-eye calibration determines the precise geometric relationship between the camera and the end effector. For example, it allows for the homogeneous transformation from the camera coordinate system to the robotic arm's base coordinate system. This homogeneous transformation can be used for coordinate transformation when outputting prior knowledge.

[0039] like Figure 2 As shown in box 220, the control device 110 can determine the descriptive information associated with the object to be grasped by the robotic arm by parsing the control commands.

[0040] In some embodiments, the natural language representation of the control instructions may include information from multiple angles regarding the control of the robotic arm. For example, in the natural language representation "Take a container that can be poured into, and grasp the handle, avoiding touching the rim, not the lid, and not a disposable paper cup," not only is the type of object the robotic arm is expected to grasp described as "a container that can be poured into," but the specific part of the object that the robotic arm is expected to grasp, "the handle," is also described, as well as the objects that the robotic arm is not expected to grasp, such as "the rim," "the lid," and "a disposable paper cup." Due to the complexity of the control instructions, they need to be parsed to better understand the user's precise needs.

[0041] In some embodiments, the descriptive information includes first information, second information, and third information. The first information describes the type of object, the second information describes the parts of the object used for grasping, and the third information describes objects or parts that the robotic arm avoids grasping during its operation. For example, refer to... Figure 3A In step 303, the control instruction "Take a container that can be poured into, and grasp the handle, avoiding pressing the rim of the cup, not the lid, and not a disposable paper cup" can be parsed to generate descriptive information associated with the object to be grasped by the robotic arm. This descriptive information may include three parts: first information, second information, and third information. Furthermore, this embodiment of the disclosure does not limit the model used to parse the control instruction; any model capable of understanding text can be applied.

[0042] To accurately understand control commands, richer descriptive information is needed. Therefore, the first information can be represented as a set of phrases describing the type of object, the second information can be represented as a set of phrases describing the gripping parts within the object, and the third information can be represented as a set of phrases describing objects or parts that the robotic arm should avoid gripping during its operation. For example, the first information could include "a container that can pour water," the second information could include "gripper handle," and the third information could include "cup lid," "disposable paper cup," and "cup rim."

[0043] In some embodiments, since the first information is information used to describe the type of the object, for the control instruction "take a container that can be poured with water, and grab the handle, avoid pressing the rim of the cup, not the lid, and not a disposable paper cup", the information used to describe the type of the object only includes "a container that can be poured with water". Such a description is too simplistic and will have a certain impact on the subsequent processing. Therefore, the first information can be enriched by adding synonyms.

[0044] Specifically, the control device 110 can provide control instructions to the text parsing model, determining a first expression in the control instructions that indicates the type of object to be crawled. The first expression is then provided to the language model to obtain at least one second expression, the semantic similarity between the first expression and the at least one second expression being greater than a threshold. Next, based on the first expression and the at least one second expression, first information is determined. As an example, the text parsing model may include, for instance, a suitable machine learning model such as a language model, which can be used to parse text content to extract specified information.

[0045] For example, a control instruction like "take a container that can be poured into, and grasp the handle, avoiding touching the rim, not the lid, and not a disposable paper cup" can be provided to the text parsing model to determine the first expression "a container that can be poured into" indicating the type of object to be grasped in the control instruction. Then, the first expression "a container that can be poured into" can be provided to the language model to obtain synonyms for the first expression, such as "mug" or "cup," and these synonyms can be identified as the second expression. Next, the first information can be determined based on the first and second expressions, which can include "mug," "cup," "a container that can be poured into," etc.

[0046] This method allows for the acquisition of richer primary information, thereby improving the accuracy of controlling the robotic arm. Furthermore, a similar method can be applied to expand the secondary information; for example, expanding "grip handle" in the secondary information to include "grip handle" and "cup handle," etc.

[0047] In some embodiments, the text features include a first feature portion corresponding to the first information, a second feature portion corresponding to the second information, and a third feature portion corresponding to the third information. For example, refer to... Figure 3A In step 304, after obtaining the first, second, and third information, a text encoder can be applied to encode the first, second, and third information into semantic vectors respectively. Specifically, the text encoder can be applied to encode the phrase t in the description information into an embedding E(t). These are then stored in three types of word vector tables for subsequent cross-modal matching calls.

[0048] like Figure 2As shown in box 230, the control device 110 can determine the evaluation information of multiple candidate regions in the image content based on the text features corresponding to the description information.

[0049] In some embodiments, multiple candidate regions correspond to multiple candidate objects, and the image content is captured by a camera mounted on a robotic arm. For example, since descriptive information has already been obtained, corresponding text features can be extracted based on the descriptive information, and multiple candidate regions can be determined in the image content captured by the camera based on the text features. Each candidate region includes a corresponding candidate object. These candidate regions are then evaluated to obtain evaluation information for multiple candidate regions.

[0050] In some embodiments, the control device 110 may provide image content and first information to the detection model to determine multiple candidate boxes matching the first information. The multiple candidate boxes are then provided to the segmentation model to determine multiple pixel-level masks, which indicate multiple candidate regions corresponding to the multiple candidate objects. For example, refer to... Figure 3A In step 305, the detection model can use an open vocabulary detector, and input multiple phrases from the first information into the open vocabulary detector in batches so that the open vocabulary detector can be used to identify multiple candidate boxes {r} from the image content, where each candidate box {r} corresponds to a corresponding text matching score. This embodiment of the disclosure does not limit the specific model structure of the detection model; the detection model only needs to have image detection functionality.

[0051] Because a mask can pinpoint the precise outline and internal pixels of a specific object within an image, a more accurate candidate region can be extracted by determining the mask. Specifically, refer to... Figure 3A In step 306, multiple candidate boxes {r} can be provided to the segmentation model, and these candidate boxes {r} can be used as cue information to determine multiple pixel-level masks. Through multiple pixel-level masks It can indicate multiple candidate regions corresponding to multiple candidate objects, such as candidate region 1 corresponding to candidate cup 1, candidate region 2 corresponding to candidate cup 2, and so on.

[0052] In some embodiments, since the control instructions include information related to the object type, information related to the grasping component, and information related to the object or component to be avoided, the evaluation information of the candidate region can be determined based on the degree of matching between these three types of information and the corresponding regions identified in the image content. This approach improves the accuracy of the evaluation information for the candidate region.

[0053] Specifically, the control device 110 can determine a first image feature corresponding to a corresponding candidate region among multiple candidate regions. Then, based on a first similarity between the first feature portion and the first image feature, a first evaluation factor is determined. Next, based on a second similarity between a second feature portion and a second image feature, where the second image feature indicates a component region within the corresponding candidate region, a second evaluation factor is determined. Then, based on a third similarity between a third feature portion and the first image feature, a third evaluation factor is determined. Finally, based on the first evaluation factor, the second evaluation factor, and the third evaluation factor, evaluation information for the corresponding candidate region is determined.

[0054] For example, refer to Figure 3B In step 320, since multiple candidate regions have been identified from the image content, a visual encoder can be applied to extract multiple first image features from the multiple candidate regions. Then, based on the first feature portion corresponding to the first information and the first image features... The first similarity between the two images determines the first evaluation factor. That is, if the first information includes the phrase "cup," then the first image feature... It is generated based on the candidate region corresponding to the "cup" in the image. Therefore, by calculating the first feature part and the first image feature... The initial similarity score between candidates effectively determines the degree of match between the candidate objects in that region and the user's needs. A high similarity score indicates a good match, while a low similarity score indicates a poor match. (Reference) Figure 3B Step 322, First Evaluation Factor The formula for calculating it is as follows:

[0055] ;

[0056] The first image feature, The first feature is τ, where τ is the temperature coefficient and C is the number of phrases in the first information.

[0057] In some embodiments, to calculate the second evaluation factor, multiple component masks may first be determined. The control device 110 may determine the multiple component masks based on second information, whereby the component masks indicate component regions corresponding to components used for grasping in the respective candidate objects. Then, second image features corresponding to the multiple component masks are determined. For example, referring to… Figure 3B In step 314, the second information can be used to determine multiple component masks in the image content. The component mask can indicate the part of the image that can be used for grasping, such as the "handle of the cup". This disclosure does not limit the specific method of determining the component mask; any method that can find and segment the corresponding component region in the image based on the text description is acceptable.

[0058] In some embodiments, if the identified grasping portion is too small, it may affect the grasping action of the robotic arm. Therefore, this portion of the grasping area can be deleted in advance to improve the overall task execution performance. The control device 110 can determine multiple component regions based on the second information. Then, it determines the first coverage degree of the multiple component regions to the corresponding candidate regions. Next, it removes at least one component region with a coverage degree less than a threshold from the multiple component regions to determine multiple component masks.

[0059] For example, refer to Figure 3B In step 316, after determining multiple component regions, a decision can be made on whether to delete a component region based on its coverage within the corresponding candidate regions. First coverage level... The formula for calculating it is as follows:

[0060] ;

[0061] This indicates the overlapping area between the component region and the corresponding candidate region. This indicates the proportion of the overlapping region to the corresponding candidate region. Then, regions with coverage less than a threshold can be deleted. The component area is defined using this method. On the one hand, excessively small components can be removed, such as handles that are too small to be grasped by the robotic arm. On the other hand, falsely detected components can be eliminated, such as protrusions on the cup wall that are mistaken for handles. This ensures that the candidate area truly includes the components used for grasping, thus improving the accuracy of component recognition.

[0062] Since multiple component regions have been identified from the image content, and these component regions may not necessarily be entirely within their corresponding candidate regions, a visual encoder can be applied to extract multiple second image features corresponding to the overlapping areas between the multiple component regions and their corresponding candidate regions. Then, based on the second feature portion corresponding to the second information and the second image features... The second similarity between the two is used to determine the second evaluation factor. That is, if the second information includes the phrase "cup handle", the second image feature... It is generated based on the overlapping area between the component region corresponding to the "cup handle" in the image and the corresponding candidate region "cup". Then, by calculating the second feature part and the second image feature... The second similarity between components can effectively determine the degree of matching between the components in that component area and the user's needs. (Reference) Figure 3B Step 315, Second Evaluation Factor The formula for calculating it is as follows:

[0063] ;

[0064] For the second image feature, This is the second characteristic part. Let A be the temperature coefficient, and A be the number of phrases in the second information.

[0065] In some embodiments, the control device 110 may determine a plurality of negative masks based on third information. Then, it determines a second degree of coverage of the regions corresponding to the plurality of negative masks to the corresponding candidate regions. Next, it determines a third evaluation factor based on the third similarity and the second coverage.

[0066] For example, refer to Figure 3B In step 317, the third information can be used to determine multiple negative masks in the image content. 'k' represents the sequence number of the phrase in the third piece of information. Negative masks can be used to indicate objects or parts that the robotic arm should avoid grasping during its operation, such as "cup rim," "disposable paper cup," "cup lid," etc. To integrate the effects of multiple negative examples, these negative masks can be combined in a union manner as follows:

[0067] .

[0068] Next, refer to Figure 3B In step 318, the second coverage degree of the regions corresponding to multiple negative masks to the corresponding candidate regions can be determined. The formula for calculating it is as follows:

[0069] ;

[0070] This indicates the overlapping region between the negative example region and the corresponding candidate region. This indicates the proportion of the overlapping region to the corresponding candidate region, that is, This indicates the percentage of objects or components in the candidate area that need to be avoided from being crawled.

[0071] Then, based on the third feature part corresponding to the third information and the first image features... The third similarity score is used to effectively determine whether a candidate object in a candidate region is similar to the description in the third information. For example, if the third information includes the phrase "cup lid," then the more similar the candidate object is to the description of "cup lid," the more likely the candidate object is to be "cup lid," and it should be avoided from being crawled; conversely, the more similar the candidate object is to the description of "cup lid," the less likely the candidate object is to be "cup lid." (Reference) Figure 3B Step 321, third similarity The formula for calculating it is as follows:

[0072] ;

[0073] The first image feature, This is the third characteristic part. Here, N represents the temperature coefficient, and N represents the number of phrases in the third piece of information.

[0074] Specifically, in order to fully consider the parts or objects that need to be avoided during the evaluation of candidate regions, a negative example gating coefficient can be used to adjust the final evaluation information. (Reference) Figure 3B Step 319, the third evaluation factor The formula for calculating it is as follows:

[0075] ;

[0076] Indicates the third similarity. The larger the value, the greater the likelihood that the candidate object is one that needs to be avoided from being crawled. This indicates the percentage of objects or components in the candidate region that should be avoided from being crawled. The larger the value, the more areas in the candidate region need to be avoided from being crawled. (Third evaluation factor) Let represent the probability of passing through the negative example gate, and Evaluation information can be adjusted through multiplicative methods.

[0077] In some embodiments, the control device 110 can determine multiple weight parameters corresponding to the first evaluation factor, the second evaluation factor, and the third evaluation factor. These weight parameters are then applied to the first, second, and third evaluation factors respectively to determine the evaluation information for the corresponding candidate regions. For example, different weight parameters can be assigned to each of the multiple evaluation factors to generate more accurate evaluation information.

[0078] In some embodiments, reference Figure 3A Step 307: Evaluation information corresponding to the candidate region The formula for calculating it is as follows:

[0079] ;

[0080] These are calibrable parameters. The corresponding weight parameters are The higher the proportion of a component region in the candidate regions, the better that component region is. Embodiments of this disclosure can perform offline calibration of temperature and weights on a very small number of representative samples, thereby stabilizing the threshold.

[0081] because It originates from the maximum similarity of multiple synonyms. Therefore, there is a lack of comparability between different images or different candidate regions. This can be addressed by... conduct Normalization preserves information more similar to the object category in the first information, improving candidate comparability and reducing the original similarity scale differences between different images. Specifically, the normalization formula can be as follows:

[0082] ;

[0083] T represents temperature. Furthermore, it can also be analyzed along the lines of synonyms. After normalization, the probability of the word with the highest probability is taken.

[0084] because The calculations are performed on the component area, where the distribution is more concentrated, therefore, by... conduct Normalization, which normalizes the values ​​to the [0,1] interval, achieves stable fusion. Various specific methods can be used for this process. Normalization can be performed, for example, using cosine mapping. Normalization processing For example, it can also be done using the temperature sigmoid method. Normalization processing ,in, and A small number of samples can be used for calibration.

[0085] like Figure 2 As shown in box 240, control device 110 can determine the target object from multiple candidate objects based on evaluation information.

[0086] For example, refer to Figure 3A In step 308, multiple evaluation information corresponding to multiple candidate regions is obtained. After that, you can base your evaluation on the information. The numerical values ​​are used to sort multiple candidate regions. Then, mask-based non-maximum suppression is performed to select the top-k candidate regions. Specifically, non-maximum suppression is a step to remove duplicates from multiple highly overlapping candidate regions, which can obtain... The candidate region with the largest value is selected, and candidate regions with an overlap exceeding a threshold are removed based on the candidate mask. This process is then iterated to select the Top-k candidate regions. This method avoids the same candidate object being output multiple times, maintains the compactness and interpretability of the information in the Top-k candidate regions, and reduces subsequent branches in the crawling plan.

[0087] In some embodiments, since occlusion or stacking may occur in the image content, additional processing is required for these occlusion or stacking situations to determine a more reasonable target object. The control device 110 can remove the first sub-image corresponding to the occluded object from the image content. Then, it fills the empty areas created by removing the first sub-image in multiple second sub-images corresponding to multiple candidate objects. Based on the filled multiple second sub-images, it determines the revenue information corresponding to moving the occluded object. Next, based on the revenue information and the cost information corresponding to moving the occluded object, it determines whether to use a robotic arm to move the occluded object. After the occluded object is removed, the target object is determined based on the updated evaluation information of multiple candidate objects.

[0088] For example, in scenes with stacking or occlusion, the occluded object captured first will affect the visible portion after capture. Therefore, the color image corresponding to the occluded object can be removed from the image content. Then, the depth image corresponding to the occluded object is repaired, for example, by applying local planar extrapolation or guided filtering for hole filling. Finally, a graph-based image model is used for further repair to obtain multiple filled second sub-images. The occluded object can be a candidate object or not; there is no restriction on this.

[0089] The part coverage can then be recalculated based on the multiple filled second sub-images. Or simply recalculate. Where s represents the candidate regions corresponding to multiple second sub-images after removing the occluded object. Next, a quantification metric primarily focused on improving component visibility can be defined. Or quantitative indicators that primarily improve the final evaluation score. The formula is as follows:

[0090] ;

[0091] .

[0092] Since capturing occluded objects takes time and carries some risk, a cost can be defined. .cost This can be determined through historical success rate, gap accessibility, and crawling time estimation. Then, ranking metrics can be calculated. :

[0093] or .

[0094] Obtain ranking index Next, the occluded objects to be crawled first can be determined based on the ranking metric values. This method prioritizes removing occluded objects that most significantly impact the task's progress. Removing these occluded objects makes the candidate objects more aligned with the user's crawling needs, leading to faster task convergence and improved overall efficiency.

[0095] like Figure 2 As shown in box 250, control device 110 can control the robotic arm to grasp the target object.

[0096] For example, after identifying the target object, the position information of the robotic arm when grasping the target object can be determined based on the position information of the target object, such as the grasping position, grasping posture, gripper opening, etc., thereby controlling the robotic arm to grasp the target object.

[0097] In some embodiments, the control device 110 can determine a set of three-dimensional points corresponding to the target object. Then, based on the intersection of the three-dimensional point set and the component region corresponding to the target object, the grasping position of the target object is determined. The robotic arm is then controlled to grasp the target object according to the grasping position.

[0098] For example, refer to Figure 3A In step 309, the candidate mask corresponding to the target object can be projected onto three-dimensional space to obtain the three-dimensional point set corresponding to the target object. The grab position can be determined using a 3D point set. Then refer to... Figure 3A In step 310, the component subset corresponding to the target object can be selected. The contact normal *n* and principal direction axis are obtained through geometric estimation, and the gripping posture can be determined using the contact normal and principal direction axis. Next, the 3D point set is projected onto the normal tangent plane, and the minimum bounding rectangle that can contain the projected points is calculated on this plane. The opening of the gripper can be determined based on this minimum bounding rectangle.

[0099] refer to Figure 3A In step 311, then along the direction of the contact normal. Rays are emitted to detect obstacles between the outside of the gripper and the gripping point. An accessibility score is calculated based on ray penetration, which assesses whether the gripper can reach the gripping point without collision. Next, refer to... Figure 3A In step 312, the obtained capture position can be... The system outputs the grasping posture R, estimated opening, candidate mask, candidate bounding box, and evaluation information together to control the robotic arm to grasp the target object. Additionally, refer to... Figure 3A In step 313, if hand-eye calibration exists, then it needs to be applied. The capture pose is converted from the camera coordinate system to the robot arm's base coordinate system so that the robot arm can directly use these coordinates for motion planning.

[0100] This method parses control commands into information including the target to be grasped, the grasping area, and constraints. This allows for understanding not only the type of object to be grasped but also the specific areas to be grasped and those to be avoided, thus achieving a full understanding of the control command's intent. Furthermore, when calculating the evaluation information for each candidate region, these three levels of factors are fully considered, resulting in more accurate evaluation information for the candidate regions. Additionally, visibility compensation evaluation through removal simulation in stacked or occluded scenarios addresses the difficulty in handling grasping problems caused by stacking or occlusion.

[0101] Example devices and equipment:

[0102] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example device 400 for controlling a robotic arm according to certain embodiments of the present disclosure is shown. Device 400 may be implemented as or included in control device 110. Various modules / components in device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0103] like Figure 4 As shown, the device 400 includes: a receiving module 410 configured to receive control commands for a robotic arm, the control commands corresponding to natural language representations; a parsing module 420 configured to determine descriptive information associated with an object to be grasped by the robotic arm by parsing the control commands, the descriptive information including first information, second information, and third information, the first information describing the type of the object, the second information describing the grasping components in the object, and the third information describing objects or components to be avoided from being grasped during the operation of the robotic arm; an evaluation module 430 configured to determine evaluation information of multiple candidate regions in an image based on text features corresponding to the descriptive information, the multiple candidate regions corresponding to multiple candidate objects, the image content being captured by a camera mounted on the robotic arm; a determination module 440 configured to determine a target object from the multiple candidate objects based on the evaluation information; and a control module 450 configured to control the robotic arm to grasp the target object.

[0104] In some embodiments, the parsing module 420 is further configured to provide control instructions to a text parsing model, determine a first expression in the control instructions indicating the type of the object to be crawled; provide the first expression to a language model to obtain at least one second expression, wherein the semantic similarity between the first expression and at least one second expression is greater than a threshold; and determine first information based on the first expression and at least one second expression.

[0105] In some embodiments, the apparatus 400 is further configured to provide image content and first information to a detection model to determine a plurality of candidate boxes matching the first information; and to provide the plurality of candidate boxes to a segmentation model to determine a plurality of pixel-level masks, wherein the plurality of pixel-level masks indicate a plurality of candidate regions corresponding to the plurality of candidate objects.

[0106] In some embodiments, the text features include a first feature portion corresponding to the first information, a second feature portion corresponding to the second information, and a third feature portion corresponding to the third information.

[0107] In some embodiments, the evaluation module 430 is further configured to: determine a first image feature corresponding to a corresponding candidate region among a plurality of candidate regions; determine a first evaluation factor based on a first similarity between a first feature portion and a first image feature; determine a second evaluation factor based on a second similarity between a second feature portion and a second image feature, wherein the second image feature indicates a component region in the corresponding candidate region; determine a third evaluation factor based on a third similarity between a third feature portion and a first image feature; and determine evaluation information for the corresponding candidate region based on the first evaluation factor, the second evaluation factor, and the third evaluation factor.

[0108] In some embodiments, the evaluation module 430 is further configured to determine, based on second information, a plurality of component masks, wherein the component masks indicate component regions corresponding to components used for capture in the corresponding candidate objects; and to determine second image features corresponding to the plurality of component masks.

[0109] In some embodiments, the evaluation module 430 is further configured to determine a plurality of component regions based on second information; determine a first degree of coverage of the plurality of component regions to corresponding candidate regions; and remove at least one component region from the plurality of component regions whose coverage is less than a threshold, in order to determine a plurality of component masks.

[0110] In some embodiments, the evaluation module 430 is further configured to determine a plurality of negative masks based on third information; determine a second coverage degree of the region corresponding to the plurality of negative masks to the corresponding candidate region; and determine a third evaluation factor based on the third similarity and the second coverage degree.

[0111] In some embodiments, the evaluation module 430 is further configured to determine multiple weight parameters corresponding to the first evaluation factor, the second evaluation factor, and the third evaluation factor; and to apply the multiple weight parameters to the first evaluation factor, the second evaluation factor, and the third evaluation factor respectively to determine the evaluation information of the corresponding candidate region.

[0112] In some embodiments, the control module 450 is further configured to determine a set of three-dimensional points corresponding to the target object; determine the grasping position of the target object based on the intersection of the three-dimensional point set and the component region corresponding to the target object; and control the robotic arm to grasp the target object through the grasping position.

[0113] In some embodiments, the determining module 440 is further configured to: remove a first sub-image corresponding to an occluding object from the image content; fill the empty areas in a plurality of second sub-images corresponding to a plurality of candidate objects caused by the removal of the first sub-image; determine revenue information corresponding to the moved occluding object based on the filled plurality of second sub-images; determine whether to move the occluding object using a robotic arm based on the revenue information and the cost information corresponding to the moved occluding object; and after the occluding object is removed, determine the target object based on the updated evaluation information of the plurality of candidate objects.

[0114] The units included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0115] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Control device 110.

[0116] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0117] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0118] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0119] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0120] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0121] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0122] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0124] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for controlling a robotic arm, characterized in that, The method includes: Receive control commands for the robotic arm, the control commands corresponding to natural language representations; By parsing the control commands, descriptive information associated with the object to be grasped by the robotic arm is determined. The descriptive information includes first information, second information, and third information. The first information describes the type of the object, the second information describes the grasping components in the object, and the third information describes the objects or components that the robotic arm should avoid grasping during its operation. Based on the text features corresponding to the description information, evaluation information of multiple candidate regions in the image content is determined. The multiple candidate regions correspond to multiple candidate objects. The image content is acquired by a camera mounted on the robotic arm. Based on the evaluation information, a target object is determined from the plurality of candidate objects; and Control the robotic arm to grasp the target object. The text features include a first feature portion corresponding to the first information, a second feature portion corresponding to the second information, and a third feature portion corresponding to the third information. Furthermore, the evaluation information for determining multiple candidate regions in the image content based on the text features corresponding to the description information includes: Determine the first image feature corresponding to the corresponding candidate region among the plurality of candidate regions; A first evaluation factor is determined based on the first similarity between the first feature portion and the first image feature; A second evaluation factor is determined based on the second similarity between the second feature portion and the second image feature, wherein the second image feature indicates the component region in the corresponding candidate region; Based on the third similarity between the third feature portion and the first image feature, a third evaluation factor is determined; and The evaluation information of the corresponding candidate region is determined based on the first evaluation factor, the second evaluation factor, and the third evaluation factor.

2. The method according to claim 1, characterized in that, The step of determining the descriptive information associated with the object to be grasped by the robotic arm by parsing the control commands includes: The control instructions are provided to the text parsing model, and a first expression in the control instructions indicating the type of the object to be crawled is determined; The first expression is provided to a language model to obtain at least one second expression, wherein the semantic similarity between the first expression and the at least one second expression is greater than a threshold; and The first information is determined based on the first expression and the at least one second expression.

3. The method according to claim 1, characterized in that, The method further includes: The image content and the first information are provided to the detection model to determine multiple candidate boxes that match the first information; The plurality of candidate boxes are provided to the segmentation model to determine a plurality of pixel-level masks, the plurality of pixel-level masks indicating the plurality of candidate regions corresponding to the plurality of candidate objects.

4. The method according to claim 1, characterized in that, The method further includes: Based on the second information, multiple component masks are determined, each component mask indicating a component region corresponding to a component used for grasping in a corresponding candidate object; and Determine the second image feature corresponding to the plurality of component masks.

5. The method according to claim 4, characterized in that, The determination of multiple component masks based on the second information includes: Based on the second information, multiple component regions are identified; Determine the first coverage degree of the plurality of component regions to the corresponding candidate regions; and Remove at least one component region whose coverage is less than a threshold from the plurality of component regions to determine the plurality of component masks.

6. The method according to claim 1, characterized in that, The determination of the third evaluation factor based on the third similarity between the third feature portion and the first image feature includes: Based on the aforementioned third information, multiple negative masks are determined; Determine the second coverage degree of the regions corresponding to the plurality of negative masks to the corresponding candidate regions; and The third evaluation factor is determined based on the third similarity and the second coverage.

7. The method according to claim 1, characterized in that, The evaluation information for determining the corresponding candidate region based on the first evaluation factor, the second evaluation factor, and the third evaluation factor includes: Determine multiple weight parameters corresponding to the first evaluation factor, the second evaluation factor, and the third evaluation factor; The weight parameters are applied to the first evaluation factor, the second evaluation factor, and the third evaluation factor respectively to determine the evaluation information of the corresponding candidate regions.

8. The method according to claim 1, characterized in that, The process of controlling the robotic arm to grasp the target object includes: Determine the three-dimensional point set corresponding to the target object; Based on the intersection of the three-dimensional point set and the component region corresponding to the target object, the grab position of the target object is determined; and The robotic arm is controlled to grasp the target object through the grasping position.

9. The method according to claim 1, characterized in that, The step of determining the target object from the plurality of candidate objects based on the evaluation information includes: Remove the first sub-image corresponding to the occluded object from the image content; Fill the empty regions in the multiple second sub-images corresponding to the multiple candidate objects caused by the removal of the first sub-image; Based on the filled plurality of second sub-images, revenue information corresponding to moving the occluded object is determined; Based on the revenue information and the cost information corresponding to moving the obstructing object, determine whether to use the robotic arm to move the obstructing object; and After the occluding object is removed, the target object is determined based on the updated evaluation information of the multiple candidate objects.

10. A device for controlling a robotic arm, characterized in that, The device includes: The receiving module is configured to receive control commands for the robotic arm, the control commands corresponding to natural language representations; The parsing module is configured to determine descriptive information associated with the object to be grasped by the robotic arm by parsing the control instructions. The descriptive information includes first information, second information and third information. The first information describes the type of the object, the second information describes the parts in the object used for grasping, and the third information describes the objects or parts that the robotic arm should avoid grasping during operation. The evaluation module is configured to determine the evaluation information of multiple candidate regions in the image content based on the text features corresponding to the description information. The multiple candidate regions correspond to multiple candidate objects. The image content is acquired by a camera mounted on the robotic arm. The determination module is configured to determine a target object from the plurality of candidate objects based on the evaluation information; and The control module is configured to control the robotic arm to grasp the target object. The text features include a first feature portion corresponding to the first information, a second feature portion corresponding to the second information, and a third feature portion corresponding to the third information. The evaluation module is further configured to: determine a first image feature corresponding to a corresponding candidate region among the plurality of candidate regions; determine a first evaluation factor based on a first similarity between the first feature portion and the first image feature; determine a second evaluation factor based on a second similarity between the second feature portion and the second image feature, wherein the second image feature indicates a component region in the corresponding candidate region; determine a third evaluation factor based on a third similarity between the third feature portion and the first image feature; and determine the evaluation information of the corresponding candidate region based on the first evaluation factor, the second evaluation factor, and the third evaluation factor.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images