Intelligent control method, device and electronic equipment for robotic arm

Abstract

The present invention provides a method, device and electronic device for intelligent control of a robotic arm, which are applied to the fields of artificial intelligence and computer vision. The method comprises: obtaining an environmental observation image and an environmental depth image; inputting a target language token sequence into a large language model to obtain a target object description text and a target task description text; inputting the target object description text and the environmental observation image into an open world environmental perception model to obtain a first mask image of the target object; obtaining a point cloud image of the target object based on the first mask image and the environmental depth image; inputting the target object description text, the target task description text and the point cloud image into the large language model to obtain the pose estimation information of the robotic arm; and controlling the robotic arm to perform corresponding operations corresponding to the target task description text on the target object based on the pose estimation information.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and more specifically, to a method, device and electronic equipment for intelligent control of a robotic arm. Background Art

[0002] Intelligent control of robotic arms requires the intelligent agent to understand the operation instructions given by humans, perceive the external environment, plan the operation trajectory, and finally control the robotic arm to complete the task. It has a wide range of applications in multiple industries, from industrial manufacturing to service industries, and to space exploration. Its application scope is constantly expanding. Traditional robotic arm control methods design special control solutions for fixed scenes, fixed tasks, and limited categories, which cannot achieve convenient human-computer interaction and more generalized application scenarios. Summary of the invention

[0003] In view of this, the present invention provides a method, device and electronic equipment for intelligent control of a robotic arm.

[0004] One aspect of the present invention provides a method for intelligent control of a robotic arm, comprising: acquiring an environmental observation image and an environmental depth image, wherein the environmental observation image and the environmental depth image both contain a target object, and the environmental observation image and the environmental depth image are obtained by taking a depth camera; inputting a target language token sequence into a large language model to obtain a target object description text and a target task description text, wherein the target task description text represents text for controlling the robotic arm to perform corresponding operations on the target object; inputting the target object description text and the environmental observation image into an open world environment perception model to obtain a first mask image of the target object; obtaining a point cloud image of the target object based on the first mask image and the environmental depth image; inputting the target object description text, the target task description text and the point cloud image into the large language model to obtain posture estimation information of the robotic arm; and controlling the robotic arm to perform corresponding operations corresponding to the target task description text on the target object based on the posture estimation information.

[0005] Another aspect of the present invention provides an intelligent control device for a robotic arm, comprising: an image acquisition module, used to acquire an environmental observation image and an environmental depth image, wherein the environmental observation image and the environmental depth image both contain a target object, and the environmental observation image and the environmental depth image are obtained by taking a depth camera; a text acquisition module, used to input a target language token sequence into a large language model to obtain a target object description text and a target task description text, wherein the target task description text represents the text for controlling the robotic arm to perform corresponding operations on the target object; a mask image acquisition module, used to input the target object description text and the environmental observation image into an open world environment perception model to obtain a first mask image of the target object; a point cloud image acquisition module, used to obtain a point cloud image of the target object based on the first mask image and the environmental depth image; a pose estimation information acquisition module, used to input the target object description text, the target task description text and the point cloud image into the large language model to obtain pose estimation information of the robotic arm; and a control module, used to control the robotic arm to perform corresponding operations corresponding to the target task description text on the target object based on the pose estimation information.

[0006] Another aspect of the present invention provides an electronic device, comprising:

[0007] one or more processors;

[0008] a memory for storing one or more programs,

[0009] When one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0010] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0011] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when being executed.

[0012] According to an embodiment of the present invention, a target object description text and a target task description text are obtained by inputting a target language token sequence into a large language model, and an open world environment perception model is used to explicitly perceive the target object based on the target object description text and the environment sensing image to obtain a first mask image of the target object; the first mask image and the environment depth image are then used to obtain a point cloud image of the target object; the target object description text, the target task description text and the point cloud image are then input into the large language model to obtain position estimation information of the robotic arm, so as to control the robotic arm to perform corresponding operations on the target object corresponding to the target task description text. This technical solution solves the problem that the existing method based on language instructions allows the model to perceive implicitly. When observing the target object in the environment image, the ability to perceive the target object is weak due to the background interference in the observed environment image, resulting in low accuracy in perceiving the target object category and spatial position. In addition, the existing language instruction-based method uses a single-stage method to generate control instructions for various operation tasks, resulting in the robot arm being not flexible enough and unable to meet the diverse operation task requirements. Further implementation adopts a unified target language token sequence, utilizes the combination of a large language model and an open world environment perception model, improves the robot arm's natural language instruction comprehension ability and the accuracy of target perception category and spatial position, achieves multi-stage generation of control instructions, and meets the diverse operation task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0014] Figure 1 An exemplary system architecture to which the robot arm intelligent control method and device according to an embodiment of the present invention can be applied is shown;

[0015] Figure 2 A flowchart of a method for intelligently controlling a robotic arm according to an embodiment of the present invention is shown;

[0016] Figure 3 A flow chart of a method for obtaining a first mask image of a target object according to an embodiment of the present invention is shown;

[0017] Figure 4 A flow chart of a method for obtaining position and posture estimation information of a robotic arm according to an embodiment of the present invention is shown;

[0018] Figure 5 A flowchart of a method for controlling a robotic arm to perform a corresponding operation on a target object corresponding to a target task description text according to an embodiment of the present invention is shown;

[0019] Fig. 6AA schematic diagram of obtaining a point cloud image of a target object according to an embodiment of the present invention is shown;

[0020] Figure 6B A schematic diagram of an intelligent control method for a robotic arm according to an embodiment of the present invention is shown;

[0021] Figure 6C A schematic diagram showing a method for intelligently controlling a robotic arm according to another embodiment of the present invention is shown;

[0022] Figure 7 A block diagram of an intelligent control device for a robotic arm according to an embodiment of the present invention is shown; and

[0023] Figure 8 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0029] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0030] In related technologies, in order to improve the ability of intelligent human interaction, intelligent control methods for robotic arms based on language instructions have emerged. These methods parse the operation instructions based on the existing large language model and generate the underlying instructions for the control of the robotic arm. Specifically, most of the existing methods based on language instructions input text instructions, observation data (images, point clouds), etc. into the model as a whole, allowing the model to implicitly perceive the target objects in the observation data and generate operation instructions.

[0031] However, the existing language instruction-based methods use a single-stage approach to generate control instructions for various operation tasks, which makes the robotic arm not flexible and accurate enough when performing operation tasks and cannot meet the diverse operation task requirements; in addition, based on language instructions, the model implicitly perceives the target objects in the observation data. This implicit perception method will be affected by the background interference in the observation data and has a weak perception ability for the target objects.

[0032] In view of this, an embodiment of the present invention provides a method for intelligent control of a robotic arm, the method comprising: acquiring an environmental observation image and an environmental depth image, wherein the environmental observation image and the environmental depth image both contain a target object, and the environmental observation image and the environmental depth image are obtained by taking a depth camera; inputting a target language token sequence into a large language model to obtain a target object description text and a target task description text, wherein the target task description text represents the text for controlling the robotic arm to perform corresponding operations on the target object; inputting the target object description text and the environmental observation image into an open world environment perception model to obtain a first mask image of the target object; obtaining a point cloud image of the target object based on the first mask image and the environmental depth image; inputting the target object description text, the target task description text and the point cloud image into the large language model to obtain posture estimation information of the robotic arm; and controlling the robotic arm to perform corresponding operations corresponding to the target task description text on the target object based on the posture estimation information.

[0033] Figure 1 The exemplary system architecture to which the robot arm intelligent control method and device according to the embodiment of the present invention can be applied is shown. It should be noted that: Figure 1The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the robot arm intelligent control method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the robot arm intelligent control device provided in the embodiment of the present invention can generally be set in the server 105. The robot arm intelligent control method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the robot arm intelligent control device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the robot arm intelligent control method provided in the embodiment of the present invention can also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or can also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the intelligent control device for the robotic arm provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Figure 2 A flow chart of a method for intelligently controlling a robotic arm according to an embodiment of the present invention is shown.

[0041] like Figure 2 As shown, the method 200 includes operations S210 to S260.

[0042] In operation S210, an environment observation image and an environment depth image are acquired, wherein both the environment observation image and the environment depth image contain a target object, and the environment observation image and the environment depth image are acquired by taking a depth camera.

[0043] According to an embodiment of the present invention, the environment observation image can be a visual feature that can intuitively display the image so that it can be directly observed and understood. The visual feature can be, for example, color, brightness, etc. It can be obtained by shooting with a color camera of a depth camera.

[0044] According to an embodiment of the present invention, the environment depth image may be an image capable of capturing the three-dimensional structural information of an object in the image. The environment depth image includes the distance information between each pixel in the image and the depth camera, and may be obtained by shooting with a depth camera on the depth camera.

[0045] According to an embodiment of the present invention, both the environment observation image and the environment depth image contain a target object, which is an operation object when the robot arm performs a corresponding operation.

[0046] In operation S220, the target language token sequence is input into the large language model to obtain a target object description text and a target task description text, wherein the target task description text represents the text for controlling the robot arm to perform a corresponding operation on the target object.

[0047] According to an embodiment of the present invention, the target language token sequence can be used to describe the language sequence for controlling the robot arm to perform corresponding operations on the operation object. For example, it can be "lift the bottle with a white cap to a height of 0.5 meters from the ground and rotate it to an angle of 45 degrees with the vertical upward direction".

[0048] According to an embodiment of the present invention, a large language model can be used to perform instruction parsing on a target language token, and a target object description text and a target task description text can be obtained. For example, the target object description text can be "a bottle with a white lid"; the target task description text can be "a height of 0.5 meters, rotated to an angle of 45 degrees with the vertical upward direction".

[0049] According to an embodiment of the present invention, the target task description text may represent text for controlling the robotic arm to perform corresponding operations on the target object.

[0050] In operation S230 , the target object description text and the environment observation image are input into the open world environment perception model to obtain a first mask image of the target object.

[0051] According to an embodiment of the present invention, the open world environment perception model can identify and locate computer vision technology of different objects in an image.

[0052] According to an embodiment of the present invention, an open world environment perception model can be used to perform target object detection on target object description text and environment observation images, and the target object can be detected from the environment observation image to obtain a first mask image of the target object.

[0053] In operation S240 , a point cloud image of the target object is obtained according to the first mask image and the environment depth image.

[0054] According to the embodiment of the present invention, the first mask image and the environment depth image may be subjected to a product operation, and a second mask image of the target object may be acquired from the environment depth image.

[0055] According to the embodiment of the present invention, the second mask image of the target object is three-dimensionally reconstructed using the camera parameters of the depth camera, so as to obtain a point cloud image of the target object.

[0056] According to an embodiment of the present invention, a point cloud image may be a three-dimensional data set of a large number of points, which is usually used to express surface features of an object, such as shape or texture. Each point contains three-dimensional coordinate information and may also contain color information.

[0057] In operation S250, the target object description text, the target task description text and the point cloud image are input into the large language model to obtain the pose estimation information of the robot arm.

[0058] According to an embodiment of the present invention, a large language model can be used to extract the estimated gripping posture information of the robot arm from the target object description text and point cloud images; and a large language model can be used to extract the estimated posture information of the robot arm when completing the target task from the target task description text and point cloud images.

[0059] According to an embodiment of the present invention, a large language model can be used to extract intermediate pose estimation information of the robotic arm between the two poses of grasping and completing the task from the target object description problem, the target task description text, the grasping pose estimation information and the pose estimation information when completing the target task.

[0060] In operation S260, the robot arm is controlled to perform a corresponding operation corresponding to the target task description text on the target object according to the pose estimation information.

[0061] According to an embodiment of the present invention, inverse kinematics can be solved for each pose estimation information in each stage, and the motion state of the robot arm at each moment can be obtained. Based on the motion state of the robot arm at each moment, the complete robot arm state can be obtained, so as to control the robot arm to perform corresponding operations corresponding to the target task description text on the target object.

[0062] According to an embodiment of the present invention, a target object description text and a target task description text are obtained by inputting a target language token sequence into a large language model, and an open world environment perception model is used to explicitly perceive the target object based on the target object description text and the environment sensing image to obtain a first mask image of the target object; the first mask image and the environment depth image are then used to obtain a point cloud image of the target object; the target object description text, the target task description text and the point cloud image are then input into the large language model to obtain position estimation information of the robotic arm, so as to control the robotic arm to perform corresponding operations on the target object corresponding to the target task description text. This technical solution solves the problem that the existing method based on language instructions allows the model to implicitly perceive the object. When measuring the target object in the environmental image, the ability to perceive the target object is weak due to the background interference in the observed environmental image, resulting in low accuracy in perceiving the target object category and spatial position. In addition, the existing language instruction-based method uses a single-stage method to generate control instructions for various operation tasks, resulting in the robot arm being not flexible enough and unable to meet the technical problems of diverse operation task requirements. Further implementation adopts a unified target language token sequence, utilizes the combination of a large language model and an open world environment perception model, improves the robot arm's natural language instruction comprehension ability and the accuracy of target perception category and spatial position, achieves multi-stage generation of control instructions, and meets the technical effect of diverse operation task requirements.

[0063] According to an embodiment of the present invention, a target language token sequence is input into a large language model to obtain a target object description text and a target task description text, including: vector encoding the target language token sequence to obtain a target language word embedding vector; position encoding the target language token sequence to obtain a target language position vector; summing the target language embedding vector and the target language position vector to obtain a target language feature vector; and feature extraction of the target language feature vector to obtain a target object description text and a target task description text.

[0064] According to an embodiment of the present invention, for example, the length of the target language token sequence is L, which can be expressed as For each language token in the target language token sequence Perform vector encoding and query the target language word embedding vector of each language token from the word embedding matrix Tokenizer , that is, obtained by formula (1).

[0065] According to an embodiment of the present invention, for each language token in the target language token sequence Perform position encoding on the input position to obtain the target language position vector for each language token , that is, obtained by formula (2).

[0066] According to an embodiment of the present invention, the target language word embedding vector and the target language position vector can be summed to obtain the target language feature vector of each language token: , that is, obtained by formula (3).

[0067] (1);

[0068] (2);

[0069] (3).

[0070] According to an embodiment of the present invention, the target language feature vector Perform feature extraction to output the word embedding vector of the next predicted word , query the corresponding word in the word embedding matrix Tokenizer You can embed the word into the vector The output is restored to the natural language representing the target object description text and the target task description text. In the present invention, the above-mentioned embedded parts of the input and output can be respectively recorded as and .

[0071] According to an embodiment of the present invention, in addition to the embedding part of the input and output, the large language model is generally stacked by r layers of decoders D, where r can range from 12 to 120 layers. Assume that the output sequence of the large language model is , length is T, predict the jth word The task can be expressed as formula (3-1), that is:

[0072] (3-1);

[0073] in, Represents the predicted words from 0th to j-1th.

[0074] According to an embodiment of the present invention, in the decoder part, the input vector corresponding to the dth layer of the jth word After the masked self-attention mechanism, the key corresponding to the token before it is saved in this layer Query and get the value corresponding to each key , and the input vector Similarity score to each key , and finally the scores will be weighted to get the first The output vector of the layer , which will also serve as the input of the next layer of decoder.

[0075] The process of generating instruction parsing content from a large language model is consistent with the above process of decoding language tokens, and can be modeled as a multi-step Markov decision process, whose action and state spaces are both natural language spaces. The large language model is used as a strategy , sampling language token sequences as actions. Assume that the lengths of the input sequence and the output sequence are and , when the input language token sequence is In the case of The probability is From the properties of the above large language model, it can be seen that the design of the input language token sequence is crucial to achieve high-quality instruction parsing.

[0076] According to an embodiment of the present invention, in order to enable the large language model to accurately parse the target language token sequence representing the natural language instruction into the target object description text and the target task description text, the present invention designs the input sequence format of the robot arm manipulation task, thereby instructing the large language model to generate an output sequence y that meets the requirements. That is, the composition of the input sequence x representing the target language token sequence is as follows (4):

[0077] (4);

[0078] Where I is the target language token sequence, representing the language sequence describing the control of the robot arm to perform the corresponding operation on the operation object; To supplement the requirement, it is used to instruct the large language model to extract key information of the target object description text and the target task description text from I; is a set of s examples, which helps the large language model understand the task, where is a sample sequence of target language tokens And the corresponding manually annotated output sample sequence Composition, s is an integer ≥ 1, 0≤i≤s.

[0079] According to an embodiment of the present invention, based on the above method of decoding language tokens using a large language model, the large language model can parse the target object description text and the target task description text from the target language token sequence I in the input sequence x of the above formula (4).

[0080] Figure 3 A flow chart of a method for obtaining a first mask image of a target object according to an embodiment of the present invention is shown.

[0081] like Figure 3 As shown, the method 300 may include operations S310 to S320.

[0082] In operation S310 , the target object description text and the environment observation image are input into an open-world object detector network to obtain the bounding box information of the target object.

[0083] In operation S320, the bounding box information of the target object and the environment observation image are input into an image segmentation network to obtain a first mask image of the target object.

[0084] According to an embodiment of the present invention, an open-world environment perception model includes an open-world object detector network and an image segmentation network.

[0085] According to an embodiment of the present invention, an open-world target detector network can be used to detect target objects in environmental observation images to obtain bounding box information of the target objects.

[0086] According to an embodiment of the present invention, an image segmentation network can be used to encode and decode an environment observation image based on bounding box information of a target object to obtain a first mask image of the target object.

[0087] According to an embodiment of the present invention, the target object description text and the environment observation image are input into an open-world target detector network to obtain the bounding box information of the target object, including: performing feature extraction on the environment observation image to obtain a first image feature vector; performing feature extraction on the target object description text to obtain a text feature vector; performing feature interaction enhancement processing on the first image feature vector and the text feature vector to obtain an enhanced image feature vector and an enhanced text feature vector; and obtaining the bounding box information of the target object based on the enhanced image feature vector and the enhanced text feature vector.

[0088] According to an embodiment of the present invention, an open world object detection network may include an image backbone module , Text backbone module , Feature Enhancement Layer Module , language-guided query selection module and cross-modal decoding layer module .

[0089] According to an embodiment of the present invention, an image backbone module can be used Perform image feature extraction on the environmental observation image to obtain the first image feature vector , which can be obtained by formula (5), namely:

[0090] (5);

[0091] According to an embodiment of the present invention, a text backbone module can be used Extract text features from the target object description text w to obtain the text feature vector , which can be obtained by formula (6), namely:

[0092] (6);

[0093] According to an embodiment of the present invention, a feature enhancement layer module can be used based on an attention mechanism. Perform feature interaction enhancement on the first image feature vector and the text feature vector to obtain an enhanced image feature vector And the enhanced text feature vector , which can be obtained by formula (7), namely:

[0094] (7);

[0095] According to an embodiment of the present invention, a query selection module guided by language is used. , using the enhanced text feature vector as the query guide, querying the feature q corresponding to the text feature vector from the enhanced image feature vector, and using the cross-modal decoding layer module For this feature q, the enhanced image feature vector And the enhanced text feature vector Perform feature extraction to obtain the bounding box information b of the target object in the environment observation image. It can be obtained through equations (8) and (9):

[0096] (8);

[0097] (9);

[0098] Among them, h is the environment observation image; w is the target object description text.

[0099] According to an embodiment of the present invention, the bounding box information of the target object and the environment observation image are input into an image segmentation network to obtain a first mask image of the target object, including: encoding the bounding box information of the target object to obtain a bounding box feature vector; performing image encoding on the environment observation image to obtain a second image feature vector; and decoding the bounding box feature vector and the second image feature vector to obtain the first mask image of the target object.

[0100] According to an embodiment of the present invention, the image segmentation network may include an image encoder , Prompt Encoder and mask decoder .

[0101] According to an embodiment of the present invention, an image encoder of an image segmentation network can be used Perform image encoding on the environment observation image h to obtain the second image feature vector , which can be obtained by formula (10), namely:

[0102] (10);

[0103] According to an embodiment of the present invention, the bounding box information of the target object obtained above can be input into the image segmentation network as the target prompt, and the prompt encoder of the image segmentation network can be used to generate the target object. Encode the bounding box information of the target object to obtain the bounding box feature vector , which can be obtained by formula (11), namely:

[0104] (11);

[0105] According to an embodiment of the present invention, the mask decoder of the image segmentation network can be used , using the bounding box feature vector as a hint, the second image feature vector is masked to obtain the first mask image m of the target object in the environment observation image, which can be obtained by formula (12), namely:

[0106] (12).

[0107] According to an embodiment of the present invention, through the powerful prior knowledge of the open world environment perception model, the target objects in the environment observation image are explicitly perceived to improve the accuracy of the target object category perception and the precision of the target object spatial position.

[0108] According to an embodiment of the present invention, a point cloud image of a target object is obtained based on a first mask image and an environmental depth image, including: binary encoding the first mask image to obtain a binary image of the first mask image; multiplying the binary image and the environmental depth image to obtain a second mask image of the target object; obtaining two-dimensional pixel information of the second mask image; obtaining intrinsic parameter information and extrinsic parameter information of a depth camera; and performing three-dimensional reconstruction of the second mask image using the two-dimensional pixel information, the intrinsic parameter information and the extrinsic parameter information to obtain a point cloud image of the target object.

[0109] According to an embodiment of the present invention, the second mask image of the target object may be acquired from an environment depth image.

[0110] According to an embodiment of the present invention, the binary image of the first mask image may be a binary image in which each pixel on the first mask image has only two possible values ​​or grayscale states.

[0111] According to an embodiment of the present invention, the first mask image can be used as a prompt to perform binary encoding on the first mask image to obtain a binary image of the first mask image, and then the binary image of the first mask image and the environmental depth image are multiplied to obtain a second mask image of the target object depth information obtained from the environmental depth image.

[0112] According to an embodiment of the present invention, the two-dimensional pixel information of the second mask image may include the position information (u, v) of the two-dimensional pixel and the depth information of the two-dimensional pixel. .

[0113] According to an embodiment of the present invention, the two-dimensional pixel position information of the second mask image of the target object is (u, v), and the two-dimensional pixel position information can be represented as its corresponding homogeneous coordinates: , that is, it can be expressed as formula (13):

[0114] (13);

[0115] Wherein, u is the horizontal coordinate of the two-dimensional pixel point position information, and v is the vertical coordinate of the two-dimensional pixel point position information.

[0116] According to an embodiment of the present invention, the intrinsic parameter information K of the depth camera is used to describe the internal properties of the camera, such as focal length, principal point coordinates, distortion parameters, etc. The extrinsic parameter information (T, R) of the depth camera is composed of a translation vector T and a rotation matrix R, which is used to describe the position and posture of the camera in the world coordinate system.

[0117] According to an embodiment of the present invention, the homogeneous coordinates of the two-dimensional pixel points and the depth information of the two-dimensional pixel points can be used. And the internal parameter information K of the depth camera, get the coordinates in the camera coordinate system , which can be obtained by formula (14), namely:

[0118] (14);

[0119] in, is the transpose of the intrinsic parameter information of the depth camera.

[0120] According to an embodiment of the present invention, the coordinates in the camera coordinate system can be used , the camera's external parameter information (T, R), and obtain the coordinates in the world coordinate system , represents the point cloud image of the target object, which can be obtained by formula (15), that is:

[0121] (15);

[0122] According to an embodiment of the present invention, the above formulas (13) to (15) can be used to obtain a mapping from two-dimensional pixel points to three-dimensional pixel space points, so as to realize three-dimensional reconstruction of all pixel points in the second mask image of the target object, obtain the point cloud coordinates of the target object in the world coordinate system, and thus obtain a point cloud image of the target object, thereby providing comprehensive and accurate information for subsequent intelligent autonomous planning and control of the robotic arm.

[0123] Figure 4 A flow chart of a method for obtaining posture estimation information of a robotic arm according to an embodiment of the present invention is shown.

[0124] like Figure 4 As shown, the method 400 may include: operations S410 to S460.

[0125] In operation S410 , size information of the point cloud image and initial pose estimation information of the robot arm are acquired.

[0126] In operation S420, sparse sampling is performed layer by layer on the point cloud image to determine point cloud information for each layer of the point cloud image, thereby obtaining a point cloud set of the point cloud image.

[0127] In operation S430, first position estimation information of the manipulator is determined according to the description text, size information and point cloud set of the target object, wherein the first position estimation information represents the position estimation of the manipulator when it starts to grasp the target object.

[0128] In operation S440, second pose estimation information of the robotic arm is determined based on the first pose estimation information and the target task description text, wherein the second pose estimation information represents the pose estimation of the robotic arm when completing a corresponding operation corresponding to the target task description text.

[0129] In operation S450, at least one third pose estimation information of the robotic arm is determined based on the target object description text, the target task description text, the initial pose estimation information and the second pose estimation information, wherein the third pose estimation information represents an intermediate pose estimation of the robotic arm from the initial pose estimation to the second pose estimation.

[0130] In operation S460, pose estimation information of the robot arm is obtained according to the first pose estimation information, the second pose estimation information, and the third pose estimation information.

[0131] According to an embodiment of the present invention, all pixel points of the second mask image of the target object are three-dimensionally reconstructed by the above equations (13) to (15) to obtain a point cloud image of the target object, and the size information of the point cloud image of the target object can be obtained. , that is, the length of the point cloud image ,Width and high .

[0132] According to an embodiment of the present invention, the point cloud image of the target object may be sparsely sampled layer by layer, and each layer may store point coordinate information of the height of the layer in the form of a tuple to obtain a point cloud set of the point cloud image.

[0133] According to an embodiment of the present invention, for example, the point cloud image of the target object with the target object description text w is divided into Z layers, and the point cloud image of the target object is sparsely sampled layer by layer from high to low, so that the point coordinate information for each layer height can be obtained. , the point cloud set of the obtained point cloud image can be expressed as . Wherein, Z is an integer greater than or equal to 1, and 0≤z≤Z.

[0134] According to an embodiment of the present invention, the first pose estimation information may represent the pose estimation of the robot arm when it starts to grasp the target object. and rotation .

[0135] According to an embodiment of the present invention, in order to enable the large language model to accurately obtain the first pose estimation information when the robot arm starts to perform the operation of grasping the target object, the input sequence of the large language model can be The composition is as follows (16):

[0136] (16);

[0137] in, To capture the pose requirements, instruct the large language model to describe the target object through the text w and the point cloud image Analyze the first pose estimation information when the robot arm starts to grasp; is a set of s examples, which helps the large language model understand the task, where is the input sample sequence of the large language model And the corresponding manually annotated output of the first pose estimation sample information composition.

[0138] According to an embodiment of the present invention, the input sequence of the above formula (16) is input into the large language model for analysis, and a translation and rotation tuple describing the moment when the robot arm starts grasping (i.e., moment M) can be output, i.e., the first pose estimation information .

[0139] According to an embodiment of the present invention, the large language model can use the target task description text g, the first pose estimation information , and the input sequence of the above formula (16), the second pose estimation information of the robot arm at the time when it completes the corresponding operation corresponding to the target task description text (i.e., time N) is obtained .

[0140] According to an embodiment of the present invention, based on the above description, when calculating the second pose estimation information of the robot arm at the time corresponding to the completion of the task, the large language model input sequence The composition can be expressed as formula (17), that is:

[0141] (17);

[0142] in, To complete the task pose requirement, instruct the large language model to analyze the second pose estimation information of the robot arm when completing the task through the point cloud image, the target task description text and the first pose estimation information; g is the target task description text; The input sequence for calculating the first pose estimation information when the robot arm starts to grasp is a point cloud image.

[0143] According to an embodiment of the present invention, the large language model analyzes the input sequence of the above formula (17) to obtain the translation and rotation tuples describing the robot arm at the corresponding operation moment (i.e., moment N) corresponding to the target task description text, i.e., the second pose estimation information .

[0144] According to an embodiment of the present invention, after determining the first pose estimation information and the second pose estimation information, the manipulator system records two states of the manipulator: and .

[0145] According to an embodiment of the present invention, the initial pose estimation information of the robot arm may be the pose estimation at time 0 before the robot arm starts to perform the grasping task, which may be expressed as .

[0146] According to an embodiment of the present invention, the initial pose estimation of the robot arm at time 0 can be set The second pose estimation of the manipulator from the initial state to the time N The status of completing the task has An intermediate pose estimate.

[0147] According to an embodiment of the present invention, a large language model can be used to describe the target object text w, the target task description text g, and the initial pose estimation information. and the second pose estimation information By analyzing, we can get An intermediate pose estimate, that is, The third pose estimation information, where .

[0148] According to an embodiment of the present invention, a large language model is used to obtain When the third pose estimation information is obtained, the input sequence of the large language model can be expressed as formula (18), that is:

[0149] (18);

[0150] According to an embodiment of the present invention, the output sequence may be The pose estimation information represents all the pose estimation information of the robot arm from the initial state to the task completion state, so as to obtain a motion trajectory suitable for the robot arm.

[0151] According to an embodiment of the present invention, the output sequence of the large language model is all the posture estimation information of the robot arm from the initial state to the completion of the task, which can represent the running trajectory of the robot arm, that is, it can be expressed as formula (19):

[0152] (19);

[0153] in, Represents the pose estimation information at time 1; ...; Indicates time The pose estimation information.

[0154] Figure 5 A flow chart of a method for controlling a robotic arm to perform corresponding operations on a target object corresponding to a target task description text according to an embodiment of the present invention is shown.

[0155] like Figure 5 As shown, the method 500 may include: operations S510 to S530.

[0156] In operation S510, kinematic analysis is performed on the pose estimation information to obtain joint driving angle information of the robot arm corresponding to the pose estimation information.

[0157] In operation S520, motion trajectory information of the robot arm when performing a corresponding operation corresponding to the target task description text is obtained according to the joint driving angle information.

[0158] In operation S530, the robot arm is controlled to perform a corresponding operation corresponding to the target task description text on the target object according to the motion trajectory information.

[0159] According to an embodiment of the present invention, after the motion trajectory of the robot arm is obtained, the inverse kinematics solution algorithm may be used to obtain the joint states of the robot arm, that is, the joint drive angle information of the robot arm.

[0160] According to an embodiment of the present invention, for the motion trajectory of the robot arm For each pose estimation in , the inverse kinematics solution algorithm can be used to obtain the joint state of the robot arm at each moment.

[0161] According to an embodiment of the present invention, the initial state at time 0 corresponding to the initial posture estimation information of the robot arm For example, the calculation The joint state of the robot arm at the moment When , it can be obtained by the following formula (20):

[0162] (20);

[0163] Among them, Motion is the inverse kinematics solution algorithm; For The joint status of the robot arm at time .

[0164] According to an embodiment of the present invention, the above formula (20) can be used to calculate the joint state of the robot arm at the corresponding time corresponding to each pose estimation information in the robot arm motion trajectory, and the joint state of the robot arm corresponding to time 0 can be obtained. The joint state of the robot arm corresponding to time N , that is, the joint state of the complete robotic arm can be expressed as .

[0165] According to an embodiment of the present invention, the initial pose estimation corresponding to the initial state of the robot arm can be obtained based on the states of the robot arm's joints at various times. The second pose estimation corresponding to the state when the task is completed Specifically, the motion planning of the robot arm can be obtained through two implementation methods.

[0166] One implementation method may be to use the above formula (20) to respectively calculate the initial pose estimation corresponding to the initial state of the robot arm: The first pose estimate at time M corresponding to when the robot arm starts to grasp The joint state of the robot arm between , and the first pose estimate at time M corresponding to when the robot arm starts to grasp The second pose estimation at time N corresponding to when the robot completes the task The joint states of the robotic arm between the two are obtained to obtain the joint states of the complete robotic arm, that is, to obtain the complete motion plan of the robotic arm.

[0167] Calculate the initial pose estimate corresponding to the initial state of the robot arm The first pose estimate at time M corresponding to when the robot arm starts to grasp The joint state of the robot arm , that is, the motion planning from the initial state to the start of grasping state can be obtained by formula (21), that is:

[0168] (twenty one);

[0169] Among them, M+1 represents the total number of moments from time 0 to time M.

[0170] Calculate the first pose estimate at time M corresponding to when the robot starts to grasp The second pose estimation at time N corresponding to when the robot completes the task The joint state of the robot arm , that is, the motion planning from the start of the grasping state to the state when the task is completed can be obtained by formula (22), that is:

[0171] (twenty two);

[0172] Among them, N-M+1 represents the total number of moments from moment M to moment N.

[0173] According to an embodiment of the present invention, the above equations (21) and (22) can be combined to obtain the initial pose estimation of the robot arm corresponding to the initial state: The second pose estimation corresponding to the state when the task is completed The complete motion plan between , which can be obtained by equation (21) and equation (22), as shown in equation (23):

[0174] (twenty three);

[0175] in, Indicates the total number of moments from time 0 to time N.

[0176] Since the state of the robot at time M is calculated in the calculation process of equations (21) and (22), when splicing them to obtain a complete motion plan, the total number of moments needs to be calculated one less time. Therefore, the total number of moments in equation (23) is N+1.

[0177] Another implementation method is to directly use the above (23) to calculate the initial pose estimate corresponding to the initial state of the robot arm The second pose estimation corresponding to the state when the task is completed The complete motion plan between , I will not go into details here.

[0178] According to an embodiment of the present invention, based on the above method, the initial pose estimation corresponding to the initial state of the robot arm can be obtained. The second pose estimation corresponding to the state when the task is completed The complete motion plan between , that is, the movement trajectory information of the robot arm. According to the movement trajectory information of the robot arm, based on the target object description text w and the target task description text g, the robot arm can be controlled to perform corresponding operations on the target object corresponding to the target task description text, thereby realizing the entire process of the robot arm from the initial state to the grasping state and then to the task completion state.

[0179] Fig. 6A A schematic diagram of obtaining a point cloud image of a target object according to an embodiment of the present invention is shown.

[0180] like Fig. 6A As shown, the target language token sequence 601 is input into the large language model 602 to obtain the target object description text 603 and the target task description text 604. The target object description text 603 and the environment observation image 605 are input into the open world object detector network 606-1 of the open world environment perception model 606 to obtain the bounding box information 607 of the target object. The bounding box information 607 of the target object and the environment observation image 605 are then input into the image segmentation network 606-2 of the open world environment perception model 606 to obtain the first mask image 608 of the target object. The environment depth image 609 and the first mask image 608 are multiplied to obtain the second mask image 610 of the target object. The second mask image is three-dimensionally reconstructed using the intrinsic and extrinsic information of the depth camera to obtain the point cloud image 611 of the target object.

[0181] Figure 6B A schematic diagram of a robot arm intelligent control method according to an embodiment of the present invention is shown.

[0182] like Figure 6BAs shown, on the basis of the point cloud image 611 obtained above, the point cloud image 611 is sparsely sampled layer by layer, the point cloud information for each layer of the point cloud image is determined, and the point cloud set 612 of the point cloud image is obtained. The size information 613 of the point cloud image is obtained, and the first pose estimation information 614 of the robot arm is determined according to the target object description text 603, the point cloud set 612 and the size information 613 of the point cloud image. According to the first pose estimation information 614 and the target task description text 604, the second pose estimation information 615 of the robot arm is determined. According to the target object description text 603, the target task description text 604, the initial pose estimation information 616 of the robot arm and the second pose estimation information 615, the third pose estimation information 617 of the robot arm is determined. According to the first pose estimation information 614, the second pose estimation information 615 and the third pose estimation information 617, the pose estimation information 618 of the robot arm is obtained. The posture estimation information 618 of the robot arm is kinematically analyzed to obtain the joint drive angle information 619 of the robot arm corresponding to the posture estimation information. Based on the joint drive angle information 619 of the robot arm, the motion trajectory information 620 when the robot arm performs the corresponding operation corresponding to the target task description text is obtained. Based on the motion trajectory information 620, the robot arm is controlled to perform the corresponding operation 621 corresponding to the target task description text on the target object.

[0183] Figure 6C A schematic diagram of a robot arm intelligent control method according to another embodiment of the present invention is shown.

[0184] like Figure 6C As shown, for example, the target language token sequence is represented as "lift the bottle with a white lid to a height of 0.5 meters from the ground and rotate it to a 45-degree angle with the vertical upward direction" 622, and the target language token sequence 622 is parsed by using the large language model to obtain the target object description text "bottle with a white lid" 623 and the target task description text "0.5 meters from the ground, rotate it to a 45-degree angle with the vertical upward direction" 624. According to the point cloud image information 625 of the target object determined by the open world environment perception model and the target object description text "bottle with a white lid" 623, the first pose estimation information 626 of the manipulator is determined. According to the point cloud image information 625 of the target object determined by the open world environment perception model and the target task description text "0.5 meters from the ground, rotate it to a 45-degree angle with the vertical upward direction" 624, the second pose estimation information 627 of the manipulator is determined. Based on the first pose estimation information 626, the second pose estimation information 627 and the initial pose estimation information 628, the motion trajectory information 629 of the robotic arm is determined, and according to the motion trajectory information 629 of the robotic arm, the robotic arm is controlled to perform corresponding operations 630 corresponding to the target task description text on the target object.

[0185] Figure 7 A block diagram of an intelligent control device for a robotic arm according to an embodiment of the present invention is shown.

[0186] like Figure 7 As shown, the device 700 may include: an image acquisition module 710 , a text acquisition module 720 , a mask image acquisition module 730 , a point cloud image acquisition module 740 , a pose estimation information acquisition module 750 and a control module 760 .

[0187] The image acquisition module 710 is used to acquire an environment observation image and an environment depth image, wherein the environment observation image and the environment depth image both contain a target object and are obtained by taking pictures with a depth camera.

[0188] The text acquisition module 720 is used to input the target language token sequence into the large language model to obtain the target object description text and the target task description text, wherein the target task description text represents the text for controlling the robot arm to perform corresponding operations on the target object.

[0189] The mask image acquisition module 730 is used to input the target object description text and the environment observation image into the open world environment perception model to obtain a first mask image of the target object.

[0190] The point cloud image acquisition module 740 is used to obtain a point cloud image of the target object according to the first mask image and the environment depth image.

[0191] The pose estimation information acquisition module 750 is used to input the target object description text, the target task description text and the point cloud image into the large language model to obtain the pose estimation information of the robot arm.

[0192] The control module 760 is used to control the robotic arm to perform corresponding operations corresponding to the target task description text on the target object according to the posture estimation information.

[0193] According to an embodiment of the present invention, an open-world environment perception model includes an open-world object detector network and an image segmentation network.

[0194] The mask image obtaining module 730 may include: a bounding box obtaining submodule and a first mask image obtaining submodule.

[0195] The bounding box acquisition submodule is used to input the target object description text and the environment observation image into the open world object detector network to obtain the bounding box information of the target object.

[0196] The first mask image acquisition submodule is used to input the bounding box information of the target object and the environmental observation image into the image segmentation network to obtain the first mask image of the target object.

[0197] According to an embodiment of the present invention, the bounding box obtaining submodule may include: a first image feature vector obtaining unit, a text feature vector obtaining unit, a feature interaction enhancement processing unit and a bounding box obtaining unit.

[0198] The first image feature vector obtaining unit is used to extract features from the environment observation image to obtain a first image feature vector.

[0199] The text feature vector obtaining unit is used to extract features from the target object description text to obtain a text feature vector.

[0200] The feature interaction enhancement processing unit is used to perform feature interaction enhancement processing on the first image feature vector and the text feature vector to obtain an enhanced image feature vector and an enhanced text feature vector.

[0201] The bounding box obtaining unit is used to obtain the bounding box information of the target object according to the enhanced image feature vector and the enhanced text feature vector.

[0202] According to an embodiment of the present invention, the first mask image obtaining submodule may include: a bounding box feature vector obtaining unit, a second image feature vector obtaining unit and a first mask image obtaining unit.

[0203] The bounding box feature vector obtaining unit is used to encode the bounding box information of the target object to obtain a bounding box feature vector.

[0204] The second image feature vector obtaining unit is used to perform image encoding on the environment observation image to obtain a second image feature vector.

[0205] The first mask image obtaining unit is used to decode the boundary box feature vector and the second image feature vector to obtain a first mask image of the target object.

[0206] According to an embodiment of the present invention, the pose estimation information acquisition module 750 may include: an information acquisition submodule, a point cloud set acquisition submodule, a first pose estimation information determination submodule, a second pose estimation information determination submodule, a third pose estimation information determination submodule and a robotic arm pose estimation information acquisition submodule.

[0207] The information acquisition submodule is used to obtain the size information of the point cloud image and the initial pose estimation information of the robotic arm.

[0208] The point cloud set acquisition submodule is used to perform sparse sampling on the point cloud image layer by layer, determine the point cloud information for each layer of the point cloud image, and obtain the point cloud set of the point cloud image.

[0209] The first pose estimation information determination submodule is used to determine the first pose estimation information of the robotic arm based on the target object description text, size information and point cloud set, wherein the first pose estimation information represents the pose estimation when the robotic arm starts to perform grasping the target object.

[0210] The second pose estimation information determination submodule is used to determine the second pose estimation information of the robot arm based on the first pose estimation information and the target task description text, wherein the second pose estimation information represents the pose estimation of the robot arm when completing the corresponding operation corresponding to the target task description text.

[0211] The third pose estimation information determination submodule is used to determine at least one third pose estimation information of the robotic arm based on the target object description text, the target task description text, the initial pose estimation information and the second pose estimation information, wherein the third pose estimation information represents the intermediate pose estimation of the robotic arm from the initial pose estimation to the second pose estimation.

[0212] The robot arm posture estimation information acquisition submodule is used to obtain the posture estimation information of the robot arm according to the first posture estimation information, the second posture estimation information and the third posture estimation information.

[0213] According to an embodiment of the present invention, the control module 760 may include: a parsing submodule, a motion trajectory information obtaining submodule and a control submodule.

[0214] The analysis submodule is used to perform kinematic analysis on the pose estimation information to obtain the joint drive angle information of the robot arm corresponding to the pose estimation information.

[0215] The motion trajectory information acquisition submodule is used to obtain the motion trajectory information when the robot arm performs the corresponding operation corresponding to the target task description text according to the joint drive angle information.

[0216] The control submodule is used to control the robotic arm to perform corresponding operations on the target object corresponding to the target task description text according to the motion trajectory information.

[0217] According to an embodiment of the present invention, the point cloud image acquisition module 740 may include: a binary image acquisition submodule, a second mask image acquisition submodule, a two-dimensional pixel acquisition submodule, an internal and external parameter information acquisition submodule and a point cloud image acquisition submodule.

[0218] The binary image obtaining submodule is used to perform binary encoding on the first mask image to obtain a binary image of the first mask image.

[0219] The second mask image obtaining submodule is used to perform a product operation on the binary image and the environment depth image to obtain a second mask image of the target object.

[0220] The two-dimensional pixel point acquisition submodule is used to acquire the two-dimensional pixel point information of the second mask image.

[0221] The internal and external parameter information acquisition submodule is used to obtain the internal and external parameter information of the depth camera.

[0222] The point cloud image acquisition submodule is used to perform three-dimensional reconstruction on the second mask image using two-dimensional pixel information, internal reference information and external reference information to obtain a point cloud image of the target object.

[0223] According to an embodiment of the present invention, the text obtaining module 720 may include: a word embedding vector obtaining submodule, a position vector obtaining submodule, a target language feature vector obtaining submodule and a text obtaining submodule.

[0224] The word embedding vector acquisition submodule is used to vector encode the target language token sequence to obtain the target language word embedding vector.

[0225] The position vector acquisition submodule is used to position encode the target language token sequence to obtain the target language position vector.

[0226] The target language feature vector acquisition submodule is used to sum the target language embedding vector and the target language position vector to obtain the target language feature vector.

[0227] The text acquisition submodule is used to extract features from the target language feature vector to obtain the target object description text and the target task description text.

[0228] According to an embodiment of the present invention, any multiple of the image acquisition module 710, the text acquisition module 720, the mask image acquisition module 730, the point cloud image acquisition module 740, the pose estimation information acquisition module 750 and the control module 760 can be combined in one module / submodule / unit / subunit for implementation, or any one of the modules / submodules / units / subunits can be split into multiple modules / submodules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / submodules / units / subunits can be combined with at least part of the functions of other modules / submodules / units / subunits and implemented in one module / submodule / unit / subunit. According to an embodiment of the present invention, at least one of the image acquisition module 710, the text acquisition module 720, the mask image acquisition module 730, the point cloud image acquisition module 740, the pose estimation information acquisition module 750 and the control module 760 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a proper combination of any of them. Alternatively, at least one of the image acquisition module 710, the text acquisition module 720, the mask image acquisition module 730, the point cloud image acquisition module 740, the pose estimation information acquisition module 750 and the control module 760 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0229] It should be noted that the part of the robot arm intelligent control device in the embodiment of the present invention corresponds to the part of the robot arm intelligent control method in the embodiment of the present invention. The description of the part of the robot arm intelligent control device specifically refers to the part of the robot arm intelligent control method, which will not be repeated here.

[0230] Figure 8 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is shown. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0231] like Figure 8As shown, the electronic device 800 according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0232] In RAM 803, various programs and data required for the operation of electronic device 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other via bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in one or more memories.

[0233] According to an embodiment of the present invention, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the input / output (I / O) interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read therefrom is installed into the storage portion 808 as needed.

[0234] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0235] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0236] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0237] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 802 and / or the RAM 803 described above and / or one or more memories other than the ROM 802 and the RAM 803 .

[0238] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the robotic arm intelligent control method provided by the embodiment of the present invention.

[0239] When the computer program is executed by the processor 801, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0240] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0241] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0242] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0243] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for intelligent control of a robotic arm, characterized in that: The method comprises: Acquire an environment observation image and an environment depth image, wherein both the environment observation image and the environment depth image contain a target object, and the environment observation image and the environment depth image are obtained by taking a depth camera; Inputting the target language token sequence into the large language model to obtain a target object description text and a target task description text, wherein the target task description text represents the text for controlling the robot arm to perform a corresponding operation on the target object; Inputting the target object description text and the environment observation image into an open world environment perception model to obtain a first mask image of the target object; Obtaining a point cloud image of the target object according to the first mask image and the environment depth image; Inputting the target object description text, the target task description text and the point cloud image into the large language model to obtain the pose estimation information of the robotic arm; According to the pose estimation information, controlling the robotic arm to perform a corresponding operation on the target object corresponding to the target task description text; Wherein, the open world environment perception model includes an open world object detector network and an image segmentation network; The step of inputting the target object description text and the environment observation image into an open world environment perception model to obtain a first mask image of the target object includes: Inputting the target object description text and the environment observation image into the open world object detector network to obtain the bounding box information of the target object; The bounding box information of the target object and the environment observation image are input into the image segmentation network to obtain a first mask image of the target object.

2. The method according to claim 1, characterized in that The step of inputting the target object description text and the environment observation image into the open world object detector network to obtain the bounding box information of the target object includes: Extracting features from the environment observation image to obtain a first image feature vector; Extracting features from the target object description text to obtain a text feature vector; Performing feature interaction enhancement processing on the first image feature vector and the text feature vector to obtain an enhanced image feature vector and an enhanced text feature vector; The bounding box information of the target object is obtained according to the enhanced image feature vector and the enhanced text feature vector.

3. The method according to claim 1, characterized in that The step of inputting the bounding box information of the target object and the environment observation image into the image segmentation network to obtain a first mask image of the target object includes: Encoding the bounding box information of the target object to obtain a bounding box feature vector; Performing image encoding on the environment observation image to obtain a second image feature vector; The bounding box feature vector and the second image feature vector are decoded to obtain a first mask image of the target object.

4. The method according to claim 1, characterized in that: The step of inputting the target object description text, the target task description text and the point cloud image into the large language model to obtain the pose estimation information of the robotic arm includes: Acquire size information of the point cloud image and initial pose estimation information of the robotic arm; Performing sparse sampling on the point cloud image layer by layer, determining point cloud information for each layer of the point cloud image, and obtaining a point cloud set of the point cloud image; Determine, according to the target object description text, the size information and the point cloud set, first pose estimation information of the robotic arm, wherein the first pose estimation information represents the pose estimation of the robotic arm when it starts to grasp the target object; Determine second pose estimation information of the robotic arm according to the first pose estimation information and the target task description text, wherein the second pose estimation information represents the pose estimation of the robotic arm when completing a corresponding operation corresponding to the target task description text; Determine at least one third pose estimation information of the robotic arm according to the target object description text, the target task description text, the initial pose estimation information and the second pose estimation information, wherein the third pose estimation information represents an intermediate pose estimation of the robotic arm from the initial pose estimation to the second pose estimation; The pose estimation information of the robotic arm is obtained according to the first pose estimation information, the second pose estimation information and the third pose estimation information.

5. The method according to claim 4, characterized in that The step of controlling the robot arm to perform a corresponding operation on the target object corresponding to the target task description text according to the pose estimation information includes: Performing kinematic analysis on the pose estimation information to obtain joint drive angle information of the robotic arm corresponding to the pose estimation information; According to the joint driving angle information, obtaining motion trajectory information when the robot arm performs a corresponding operation corresponding to the target task description text; According to the motion trajectory information, the robotic arm is controlled to perform a corresponding operation on the target object corresponding to the target task description text.

6. The method according to claim 1, characterized in that The step of obtaining a point cloud image of the target object according to the first mask image and the environment depth image includes: performing binary encoding on the first mask image to obtain a binary image of the first mask image; Performing a product operation on the binary image and the environment depth image to obtain a second mask image of the target object; Acquire two-dimensional pixel information of the second mask image; Obtaining intrinsic and extrinsic information of the depth camera; The second mask image is three-dimensionally reconstructed using the two-dimensional pixel information, the internal reference information, and the external reference information to obtain a point cloud image of the target object.

7. The method according to claim 1, characterized in that The target language token sequence is input into the large language model to obtain the target object description text and the target task description text, including: Performing vector encoding on the target language token sequence to obtain a target language word embedding vector; Performing position encoding on the target language token sequence to obtain a target language position vector; Summing the target language embedding vector and the target language position vector to obtain a target language feature vector; Feature extraction is performed on the target language feature vector to obtain a target object description text and a target task description text.

8. A robot arm intelligent control device, characterized in that: The device comprises: An image acquisition module, used to acquire an environment observation image and an environment depth image, wherein both the environment observation image and the environment depth image contain a target object, and the environment observation image and the environment depth image are obtained by taking a depth camera; A text acquisition module, used for inputting a target language token sequence into a large language model to obtain a target object description text and a target task description text, wherein the target task description text represents a text for controlling the robot arm to perform a corresponding operation on the target object; A mask image acquisition module, used for inputting the target object description text and the environment observation image into an open world environment perception model to obtain a first mask image of the target object, wherein the open world environment perception model includes an open world object detector network and an image segmentation network; A point cloud image acquisition module, used to obtain a point cloud image of the target object according to the first mask image and the environment depth image; a pose estimation information acquisition module, used to input the target object description text, the target task description text and the point cloud image into the large language model to obtain the pose estimation information of the robotic arm; A control module, configured to control the robotic arm to perform a corresponding operation on the target object corresponding to the target task description text according to the pose estimation information; Wherein, the mask image acquisition module includes: A bounding box acquisition submodule, used for inputting the target object description text and the environment observation image into the open world object detector network to obtain the bounding box information of the target object; The first mask image obtaining submodule is used to input the bounding box information of the target object and the environmental observation image into the image segmentation network to obtain a first mask image of the target object.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images