Method and system for constructing large manipulation model of robot long-range task and intelligent robot

Abstract

The invention discloses a robot long-range task manipulation large model construction method and system and an intelligent robot, and belongs to the technical field of robot navigation technologies and multi-mode large models. According to the method, the geometrical characteristics of the to-be-controlled object are recognized, the operable graph corresponding to the to-be-controlled object is recognized according to the control action, comprehensive understanding of the to-be-controlled object is provided for the intelligent robot in the operation process, and the posture accuracy of the end effector of the intelligent robot is improved; and under the supervision of the objective function, training the multi-modal large model according to the operable diagram to obtain a control large model so as to control the attitude of the end effector of the intelligent robot, and further improving the attitude accuracy of the end effector of the intelligent robot.

Description

Technical Field

[0001] The present invention relates to the fields of robot navigation technology and multi-modal large model technology, and in particular to a method and system for constructing a large manipulation model for a robot long-range task, and an intelligent robot. Background Art

[0002] When the robot performs long-range mobile manipulation tasks, it mainly uses a pre-trained large language model as an agent for navigation tasks, and uses a multimodal large model for manipulation and operation of the object to be manipulated.

[0003] The prior art still has the following problems:

[0004] Multimodal large models still cannot realize the autonomous action of robots, especially in the long-range task processing of robots in open and complex scenes. The calculation ability of the posture of the end effector of the intelligent robot is poor, which makes the robot unable to accurately operate the manipulated objects during the operation, and cannot meet the needs of practical applications for intelligent robots. Summary of the invention

[0005] In order to solve the problems of the prior art, the embodiments of the present invention provide a method, system and intelligent robot for constructing a large model of robot long-range tasks, including:

[0006] On the one hand, a method for constructing a large manipulation model for a robot long-range task is provided, the method comprising:

[0007] Identify the geometric characteristics of the object to be manipulated;

[0008] generating an operable graph corresponding to an operation action according to the geometric characteristics, wherein the operation action includes a rotation operation and a translation operation facing the object to be manipulated;

[0009] Set the objective function;

[0010] Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain a manipulation large model to control the posture of the end effector of the intelligent robot.

[0011] Optionally, the identifying geometric characteristics of the object to be manipulated includes:

[0012] Identify the pixels of the object to be manipulated from the captured image;

[0013] A geometric property indicated by the pixel is identified, where the geometric property is used to indicate a shape of an object to be manipulated.

[0014] Optionally, generating an operable graph corresponding to the operation action according to the geometric characteristic includes:

[0015] generating a first operable graph corresponding to a rotation operation according to the geometric characteristic;

[0016] A second operable map corresponding to the direct-acting operation is generated according to the geometric characteristic.

[0017] Optionally, generating a first operable operation corresponding to a rotation operation according to the geometric characteristic includes:

[0018] According to the geometric characteristics, identifying the axis of the object to be manipulated;

[0019] Simulating the rotation of the object to be controlled along the axis;

[0020] A first operable graph of the object to be controlled after rotation is calculated, wherein the first operable graph includes operability probabilities of various regions in the graph.

[0021] Optionally, generating a second operable graph corresponding to the direct-acting operation according to the geometric characteristic includes:

[0022] Identifying a direct motion area of ​​the object to be manipulated;

[0023] A second operable graph of the object to be manipulated is calculated, and the operability probability of the direct motion area in the second operable graph is set to a preset value.

[0024] Optionally, setting the objective function includes:

[0025] Using a first cross entropy loss function, training the multimodal large model to learn to distinguish control points, and recording a first loss tensor corresponding to the first cross entropy loss function;

[0026] Obtain the RGB image when the operation is successful and the posture of the end effector of the intelligent robot through the second cross entropy loss function, and record the second loss tensor corresponding to the second cross entropy loss function;

[0027] Obtain the predicted value and the true value of the multimodal large model through a third cross entropy loss function, and record a third loss tensor corresponding to the third cross entropy loss function;

[0028] An objective function is set according to the first loss tensor, the second loss tensor, and the third loss tensor.

[0029] Optionally, the step of training the multimodal large model according to the operable graph under the supervision of the objective function to obtain the manipulation large model comprises:

[0030] Set the training steps;

[0031] Under the supervision of the objective function, according to the operational graph and in accordance with the training steps, the multimodal large model is trained to obtain the manipulation large model.

[0032] Optionally, the step of setting training includes:

[0033] The multimodal large model outputs the predicted coordinate values ​​of the operable points;

[0034] Determining a direction vector of the end effector of the intelligent robot;

[0035] The coordinate values ​​of the operable points are projected from the image coordinate system to the three-dimensional space coordinate system through the depth image data.

[0036] On the other hand, a system for constructing a large-scale manipulation model for a robot long-range task is provided, the system comprising an intelligent robot and a large-scale manipulation model construction device, wherein:

[0037] The intelligent robot is used for:

[0038] Acquire image data and three-dimensional space data of the environment; the image and the three-dimensional space at least include the object to be manipulated;

[0039] The large model construction device is used for:

[0040] Identify the geometric characteristics of the object to be manipulated;

[0041] generating an operable graph corresponding to an operation action according to the geometric characteristics, wherein the operation action includes a rotation operation and a translation operation facing the object to be manipulated;

[0042] Set the objective function;

[0043] Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain a manipulation large model to control the posture of the end effector of the intelligent robot.

[0044] On the other hand, an intelligent robot is provided, comprising:

[0045] A camera, used to obtain multiple images of the environment, wherein the multiple images at least include the object to be manipulated;

[0046] A calculation module is used to calculate the image data and three-dimensional space data of the environment through the multiple images; so that the manipulation large model construction device can identify the geometric characteristics of the object to be manipulated through the image data and the three-dimensional space data; generate an operable graph corresponding to the operation action according to the geometric characteristics, and the operation action includes a rotation operation and a direct motion operation facing the object to be manipulated; set an objective function; under the supervision of the objective function, train the multimodal large model according to the operable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot.

[0047] The present invention has at least the following beneficial effects:

[0048] 1. By identifying the geometric characteristics of the object to be manipulated and identifying the operable graph corresponding to the object to be manipulated according to the manipulation action, the intelligent robot is provided with a comprehensive understanding of the object to be manipulated during the operation process, thereby improving the accuracy of the posture of the end effector of the intelligent robot;

[0049] 2. Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot, further improving the accuracy of the posture of the end effector of the intelligent robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic flow chart of a method for constructing a large manipulation model for a robot long-range task provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a method for constructing a large manipulation model for a robot long-range task provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a large-scale model construction system for manipulating a robot for long-range tasks provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of the structure of an intelligent robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] Reference Figure 1 As shown, on the one hand, a method for constructing a large manipulation model for a robot long-range task is provided, the method comprising:

[0057] 101. Identify the geometric characteristics of the object to be manipulated;

[0058] 102. Generate an operable graph corresponding to an operation action according to the geometric characteristics, where the operation action includes a rotation operation and a direct motion operation facing the object to be controlled;

[0059] 103. Set the objective function;

[0060] 104. Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot.

[0061] It should be noted that the robot model is a robot model that integrates an instruction recognition model, an environmental perception model, a navigation model and a manipulation model. The multimodal model described in the embodiment of the present invention is the manipulation model, that is, after obtaining the operation instruction information, the operation model that executes the robot to operate the object according to the operation instruction.

[0062] Optionally, identifying the geometric characteristics of the object to be manipulated includes:

[0063] Identify the pixels of the object to be manipulated from the captured image;

[0064] The geometric properties indicated by the pixels are identified, and the geometric properties are used to indicate the shape of the object to be manipulated.

[0065] In practical applications, the process of identifying the pixels of the object to be controlled from the captured image can be implemented by an image depth recognition algorithm, such as a convolutional neural network, and the process can be:

[0066] Through the image depth recognition algorithm, the initial shape recognition model is constructed;

[0067] From the images taken historically, multiple images containing the objects to be manipulated are selected as training sets;

[0068] The initial shape recognition model is trained through the training set to obtain a shape recognition model;

[0069] The captured image is input into the shape recognition model, and the pixels of the object to be manipulated are output;

[0070] Based on the preset object category recognition model, the shape of the object to be manipulated is obtained through the pixels of the object to be manipulated;

[0071] The shape is used to indicate whether the object to be controlled is rotatable or linearly movable;

[0072] The object category recognition model can be constructed by a probabilistic neural network; that is, the input of the probabilistic neural network is set to be the pixels of the object to be manipulated, and the output is either rotatable or linear;

[0073] It should be noted that the update operation for the object category recognition model and the shape recognition model is not performed to avoid the reduction of the stability of object shape recognition and the stability of category recognition ability, so as to provide reliable object shape recognition results and category recognition results for subsequent steps.

[0074] Optionally, generating an operable graph corresponding to the operation action according to the geometric characteristics includes:

[0075] generating a first operable graph corresponding to a rotation operation according to the geometric characteristics;

[0076] A second operable map corresponding to the direct-acting operation is generated according to the geometric characteristics.

[0077] In practical applications, the above-mentioned operable graph indicates the manipulation potential of each pixel point of the object to be manipulated in the form of probability based on the geometric characteristics of the object to be manipulated. The manipulation potential may be the displacement distance that may be achieved by manipulating certain pixels.

[0078] Optionally, generating a first operable graph corresponding to the rotation operation according to the geometric characteristics includes:

[0079] According to the geometric characteristics, identify the axis line of the object to be manipulated;

[0080] Simulate the rotation of the object to be controlled along the axis;

[0081] A first operable graph of the rotated object to be manipulated is calculated, where the first operable graph includes operability probabilities of various regions in the graph.

[0082] In practical applications, the above process can be specifically as follows:

[0083] Determine the axis line of the object to be manipulated according to the pixels of the object to be manipulated and the shape of the object to be manipulated;

[0084] The rotation of the object to be manipulated along the axis is simulated, and the first operable graph is calculated based on this. For any pixel point, the operability probability A is:

[0085]

[0086] Where D is the displacement distance of the pixel point when it rotates along the axis, max(D) is the maximum displacement distance, min(D) is the minimum displacement distance, and the value of A is between 0 and 1;

[0087] After performing the above operations on all pixels, a first operable graph can be obtained.

[0088] Optionally, generating a second operable graph corresponding to the direct-acting operation according to the geometric characteristic includes:

[0089] According to the geometric characteristics, the direct motion area of ​​the object to be manipulated is identified;

[0090] A second operable graph of the object to be manipulated is calculated, and the operability probability of the direct motion area in the second operable graph is set to a preset value.

[0091] In practical applications, the process of identifying the direct motion area of ​​the object to be manipulated can be specifically as follows based on the geometric characteristics:

[0092] Identify the operating surface targeted by the direct-action operation of the object to be controlled, and the operating surface is the direct-action area;

[0093] The above recognition process can be realized by an image depth recognition algorithm, namely:

[0094] Identify all parts of the object to be controlled through image depth recognition algorithm;

[0095] An operation surface is selected from all the components; the selection process is related to the operation instruction, that is, the component targeted by the operation instruction is identified, and the operation area where the component is located is the operation surface;

[0096] Taking the opening and closing of a drawer as an example, if the operation instruction is to open the drawer, the recognized operation is to open and close the drawer; the above process can be:

[0097] Identify the component (drawer) to which the operation instruction (opening the drawer) is directed from the operable object (such as furniture containing the drawer) where the drawer is located;

[0098] The operating surface of the drawer is the drawer surface, that is, the direct-acting area is the drawer surface.

[0099] The above process of calculating the second operable graph of the object to be controlled and setting the operability probability of the direct motion area in the second operable graph to a preset value may be:

[0100] Since any point in the direct motion area can cause direct motion displacement, all pixel points in the second operable graph can cause direct motion displacement, so the probability values ​​of all pixel points in the second operable graph are set to 1, that is, each pixel point has the same control potential.

[0101] Optional, see Figure 2 As shown, setting the objective function includes:

[0102] 201. Train the multimodal large model to learn to distinguish the control points through the first cross entropy loss function, and record the first loss tensor corresponding to the first cross entropy loss function;

[0103] Specifically, after obtaining the operability graphs, these visualized operability graphs are converted into machine-interpretable language descriptions. The process can be:

[0104] A plurality of positive pixels and a plurality of negative pixels are randomly selected from the operability graph, and the plurality of positive pixels and the plurality of negative pixels are used as training samples; the probability of the positive pixels is greater than or equal to a preset value (such as 0.5), and the probability of the negative pixels is less than the preset value;

[0105] The manipulation model is trained through the training sample;

[0106] Set the text prompt of the positive pixel point as controllable, and the text prompt of the negative pixel point as uncontrollable; the output of the model is "yes" or "no", "yes" is used to indicate controllable, and "no" is used to indicate uncontrollable;

[0107] Under the guidance of the first cross entropy loss function, the model is manipulated to learn the controllable point distinction, and the first loss tensor corresponding to the first cross entropy loss function is recorded.

[0108] Through the above steps, the manipulation model can predict the contact position that can effectively promote the movement of the object.

[0109] It should be noted that during the training process, at least the coordinate values ​​of multiple positive pixel points and multiple negative pixel points need to be input into the manipulation model;

[0110] 202. Obtain the RGB image when the operation is successful and the posture of the end effector of the intelligent robot through the second cross entropy loss function, and record the second loss tensor corresponding to the second cross entropy loss function;

[0111] During the generation of the maneuverability graph, the following steps are required to train the model to accurately generate the posture of the robot end effector:

[0112] Obtain the RGB image and the corresponding end-effector posture data when the robot successfully operates the object to be operated. These data are used as the basis for model input and output verification.

[0113] Based on the above RGB image, generate an operability map when the operation is successful;

[0114] Generate, according to the end effector posture data, posture data for the controllable point when the operation is generated, the posture data including a direction vector; the direction vector including an upward direction vector and a forward direction vector;

[0115] Set the text prompt of the controllable point to controllable, and the output of the manipulation model to "yes" and the actuator posture data. "yes" is used to indicate controllable;

[0116] In the above process, in order to improve the training efficiency, the continuous values ​​of the direction vector are divided into 100 intervals ranging from [-50, 50], and the span of each interval is 0.02; the posture data output by the manipulation model is divided into 100 categories according to the 100 intervals;

[0117] Under the guidance of the second cross entropy loss function, the model is manipulated to learn the controllable point distinction and end effector posture data, and the second loss tensor corresponding to the second cross entropy loss function is recorded.

[0118] It should be noted that during the training process, at least the coordinate values ​​of the controllable points and the direction vector need to be input into the manipulation model;

[0119] Among them, the upward direction vector and the forward direction vector of the gripper not only define the spatial direction of the gripper, but also constitute the posture of the end effector in terms of rotation angle. These two direction vectors and the coordinates of the operable point together determine the initial interaction mode between the end effector and the object, providing the basis for subsequent precise manipulation.

[0120] 203. Obtain the predicted value and the true value of the multimodal large model through the third cross entropy loss function, and record the third loss tensor corresponding to the third cross entropy loss function;

[0121] In practical applications, the above process can be:

[0122] In step 202, perform the following operations:

[0123] The coordinate value or direction vector of the controllable point is shielded, that is, the coordinate value and direction vector of the controllable point are not input into the manipulation model; that is:

[0124] Inputting an operable diagram and an operation instruction into the manipulation model; the manipulation model predicts an operable point and a direction vector;

[0125] The third loss tensor is calculated by the third cross entropy loss function through the actual operable point of the object to be manipulated and the actual direction vector when the operation is successful, and the predicted result and the true value of the unmasked answer are compared.

[0126] Through the above steps, the accuracy of posture data prediction can be improved, and the ability of posture data prediction can be enhanced.

[0127] 204. Set an objective function according to the first loss tensor, the second loss tensor, and the third loss tensor.

[0128] In practical applications, during the model training phase, the objective function is set through the first loss tensor, the second loss tensor, and the third loss tensor, and is recorded as:

[0129]

[0130] The training of the manipulation model is supervised through the above objective function.

[0131] Optionally, under the supervision of the objective function, the multimodal large model is trained according to the operable graph, and the manipulation large model includes:

[0132] Set the training steps;

[0133] Under the supervision of the objective function, the multimodal large model is trained according to the manipulable graph and the training steps to obtain the manipulated large model.

[0134] In practical applications, this objective function will coordinate the loss functions of different training tasks to optimize the model's performance in generating the robot's end-effector pose.

[0135] Optionally, set the training steps to include:

[0136] The coordinate values ​​of the operable points predicted by the multimodal large model output;

[0137] Determine the direction vector of the end effector of the intelligent robot; this process is also predicted by the multi-modal large model;

[0138] The coordinate values ​​of the operable points are projected from the image coordinate system to the three-dimensional space coordinate system through the depth image data.

[0139] That is, in the training stage, the multimodal large model predicts the coordinate values ​​of the output operable points and the direction vector of the end effector, and uses the objective function, the coordinate values ​​of the actual operable points, and the actual direction vector of the end effector as supervision to perform training and obtain the manipulation large model.

[0140] Reference Figure 3As shown, a large-scale manipulation model construction system for a robot long-range task is provided, the system includes an intelligent robot and a large-scale manipulation model construction device, wherein:

[0141] Intelligent robots are used for:

[0142] Acquire image data and three-dimensional space data of the environment; the image and the three-dimensional space at least include the object to be manipulated;

[0143] Manipulate large model building devices for:

[0144] Identify the geometric characteristics of the object to be manipulated;

[0145] According to the geometric characteristics, an operable graph corresponding to the operation action is generated, and the operation action includes a rotation operation and a direct motion operation facing the object to be manipulated;

[0146] Set the objective function;

[0147] Under the supervision of the objective function, the multimodal large model is trained according to the manipulable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot.

[0148] Optionally, the large model building device is specifically used to:

[0149] Identify the pixels of the object to be manipulated from the captured image;

[0150] The geometric properties indicated by the pixels are identified, and the geometric properties are used to indicate the shape of the object to be manipulated.

[0151] Optionally, the large model building device is specifically used to:

[0152] generating a first operable graph corresponding to a rotation operation according to the geometric characteristics;

[0153] A second operable map corresponding to the direct-acting operation is generated according to the geometric characteristics.

[0154] Optionally, the large model building device is specifically used to:

[0155] According to the geometric characteristics, identify the axis line of the object to be manipulated;

[0156] Simulate the rotation of the object to be controlled along the axis;

[0157] A first operable graph of the rotated object to be manipulated is calculated, where the first operable graph includes operability probabilities of various regions in the graph.

[0158] Optionally, the large model building device is specifically used to:

[0159] Identify the direct motion area of ​​the object to be manipulated;

[0160] A second operable graph of the object to be manipulated is calculated, and the operability probability of the direct motion area in the second operable graph is set to a preset value.

[0161] Optionally, the large model building device is specifically used to:

[0162] Through the first cross entropy loss function, the multimodal large model is trained to learn to distinguish the control points, and the first loss tensor corresponding to the first cross entropy loss function is recorded;

[0163] Through the second cross entropy loss function, the RGB image when the operation is successful and the posture of the end effector of the intelligent robot are obtained, and the second loss tensor corresponding to the second cross entropy loss function is recorded;

[0164] Obtain the predicted value and true value of the multimodal large model through the third cross entropy loss function, and record the third loss tensor corresponding to the third cross entropy loss function;

[0165] According to the first loss tensor, the second loss tensor, and the third loss tensor, an objective function is set.

[0166] Optionally, the large model building device is specifically used to:

[0167] Set the training steps;

[0168] Under the supervision of the objective function, the multimodal large model is trained according to the manipulable graph and the training steps to obtain the manipulated large model.

[0169] Optionally, the large model building device is specifically used to:

[0170] The coordinate values ​​of the operable points predicted by the multimodal large model output;

[0171] Determine the direction vector of the end effector of the intelligent robot;

[0172] The coordinate values ​​of the operable points are projected from the image coordinate system to the three-dimensional space coordinate system through the depth image data.

[0173] Reference Figure 4 As shown, an intelligent robot is provided, and the intelligent robot includes:

[0174] A camera, used to obtain multiple images of the environment, wherein the multiple images at least include the object to be manipulated;

[0175] The computing module is used to compute the image data and three-dimensional space data of the environment through multiple images; so that the manipulation large model construction device can identify the geometric characteristics of the object to be manipulated through the image data and the three-dimensional space data; generate an operable graph corresponding to the operation action according to the geometric characteristics, and the operation action includes a rotation operation and a direct motion operation facing the object to be manipulated; set an objective function; under the supervision of the objective function, train the multimodal large model according to the operable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot.

[0176] The above specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0177] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). To make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

[0178] The present invention is described in more detail through general description and specific embodiments. It should be noted that, without departing from the concept of the present invention, it is obvious that several variations and improvements can be made to these specific embodiments, which all belong to the protection scope of this application. Therefore, the protection scope of the patent of this application shall be subject to the attached claims.

[0179] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a large-scale manipulation model for a robot's long-range task, characterized in that: The method comprises: Identify the geometric characteristics of the object to be manipulated; generating an operable graph corresponding to an operation action according to the geometric characteristics, wherein the operation action includes a rotation operation and a translation operation facing the object to be manipulated; Set the objective function; Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain a manipulation large model to control the posture of the end effector of the intelligent robot.

2. The method according to claim 1, characterized in that The identification of the geometric characteristics of the object to be manipulated includes: Identify the pixels of the object to be manipulated from the captured image; A geometric property indicated by the pixel is identified, where the geometric property is used to indicate a shape of an object to be manipulated.

3. The method according to claim 2, characterized in that The step of generating an operable graph corresponding to the operation action according to the geometric characteristics comprises: generating a first operable graph corresponding to a rotation operation according to the geometric characteristic; A second operable map corresponding to the direct-acting operation is generated according to the geometric characteristic.

4. The method according to claim 3, characterized in that The generating, according to the geometric characteristic, a first operable operation corresponding to the rotation operation comprises: According to the geometric characteristics, identifying the axis of the object to be manipulated; Simulating the rotation of the object to be controlled along the axis; A first operable graph of the object to be controlled after rotation is calculated, wherein the first operable graph includes operability probabilities of various regions in the graph.

5. The method according to claim 3, characterized in that: The step of generating a second operable diagram corresponding to the direct-acting operation according to the geometric characteristic comprises: According to the geometric characteristics, identifying the direct motion area of ​​the object to be manipulated; A second operable graph of the object to be controlled is calculated, and the operability probability of the direct motion area in the second operable graph is set to a preset value.

6. The method according to claim 4 or 5, characterized in that: The objective function setting includes: Using a first cross entropy loss function, training the multimodal large model to learn to distinguish control points, and recording a first loss tensor corresponding to the first cross entropy loss function; Obtain the RGB image when the operation is successful and the posture of the end effector of the intelligent robot through the second cross entropy loss function, and record the second loss tensor corresponding to the second cross entropy loss function; Obtain the predicted value and the true value of the multimodal large model through a third cross entropy loss function, and record a third loss tensor corresponding to the third cross entropy loss function; An objective function is set according to the first loss tensor, the second loss tensor, and the third loss tensor.

7. The method according to claim 6, characterized in that: The step of training the multimodal large model according to the operable graph under the supervision of the objective function to obtain the manipulation large model comprises: Set the training steps; Under the supervision of the objective function, according to the operational graph and in accordance with the training steps, the multimodal large model is trained to obtain the manipulation large model.

8. The method according to claim 7, characterized in that The step of setting up training comprises: The multimodal large model outputs the predicted coordinate values ​​of the operable points; Determining a direction vector of the end effector of the intelligent robot; The coordinate values ​​of the operable points are projected from the image coordinate system to the three-dimensional space coordinate system through the depth image data.

9. A large-scale model construction system for robot long-range tasks, characterized in that: The system includes an intelligent robot and a large model construction device, wherein: The intelligent robot is used for: Acquire image data and three-dimensional space data of the environment; the image and the three-dimensional space at least include the object to be manipulated; The large model construction device is used for: Identify the geometric characteristics of the object to be manipulated; generating an operable graph corresponding to an operation action according to the geometric characteristics, wherein the operation action includes a rotation operation and a translation operation facing the object to be manipulated; Set the objective function; Under the supervision of the objective function, the multimodal large model is trained according to the operable graph to obtain a manipulation large model to control the posture of the end effector of the intelligent robot.

10. An intelligent robot, characterized in that: The intelligent robot comprises: A camera, used to obtain multiple images of the environment, wherein the multiple images at least include the object to be manipulated; A calculation module is used to calculate the image data and three-dimensional space data of the environment through the multiple images; so that the manipulation large model construction device can identify the geometric characteristics of the object to be manipulated through the image data and the three-dimensional space data; generate an operable graph corresponding to the operation action according to the geometric characteristics, and the operation action includes a rotation operation and a direct motion operation facing the object to be manipulated; set an objective function; under the supervision of the objective function, train the multimodal large model according to the operable graph to obtain the manipulation large model to control the posture of the end effector of the intelligent robot.

Images