Robot task execution method, device, robot and active perspective selection system

Abstract

The present invention provides a robot task execution method, device, robot, and active viewpoint selection system, relating to the field of robotics. A robot is connected to a mobile single-camera system. When executing a target task, the robot can obtain first observation data from the mobile single-camera system at its current viewpoint. Based on the first observation data and a preset target task, the robot determines a target action for its manipulator arm and a target viewpoint corresponding to the mobile single-camera system. The manipulator arm is controlled to execute the target action, and the mobile single-camera system is controlled to move to the target viewpoint to complete the target task. This allows the viewpoint of the mobile single-camera system to be dynamically adjusted during task execution, enabling the system to observe more meaningful portions, thereby avoiding the limited field of view of the single-camera system. This reduces noise in the observation data, improves decision-making, and thus enhances task execution efficiency.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot task execution method, device, robot, and active viewpoint selection system. Background Art

[0002] Robot manipulation is a core challenge in robotics and is crucial for a wide range of applications, from industrial automation to healthcare. Currently, visual robotic manipulation is primarily based on imitation learning (IL). Imitation learning enables robots to learn complex tasks by observing expert demonstrations and mapping the observations to robotic arm movements. Therefore, visual observation is crucial for the efficiency of imitation learning. Existing methods rely on fixed camera setups, which can be single-camera or multi-camera systems. Single-camera systems use only one camera for observation, typically in an "eye-in-hand" setup, where the camera is mounted near the robot's end effector (e.g., a wrist camera) or fixed in the external scene to cover the entire task area. Multi-camera systems consist of multiple fixed external cameras or a combination of wrist cameras.

[0003] However, single-camera systems have a limited field of view and may occlude critical parts or objects in the environment, negatively impacting task performance. While multi-camera systems offer more comprehensive scene coverage, they introduce complexity: a large amount of redundant or irrelevant information can constrain learning algorithms and reduce efficiency. Furthermore, these passive, static multi-camera setups do not always provide the most relevant task information, leading to suboptimal decisions and impacting task execution efficiency. Summary of the Invention

[0004] The purpose of the present invention is to provide a robot task execution method, device, robot and active view selection system to improve task execution efficiency.

[0005] In a first aspect, the present invention provides a robot task execution method, which is applied to a robot connected to a mobile single-camera system; the robot task execution method comprises:

[0006] Obtaining the first observation data of the mobile single camera system at the current viewing angle;

[0007] Determining a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task;

[0008] Control the robotic arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0009] Furthermore, determining a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task includes:

[0010] Predicting the motion of the robot arm in the current time block based on the first observation data and the trained control model to obtain the target motion;

[0011] Predicting the view angle of the next time block of the mobile single-camera system based on the first observation data, the target motion, and the trained view angle model to obtain the target view angle;

[0012] Among them, the control model and the perspective model both correspond to the target task. The control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks. The sample data includes an observation data set, robotic arm position data and robotic arm motion data. The observation data set includes observation data from multiple perspectives.

[0013] Furthermore, the control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features of the input first observation data, and the decoder is used to convert the first target feature output by the encoder into a target action.

[0014] Furthermore, the perspective model includes a Transformer model and a SoftMax activation function. The Transformer model is used to extract features of the input first observation data and target action, and the SoftMax activation function is used to convert the second target features output by the Transformer model into a target perspective.

[0015] Furthermore, the robot task execution method further includes:

[0016] Acquire sample data in multiple adjacent time blocks, where the sample data includes an observation data set, robotic arm position data, and robotic arm motion data, and the observation data set includes observation data from multiple perspectives;

[0017] Based on the sample data, the control model and the view model are jointly trained.

[0018] Furthermore, based on each sample data, the control model and the view model are jointly trained, including:

[0019] Randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data;

[0020] Inputting second observation data randomly selected from the first sample data at the first perspective into the control model to perform motion prediction in the first time block, obtaining a first predicted motion output by the control model, and updating parameters of the control model based on a first loss between the first predicted motion and the robot arm motion data in the first sample data;

[0021] The second observation data and the manipulator motion data in the first sample data are input into the perspective model to perform perspective prediction for the second time block, and a predicted camera perspective output by the perspective model is obtained; the third observation data under the predicted camera perspective in the second sample data is input into the control model to perform motion prediction in the second time block, and a second predicted motion is obtained as output by the control model, and the parameters of the control model are updated according to a second loss between the second predicted motion and the manipulator motion data in the second sample data;

[0022] Use the second loss of the second time block as supervision to update the parameters of the view model;

[0023] The step of randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data is re-executed until the control model and the view model converge.

[0024] In a second aspect, the present invention further provides a robot task execution device, which is applied to a robot connected to a mobile single-camera system; the robot task execution device comprises:

[0025] An acquisition module, configured to acquire first observation data of the mobile single camera system at a current viewing angle;

[0026] a determination module, configured to determine a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task;

[0027] The control module is used to control the robotic arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0028] In a third aspect, the present invention further provides a robot comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the robot task execution method of the first aspect is implemented.

[0029] In a fourth aspect, the present invention further provides an active perspective selection system, comprising the robot of the third aspect and a mobile single-camera system connected to the robot.

[0030] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is run by a processor, the robot task execution method of the first aspect is executed.

[0031] The present invention provides a robot task execution method, device, robot, and active viewpoint selection system. A robot is connected to a mobile single-camera system. When executing a target task, the robot obtains first observation data from the mobile single-camera system at its current viewpoint. Based on the first observation data and a preset target task, the robot determines a target action for its manipulator arm and a target viewpoint corresponding to the mobile single-camera system. The manipulator arm is controlled to execute the target action and the mobile single-camera system is controlled to move to the target viewpoint to complete the target task. This allows the mobile single-camera system's viewpoint to be dynamically adjusted during task execution, enabling it to observe more meaningful areas. This avoids the limited field of view of the single-camera system, reduces noise in the observation data, improves decision-making, and thus enhances task execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A schematic diagram of the scene of the existing robot task execution process;

[0034] Figure 2 A schematic diagram of a flow chart of a robot task execution method provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of a scenario of a robot task execution process provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of a model training process provided by an embodiment of the present invention;

[0037] Figure 5 A schematic flow chart of another robot task execution method provided by an embodiment of the present invention;

[0038] Figure 6 A schematic structural diagram of a robot task execution device provided by an embodiment of the present invention;

[0039] Figure 7 A schematic structural diagram of a robot provided by an embodiment of the present invention;

[0040] Figure 8 A schematic structural diagram of an active viewing angle selection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] like Figure 1 As shown, existing robots usually use a fixed single-camera system or a fixed multi-camera system, that is, the viewing angle of the single-camera system or the multi-camera system does not change during the task execution; after the control model is obtained through expert data training, the control model can convert the observation data of the single-camera system or the multi-camera system into the robot arm action output.

[0043] However, the field of view of a single-camera system is limited, which affects task performance; a multi-camera system contains a large amount of redundant or irrelevant information, which results in low computational efficiency of the control model and affects decision-making results, resulting in low task execution efficiency. Based on this, an embodiment of the present invention provides a robot task execution method, device, robot, and active perspective selection system, which adopts a movable single-camera system (i.e., a mobile single-camera system) to allow the mobile single-camera system to be transformed at different positions to change the perspective of the mobile single-camera system; at the same time, an active perspective selection strategy of the robot based on imitation learning is adopted. By actively selecting the optimal perspective of the mobile single-camera system in the next time block, the problem of limited field of view of the single-camera system can be avoided, and the influence of redundant or irrelevant information in the multi-camera system on decision-making results can be avoided, thereby improving task execution efficiency.

[0044] To facilitate understanding of this embodiment, a robot task execution method disclosed in an embodiment of the present invention is first introduced in detail.

[0045] An embodiment of the present invention provides a robot task execution method, which is applied to a robot connected to a mobile single-camera system and can be executed by a main control device such as a robot controller; wherein the mobile single-camera system can change the viewing angle by changing the position and / or angle, and the mobile single-camera system can use an RGB camera. Figure 2 The flowchart of a robot task execution method shown in FIG. 1 mainly includes the following steps S210 to S230:

[0046] Step S210: Acquire first observation data of the mobile single camera system at the current viewing angle.

[0047] The mobile single-camera system is in communication with the robot. During task execution, the mobile single-camera system can transmit collected observation data to the robot and can also move under the robot's control to change the viewing angle. This observation data can be image data captured by the mobile single-camera system's camera, such as video or pictures. The observation data transmitted to the robot by the mobile single-camera system during the current time block is referred to as the first observation data.

[0048] Step S220 : determining a target motion of the robot arm and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task.

[0049] The target motion of the robotic arm in the current time block can be predicted based on the first observation data. The optimal viewing angle for the next time block, i.e., the target viewing angle, can be determined based on the combination of the first observation data and the predicted target motion. The target motion can include the motion of one robotic arm or two robotic arms.

[0050] In some possible embodiments, the target action and target perspective can be predicted by pre-trained control models and perspective models, see Figure 3 The schematic diagram of a scenario illustrating a robot task execution process is shown. The control model can convert the observation input of a mobile single-camera system into a prediction of the robot arm's motion, and the perspective model outputs the optimal perspective for the next time block based on the observation and predicted robot arm motion of the mobile single-camera system. Based on this, step S220 may include: predicting the robot arm's motion in the current time block based on first observation data and the trained control model to obtain a target motion; and predicting the perspective of the mobile single-camera system in the next time block based on the first observation data, the target motion, and the trained perspective model to obtain a target perspective. Both the control model and the perspective model correspond to the target task and are jointly trained based on sample data from multiple adjacent time blocks. The sample data includes an observation data set, robot arm position data, and robot arm motion data. The observation data set includes observation data from multiple perspectives.

[0051] In this embodiment, the robot can perform one or more tasks, and each task type corresponds to a set of control models and perspective models. If the robot can perform multiple tasks, the corresponding control model and perspective model can be selected based on the task type of the target task currently being performed.

[0052] Optionally, the control model can consist of an encoder and a decoder. The encoder can use a pretrained multi-view masked autoencoder, and the decoder can use a Transformer model. The encoder is used to extract features from the input first observation data, and the decoder is used to convert the first target feature output by the encoder into a target action. The first target feature is obtained by the encoder through feature extraction of the first observation data. The decoder can be a diffusion-based Transformer model or a classic Transformer decoder model. This control model based on a pretrained multi-view masked autoencoder has better ability to extract view features.

[0053] Optionally, the perspective model can be composed of a Transformer model and a SoftMax activation function. The Transformer model is used to extract features from the input first observation data and target action, and the SoftMax activation function is used to convert the second target feature output by the Transformer model into a target perspective. The second target feature is obtained by the Transformer model through feature extraction of the first observation data and target action. This perspective model can better extract features of the current observation data and action to predict the subsequent target perspective.

[0054] Step S230 , controlling the robotic arm to perform a target action, and controlling the mobile single camera system to move to a target viewing angle to complete the target task.

[0055] After obtaining the target action and target viewing angle in step S220, the robot controls the manipulator arm to perform the target action and issues a control command corresponding to the target viewing angle to the mobile single-camera system, causing the mobile single-camera system to move to the target viewing angle based on the control command. Steps S210 to S230 are then executed repeatedly until the target task is completed.

[0056] The robot task execution method provided by the embodiment of the present invention can not only complete the grasping task, but also complete other operation tasks, such as placing an object in a specified position and dual-arm interactive operation (such as two robotic arms grasping two objects respectively and assembling them).

[0057] The robot task execution method provided by an embodiment of the present invention can obtain first observation data from a mobile single-camera system at its current viewing angle; based on the first observation data and a preset target task, determine the target motion of the robot's manipulator arm and the corresponding target viewing angle of the mobile single-camera system; control the manipulator arm to execute the target motion and control the mobile single-camera system to move to the target viewing angle to complete the target task. This allows the mobile single-camera system's viewing angle to be dynamically adjusted during task execution, enabling it to observe more meaningful areas, avoiding the limited field of view of the single-camera system. This also reduces noise in the observation data, improves decision-making, and thus enhances task execution efficiency.

[0058] An embodiment of the present invention also provides a training process for the above-mentioned control model and perspective model, as follows: first obtain sample data under multiple adjacent time blocks, the sample data including an observation data set, robotic arm position data and robotic arm motion data, the observation data set including observation data under multiple perspectives, and the robotic arm position data can be the robotic arm joint position; then based on each sample data, jointly train to obtain the control model and perspective model.

[0059] In some possible embodiments, the above joint training process may include:

[0060] 1. Randomly select first sample data and second sample data corresponding to adjacent first and second time blocks from each sample data, wherein the first sample data is the sample data corresponding to the first time block, and the second sample data is the sample data corresponding to the second time block.

[0061] 2. Input the second observation data from the first perspective randomly selected from the first sample data into the control model to predict the action in the first time block, obtain the first predicted action output by the control model, and update the parameters of the control model according to the first loss between the first predicted action and the robotic arm action data in the first sample data.

[0062] Specifically, a perspective (that is, a camera) can be randomly selected from the first sample data, which is referred to as the first perspective; the observation data under the first perspective is referred to as the second observation data; after the second observation data is input into the control model, the action output by the control model is referred to as the first predicted action; the action loss (that is, the first loss) can be calculated based on the first predicted action and the robotic arm action data in the first sample data, and the control model can be updated based on the action loss.

[0063] 3. Input the second observation data and the robotic arm motion data in the first sample data into the perspective model to perform perspective prediction for the second time block, and obtain the predicted camera perspective output by the perspective model; input the third observation data under the predicted camera perspective in the second sample data into the control model to perform action prediction in the second time block, and obtain the second predicted action output by the control model, and update the parameters of the control model according to the second loss between the second predicted action and the robotic arm motion data in the second sample data.

[0064] Specifically, after the second observation data and the robotic arm motion data in the first sample data are input into the perspective model, the perspective output by the perspective model is referred to as the predicted camera perspective; the observation data under the predicted camera perspective is referred to as the third observation data; after the third observation data is input into the control model, the action output by the control model is referred to as the second predicted action; the action loss (i.e., the second loss) can be calculated based on the second predicted action and the robotic arm motion data in the second sample data, and the control model can be further updated based on the action loss.

[0065] 4. Use the second loss of the second time block as supervision to update the parameters of the view model.

[0066] In specific implementations, the second loss mentioned above can be used as supervision to update the view model using a gradient descent algorithm. The gradient descent algorithm iteratively adjusts model parameters, moving along the negative gradient of the loss function to minimize the loss function.

[0067] The above steps are repeated, that is, step 1 is repeated, until the control model and the view model converge.

[0068] For ease of understanding, refer to Figure 4 The above model training process is introduced. Figure 4 As shown in the figure, the model training process includes the following steps:

[0069] Step S410: randomly sample two adjacent time blocks in the expert data.

[0070] The expert data here refers to the sample data used for training. The expert data corresponding to the two time blocks can be expressed as D t:t+T ={O t ,s t ,a t：t+T},D t+T:t+2T ={O t+T ,s t+T ,a t+T:t+2T}, where t represents the beginning of the first time block, t+T represents the beginning of the second time block, and t+2T represents the end of the second time block. t Represents the observation data from each perspective corresponding to the beginning of the first time block t, st represents the joint position of the robot arm at the beginning of the first time block t, a t:t+T Represents the robot arm action (i.e., expert action) between the beginning of the first time block t and the beginning of the second time block t+T.

[0071] Step S420: Randomly select the camera of the first time block, and input its observation into the control model to predict the action of the robotic arm in the current time block. Calculate the motion loss with the corresponding expert action to update the control model. At the same time, input the observation and expert action into the perspective model to predict the optimal camera perspective for the next time block.

[0072] Randomly select the camera of the first time block, assume the viewing angle is i, and observe it The input control model predicts the action of the robot arm in the current time block With expert action a t:t+T Calculate the action loss L t Update the control model; at the same time, the observation and expert action a t:t+T The input view model predicts the optimal camera view for the next time block, assuming it is j.

[0073] Step S430: For the second time block, the observation of the optimal camera angle predicted previously is input into the control model to predict the movement of the robotic arm, and the movement loss is calculated with the corresponding expert movement to update the control model.

[0074] For the second time block, the observation of the optimal camera angle j predicted previously is Input control model predicts the robot's movements With expert action a t+T:t+2T Calculate the action loss L t+T Update the control model.

[0075] In step S440 , the view model is updated using the motion loss of the second time block as supervision.

[0076] Use the action loss L for the second time block t+T As supervision, update the perspective model.

[0077] Step S450: Determine whether the control model and the view model have converged. If not, re-execute step S410; if yes, the process ends.

[0078] For ease of understanding, refer to Figure 5 The active perspective selection reasoning process of the above robot task execution method is introduced. Figure 5 As shown in Figure 2, the active perspective selection reasoning process includes the following steps:

[0079] Step S510 , the current camera view observation input control model predicts the motion of the robotic arm in the current time block.

[0080] Current camera perspective observation o t The input control model predicts the action of the robot arm in the current time block

[0081] Step S520 , the current camera perspective observation and the predicted action input perspective model predict the optimal camera perspective for the next time block.

[0082] Current camera perspective observation o t and predict actions The input view model predicts the optimal camera view j for the next time block.

[0083] Step S530 , executing a robotic arm action and moving the camera to the predicted optimal camera angle.

[0084] Step S540: Determine whether the target task is completed. If not, re-execute step S510; if yes, the process ends.

[0085] The above steps S510 to S530 are repeated until the target task is completed.

[0086] This embodiment of the present invention utilizes motion loss as supervision during imitation learning to achieve label-free optimal viewpoint selection for robots. By dynamically adjusting the robot's viewpoint, the camera can observe more meaningful areas, avoiding the limited field of view of previous single-camera systems. This also reduces noise input to the control model, achieving performance comparable to, and even exceeding, multi-camera systems in some scenarios.

[0087] Corresponding to the above-mentioned robot task execution method, an embodiment of the present invention further provides a robot task execution device, which is applied to a robot connected to a mobile single camera system. Figure 6 The schematic diagram of the structure of a robot task execution device shown in FIG. 1 includes:

[0088] An acquisition module 601 is configured to acquire first observation data of a mobile single-camera system at a current viewing angle;

[0089] A determination module 602 is configured to determine a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task;

[0090] The control module 603 is used to control the robotic arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0091] The robot task execution device provided in an embodiment of the present invention can obtain first observation data from a mobile single-camera system at its current viewing angle. Based on this first observation data and a preset target task, it determines the target action of the robot's manipulator arm and the corresponding target viewing angle of the mobile single-camera system. The manipulator arm is controlled to execute the target action, and the mobile single-camera system is controlled to move to the target viewing angle to complete the target task. This allows the mobile single-camera system's viewing angle to be dynamically adjusted during task execution, enabling it to observe more meaningful areas. This avoids the limited field of view of the single-camera system, reduces noise in the observation data, improves decision-making effectiveness, and thus enhances task execution efficiency.

[0092] Furthermore, the above-mentioned determination module 602 is specifically used to: predict the action of the robot arm in the current time block based on the first observation data and the trained control model to obtain the target action; predict the perspective of the mobile single camera system in the next time block based on the first observation data, the target action and the trained perspective model to obtain the target perspective; wherein, the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm motion data, and the observation data set includes observation data under multiple perspectives.

[0093] Furthermore, the above-mentioned control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features of the input first observation data, and the decoder is used to convert the first target feature output by the encoder into a target action.

[0094] Furthermore, the above-mentioned perspective model includes a Transformer model and a SoftMax activation function. The Transformer model is used to extract features of the input first observation data and target action, and the SoftMax activation function is used to convert the second target features output by the Transformer model into a target perspective.

[0095] Furthermore, the robot task execution device further includes a training module for:

[0096] Acquire sample data in multiple adjacent time blocks, where the sample data includes an observation data set, robotic arm position data, and robotic arm motion data, and the observation data set includes observation data from multiple perspectives;

[0097] Based on the sample data, the control model and the view model are jointly trained.

[0098] Furthermore, the above-mentioned training module is specifically used to: randomly select first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data; input the second observation data under the first perspective randomly selected from the first sample data into the control model to perform action prediction under the first time block, obtain the first predicted action output by the control model, and update the parameters of the control model according to the first loss between the first predicted action and the robotic arm action data in the first sample data; input the second observation data and the robotic arm action data in the first sample data into the perspective model to perform perspective prediction of the second time block, and obtain the predicted camera perspective output by the perspective model; input the third observation data under the predicted camera perspective in the second sample data into the control model to perform action prediction under the second time block, obtain the second predicted action output by the control model, and update the parameters of the control model according to the second loss between the second predicted action and the robotic arm action data in the second sample data; use the second loss of the second time block as supervision to update the parameters of the perspective model; re-execute the steps of randomly selecting the first sample data and second sample data corresponding to the adjacent first time blocks and second time blocks from each sample data until the control model and the perspective model converge.

[0099] The robot task execution device provided in this embodiment has the same implementation principle and technical effects as those in the aforementioned robot task execution method embodiment. For the sake of brief description, for matters not mentioned in the robot task execution device embodiment, reference may be made to the corresponding content in the aforementioned robot task execution method embodiment.

[0100] like Figure 7 As shown, an embodiment of the present invention provides a robot 700, including: a processor 701, a memory 702 and a bus. The memory 702 stores a computer program that can be run on the processor 701. When the robot 700 is running, the processor 701 and the memory 702 communicate through the bus, and the processor 701 executes the computer program to implement the above-mentioned robot task execution method.

[0101] Specifically, the memory 702 and processor 701 can be general-purpose memories and processors, which are not specifically limited here.

[0102] The embodiment of the present invention also provides an active viewing angle selection system, such as Figure 8 As shown, the active view selection system includes the above-mentioned robot 801 and a mobile single camera system 802 , which is connected to the robot 801 .

[0103] The active perspective selection system provided in this embodiment has the same implementation principle and technical effects as those in the aforementioned robot task execution method embodiment. For the sake of brief description, for matters not mentioned in the active perspective selection system embodiment, reference may be made to the corresponding content in the aforementioned robot task execution method embodiment.

[0104] Embodiments of the present invention further provide a computer-readable storage medium storing a computer program that, when executed by a processor, executes the robot task execution method described in the preceding method embodiments. The computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk.

[0105] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0106] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0107] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, the indirect coupling or communication connection of the device or module can be electrical, mechanical or other forms.

[0109] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0110] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot task execution method, characterized in that: Applicable to robots communicating with mobile single-camera systems; The robot task execution method includes: Acquiring first observation data of the mobile single camera system at a current viewing angle; Determining a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task; Controlling the robotic arm to perform the target action and controlling the mobile single-camera system to move to the target viewing angle to complete the target task; wherein the mobile single-camera system changes the viewing angle by changing the position and / or angle; The determining, based on the first observation data and a preset target task, a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system includes: Predicting the motion of the robotic arm in the current time block based on the first observation data and the trained control model to obtain the target motion; Predicting the view angle of the next time block of the mobile single-camera system based on the first observation data, the target motion, and the trained view angle model to obtain the target view angle; In which, the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data under multiple adjacent time blocks. The sample data includes an observation data set, robotic arm position data and robotic arm motion data, and the observation data set includes observation data under multiple perspectives.

2. The robot task execution method according to claim 1, characterized in that: The control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features of the first observation data input, and the decoder is used to convert the first target feature output by the encoder into the target action.

3. The robot task execution method according to claim 1, characterized in that: The perspective model includes a Transformer model and a SoftMax activation function. The Transformer model is used to extract features of the input first observation data and the target action, and the SoftMax activation function is used to convert the second target feature output by the Transformer model into the target perspective.

4. The robot task execution method according to claim 1, characterized in that: The robot task execution method further includes: Acquire sample data in a plurality of adjacent time blocks, the sample data including an observation data set, robotic arm position data, and robotic arm motion data, the observation data set including observation data from a plurality of perspectives; Based on each of the sample data, the control model and the viewing angle model are obtained through joint training.

5. The robot task execution method according to claim 4, characterized in that: The step of jointly training the control model and the viewing angle model based on the sample data includes: Randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each of the sample data; Inputting second observation data randomly selected from the first sample data at the first perspective into the control model to perform motion prediction in the first time block, obtaining a first predicted motion output by the control model, and updating parameters of the control model based on a first loss between the first predicted motion and the robot arm motion data in the first sample data; Inputting the second observation data and the manipulator motion data in the first sample data into the perspective model to perform perspective prediction for the second time block, and obtaining a predicted camera perspective output by the perspective model; inputting the third observation data under the predicted camera perspective in the second sample data into the control model to perform motion prediction for the second time block, and obtaining a second predicted motion output by the control model; and updating the parameters of the control model according to a second loss between the second predicted motion and the manipulator motion data in the second sample data; Using the second loss of the second time block as supervision, updating the parameters of the view model; The step of randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each of the sample data is re-executed until the control model and the viewing angle model converge.

6. A robot task execution device, characterized in that: Applicable to robots communicating with mobile single-camera systems; The robot task execution device includes: An acquisition module, configured to acquire first observation data of the mobile single-camera system at a current viewing angle; a determination module, configured to determine a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system based on the first observation data and a preset target task; a control module, configured to control the robotic arm to perform the target action and control the mobile single-camera system to move to the target viewing angle to complete the target task; wherein the mobile single-camera system changes the viewing angle by changing its position and / or angle; The determination module is specifically used to: predict the action of the robotic arm in the current time block based on the first observation data and the trained control model to obtain the target action; predict the perspective of the next time block of the mobile single camera system based on the first observation data, the target action and the trained perspective model to obtain the target perspective; wherein, the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks, the sample data includes an observation data set, robotic arm position data and robotic arm motion data, and the observation data set includes observation data from multiple perspectives.

7. A robot comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the robot task execution method according to any one of claims 1 to 5 is implemented.

8. An active perspective selection system, characterized in that: The robot of claim 7 further comprising a mobile single-camera system connected to the robot.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot task execution method according to any one of claims 1 to 5 is executed.

Images