Robot task execution method and device, robot and active view angle selection system

Abstract

The invention provides a robot task execution method and device, a robot and an active visual angle selection system, and relates to the technical field of robots, the robot is connected with a mobile single camera system, and when the robot executes a target task, the robot can obtain first observation data of the mobile single camera system at a current visual angle; according to the first observation data and a preset target task, determining a target action of the robot mechanical arm and a target view angle corresponding to the mobile single-camera system; and controlling the mechanical arm to execute the target action, and controlling the mobile single-camera system to move to the target visual angle so as to complete the target task. Therefore, in the task execution process, the visual angle of the mobile single-camera system can be dynamically adjusted, so that the mobile single-camera system can observe a more meaningful part, the problem that the visual field range of the single-camera system is limited is avoided, meanwhile, noise of observation data is reduced, the decision-making effect is improved, and the task execution efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot task execution method, device, robot and active viewpoint selection system. Background Art

[0002] Robot manipulation is a core challenge in robotics and is critical for a variety of applications, from industrial automation to healthcare. Visual robotic manipulation is currently mainly based on imitation learning (IL). Imitation learning enables robots to learn complex tasks by observing expert demonstrations and mapping the observations to the movements of the robotic arm. Therefore, visual observation is crucial for the efficiency of imitation learning. Existing methods rely on fixed camera setups, which can be single-camera systems or multi-camera systems; among them, single-camera systems use only one camera for observation, usually in an "eye-in-hand" setup, that is, the camera is installed near the robot's end effector (such as a wrist camera) or fixed in the external scene to cover the entire task area; multi-camera systems are composed of multiple fixed external cameras or in combination with wrist cameras.

[0003] However, single-camera systems have a limited field of view and may block key parts or objects in the environment, which can negatively affect task performance. While multi-camera systems can provide more comprehensive scene coverage, they introduce complexity: a large amount of redundant or irrelevant information may limit learning algorithms and reduce efficiency. In addition, these passive and static multi-camera system settings do not always provide the most relevant task information, resulting in suboptimal decisions and affecting task execution efficiency. Summary of the invention

[0004] The object of the present invention is to provide a robot task execution method, device, robot and active view selection system to improve task execution efficiency.

[0005] In a first aspect, the present invention provides a robot task execution method, which is applied to a robot connected to a mobile single camera system; the robot task execution method comprises:

[0006] Acquire the first observation data of the mobile single camera system at the current viewing angle;

[0007] Determine, according to the first observation data and the preset target task, a target action of the robot manipulator and a target viewing angle corresponding to the mobile single camera system;

[0008] Control the robotic arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0009] Further, according to the first observation data and the preset target task, determining the target action of the robot manipulator and the target viewing angle corresponding to the mobile single camera system includes:

[0010] Predicting the action of the robot arm in the current time block according to the first observation data and the trained control model to obtain the target action;

[0011] Predicting the view angle of the next time block of the mobile single camera system according to the first observation data, the target action and the trained view angle model to obtain the target view angle;

[0012] Among them, the control model and the perspective model both correspond to the target task. The control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks. The sample data includes an observation data set, robotic arm position data and robotic arm motion data. The observation data set includes observation data from multiple perspectives.

[0013] Furthermore, the control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features from the input first observation data, and the decoder is used to convert the first target features output by the encoder into a target action.

[0014] Furthermore, the perspective model includes a Transformer model and a SoftMax activation function, the Transformer model is used to extract features of the input first observation data and target action, and the SoftMax activation function is used to convert the second target features output by the Transformer model into a target perspective.

[0015] Furthermore, the robot task execution method further includes:

[0016] Acquire sample data under multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm motion data, and the observation data set includes observation data under multiple perspectives;

[0017] Based on the sample data, the control model and the view model are jointly trained.

[0018] Furthermore, based on each sample data, a control model and a view model are obtained by joint training, including:

[0019] Randomly select first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data;

[0020] Inputting the second observation data at the first perspective randomly selected from the first sample data into the control model to perform motion prediction in the first time block, obtaining a first predicted motion output by the control model, and updating the parameters of the control model according to a first loss between the first predicted motion and the robot arm motion data in the first sample data;

[0021] The second observation data and the robot arm motion data in the first sample data are input into the perspective model to perform perspective prediction of the second time block, and obtain the predicted camera perspective output by the perspective model; the third observation data under the predicted camera perspective in the second sample data is input into the control model to perform action prediction under the second time block, and obtain the second predicted action output by the control model, and the parameters of the control model are updated according to the second loss between the second predicted action and the robot arm motion data in the second sample data;

[0022] Use the second loss of the second time block as supervision to update the parameters of the view model;

[0023] The step of randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data is re-executed until the control model and the viewing angle model converge.

[0024] In a second aspect, the present invention further provides a robot task execution device, which is applied to a robot connected to a mobile single camera system; the robot task execution device comprises:

[0025] An acquisition module, used to acquire first observation data of the mobile single camera system at a current viewing angle;

[0026] A determination module, used to determine a target action of the robot manipulator and a target viewing angle corresponding to the mobile single camera system according to the first observation data and a preset target task;

[0027] The control module is used to control the robotic arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0028] In a third aspect, the present invention further provides a robot, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the robot task execution method of the first aspect is implemented.

[0029] In a fourth aspect, the present invention further provides an active perspective selection system, comprising the robot of the third aspect, and a mobile single-camera system connected to the robot.

[0030] In a fifth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot task execution method of the first aspect is executed.

[0031] In the robot task execution method, device, robot and active viewpoint selection system provided by the present invention, the robot is connected to the mobile single camera system, and when the robot executes the target task, it can obtain the first observation data of the mobile single camera system at the current viewpoint; according to the first observation data and the preset target task, the target action of the robot's mechanical arm and the target viewpoint corresponding to the mobile single camera system are determined; the mechanical arm is controlled to execute the target action, and the mobile single camera system is controlled to move to the target viewpoint to complete the target task. In this way, during the execution of the task, the viewpoint of the mobile single camera system can be dynamically adjusted, so that the mobile single camera system can observe more meaningful parts, avoiding the problem of limited field of view of the single camera system, while reducing the noise of the observation data, improving the decision-making effect, and thus improving the efficiency of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of the scene of the existing robot task execution process;

[0034] Figure 2 A schematic diagram of a flow chart of a robot task execution method provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of a scenario of a robot task execution process provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of a model training process provided by an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of a flow chart of another robot task execution method provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of the structure of a robot task execution device provided by an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of the structure of a robot provided by an embodiment of the present invention;

[0040] Figure 8 A schematic diagram of the structure of an active viewing angle selection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] like Figure 1 As shown, existing robots usually use a fixed single-camera system or a fixed multi-camera system, that is, the viewing angle of the single-camera system or the multi-camera system will not change during the task execution; after the control model is obtained through expert data training, the control model can convert the observation data of the single-camera system or the multi-camera system into the robot arm motion output.

[0043] However, the field of view of a single-camera system is limited, which affects the task performance; a multi-camera system has a large amount of redundant or irrelevant information, the computational efficiency of the control model is low, and it affects the decision-making effect, resulting in low task execution efficiency. Based on this, an embodiment of the present invention provides a robot task execution method, device, robot and active perspective selection system, which adopts a movable single-camera system (i.e., a mobile single-camera system) to allow the mobile single-camera system to be transformed at different positions to change the perspective of the mobile single-camera system; at the same time, a robot active perspective selection strategy based on imitation learning is adopted, which can avoid the problem of limited field of view of the single-camera system by actively selecting the optimal perspective of the mobile single-camera system in the next time block, and at the same time avoid the influence of redundant or irrelevant information in the multi-camera system on the decision-making effect, thereby improving the task execution efficiency.

[0044] To facilitate understanding of this embodiment, a robot task execution method disclosed in an embodiment of the present invention is first introduced in detail.

[0045] The embodiment of the present invention provides a robot task execution method, which is applied to a robot connected to a mobile single camera system and can be executed by a main control device such as a robot controller; wherein the mobile single camera system can change the viewing angle by changing the position and / or angle, and the mobile single camera system can use an RGB camera. Figure 2 The flowchart of a robot task execution method shown in FIG. 1 mainly includes the following steps S210 to S230:

[0046] Step S210: acquiring first observation data of the mobile single camera system at a current viewing angle.

[0047] The mobile single camera system is connected to the robot in communication. During the task execution, the mobile single camera system can transmit the collected observation data to the robot, or move based on the control of the robot to change the viewing angle. The observation data can be image data captured by the camera in the mobile single camera system, such as video or picture. Here, the observation data transmitted to the robot by the mobile single camera system in the current time block is referred to as the first observation data.

[0048] Step S220, determining a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system according to the first observation data and a preset target task.

[0049] The target action of the robot arm in the current time block can be predicted based on the first observation data, and then the optimal viewing angle of the next time block, i.e., the target viewing angle, can be determined based on the first observation data combined with the predicted target action. The target action can include the action of one robot arm or the action of two robot arms.

[0050] In some possible embodiments, the target action and the target perspective can be predicted by using a pre-trained control model and a perspective model, see Figure 3 The schematic diagram of a scene of a robot task execution process shown in the figure, the control model can convert the observation input of the mobile single camera system into the prediction of the robot arm movement, and the perspective model outputs the optimal perspective of the next time block according to the observation of the mobile single camera system and the predicted robot arm movement. Based on this, the above step S220 may include: predicting the movement of the robot arm in the current time block according to the first observation data and the trained control model to obtain the target movement; predicting the perspective of the next time block of the mobile single camera system according to the first observation data, the target movement and the trained perspective model to obtain the target perspective. Among them, the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data under multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm movement data, and the observation data set includes observation data under multiple perspectives.

[0051] In this embodiment, the robot can perform one or more tasks, and each task type corresponds to a set of control models and perspective models. If the robot can perform multiple tasks, the corresponding control model and perspective model can be selected according to the task type of the target task currently being performed.

[0052] Optionally, the control model can be composed of an encoder and a decoder. The encoder can use a pre-trained multi-view mask autoencoder, and the decoder can use a Transformer model. The encoder is used to extract features from the input first observation data, and the decoder is used to convert the first target feature output by the encoder into a target action, where the first target feature is obtained by the encoder extracting features from the first observation data. The decoder can be a diffusion-based Transformer model or a classic Transformer decoder model. This control model based on a pre-trained multi-view mask autoencoder has a better ability to extract view features.

[0053] Optionally, the perspective model can be composed of a Transformer model and a SoftMax activation function, wherein the Transformer model is used to extract features from the input first observation data and target action, and the SoftMax activation function is used to convert the second target feature output by the Transformer model into a target perspective, wherein the second target feature is obtained by extracting features from the input first observation data and target action by the Transformer model. This perspective model can better extract features of the current observation data and action to predict the subsequent target perspective.

[0054] Step S230, controlling the robotic arm to perform a target action, and controlling the mobile single camera system to move to a target viewing angle to complete a target task.

[0055] After obtaining the target action and the target viewing angle through step S220, the robot can control the mechanical arm to perform the target action and send a control instruction corresponding to the target viewing angle to the mobile single camera system, so that the mobile single camera system moves to the target viewing angle based on the control instruction. Then, steps S210 to S230 can be executed repeatedly until the target task is completed.

[0056] The robot task execution method provided in the embodiment of the present invention can not only complete the grasping task, but also complete other operation tasks, such as placing an object at a specified location and dual-arm interactive operation (such as two robotic arms grasping two objects respectively and assembling them).

[0057] The robot task execution method provided by the embodiment of the present invention can obtain the first observation data of the mobile single camera system at the current viewing angle; determine the target action of the robot manipulator and the target viewing angle corresponding to the mobile single camera system according to the first observation data and the preset target task; control the manipulator to execute the target action, and control the mobile single camera system to move to the target viewing angle to complete the target task. In this way, during the task execution process, the viewing angle of the mobile single camera system can be dynamically adjusted, so that the mobile single camera system can observe more meaningful parts, avoiding the problem of limited field of view of the single camera system, while reducing the noise of the observation data, improving the decision-making effect, and thus improving the efficiency of task execution.

[0058] An embodiment of the present invention also provides a training process for the above-mentioned control model and perspective model, as follows: first obtain sample data under multiple adjacent time blocks, the sample data including an observation data set, robotic arm position data and robotic arm motion data, the observation data set including observation data under multiple perspectives, and the robotic arm position data may be the robotic arm joint position; then based on each sample data, jointly train to obtain the control model and the perspective model.

[0059] In some possible embodiments, the above joint training process may include:

[0060] 1. Randomly select first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data, wherein the first sample data is sample data corresponding to the first time block, and the second sample data is sample data corresponding to the second time block.

[0061] 2. Input the second observation data from the first perspective randomly selected from the first sample data into the control model to predict the action in the first time block, obtain the first predicted action output by the control model, and update the parameters of the control model according to the first loss between the first predicted action and the robotic arm action data in the first sample data.

[0062] Specifically, a perspective (that is, a camera) can be randomly selected from the first sample data, which is referred to as the first perspective; the observation data under the first perspective is referred to as the second observation data; after the second observation data is input into the control model, the action output by the control model is referred to as the first predicted action; the action loss (that is, the first loss) can be calculated based on the first predicted action and the robotic arm action data in the first sample data, and the control model can be updated based on the action loss.

[0063] 3. Input the second observation data and the robotic arm motion data in the first sample data into the perspective model to perform perspective prediction for the second time block, and obtain the predicted camera perspective output by the perspective model; input the third observation data under the predicted camera perspective in the second sample data into the control model to perform action prediction in the second time block, and obtain the second predicted action output by the control model, and update the parameters of the control model according to the second loss between the second predicted action and the robotic arm motion data in the second sample data.

[0064] Specifically, after the second observation data and the robotic arm motion data in the first sample data are input into the perspective model, the perspective output by the perspective model is referred to as the predicted camera perspective; the observation data under the predicted camera perspective is referred to as the third observation data; after the third observation data is input into the control model, the action output by the control model is referred to as the second predicted action; the action loss (i.e., the second loss) can be calculated based on the second predicted action and the robotic arm motion data in the second sample data, and the control model can be further updated based on the action loss.

[0065] 4. Use the second loss of the second time block as supervision to update the parameters of the view model.

[0066] In specific implementation, the second loss mentioned above can be used as supervision to update the view model using a gradient descent algorithm, which iteratively adjusts model parameters and moves along the negative gradient direction of the loss function to minimize the loss function.

[0067] The above steps are repeated, that is, step 1 is executed again, until the control model and the view model converge.

[0068] For ease of understanding, refer to Figure 4 The above model training process is introduced. Figure 4 As shown in the figure, the model training process includes the following steps:

[0069] Step S410: randomly sample two adjacent time blocks in the expert data.

[0070] The expert data here refers to the sample data used for training. The expert data corresponding to the two time blocks can be expressed as D t:t+T = {O t ,s t ,a t：t+T},D t+T:t+2T = {O t+T ,s t+T ,a t+T:t+2T}, where t represents the beginning of the first time block, t+T represents the beginning of the second time block, and t+2T represents the end of the second time block. t represents the observation data from each perspective corresponding to the beginning of the first time block t, st represents the robot joint position at the beginning of the first time block t, a t:t+T Represents the robot arm action (i.e., expert action) between the beginning of the first time block t and the beginning of the second time block t+T.

[0071] Step S420, randomly select the camera of the first time block, and input its observation into the control model to predict the action of the robot arm in the current time block, calculate the action loss with the corresponding expert action and update the control model; at the same time, input the observation and expert action into the perspective model to predict the optimal camera perspective of the next time block.

[0072] Randomly select the camera of the first time block, assuming the viewing angle is i, and observe it The input control model predicts the action of the robot arm in the current time block With expert action a t:t+T Calculate the action loss L t Update the control model; at the same time, the observation and expert action a t:t+T The input view model predicts the optimal camera view for the next time block, assuming it is j.

[0073] Step S430, for the second time block, the observation of the optimal camera angle predicted previously is input into the control model to predict the movement of the robotic arm, and the movement loss is calculated with the corresponding expert movement to update the control model.

[0074] For the second time block, the observation of the optimal camera angle j predicted previously is Input control model predicts the robot's movements With expert action a t+T:t+2T Calculate the action loss L t+T Update the control model.

[0075] Step S440, using the motion loss of the second time block as supervision, updating the view model.

[0076] Use the action loss L for the second time chunk t+T As supervision, update the perspective model.

[0077] Step S450, determine whether the control model and the view model are converged. If not, re-execute step S410; if yes, the process ends.

[0078] For ease of understanding, refer to Figure 5 The active perspective selection reasoning process of the above robot task execution method is introduced. Figure 5 As shown in Figure 2, the active perspective selection reasoning process includes the following steps:

[0079] Step S510, the current camera perspective observation input control model predicts the action of the robotic arm in the current time block.

[0080] Current camera perspective observation o t The input control model predicts the action of the robot arm in the current time block

[0081] Step S520, the current camera perspective observation and the predicted action input perspective model predict the optimal camera perspective for the next time block.

[0082] Current camera perspective observation o t and predict actions The input view model predicts the optimal camera view j for the next time block.

[0083] Step S530, executing the robot arm action and moving the camera to the predicted optimal camera angle.

[0084] Step S540, determine whether the target task is completed. If not, re-execute step S510; if yes, the process ends.

[0085] The above steps S510 to S530 are repeated until the target task is completed.

[0086] The embodiment of the present invention uses motion loss as supervision in the process of imitation learning to achieve optimal robot perspective selection without labels. By dynamically adjusting the robot's perspective, the camera can observe more meaningful parts, avoiding the problem of limited field of view of the previous single-camera system, while reducing the noise of the input control model, achieving performance comparable to that of a multi-camera system, and even surpassing the multi-camera system in some scenarios.

[0087] Corresponding to the above-mentioned robot task execution method, an embodiment of the present invention further provides a robot task execution device, which is applied to a robot connected to a mobile single camera system. Figure 6 The schematic diagram of the structure of a robot task execution device shown in FIG. 1 includes:

[0088] An acquisition module 601 is used to acquire first observation data of the mobile single camera system at a current viewing angle;

[0089] A determination module 602 is used to determine a target action of the robot manipulator and a target viewing angle corresponding to the mobile single camera system according to the first observation data and a preset target task;

[0090] The control module 603 is used to control the robot arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

[0091] The robot task execution device provided by the embodiment of the present invention can obtain the first observation data of the mobile single camera system at the current viewing angle; determine the target action of the robot mechanical arm and the target viewing angle corresponding to the mobile single camera system according to the first observation data and the preset target task; control the mechanical arm to execute the target action, and control the mobile single camera system to move to the target viewing angle to complete the target task. In this way, during the execution of the task, the viewing angle of the mobile single camera system can be dynamically adjusted, so that the mobile single camera system can observe more meaningful parts, avoiding the problem of limited field of view of the single camera system, while reducing the noise of the observation data, improving the decision-making effect, and thus improving the efficiency of task execution.

[0092] Furthermore, the above-mentioned determination module 602 is specifically used to: predict the action of the robot arm in the current time block according to the first observation data and the trained control model to obtain the target action; predict the perspective of the next time block of the mobile single camera system according to the first observation data, the target action and the trained perspective model to obtain the target perspective; wherein the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm motion data, and the observation data set includes observation data from multiple perspectives.

[0093] Furthermore, the above-mentioned control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features of the input first observation data, and the decoder is used to convert the first target features output by the encoder into target actions.

[0094] Furthermore, the above-mentioned perspective model includes a Transformer model and a SoftMax activation function, the Transformer model is used to extract features of the input first observation data and target action, and the SoftMax activation function is used to convert the second target feature output by the Transformer model into a target perspective.

[0095] Furthermore, the robot task execution device further includes a training module for:

[0096] Acquire sample data under multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm motion data, and the observation data set includes observation data under multiple perspectives;

[0097] Based on the sample data, the control model and the view model are jointly trained.

[0098] Furthermore, the above-mentioned training module is specifically used to: randomly select first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each sample data; input the second observation data under the first perspective randomly selected from the first sample data into the control model to perform action prediction under the first time block, obtain the first predicted action output by the control model, and update the parameters of the control model according to the first loss between the first predicted action and the robotic arm action data in the first sample data; input the second observation data and the robotic arm action data in the first sample data into the perspective model to perform perspective prediction of the second time block, and obtain the predicted camera perspective output by the perspective model; input the third observation data under the predicted camera perspective in the second sample data into the control model to perform action prediction under the second time block, obtain the second predicted action output by the control model, and update the parameters of the control model according to the second loss between the second predicted action and the robotic arm action data in the second sample data; use the second loss of the second time block as supervision to update the parameters of the perspective model; re-execute the steps of randomly selecting the first sample data and the second sample data corresponding to the adjacent first time blocks and second time blocks from each sample data until the control model and the perspective model converge.

[0099] The robot task execution device provided in this embodiment has the same implementation principle and technical effects as those of the aforementioned robot task execution method embodiment. For the sake of brief description, for matters not mentioned in the robot task execution device embodiment, reference may be made to the corresponding contents in the aforementioned robot task execution method embodiment.

[0100] like Figure 7 As shown, a robot 700 provided by an embodiment of the present invention includes: a processor 701, a memory 702 and a bus, the memory 702 stores a computer program that can be run on the processor 701, when the robot 700 is running, the processor 701 and the memory 702 communicate through the bus, and the processor 701 executes the computer program to implement the above-mentioned robot task execution method.

[0101] Specifically, the memory 702 and the processor 701 can be general-purpose memories and processors, which are not specifically limited here.

[0102] The embodiment of the present invention also provides an active viewing angle selection system, such as Figure 8 As shown, the active viewing angle selection system includes the above-mentioned robot 801 and also includes a mobile single camera system 802 , and the mobile single camera system 802 is connected to the robot 801 .

[0103] The active perspective selection system provided in this embodiment has the same implementation principle and technical effects as those in the aforementioned robot task execution method embodiment. For the sake of brief description, for matters not mentioned in the active perspective selection system embodiment, reference may be made to the corresponding contents in the aforementioned robot task execution method embodiment.

[0104] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the robot task execution method in the above method embodiment is executed. The computer-readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0105] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0106] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0107] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0109] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot task execution method, characterized in that: Applied to robots connected to mobile single camera systems; The robot task execution method comprises: Acquire first observation data of the mobile single camera system at a current viewing angle; Determining a target action of the robot manipulator and a target viewing angle corresponding to the mobile single camera system according to the first observation data and a preset target task; The robotic arm is controlled to perform the target action, and the mobile single camera system is controlled to move to the target viewing angle to complete the target task.

2. The robot task execution method according to claim 1, characterized in that: The determining, according to the first observation data and the preset target task, the target action of the robot manipulator and the target viewing angle corresponding to the mobile single camera system includes: Predicting the action of the robot arm in the current time block according to the first observation data and the trained control model to obtain the target action; Predicting the viewing angle of the next time block of the mobile single camera system according to the first observation data, the target action and the trained viewing angle model to obtain the target viewing angle; Among them, the control model and the perspective model both correspond to the target task, and the control model and the perspective model are obtained by joint training based on sample data in multiple adjacent time blocks, the sample data includes an observation data set, robot arm position data and robot arm motion data, and the observation data set includes observation data from multiple perspectives.

3. The robot task execution method according to claim 2, characterized in that: The control model includes an encoder and a decoder, the encoder adopts a pre-trained multi-view mask autoencoder, and the decoder includes a Transformer model; the encoder is used to extract features of the first input observation data, and the decoder is used to convert the first target features output by the encoder into the target action.

4. The robot task execution method according to claim 2, characterized in that: The perspective model includes a Transformer model and a SoftMax activation function. The Transformer model is used to extract features from the first observation data and the target action that are input, and the SoftMax activation function is used to convert the second target features output by the Transformer model into the target perspective.

5. The robot task execution method according to claim 2, characterized in that: The robot task execution method also includes: Acquire sample data in a plurality of adjacent time blocks, wherein the sample data includes an observation data set, robot position data, and robot motion data, and the observation data set includes observation data in a plurality of viewing angles; Based on each of the sample data, the control model and the viewing angle model are obtained by joint training.

6. The robot task execution method according to claim 5, characterized in that: The step of jointly training the control model and the viewing angle model based on the sample data includes: Randomly selecting first sample data and second sample data corresponding to adjacent first time blocks and second time blocks from each of the sample data; Inputting the second observation data at the first perspective randomly selected from the first sample data into the control model to perform motion prediction in the first time block, obtaining a first predicted motion output by the control model, and updating the parameters of the control model according to a first loss between the first predicted motion and the robot arm motion data in the first sample data; Input the second observation data and the robot arm motion data in the first sample data into the perspective model to perform perspective prediction of the second time block, and obtain the predicted camera perspective output by the perspective model; input the third observation data under the predicted camera perspective in the second sample data into the control model to perform action prediction under the second time block, and obtain the second predicted action output by the control model, and update the parameters of the control model according to the second loss between the second predicted action and the robot arm motion data in the second sample data; Using the second loss of the second time block as supervision, updating the parameters of the view model; The step of randomly selecting the first sample data and the second sample data corresponding to the adjacent first time blocks and the second time blocks from each of the sample data is re-executed until the control model and the viewing angle model converge.

7. A robot task execution device, characterized in that: Applied to robots connected to mobile single camera systems; The robot task execution device comprises: An acquisition module, used to acquire first observation data of the mobile single camera system at a current viewing angle; a determination module, configured to determine a target motion of the robot manipulator and a target viewing angle corresponding to the mobile single camera system according to the first observation data and a preset target task; The control module is used to control the robot arm to perform the target action and control the mobile single camera system to move to the target viewing angle to complete the target task.

8. A robot comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the robot task execution method described in any one of claims 1 to 6 is implemented.

9. An active viewing angle selection system, characterized in that: The robot of claim 8 further comprises a mobile single-camera system connected to the robot.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot task execution method according to any one of claims 1 to 6 is executed.

Images