Intelligent control method, device and equipment for body, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides an intelligent control method and device for a tool body, equipment, a storage medium and a program product. The method comprises the steps of obtaining image features and text features; inputting the image features and the text features into a high-order target planning network to obtain target hidden variables and multi-modal features; inputting the target hidden variable and the multi-modal feature into a low-order execution network to obtain an action sequence corresponding to the current subtask; the robot is controlled to execute the action sequence, an execution result of the action sequence is obtained, and whether the current subtask is completed or not is judged based on the execution result; and if the current sub-task is completed, generating an action sequence corresponding to the next sub-task based on the low-order execution network until all the sub-tasks are completed. Through the above mode, a hierarchical strategy of high-order planning and low-order execution is adopted, so that the robot can complete execution of complex tasks in a changeable real environment, and the intelligent control method is high in applicability and high in control efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an embodied intelligent control method, device, equipment, storage medium and program product. Background Art

[0002] Embodied AI is an intelligent system based on a physical robot body that achieves perception and action through interaction with the environment. The system acquires information, understands problems, makes decisions and performs corresponding actions through the interaction between the robot agent and the surrounding environment, thereby demonstrating intelligent behavior and adaptability. Embodied intelligent control refers to converting perceived information, including task instructions, target objects and current scenes, into appropriate decisions and formulating step plans. This process needs to consider the robot's execution capabilities and changes in the environment to ensure that the planned actions can be smoothly executed during implementation.

[0003] Robots are mainly divided into industrial robots and service robots. Industrial robots are mainly used in industrial manufacturing scenarios, such as automobile manufacturing, parts processing, etc. Their working environment is structured, usually in a fixed space suitable for robot arms to work. Service robots are mainly used in people's work and life scenarios, and the working environment is relatively complex, such as hotels, restaurants, office buildings, etc., and the control accuracy and repeatability of service robots are also lower than those of industrial robots. With the increasing application of robots in life, how to control robots to complete complex tasks more efficiently and quickly has become an urgent problem to be solved.

[0004] In some related technologies, embodied intelligent control solutions are mainly based on kinematic principles, usually relying on preset rules and limited reaction modes. However, existing embodied intelligent control solutions have poor applicability and are difficult to implement tasks in a changing real environment. Summary of the invention

[0005] The embodiments of the present application provide an embodied intelligent control method, device, equipment, storage medium and program product to solve the technical problem that the existing embodied intelligent control solutions have poor applicability and are difficult to implement task execution in a changing real environment.

[0006] In a first aspect, an embodiment of the present application provides an embodied intelligent control method, comprising: acquiring image features and text features; the image features are determined based on the environment images collected by the robot, and the text features are determined based on the voice commands of the user; the image features and text features are input into a high-order goal planning network to obtain target latent variables and multimodal features output by the high-order goal planning network; the target latent variables are determined based on the high-order goals of multiple subtasks, all subtasks have the same high-order goals, and the high-order goals are determined based on multimodal features; the target latent variables and multimodal features are input into a low-order execution network to obtain an action sequence corresponding to the current subtask output by the low-order execution network; the robot is controlled to execute the action sequence, obtain the execution result of the action sequence, and based on the execution result, determine whether the current subtask is completed; if the current subtask is completed, the action sequence corresponding to the next subtask is generated based on the low-order execution network until all subtasks are completed.

[0007] In one embodiment, a high-order goal planning network includes a feature fusion decoder, which includes multiple neural network layers, each of which includes a Transformer decoder layer and a cross-attention layer connected in sequence; a high-order goal planning network is used to perform feature fusion processing on image features and text features to generate multimodal features; based on the multimodal features, high-order goals for multiple subtasks are generated; based on the high-order goals, target latent variables are generated; and target latent variables and multimodal features are output.

[0008] In one embodiment, a low-order execution network includes a long short-term memory network and a multi-layer perceptron connected in sequence; inputting target latent variables and multimodal features into the low-order execution network to obtain an action sequence corresponding to the current subtask output by the low-order execution network, including: inputting the target latent variables and multimodal features into the long short-term memory network of the low-order execution network to obtain intermediate information output by the long short-term memory network; inputting the intermediate information into the multi-layer perceptron of the low-order execution network to obtain an action sequence corresponding to the current subtask output by the multi-layer perceptron.

[0009] In one embodiment, image features and text features are obtained, including: collecting environmental images based on a robot's visual sensor; inputting the environmental images into an image encoder to obtain image features output by the image encoder; obtaining user voice commands; converting the voice commands into text data; and performing feature extraction on the text data to obtain text features.

[0010] In one embodiment, an environment image includes a first image and a second image, the first image is a global perspective image of the robot, and the second image is a gripper perspective image of the robot, and the image encoder includes a Transformer layer and a resampler connected in sequence; the environment image is input into the image encoder to obtain image features output by the image encoder, including: inputting the first image and the second image into the Transformer layer of the image encoder to obtain an image feature sequence output by the Transformer layer; inputting the image feature sequence into the resampler of the image encoder to obtain image features output by the resampler.

[0011] In one embodiment, feature extraction is performed on text data to obtain text features, including: data preprocessing is performed on the text data to obtain text to be processed; data preprocessing includes data cleaning, word segmentation, word form restoration and stop word removal; the text to be processed is input into a text encoder to obtain text features output by the text encoder.

[0012] In a second aspect, an embodiment of the present application provides an embodied intelligent control device, comprising: an acquisition module, used to acquire image features and text features; the image features are determined based on the environmental images collected by the robot, and the text features are determined based on the user's voice commands; a first generation module, used to input the image features and text features into a high-order goal planning network, and obtain target latent variables and multimodal features output by the high-order goal planning network; the target latent variables are determined based on the high-order goals of multiple subtasks, all subtasks have the same high-order goals, and the high-order goals are determined based on multimodal features; a second generation module, used to input the target latent variables and multimodal features into a low-order execution network, and obtain an action sequence corresponding to the current subtask output by the low-order execution network; an execution module, used to control the robot to execute the action sequence, obtain the execution result of the action sequence, and based on the execution result, determine whether the current subtask is completed; if the current subtask is completed, then generate the action sequence corresponding to the next subtask based on the low-order execution network until all subtasks are completed.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned embodied intelligent control methods is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned embodied intelligent control methods.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned embodied intelligent control methods when executed by a processor.

[0016] The embodied intelligent control method, device, equipment, storage medium and program product provided in the embodiments of the present application adopt a hierarchical strategy of high-order planning and low-order execution. First, image features are determined based on the collected environmental image, and text features are determined based on the user's voice command. The image features and text features are input into the high-order target planning network to obtain the target latent variables and multimodal features output by the high-order target planning network. The target latent variables and multimodal features are then input into the low-order execution network. The transmission of the target latent variables enables effective information interaction between the high-order target planning network and the low-order execution network, which can ensure that the robot can accurately understand and execute each subtask and improve the control efficiency. At the same time, in the process of embodied intelligent control, the changes in the real environment are fully considered, and the current environmental information is integrated into the multimodal features, so that the low-order execution network can generate an action sequence corresponding to the current subtask according to the current environmental information integrated into the target latent variables and the multimodal features, and gradually complete each subtask according to the execution of the action sequence corresponding to the current subtask, so that the robot can complete the execution of complex tasks in a changeable real environment. The embodied intelligent control method has strong applicability and high control efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the embodied intelligent control method provided in the embodiment of the present application.

[0019] Figure 2 This is the second flow chart of the embodied intelligent control method provided in the embodiment of the present application.

[0020] Figure 3 It is a schematic diagram of the structure of the low-level execution network provided in the embodiment of the present application.

[0021] Figure 4 It is a schematic diagram of the structure of the image encoder provided in an embodiment of the present application.

[0022] Figure 5 It is a structural diagram of the embodied intelligent control device provided in an embodiment of the present application.

[0023] Figure 6It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] See also Figure 1 , Figure 1 This is one of the flow charts of the embodied intelligent control method provided in the embodiment of the present application. In the embodiment of the present application, the embodied intelligent control method includes steps S110 to S150, and each step is as follows: S110: Acquire image features and text features.

[0026] The image features are determined based on the environment images collected by the robot, and the text features are determined based on the user's voice commands.

[0027] See also Figure 2 , Figure 2 This is the second flow chart of the embodied intelligent control method provided in the embodiment of the present application.

[0028] like Figure 2 As shown, in order for the robot to adapt to complex environmental changes during task execution, it is necessary to collect environmental information.

[0029] Specifically, based on the robot's visual sensor, environmental images are collected.

[0030] The visual sensor installed on the robot can collect environmental images. The environmental images are static RGB images, which include the global perspective image of the robot and the perspective image of the robot's gripper.

[0031] Preferably, the global perspective image is used to provide an overall view of the robot's surroundings, and the image size of the global perspective image is (200, 200, 3).

[0032] Preferably, the gripper perspective image is used to provide a detailed view of the vicinity of the gripper of the robot hand, and the image size of the gripper perspective image is (84, 84, 3).

[0033] Both the global perspective image and the gripper perspective image can be stored in the central processing unit.

[0034] Furthermore, the environment image is input into an image encoder to obtain image features output by the image encoder.

[0035] Furthermore, the user's voice command is obtained and converted into text data.

[0036] Specifically, the voice recognition module built into the robot system can monitor the user's voice input, capture voice commands through a microphone, and use advanced voice recognition technology to convert voice commands into text data in real time, and store the text data in the central processing unit.

[0037] Furthermore, feature extraction is performed on the text data to obtain text features.

[0038] S120: Input the image features and text features into the high-order target planning network to obtain the target latent variables and multimodal features output by the high-order target planning network.

[0039] The target latent variable is determined based on the high-order objectives of multiple subtasks. All subtasks have the same high-order objective, which is determined based on multimodal features.

[0040] Specifically, the image features and text features are input into the High-Level Goal Planning Network, which can perform feature fusion processing on the image features and text features to generate multimodal features, and generate high-level goals for multiple subtasks based on the multimodal features, generate target latent variables based on the high-level goals, and finally output target latent variables and multimodal features.

[0041] S130: Input the target latent variable and the multimodal features into the low-order execution network to obtain the action sequence corresponding to the current subtask output by the low-order execution network.

[0042] S140: Control the robot to execute the action sequence, obtain the execution result of the action sequence, and determine whether the current subtask is completed based on the execution result.

[0043] Specifically, after determining the action sequence corresponding to the current subtask, the robot is controlled to execute the action sequence, and the task execution status is continuously monitored to obtain the execution result of the action sequence, and based on the execution result, it is determined whether the current subtask is completed.

[0044] S150: If the current subtask has been completed, an action sequence corresponding to the next subtask is generated based on the low-order execution network until all subtasks are completed.

[0045] When the current subtask is completed, the robot will output a task completion signal. If the monitoring detects that a task completion signal is generated, it means that the current subtask has been completed. If the current subtask has been completed, it is determined that the subtask goal needs to be re-planned. At this time, the action sequence corresponding to the next subtask can be generated based on the low-order execution network.

[0046] Specifically, Figure 2 As shown in the figure, after the robot completes the current subtask, the environment in which the robot is located may change. Therefore, it is necessary to re-collect the environment image and input the re-collected environment image into the image encoder to obtain the new image features output by the image encoder; the high-order goal planning network will generate new multimodal features and target latent variables based on the new image features and text features; the low-order execution network will continue to generate the action sequence corresponding to the next subtask based on the new multimodal features and target latent variables, and control the robot to continue to execute the new action sequence until all subtasks are completed.

[0047] If the current subtask is not completed, the robot is controlled to continue executing the action sequence corresponding to the current subtask generated by the low-order execution network until the current subtask is completed.

[0048] It should be noted that the embodied intelligent control method of this embodiment is a robot interactive control method based on subtask splitting. A complex long-term task is split into multiple subtasks through a high-order goal planning network. Each subtask has the same high-order goal. This splitting strategy can not only simplify the overall complexity of the task, but also significantly reduce the frequency of calling the cloud model, thereby reducing the system's response delay and improving the real-time and smoothness of the interaction. In addition, by splitting a complex long-term task into multiple subtasks, the robot can gradually complete each small goal, thereby achieving the smooth execution of the entire complex task. This method is particularly suitable for task scenarios that require long-term continuous operation, and can effectively improve the stability and reliability of the robot system.

[0049] The embodied intelligent control method provided in the embodiment of the present application adopts a hierarchical strategy of high-order planning and low-order execution. First, image features are determined based on the collected environmental images, and text features are determined based on the user's voice commands. The image features and text features are input into the high-order target planning network to obtain the target latent variables and multimodal features output by the high-order target planning network. The target latent variables and multimodal features are then input into the low-order execution network. The transmission of the target latent variables enables effective information interaction between the high-order target planning network and the low-order execution network, which can ensure that the robot can accurately understand and execute each subtask and improve the control efficiency. At the same time, in the process of embodied intelligent control, the changes in the real environment are fully considered, and the current environmental information is integrated into the multimodal features, so that the low-order execution network can generate an action sequence corresponding to the current subtask according to the current environmental information integrated into the target latent variables and the multimodal features, and gradually complete each subtask according to the execution of the action sequence corresponding to the current subtask, so that the robot can complete the execution of complex tasks in a changeable real environment. The embodied intelligent control method has strong applicability and high control efficiency.

[0050] In some embodiments, the high-order goal planning network includes a feature fusion decoder, which includes multiple neural network layers, each neural network layer includes a Transformer decoder layer and a cross-attention layer connected in sequence; the high-order goal planning network is used to perform feature fusion processing on image features and text features to generate multimodal features; based on the multimodal features, high-order goals for multiple subtasks are generated; based on the high-order goals, target latent variables are generated; and the target latent variables and multimodal features are output.

[0051] Specifically, the high-order goal planning network includes a feature fusion decoder, which includes multiple neural network layers, each of which includes a Transformer decoder layer and a cross-attention layer connected in sequence.

[0052] Among them, the Transformer decoder layer can be directly copied from the pre-trained language model (such as LlaMA model, GPT-Neox model, MPT model, etc.) and is frozen during the entire training process.

[0053] The criss-cross attention layer takes the language tokens as queries and the encoded visual tokens as keys and values, and is fine-tuned on the manipulated data via the imitation learning objective.

[0054] Specifically, the image feature X v After the text features are input into the feature fusion decoder of the high-order target planning network, the feature fusion decoder can be used to transform the image features X vThe high-order goal planning network is designed to generate high-order goals for multiple subtasks based on the fused multimodal features. The high-order goal corresponds to a subtask fragment in a complete long task, which contains multiple consecutive executable action sequences. In the high-order goal planning network, this high-order goal planning network can be represented as goal latent variables, which are used to describe the high-level goals and requirements of the task and serve as the input of the low-order execution network to guide it to generate specific action sequences.

[0055] In some embodiments, the low-order execution network includes a long short-term memory network and a multi-layer perceptron connected in sequence; the target latent variables and multimodal features are input into the low-order execution network to obtain an action sequence corresponding to the current subtask output by the low-order execution network, including: inputting the target latent variables and multimodal features into the long short-term memory network of the low-order execution network to obtain intermediate information output by the long short-term memory network; inputting the intermediate information into the multi-layer perceptron of the low-order execution network to obtain an action sequence corresponding to the current subtask output by the multi-layer perceptron.

[0056] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of the low-level execution network provided in the embodiment of the present application.

[0057] like Figure 3 As shown in the figure, the low-level execution network (Low-Level Execution Network) includes a long short-term memory network (LSTM) and a multi-layer perceptron (MLP) connected in sequence.

[0058] Specifically, the target latent variables and multimodal features output by the high-order goal planning network will be input into the long short-term memory network of the low-order execution network. The long short-term memory network can perform feature processing on the input target latent variables and multimodal features through its internal cells, capture the dependencies in the time series, and generate intermediate information.

[0059] Furthermore, the intermediate information is input into the multilayer perceptron of the low-order execution network to obtain the action sequence corresponding to the current subtask output by the multilayer perceptron.

[0060] The embodied intelligent control method provided in the embodiment of the present application combines target latent variables and multimodal features and uses a long short-term memory network to capture time series information, so that the low-order execution network can generate accurate action sequences, thereby enabling the robot to perform precise operations and task execution according to the current environmental state and target tasks.

[0061] In some embodiments, obtaining image features and text features includes: collecting environmental images based on a robot's visual sensor; inputting the environmental images into an image encoder to obtain image features output by the image encoder; obtaining user's voice commands; converting the voice commands into text data; and performing feature extraction on the text data to obtain text features.

[0062] In some embodiments, the environment image includes a first image and a second image, the first image is a global perspective image of the robot, and the second image is a gripper perspective image of the robot, and the image encoder includes a Transformer layer and a resampler connected in sequence; the environment image is input into the image encoder to obtain image features output by the image encoder, including: inputting the first image and the second image into the Transformer layer of the image encoder to obtain an image feature sequence output by the Transformer layer; inputting the image feature sequence into the resampler of the image encoder to obtain image features output by the resampler.

[0063] See also Figure 4 , Figure 4 It is a schematic diagram of the structure of the image encoder provided in an embodiment of the present application.

[0064] like Figure 4 As shown, the environment image includes a first image and a second image, the first image is the global perspective image of the robot, and the second image is the gripper perspective image of the robot; for the collected global perspective image and gripper perspective image, it is necessary to sample them separately to obtain a picture sequence, and then encode them through an image encoder to obtain image features.

[0065] The image encoder consists of a visual Transformer layer (ViT, Figure 4 VisionTransformer) and a resampler ( Figure 4 Resampler in the example).

[0066] Specifically, at each time step, the global view image I t and the gripper perspective image G t The input is sent to the Transformer layer of the image encoder. The Transformer layer can transform the global view image I t and the gripper perspective image G t Encode, generate and output an image feature sequence X containing N visual tokens n .

[0067] Furthermore, the image feature sequence X n Input to the image encoder’s resampler, which compresses the number of visual tokens from N to N r, generate and output image features X v .

[0068] Specifically, the implementation formula of the resampler is as follows: ; ; ; in, is the key vector, is the weight matrix corresponding to the key vector; is a value vector, is the weight matrix corresponding to the value vector; is the query vector; is the activation function; d is the dimension of the input data.

[0069] The meaning of the above formula is: for the image feature sequence X encoded by the Transformer layer n , the key vector and value vector can be obtained by linear mapping, and the learnable parameters of the resampler are used as the query vector. The query vector and the key vector are transposed and multiplied to calculate the feature similarity matrix. The attention mechanism is used to obtain the feature sequence X from the image. n Select the most important N r features, generate image features X v .

[0070] In some embodiments, feature extraction is performed on text data to obtain text features, including: data preprocessing is performed on the text data to obtain text to be processed; data preprocessing includes data cleaning, word segmentation, word form restoration and stop word removal; the text to be processed is input into a text encoder to obtain text features output by the text encoder.

[0071] Specifically, the text data is cleaned, segmented, restored and stop-worded to obtain the text to be processed.

[0072] Furthermore, the text to be processed is input into the text encoder, which can convert the text to be processed into word vectors, use the pre-trained language model to encode the sentences in the text to be processed, obtain sentence-level semantic representation, and combine the encoded text vectors into sequences to extract key features, generate and output text features, and text features can be used for multimodal information integration and target planning, thereby guiding the robot to perform specific tasks.

[0073] Compared with the prior art, the embodied intelligent control method provided in the embodiments of the present application has at least the following advantages: (1) In the prior art, the robot needs to re-plan its goal every time it performs a task. This process is too time-consuming and affects the real-time nature of the interaction. This embodiment reduces the frequency of planning goals by splitting a complex task into multiple subtasks, and making each subtask have the same high-level goal. This method not only simplifies the task planning process, but also improves the response speed of the system, which can significantly improve the fluency of the interaction.

[0074] (2) In the prior art, the control of robots usually relies on preset grasping points and postures, which has obvious limitations when dealing with complex environments and changing tasks. This embodiment uses a hierarchical strategy of high-level goal planning and low-level execution, so that the robot no longer relies on preset grasping points and postures, but dynamically adjusts the grasping strategy according to the real-time environment and task requirements. This method enables the robot to flexibly respond to various tasks in complex and dynamic environments, improving the adaptability and execution efficiency of the robot system.

[0075] (3) In the prior art, when there are similar objects in the environment, it is necessary to clarify the target object based on the interactive instructions, otherwise it is easy to confuse and grasp the wrong target. However, this embodiment adopts a multimodal transformer mechanism for multimodal feature fusion, which can obtain multimodal features conditioned by language instructions, and to a certain extent improves the accuracy of target locking in language interactive control.

[0076] The present application also provides an embodied intelligent control device. Figure 5 , Figure 5 5 is a schematic diagram of the structure of the embodied intelligent control device provided in the embodiment of the present application. In the embodiment of the present application, the embodied intelligent control device includes an acquisition module 510, a first generation module 520, a second generation module 530 and an execution module 540.

[0077] The acquisition module 510 is used to acquire image features and text features.

[0078] The image features are determined based on the environment images collected by the robot, and the text features are determined based on the user's voice commands.

[0079] The first generation module 520 is used to input the image features and text features into the high-order target planning network to obtain the target latent variables and multimodal features output by the high-order target planning network.

[0080] The target latent variable is determined based on the high-order objectives of multiple subtasks. All subtasks have the same high-order objective, which is determined based on multimodal features.

[0081] The second generation module 530 is used to input the target latent variable and the multimodal feature into the low-order execution network to obtain the action sequence corresponding to the current subtask output by the low-order execution network.

[0082] The execution module 540 is used to control the robot to execute the action sequence, obtain the execution result of the action sequence, and judge whether the current subtask is completed based on the execution result; if the current subtask is completed, the action sequence corresponding to the next subtask is generated based on the low-order execution network until all subtasks are completed.

[0083] In some embodiments, the high-order goal planning network includes a feature fusion decoder, which includes multiple neural network layers, each neural network layer includes a Transformer decoder layer and a cross-attention layer connected in sequence; the high-order goal planning network is used to perform feature fusion processing on image features and text features to generate multimodal features; based on the multimodal features, high-order goals for multiple subtasks are generated; based on the high-order goals, target latent variables are generated; and the target latent variables and multimodal features are output.

[0084] In some embodiments, the low-level execution network includes a long short-term memory network and a multi-layer perceptron connected in sequence.

[0085] The second generation module 530 is used to input the target latent variables and multimodal features into the long short-term memory network of the low-order execution network to obtain the intermediate information output by the long short-term memory network; input the intermediate information into the multi-layer perceptron of the low-order execution network to obtain the action sequence corresponding to the current subtask output by the multi-layer perceptron.

[0086] In some embodiments, the acquisition module 510 is used to collect environmental images based on the robot's visual sensor; input the environmental images into the image encoder to obtain image features output by the image encoder; obtain the user's voice commands; convert the voice commands into text data; perform feature extraction on the text data to obtain text features.

[0087] In some embodiments, the environment image includes a first image and a second image, the first image is a global perspective image of the robot, the second image is a gripper perspective image of the robot, and the image encoder includes a Transformer layer and a resampler connected in sequence.

[0088] The acquisition module 510 is used to input the environment image into the image encoder to obtain the image features output by the image encoder, including: inputting the first image and the second image into the Transformer layer of the image encoder to obtain the image feature sequence output by the Transformer layer; inputting the image feature sequence into the resampler of the image encoder to obtain the image features output by the resampler.

[0089] In some embodiments, the acquisition module 510 is used to perform data preprocessing on text data to obtain text to be processed; data preprocessing includes data cleaning processing, word segmentation processing, word form restoration processing and stop word removal processing; the text to be processed is input into the text encoder to obtain text features output by the text encoder.

[0090] The present application also provides an electronic device, Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the embodied intelligent control method.

[0091] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] An embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the embodied intelligent control method provided by the above-mentioned methods.

[0093] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the embodied intelligent control methods provided by the above methods.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An embodied intelligent control method, characterized in that: include: Obtain image features and text features; The image feature is determined based on the environment image collected by the robot, and the text feature is determined based on the voice command of the user; Inputting the image features and the text features into a high-order target planning network to obtain target latent variables and multimodal features output by the high-order target planning network; The target latent variable is determined based on the high-order objectives of the plurality of subtasks, all the subtasks have the same high-order objective, and the high-order objective is determined based on the multimodal features; Inputting the target latent variable and the multimodal feature into a low-order execution network, and obtaining an action sequence corresponding to the current subtask output by the low-order execution network; Controlling the robot to execute the action sequence, obtaining an execution result of the action sequence, and judging whether the current subtask is completed based on the execution result; If the current subtask has been completed, an action sequence corresponding to the next subtask is generated based on the low-order execution network until all the subtasks are completed.

2. The embodied intelligent control method according to claim 1, characterized in that: The high-order goal planning network includes a feature fusion decoder, the feature fusion decoder includes a plurality of neural network layers, each of the neural network layers includes a Transformer decoder layer and a cross attention layer connected in sequence; The high-order goal planning network is used to perform feature fusion processing on the image features and the text features to generate the multimodal features; Based on the multimodal features, multiple high-level goals of the subtasks are generated; based on the high-level goals, the target latent variables are generated; and the target latent variables and the multimodal features are output.

3. The embodied intelligent control method according to claim 1, characterized in that: The low-order execution network includes a long short-term memory network and a multi-layer perceptron connected in sequence; The step of inputting the target latent variable and the multimodal feature into a low-order execution network to obtain an action sequence corresponding to the current subtask output by the low-order execution network includes: Inputting the target latent variable and the multimodal feature into the long short-term memory network of the low-order execution network to obtain intermediate information output by the long short-term memory network; The intermediate information is input into the multilayer perceptron of the low-order execution network to obtain an action sequence corresponding to the current subtask output by the multilayer perceptron.

4. The embodied intelligent control method according to claim 1, characterized in that: The obtaining of image features and text features includes: Based on the visual sensor of the robot, collecting environmental images; Inputting the environment image into an image encoder to obtain image features output by the image encoder; Get the user's voice command; Converting the voice command into text data; Feature extraction is performed on the text data to obtain the text feature.

5. The embodied intelligent control method according to claim 4, characterized in that: The environment image includes a first image and a second image, the first image is a global perspective image of the robot, the second image is a gripper perspective image of the robot, and the image encoder includes a Transformer layer and a resampler connected in sequence; The step of inputting the environment image into an image encoder to obtain image features output by the image encoder includes: Inputting the first image and the second image into the Transformer layer of the image encoder to obtain an image feature sequence output by the Transformer layer; The image feature sequence is input into the resampler of the image encoder to obtain the image features output by the resampler.

6. The embodied intelligent control method according to claim 4, characterized in that: The step of extracting features from the text data to obtain the text features includes: Performing data preprocessing on the text data to obtain a text to be processed; the data preprocessing includes data cleaning, word segmentation, word form restoration and stop word removal; The text to be processed is input into a text encoder to obtain the text features output by the text encoder.

7. An embodied intelligent control device, characterized in that: include: An acquisition module, used to acquire image features and text features; The image feature is determined based on the environment image collected by the robot, and the text feature is determined based on the voice command of the user; A first generation module is used to input the image features and the text features into a high-order target planning network to obtain target latent variables and multimodal features output by the high-order target planning network; The target latent variable is determined based on the high-order objectives of the plurality of subtasks, all the subtasks have the same high-order objective, and the high-order objective is determined based on the multimodal features; A second generation module is used to input the target latent variable and the multimodal feature into a low-order execution network to obtain an action sequence corresponding to the current subtask output by the low-order execution network; An execution module is used to control the robot to execute the action sequence, obtain the execution result of the action sequence, and judge whether the current subtask is completed based on the execution result; if the current subtask is completed, an action sequence corresponding to the next subtask is generated based on the low-order execution network until all the subtasks are completed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the embodied intelligent control method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the embodied intelligent control method as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the embodied intelligent control method as claimed in any one of claims 1 to 6 is implemented.

Images