Behavior instruction generation method and device, equipment, robot, medium and product

Abstract

The application provides a behavior instruction generation method and device, equipment, robot, medium and product, and belongs to the technical field of robots, and comprises the following steps: processing environment perception information of a robot to generate environment description information of an environment in which the robot is located; performing robot task planning based on the environment description information and a natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination comprises a plurality of atomic instructions; matching the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions to obtain a processing result; in the case where the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and at least one atomic instruction passes the task logic verification, taking the at least one atomic instruction as a behavior instruction of the robot. The application can improve the flexibility and reliability of a body intelligence system in understanding and executing a behavior instruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a behavior instruction generation method and device, equipment, robot, medium and product. BACKGROUND

[0002] With the continuous development and progress of artificial intelligence and robot technology, the application prospect of artificial intelligence systems with physical bodies (embodied intelligence systems) in daily life, industrial automation and medical assistance, etc. is becoming more and more broad. In the related art, the embodied intelligence system mainly relies on detailed behavior instructions to gradually execute corresponding operations, and the understanding and execution of short and high-level behavior instructions (for example: the behavior instruction of "going to the kitchen to get milk" used by people in daily life) in complex and dynamic environments will not be flexible and efficient enough, and there may even be a situation where the behavior instruction output by people cannot be responded to. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a behavior instruction generation method, device, equipment, robot, medium and product, which aims to improve the flexibility and reliability of the embodied intelligence system in understanding and executing behavior instructions, so as to efficiently execute corresponding operation tasks in complex and dynamic environments.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a behavior instruction generation method, which comprises:

[0005] processing the environment perception information of the robot to generate environment description information of the environment in which the robot is located;

[0006] performing robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination comprises a plurality of atomic instructions;

[0007] matching the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions to obtain a processing result;

[0008] in the case where the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, taking the at least one atomic instruction as the behavior instruction of the robot.

[0009] In some embodiments, the matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot comprises:

[0010] matching the plurality of atomic instructions in the first atomic instruction combination with atomic instructions in an atomic instruction library of the robot;

[0011] The task logic verification on the plurality of atomic instructions comprises:

[0012] In a case where at least one atomic instruction in the plurality of atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and the at least one atomic instruction is verified in task logic.

[0013] In some embodiments, after the matching the plurality of atomic instructions in the first atomic instruction combination with atomic instructions in an atomic instruction library of the robot, the method further comprises:

[0014] In a case where a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, the first target atomic instruction is decomposed into a plurality of target sub-instructions;

[0015] The plurality of target sub-instructions are matched with atomic instructions in the atomic instruction library;

[0016] In a case where the plurality of target sub-instructions match atomic instructions in the atomic instruction library, it is confirmed that the first target atomic instruction belongs to the atomic instruction library.

[0017] In some embodiments, the task logic verification on the plurality of atomic instructions comprises:

[0018] An instruction sequence is generated based on the plurality of atomic instructions;

[0019] A verification result is obtained by verifying a possession logic state of an instruction in the instruction sequence based on a preset possession logic constraint condition;

[0020] In a case where the verification result indicates that the instruction in the instruction sequence satisfies the possession logic constraint condition, a position logic state of the instruction in the instruction sequence is verified based on a preset position logic constraint condition.

[0021] In some embodiments, the method further comprises:

[0022] in a case where the processing result indicates that the plurality of atomic instructions belong to the atomic instruction library, and a second target atomic instruction in the plurality of atomic instructions passes task logic verification, and a third target atomic instruction in the plurality of atomic instructions does not pass task logic verification, optimizing the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein a quantity of the plurality of atomic instructions is equal to a sum of a quantity of the second target atomic instructions and a quantity of the third target atomic instructions.

[0023] taking the second target atomic instruction and the optimized sub-instruction as the behavior instruction of the robot.

[0024] In some embodiments, the robot task planning based on the environment description information and the natural language command to obtain the first atomic instruction combination of the robot includes:

[0025] inputting the environment description information and the natural language command into a preset large language model to perform robot task planning based on the environment description information and the natural language command by the large language model to obtain the first atomic instruction combination of the robot.

[0026] In some embodiments, the method further includes:

[0027] in a case where the processing result indicates that a plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, and / or the plurality of atomic instructions do not pass task logic verification, adjusting the large language model;

[0028] inputting the environment description information and the natural language command into the adjusted large language model to perform robot task planning based on the environment description information and the natural language command by the adjusted large language model to obtain a second atomic instruction combination of the robot;

[0029] matching a plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions.

[0030] in a case where a fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library, and the fourth target atomic instruction passes task logic verification, taking the fourth target atomic instruction as the behavior instruction of the robot.

[0031] In some embodiments, the environment perception information includes visual input information and voice input information.

[0032] The processing of the environment perception information of the robot to generate the environment description information of the environment in which the robot is located includes:

[0033] extract features of the visual input information through a preset multi-modal large model to obtain environment visual features corresponding to the visual input information, and extract features of the speech input information through the multi-modal large model to obtain environment language features corresponding to the speech input information;

[0034] fuse the environment visual features and the environment language features through the multi-modal large model to generate environment description information of an environment in which the robot is located.

[0035] To achieve the above object, a second aspect of the embodiment of the present application provides a behavior instruction generation device, which comprises:

[0036] an environment description module configured to process environment perception information of a robot to generate environment description information of an environment in which the robot is located;

[0037] a task planning module configured to perform robot task planning based on the environment description information and a natural language command to obtain a first atomic instruction combination of the robot, wherein the first atomic instruction combination comprises a plurality of atomic instructions;

[0038] an instruction verification module configured to match the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and perform task logic verification on the plurality of atomic instructions to obtain a processing result;

[0039] an instruction generation module configured to, in a case where the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, take the at least one atomic instruction as a behavior instruction of the robot.

[0040] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0041] To achieve the above object, a fourth aspect of the embodiment of the present application provides a robot, which is configured with an electronic device, the electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0042] To achieve the above object, a fifth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the method of the first aspect when executed by a processor.

[0043] To achieve the above-mentioned purpose, the sixth aspect of the embodiments of the present application proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method provided in the first aspect above.

[0044] The behavioral instruction generation method, device, electronic device, robot, computer-readable storage medium and computer program product proposed in the present application process the robot's environmental perception information to generate environmental description information of the robot's environment; perform robot task planning based on the environmental description information and natural language commands to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes multiple atomic instructions; match the multiple atomic instructions in the first atomic instruction combination with the robot's atomic instruction library, and perform task logic verification on the multiple atomic instructions to obtain a processing result; when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is used as the behavioral instruction of the robot.

[0045] That is to say, after the embodiment of the present application performs robot task planning based on the environment description information and the natural language commands to the robot and obtains the first atomic instruction combination of the robot, it further performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination to determine whether the multiple atomic instructions belong to sub-instructions in the robot's atomic instruction library, and determine whether the atomic instructions comply with the task logic. Only when the physical constraint verification is passed to ensure that the atomic instruction is both in the robot's atomic instruction library and complies with the logic of the current task, the atomic instruction is used as the robot's behavior instruction.

[0046] Thus, when the embodiment of the present application is applied to an embodied intelligent system, the embodied intelligent system can generate atomic instructions by combining environmental description information and natural language commands, thereby being able to respond to high-level, concise natural language commands input by the user, and thus being able to efficiently plan and execute tasks in a dynamic and complex environment. Moreover, the embodied intelligent system performs physical constraint verification on the generated atomic instructions to ensure that these instructions are both in the robot's atomic instruction library and conform to the logic of the current task, thereby avoiding robot execution conflicts and erroneous operations, thereby improving the executability of these instructions and improving the success rate of executing instructions to achieve task goals. That is, the embodiment of the present application can improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavioral instructions, thereby being able to efficiently execute corresponding operational tasks in a complex and dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1The step flow diagram of the method for generating behavior instructions provided by the embodiment of the present application in some embodiments is shown in the following figure.

[0048] Figure 2 For Figure 1 The detailed step flow diagram of step S101 in the method for generating behavior instructions provided by the embodiment of the present application is shown in the following figure.

[0049] Figure 3 The step flow diagram of the method for generating behavior instructions provided by the embodiment of the present application in some other embodiments is shown in the following figure.

[0050] Figure 4 The step flow diagram of the method for generating behavior instructions provided by the embodiment of the present application in some other embodiments is shown in the following figure.

[0051] Figure 5 For Figure 1 The detailed step flow diagram of step S103 in the method for generating behavior instructions provided by the embodiment of the present application is shown in the following figure.

[0052] Figure 6 The step flow diagram of the method for generating behavior instructions provided by the embodiment of the present application in some other embodiments is shown in the following figure.

[0053] Figure 7 The overall architecture diagram of the method for generating behavior instructions provided by the embodiment of the present application in some embodiments is shown in the following figure.

[0054] Figure 8 The application flow diagram of the method for generating behavior instructions provided by the embodiment of the present application is shown in the following figure.

[0055] Figure 9 The structure diagram of the device for generating behavior instructions provided by the embodiment of the present application is shown in the following figure.

[0056] Figure 10 The hardware structure diagram of the electronic device provided by the embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0058] It should be noted that although the functional modules are divided in the device diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0060] Firstly, the professional technical terms related to the embodiments of the present application are explained.

[0061] Embodied intelligence system.

[0062] Embodied intelligence system refers to an artificial intelligence system with a physical body. The embodied intelligence system can perceive, act and interact in a real physical environment. The embodied intelligence system combines technologies from multiple fields such as computer vision, natural language processing, robotics and artificial intelligence, enabling intelligent agents such as robots to understand and adapt to their physical environment and perform complex tasks. The characteristics of the embodied intelligence system include: the embodied intelligence system can perceive information about the surrounding environment through sensors (such as cameras, microphones, etc.), the embodied intelligence system can move and manipulate objects in the environment through physical components (such as mechanical arms, wheels, etc.) to perform physical actions, and the embodied intelligence system can also understand task instructions and perform corresponding tasks in the physical environment.

[0063] Atomic instruction library.

[0064] The atomic instruction library refers to a set of basic instructions that a robot can execute. The basic instructions in the atomic instruction library are the smallest units of robot operation, and the basic qualities in the atomic instruction library can be, for example, "move to a certain position", "grab an object", "turn on a switch", etc. The atomic instruction library can be set in the form of a "predefined instruction set", that is, a set of basic operation instructions is predefined according to the functions of the robot to construct the atomic instruction library of the robot.

[0065] Next, the overall concept of the behavior instruction generation method provided by the embodiments of the present application is described.

[0066] With the continuous development and progress of artificial intelligence and robotics, the application prospect of artificial intelligence systems with physical bodies (embodied intelligence systems) in daily life, industrial automation and medical assistance is becoming more and more broad. In related technologies, the embodied intelligence system mainly relies on detailed behavior instructions to gradually perform corresponding operations, and the understanding and execution of short and high-level behavior instructions in complex and dynamic environments (for example: the behavior instruction "go to the kitchen to get milk" used by people in daily life) are not flexible and efficient enough, and there may be cases where the behavior instructions output by people cannot be responded to.

[0067] To this end, the embodiment of the present application provides a behavior instruction generation method and device, equipment, robot, medium and product, aiming to improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavior instructions, so as to efficiently execute corresponding operation tasks in a complex and dynamic environment.

[0068] The embodiment of the present application processes the environmental perception information of the robot to generate environmental description information of the environment in which the robot is located; performs robot task planning based on the environmental description information and a natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes a plurality of atomic instructions; matches the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performs task logic verification on the plurality of atomic instructions to obtain a processing result; in the case that the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is taken as a behavior instruction of the robot.

[0069] That is, after the embodiment of the present application performs robot task planning based on the environmental description information and a natural language command for the robot to obtain a first atomic instruction combination of the robot, it further performs physical constraint verification on the plurality of atomic instructions in the first atomic instruction combination, so as to determine whether the plurality of atomic instructions belong to sub-instructions in the atomic instruction library of the robot, and determine whether the atomic instructions conform to the task logic, and only in the case that the physical constraint verification is passed to ensure that the atomic instructions are both in the atomic instruction library of the robot and conform to the logic of the current task, the atomic instructions are taken as the behavior instructions of the robot.

[0070] In this way, when applied to the embodied intelligent system, the embodiment of the present application can generate atomic instructions by combining environmental description information and natural language commands, so as to respond to high-level short natural language commands input by the user, and then efficiently plan and execute tasks in a dynamic and complex environment. Moreover, the embodied intelligent system performs physical constraint verification on the generated atomic instructions to ensure that these instructions are both in the atomic instruction library of the robot and conform to the logic of the current task, avoiding conflicts and erroneous operations of the robot, so as to improve the execution of these instructions and the success rate of executing the instructions to achieve the task goal. That is, the embodiment of the present application can improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavior instructions, so as to efficiently execute corresponding operation tasks in a complex and dynamic environment.

[0071] Next, the method for generating behavior instructions provided by the embodiments of the present application, the device, the equipment, the robot, the medium and the product are specifically described through the following embodiments, and the method for generating behavior instructions provided by the embodiments of the present application is first described in detail.

[0072] It should be noted that the method for generating behavior instructions provided by the embodiments of the present application relates to the technical field of robots. The method for generating behavior instructions provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a terminal configured by a robot, or can be an electronic device such as a smartphone, a tablet computer, a notebook computer, a desktop computer, etc. associated with the robot. The terminal is associated with the robot, which means that the terminal can communicate and interact with the robot based on a network. The server end can be a background server terminal device of the robot, which can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content distribution network (CDN), and big data and artificial intelligence platform. The software can be an application, a computer program, and a storage medium carrying the computer program, etc. that implements the method for generating behavior instructions. It should be understood that, based on different design needs of actual applications, in different feasible embodiments, the terminal, the server end, and the software, etc. applying the method for generating behavior instructions provided by the embodiments of the present application can of course also be other forms not listed here, and the method for generating behavior instructions provided by the embodiments of the present application does not specifically limit this.

[0073] In addition, the present application can also be used in a plurality of general-purpose or special-purpose computer system environments or configurations. For example: embodied intelligent system, robot, personal computer, server computer, handheld device or portable device, tablet device, multi-processor system, microprocessor-based system, set-top box, programmable consumer electronics, personal computer (PC), small computer, mainframe computer, distributed computing environment including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] For the sake of understanding and elaboration, in the following, the generation method of behavior instruction provided by the embodiment of the application is taken as an example to explain the specific embodiments of the application in detail. The embodied intelligent system can control its own physical body by applying the generation method of behavior instruction provided by the embodiment of the application, or the embodied intelligent system can also control the process of planning and executing tasks of the robot by being integrated on the robot or being associated with the robot. The embodied intelligent system is referred to as system in the following. The implementation of any form of subject applying the generation method of behavior instruction provided by the embodiment of the application can refer to the process of the generation method of behavior instruction of the electronic device described in the following.

[0075] It should be noted that in each specific embodiment of the application, when it is necessary to process relevant data related to the identity or characteristics of the user according to user information, user behavior data, user historical data and user location information, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the application needs to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiment of the application will be obtained.

[0076] Please refer to Figure 1 , Figure 1 The step flowchart of the generation method of behavior instruction provided by the embodiment of the application in some embodiments. It should be understood that although Figure 1 and subsequent other step flowcharts show the execution order of some method steps, based on different design needs of actual applications, the generation method of behavior instruction provided by the embodiment of the application can of course adopt an execution order of method steps different from that shown in the figure. That is, Figure 1 The order of the method steps shown in the figure does not constitute a limitation on the execution logic order of the generation method of behavior instruction provided by the embodiment of the application. Any reasonable change based on the order of the method steps shown in the figure should be included in the protection scope of the generation method of behavior instruction provided by the embodiment of the application. Figure 1

[0077] As Figure 1 shown, in some embodiments, the system applying the generation method of behavior instruction provided by the embodiment of the application can include steps S101 to S104.

[0078] ​Step S101: processing the environment perception information of the robot to generate environment description information of the environment where the robot is located.

[0079] The system processes the environment perception information of the robot when the user issues a high-level short instruction to the robot to instruct the robot to perform a task, thereby generating environment description information of the environment where the robot is located based on the environment perception information.

[0080] In some embodiments, the system can receive the instruction input by the user to the robot through natural language by controlling the microphone of the robot, and then control the robot to perceive the information of the surrounding environment through sensors (such as a camera, a microphone, etc.) and upload to the system. In this way, the system can obtain the environment perception information of the robot and further process it to generate the corresponding environment description information.

[0081] In other embodiments, the system can also control the robot to receive the instruction manually input by the user to instruct the robot to perform a task through a visual human-computer interaction interface. Alternatively, the system can also receive the instruction input by the user on the visual human-computer interaction interface provided by the terminal device associated with the robot. After receiving the instruction input by the user, the system can further control the robot to perceive the information of the surrounding environment, thereby performing the operation of processing the environment perception information of the robot to generate the corresponding environment description information.

[0082] Step S102: robot task planning based on the environment description information and natural language commands to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes a plurality of atomic instructions.

[0083] It should be noted that the first atomic instruction combination is a combination of a series of basic instructions (atomic instructions) that the robot needs to execute in response to the high-level short instruction (natural language command) input by the user to execute the task indicated by the user.

[0084] After the system generates the environment description information of the environment where the robot is located, the system plans the task instructed by the user to the robot in combination with the environment description information and the high-level short instruction (natural language command) input by the user to the robot, thereby obtaining a plurality of atomic instructions that the robot needs to execute in response to the natural language command input by the user, and taking the plurality of atomic instructions as the first atomic instruction combination.

[0085] Step S103: matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions to obtain a processing result.

[0086] The system, after obtaining the first atomic instruction combination of the robot, does not directly take the plurality of atomic instructions in the first atomic instruction combination as the final behavior instruction of the robot, but first matches the plurality of atomic instructions with the atomic instruction library of the robot that has been set in advance, so as to confirm whether the plurality of atomic instructions are basic instructions existing in the atomic instruction library. Moreover, the system also performs task logic verification processing on the plurality of atomic instructions, so as to confirm whether each of the plurality of atomic instructions conforms to the current task logic of the robot. The system, after matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions, obtains a corresponding processing result.

[0087] In some embodiments, when the system matches the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on the plurality of atomic instructions, the system can first match all the atomic instructions one by one with the atomic instruction library, and then perform task logic verification on all the atomic instructions one by one.

[0088] In some embodiments, the system can also perform task logic verification on a target instruction in the plurality of atomic instructions simultaneously when matching the target instruction with the atomic instruction library. This is repeated for each target instruction until all the atomic instructions have been matched with the atomic instruction library and have been subjected to task logic verification.

[0089] Step S104: In the case where the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, taking the at least one atomic instruction as the behavior instruction of the robot.

[0090] It should be noted that the atomic instruction belongs to the atomic instruction library means that the system confirms that the atomic instruction is a basic instruction existing in the atomic instruction library by matching the atomic instruction with the atomic instruction library.

[0091] After the system matches the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on the plurality of atomic instructions to obtain a corresponding processing result, if the processing result indicates that at least one atomic instruction in the plurality of atomic instructions is a basic instruction existing in the atomic instruction library, and the at least one atomic instruction also passes the task logic verification performed by the system, in this case, the system takes the at least one atomic instruction as one of the behavior instructions to be executed by the robot in response to the natural language command to perform the task.

[0092] In the embodiments of the present application, when the user issues a high-level short instruction to the robot to instruct the robot to perform a task, the system processes the environment perception information of the robot to generate environment description information of the environment where the robot is located. Then, the system combines the environment description information and the high-level short instruction (natural language command) input by the user for the robot to perform robot task planning, so as to obtain a plurality of atomic instructions that the robot needs to execute in response to the natural language command input by the user, and combine the plurality of atomic instructions as a first atomic instruction combination. After that, the system matches the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on the plurality of atomic instructions to obtain a corresponding processing result. Finally, if the processing result indicates that at least one atomic instruction in the plurality of atomic instructions is a basic instruction existing in the atomic instruction library, and the at least one atomic instruction also passes the task logic verification performed by the system, in this case, the system takes the at least one atomic instruction as one of the behavior instructions that the robot needs to execute to respond to the natural language command and perform a task.

[0093] In this way, the embodiments of the present application generate atomic instructions by the system combining environment description information and natural language commands, so as to enable the robot to respond to the high-level short natural language command input by the user, and then can efficiently plan and execute tasks in a dynamic and complex environment. In addition, the system performs physical constraint verification on the generated atomic instructions to ensure that these instructions are in the atomic instruction library of the robot and also meet the logic of the current task, which can avoid the robot from performing conflicts and incorrect operations, thereby improving the executability and logical correctness of the robot for these instructions, and further improving the success rate of the robot in executing instructions to achieve the task goal. That is, the embodiments of the present application can improve the flexibility and reliability of the system in understanding and executing behavior instructions, so as to enable the robot to efficiently perform corresponding operation tasks in a complex and dynamic environment.

[0094] In some embodiments, the system can control the robot to obtain environment perception information by sensing the surrounding environment through a sensor, which can include visual input information (such as image data and / or video data, etc.) of the surrounding environment perceived by the robot through a camera, and voice input information (such as environmental noise data and / or user voice data, etc.) of the surrounding environment perceived by the robot through a microphone. In this regard, the system can introduce a multi-modal large model to process the environment perception information of the robot, so as to generate accurate and comprehensive environment description information of the environment where the robot is located. Here, the multi-modal large model is an artificial intelligence model that can process multiple input modes (such as vision and language) at the same time.

[0095] Please refer to Figure 2 , Figure 2 to Figure 1The detailed procedure of step S101 is shown in the flowchart.

[0096] As shown in some embodiments, the step S101 of processing the environment perception information of the robot to generate the environment description information of the environment where the robot is located can include steps S201 and S202 as shown below. Figure 2

[0097] Step S201: extracting features of the visual input information through a preset multi-modal large model to obtain environment visual features corresponding to the visual input information, and extracting features of the speech input information through the multi-modal large model to obtain environment language features corresponding to the speech input information.

[0098] It should be noted that the preset multi-modal large model can be an open source MINI-GEMINI multi-modal large model. It should be understood that in other different feasible embodiments, the system can of course use other multi-modal large models other than the MINI-GEMINI multi-modal large model to process the environment perception information of the robot based on different design needs of actual applications. The behavior instruction generation method provided in the embodiments of the present application is not limited to the specific type of multi-modal large model.

[0099] In the case where the environment perception information of the robot includes visual input information and speech input information, the system inputs the visual input information into the MINI-GEMINI multi-modal large model to extract features of the visual input information through the MINI-GEMINI multi-modal large model, thereby obtaining environment visual features corresponding to the visual input information. Moreover, the system also inputs the speech input information into the MINI-GEMINI multi-modal large model to extract features of the speech input information through the MINI-GEMINI multi-modal large model, thereby obtaining environment language features corresponding to the speech input information.

[0100] Step S202: fusing the environment visual features and the environment language features through the multi-modal large model to generate the environment description information of the environment where the robot is located.

[0101] ​After obtaining the environment visual features corresponding to the visual input information and the environment language features corresponding to the speech input information through the MINI-GEMINI multi-modal large model, the system further performs feature fusion processing on the environment visual features and the environment language features through the MINI-GEMINI multi-modal large model, thereby generating accurate and comprehensive environment description information of the environment in which the robot is located. Among them, the environment description information obtained by the system through the MINI-GEMINI multi-modal large model fusing the environment visual features and the environment language features can be environment state description information in the form of text.

[0102] In this embodiment, the system introduces an open-source MINI-GEMINI multi-modal large model to fuse the visual and language features in the environment perception information, thereby generating accurate and comprehensive environment description information for the environment in which the robot is located. In this way, the robot can improve the perception accuracy of the surrounding environment and the task understanding ability for high-level short instructions when responding to user input. Furthermore, the system combines the environment description information generated using the MINI-GEMINI multi-modal large model with the high-level short instruction input by the user to obtain a first atomic instruction combination for the robot through robot task planning, and combines the physical constraint verification of the multiple atomic instructions in the first atomic instruction combination, thereby generating accurate and executable navigation and operation instructions for the robot, and further improving the ability of the system to control the robot to perform tasks in a complex and dynamic environment.

[0103] In addition, in some embodiments, the system can further fine-tune and optimize the MINI-GEMINI multi-modal large model, so that the MINI-GEMINI multi-modal large model adapts to the specific task requirements of the robot, thereby further improving the accuracy of environment perception and the accuracy of subsequent generation of robot atomic instructions.

[0104] In some embodiments, the system can also use a reinforcement learning algorithm to process the environment perception information of the robot, thereby generating environment description information of the environment in which the robot is located. Therefore, with the help of reinforcement learning, the characteristics of the strategy can be gradually optimized in a dynamic and complex environment, and the perception accuracy of the environment around the robot can be improved. For example, the system continuously learns the optimal strategy by interacting with the environment through the reinforcement learning model, thereby generating detailed environment description information based on the environment perception information of the robot. Among them, the system can pre-train the reinforcement learning model using Deep Q-Learning (DQN) or Proximal Policy Optimization (PPO) and other reinforcement learning algorithms, and during the training of the reinforcement learning model, the system can optimize its action strategy through a reward mechanism.

[0105] In addition, the system can use a high-precision environment simulator to provide a training environment during training of the reinforcement learning model, so that the reinforcement learning model can be trained in a virtual environment and accumulate experience.

[0106] In some embodiments, the system can use a large language model (LLM) to generate a first combination of atomic instructions for the robot based on the environment description information and a high-level short instruction of the user.

[0107] Based on this, the above step S102 of robot task planning based on the environment description information and the natural language command to obtain the first combination of atomic instructions for the robot can include the following steps:

[0108] The environment description information and the natural language command are input into a preset large language model to perform robot task planning based on the environment description information and the natural language command by the large language model to obtain the first combination of atomic instructions for the robot.

[0109] It should be noted that the preset large language model is the above-mentioned large language model LLM, which is also called a large language model LLM or LLM large model. For unified expression, the LLM large model is used in the following embodiment description.

[0110] After the system generates the environment description information of the environment in which the robot is located, the environment description information and the natural language command input by the user for the robot are input into the LLM large model together, so that the LLM large model performs robot task planning based on the environment description information and the natural language command to generate a step-by-step instruction executable by the robot. The step-by-step instruction is the instruction (multiple atomic instructions) that the robot needs to execute to respond to the natural language command input by the user. The system takes the step-by-step instruction as the first combination of atomic instructions for the robot.

[0111] In some embodiments, the system can pre-attach a knowledge base to the LLM large model, that is, the system associates the LLM large model with a knowledge base provided based on a big data platform, so that the LLM large model can query the knowledge base for the required analysis and judgment. In this way, when the system performs robot task planning based on the environment description information and the natural language command by the LLM large model, the LLM large model can query the attached knowledge base as needed to generate a step-by-step instruction executable by the robot.

[0112] In some embodiments, in the case where the system uses the LLM large model to conduct robot task planning in combination with the environment description information and the natural language command to generate the first atomic instruction combination, if the system finds that the plurality of atomic instructions in the first atomic instruction combination all fail the physical constraint verification in a subsequent physical constraint verification of the plurality of atomic instructions in the first atomic instruction combination, the system can also adjust the LLM large model to re-conduct robot planning to generate new instructions.

[0113] Please refer to Figure 3 , Figure 3 The step flowchart of the behavior instruction generation method provided by the embodiments of the present application in other embodiments.

[0114] As Figure 3 shown, in some embodiments, in the case where the system uses the LLM large model to conduct robot task planning in combination with the environment description information and the natural language command to generate the first atomic instruction combination, the behavior instruction generation method provided by the embodiments of the present application can also include the following steps S301 to S304.

[0115] Step S301: adjusting the large language model in the case where the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, and / or the plurality of atomic instructions fail the task logic verification.

[0116] After the system uses the LLM large model to generate the first atomic instruction combination in combination with the environment description information and the natural language command, the system matches the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and conducts task logic verification on the plurality of atomic instructions, thereby obtaining a corresponding processing result. If the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, or the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination fail the task logic verification, or the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination neither belong to the atomic instruction library nor pass the task logic verification, the system confirms that the plurality of atomic instructions currently generated by the LLM large model fail the physical constraint verification, and thus adjusts the LLM large model.

[0117] It should be noted that the adjustment of the LLM large model by the system can be that the system analyzes the plurality of atomic instructions that fail the physical constraint verification, thereby determining the conflicts existing in the execution of the plurality of atomic instructions by the robot, and then feeds the conflicts back to the LLM large model so that the LLM large model avoids these conflicts when conducting new robot task planning.

[0118] Step S302: input the environment description information and the natural language command into the adjusted large language model, to obtain a second atomic instruction combination of the robot by robot task planning based on the environment description information and the natural language command through the adjusted large language model.

[0119] After the system adjusts the LLM large model, it continues to input the environment description information and the natural language command into the adjusted LLM large model, so as to recombine the environment description information and the natural language command through the adjusted LLM large model for robot task planning, thereby generating a second atomic instruction combination of the robot.

[0120] It should be noted that, since the system feeds back the conflict of the atomic instruction that has not passed the physical constraint verification when adjusting the LLM large model, the adjusted LLM large model will avoid the conflict fed back by the system when recombining the environment description information and the natural language command for robot task planning, thereby generating more accurate step-by-step instructions. The system further takes the step-by-step instructions as the second atomic instruction combination of the robot.

[0121] Step S303: match the plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the plurality of atomic instructions.

[0122] After the system obtains the second atomic instruction combination of the robot, it also matches the plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot which has been set in advance, to confirm whether the plurality of atomic instructions are basic instructions existing in the atomic instruction library. Moreover, the system also performs task logic verification on the plurality of atomic instructions in the second atomic instruction combination, to confirm whether each of the plurality of atomic instructions conforms to the current task logic of the robot. After matching the plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions, the system obtains corresponding processing results.

[0123] Step S304: in the case that the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification, taking the fourth target atomic instruction as the behavior instruction of the robot.

[0124] If the processing result indicates that at least one atomic instruction (the fourth target atomic instruction) in the second atomic instruction combination is a basic instruction in the atomic instruction library and the at least one atomic instruction (the fourth target atomic instruction) also passes the task logic verification performed by the system, in this case, the system takes the at least one atomic instruction (the fourth target atomic instruction) as one of the behavior instructions to be executed by the robot to execute the task in response to the natural language command.

[0125] In this embodiment, the system inputs the environment description information and the natural language command input by the human user into the LLM large model together for robot task planning, so as to use the LLM large model to disassemble the command in combination with the current state information, and query the externally plugged knowledge base during the reasoning process according to the situation, thereby outputting a combination of a plurality of atomic instructions. In this way, the LLM large model generates preliminary instructions according to the environment description and the user command, and then performs physical constraint verification on the instructions to ensure that they are in the robot atomic instruction library and meet the logic of the current task, thereby improving the executability of the robot for these instructions, and improving the success rate of the robot achieving the task purpose based on executing these instructions.

[0126] In some embodiments, when the system performs physical constraint verification on the plurality of atomic instructions, after confirming that a certain target sub-instruction belongs to the atomic instruction library of the robot, the system further performs task logic verification on the target sub-instruction.

[0127] Based on this, the above-mentioned "matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot" in step S103 can include the following steps:

[0128] Matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot.

[0129] When the system performs physical constraint verification on the plurality of atomic instructions in the first atomic instruction combination, the system first matches the plurality of atomic instructions one by one with the atomic instructions (basic instructions) in the atomic instruction library of the robot.

[0130] Further, the above-mentioned "task logic verification on the plurality of atomic instructions" in step S103 includes the following steps:

[0131] In a case that at least one atomic instruction in the plurality of atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and task logic verification is performed on the at least one atomic instruction.

[0132] In a case that at least one atomic instruction in the plurality of atomic instructions matches an atomic instruction in the atomic instruction library (for example, consistent or representing consistent robot actions), the system confirms that the at least one atomic instruction is an instruction belonging to the atomic instruction library, that is, the at least one atomic instruction is a basic instruction existing in the atomic instruction library. In this way, the system further performs task logic verification processing on the at least one atomic instruction, so as to confirm whether the at least one atomic instruction also conforms to the current task logic of the robot.

[0133] In some embodiments, after matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library, if one or more first target atomic instructions in the plurality of atomic instructions do not match any atomic instruction in the atomic instruction library, the system decomposes the first target atomic instruction, and then continues to match the plurality of target sub-instructions obtained after decomposition with the atomic instructions in the atomic instruction library, and in a case that the target sub-instruction matches an atomic instruction in the atomic instruction library, it is confirmed that the first target atomic instruction also belongs to the atomic instruction in the atomic instruction library.

[0134] Please refer to Figure 4 , Figure 4 The step flowchart of the behavior instruction generation method provided by the embodiments of the present application in still some embodiments.

[0135] As Figure 4 shown, in some embodiments, after the above-mentioned "matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot", the behavior instruction generation method provided by the embodiments of the present application further includes the steps S401 to S403 as shown below.

[0136] Step S401: in a case that a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, decomposing the first target atomic instruction into a plurality of target sub-instructions.

[0137] The system, when performing the physical constraint check on the plurality of atomic instructions in the first atomic instruction combination, after matching the plurality of atomic instructions with the atomic instructions in the atomic instruction library of the robot, if a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, the system performs decomposition processing on the first target atomic instruction, thereby obtaining a plurality of target sub-instructions of the first target atomic instruction.

[0138] Step S402: Matching the plurality of target sub-instructions with the atomic instructions in the atomic instruction library.

[0139] After the system decomposes the first target atomic instruction to obtain the plurality of target sub-instructions, the system further matches the plurality of target sub-instructions one by one with the atomic instructions in the atomic instruction library, thereby confirming whether the plurality of target sub-instructions are basic instructions existing in the atomic instruction library.

[0140] Step S403: In the case where the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, confirming that the first target atomic instruction belongs to the atomic instruction library.

[0141] After the system matches the plurality of target sub-instructions with the atomic instructions in the atomic instruction library, thereby confirming that the plurality of target sub-instructions are basic instructions existing in the atomic instruction library, the system confirms that the first target atomic instruction in the first atomic instruction combination also belongs to the atomic instruction library of the robot. In this way, the system can further perform task logic verification on the first target atomic instruction.

[0142] It should be noted that after the system confirms that the first target atomic instruction passes the task logic verification, thereby taking the first target atomic instruction as the behavior instruction of the robot, the robot can sequentially execute the plurality of target sub-instructions obtained by decomposing the first target atomic instruction when executing the first target atomic instruction.

[0143] The physical constraint check performed by the system on the plurality of atomic instructions in the first atomic instruction combination includes atomic instruction verification and task logic verification. The atomic instruction verification refers to matching the atomic instructions with the atomic instruction library, thereby ensuring that the atomic instructions generated by the system are in the atomic instruction library of the robot, and verifying and decomposing complex instructions (the first target atomic instruction) into atomic instructions. In addition, the task logic verification uses mathematical optimization methods (such as linear programming or integer programming optimization methods) to ensure the logical correctness of the instructions generated by the system, to avoid robot conflicts and errors. The system can significantly improve the reliability of instruction execution by performing task logic verification on a plurality of sub-instructions.

[0144] Please refer to Figure 5 , Figure 5 for Figure 1The detailed procedure of step S103 is shown in the flowchart.

[0145] As shown in some embodiments, the step S103 of "performing task logic verification on the plurality of atomic instructions" can include steps S501-S503 as shown below. Figure 5

[0146] Step S501: generating an instruction sequence based on the plurality of atomic instructions.

[0147] When the system performs task logic verification on the plurality of atomic instructions, it first generates an instruction sequence based on the plurality of atomic instructions.

[0148] Step S502: verifying the holding logic state of the instructions in the instruction sequence based on a preset holding logic constraint condition to obtain a verification result.

[0149] It should be noted that the preset holding logic condition can be: if the robot holds the object when executing the i-th instruction, then the robot must also hold the object when executing the i+1-th instruction.

[0150] After the system generates the instruction sequence based on the plurality of atomic instructions, it first verifies the holding logic state of the instructions in the instruction sequence one by one based on the holding logic condition to obtain a verification result. The verification result is used to represent whether the holding logic state of the instructions in the instruction sequence satisfies the holding logic constraint, i.e., the verification result specifically indicates whether the holding logic state of the instructions in the instruction sequence satisfies that the robot holds the object when executing the i-th instruction in the instruction sequence, and the robot also holds the object when executing the i+1-th instruction.

[0151] Step S503: in the case where the verification result indicates that the instructions in the instruction sequence satisfy the holding logic constraint condition, verifying the position logic state of the instructions in the instruction sequence based on a preset position logic constraint condition.

[0152] It should be noted that the preset position logic constraint condition can be: the robot must be located at a specific position (such as "in front of the refrigerator") before executing certain specific operations (such as "opening the refrigerator").

[0153] ​After obtaining the verification result of the possession logic state of the instruction in the instruction sequence, if the verification result indicates that the instruction in the instruction sequence satisfies the possession logic constraint condition, that is, the verification result indicates that the possession logic state of the instruction in the instruction sequence satisfies the case that the robot holds the object when executing the i-th instruction in the instruction sequence, the robot also holds the object when executing the i+1-th instruction. In this case, the system further verifies the position logic state of the instruction in the instruction sequence one by one based on the position logic constraint condition.

[0154] Exemplarily, the system defines an instruction sequence {a1, a2, …, an} based on a plurality of atomic instructions, wherein each atomic instruction ai has a corresponding logic state si, and the logic state si can include and wherein, represents whether the robot holds a certain object when executing the instruction ai, and represents the position of the robot when executing the instruction ai.

[0155] After that, the system first verifies the possession logic state of the atomic instruction ai in the instruction sequence {a1, a2, …, an}. That is, if the robot holds the object when executing the instruction ai, the robot cannot execute the “pick up the object” instruction again before executing the “put down the object” instruction.

[0156] The formula: describes that in a series of execution instructions, when the robot holds an object at a certain time step (or instruction), it must continue to hold the state at the next time step (or instruction) until the “put down the object” instruction is executed. This is a state holding constraint that ensures that the robot follows the correct logical order when completing the task. Among them, if the robot holds the object when executing the i-th instruction (i.e. is true), then the robot must also hold the object when executing the i+1-th instruction (i.e. is also true).

[0157] In addition, the system also verifies the position logic state of the atomic instruction ai in the instruction sequence {a1, a2, …, an}. That is, The robot must be located at a specific position (such as “in front of the refrigerator”) before executing certain specific operations (such as “open the refrigerator”).

[0158] In this embodiment, the system performs physical constraint verification on the first atomic instruction combination of the robot generated by combining the environment description information and the natural language command, and the physical constraint verification includes atomic instruction verification and task logic verification. Through these verifications, it can be ensured that the multiple atomic instructions in the first atomic instruction combination are both in the atomic instruction library of the robot and conform to the current task logic of the robot, thereby effectively avoiding conflicts and incorrect operations when the robot executes instructions.

[0159] In addition, since the system can fuse visual and language information by introducing the MINI-GEMINI multi-modal large model to generate more accurate and comprehensive environment description information, in this way, the system combines the multi-modal large model and the physical constraint verification, first generates accurate and comprehensive environment description information based on the multi-modal large model, then generates the first atomic instruction combination by combining the environment description information and the natural language command, and then performs physical constraint verification on the multiple atomic instructions in the first atomic instruction combination. In this way, the adaptability, flexibility and reliability of robot task planning and execution in dynamic and complex scenes can be significantly improved.

[0160] In addition to performing physical constraint verification on the multiple atomic instructions in the first atomic instruction combination, the system can also optimize the atomic instructions in combination with the physical constraint verification.

[0161] For the behavior instruction generation method provided by the embodiments of the present application, the steps of the flowchart are as follows. Figure 6 , Figure 6 For the behavior instruction generation method provided by the embodiments of the present application, the steps of the flowchart are as follows.

[0162] As shown in Figure 6 , in some embodiments, the behavior instruction generation method provided by the embodiments of the present application can further include steps S601 and S602 as shown below.

[0163] Step S601: If the processing result indicates that the multiple atomic instructions belong to the atomic instruction library, and a second target atomic instruction in the multiple atomic instructions passes the task logic verification, and a third target atomic instruction in the multiple atomic instructions does not pass the task logic verification, optimizing the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein the number of the multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions.

[0164] If the processing result indicates that the plurality of atomic instructions are all basic instructions (belonging to the atomic instruction library) in the atomic instruction library of the robot, but the second target atomic instruction in the plurality of atomic instructions passes the task logic verification, and the remaining third target atomic instructions in the plurality of atomic instructions do not pass the task logic verification, in this case, the system optimizes the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction, and then performs logic task verification on the optimized sub-instruction. Among them, the number of the plurality of atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions.

[0165] It should be noted that the optimization of the system for the third target atomic instruction can be to verify and adjust the third target atomic instruction by the above-mentioned data optimization method. For example, in the case that the third target atomic instruction does not satisfy the above-mentioned holding logic constraint, the system adjusts the holding logic state of the third target atomic instruction based on the holding logic constraint condition, so as to obtain an optimized sub-instruction, and the holding logic state of the optimized sub-instruction satisfies the holding logic constraint. Or, in the case that the third target atomic instruction does not satisfy the above-mentioned position logic constraint, the system adjusts the position logic state of the third target atomic instruction based on the position logic constraint condition, so as to obtain an optimized sub-instruction, and the position logic state of the optimized sub-instruction satisfies the position logic constraint.

[0166] Step S602: taking the second target atomic instruction and the optimized sub-instruction as the behavior instruction of the robot.

[0167] After the system optimizes the third target atomic instruction to obtain an optimized sub-instruction, in the case that the optimized sub-instruction further passes the task logic verification, the second target atomic instruction which has previously passed the task logic verification and the optimized sub-instruction are taken together as one of the behavior instructions to be executed by the robot in response to the natural language command of the user.

[0168] In this embodiment, the system optimizes the third target atomic instruction which does not pass the task logic verification by using a mathematical optimization method, so as to ensure that it is in the atomic instruction library of the robot and conforms to the logic of the current task (ensuring the logical correctness of the instruction), so as to avoid conflicts and incorrect operations when the robot executes the instruction, thereby improving the executability of the instruction and the success rate of the task.

[0169] In some embodiments, the system can also ensure the safety and logical correctness of the system behavior in combination with the rules and constraints provided by the expert system. For example, the system can pre-design a set of expert system rules to define the task logic and operation constraints when the robot executes the instructions (such as prohibiting dropping an object without holding it), and then after the system generates the first atomic instruction combination in combination with the environment description information and the natural language command, the system can use the expert system rules to perform physical constraint verification on the atomic instructions in the first atomic instruction combination, so as to ensure the logical correctness of the robot operation corresponding to the instructions by means of the explicit rules and constraints provided by the expert system rules.

[0170] Next, the generation method of the behavior instruction provided by the embodiments of the present application involves the embodied intelligent navigation algorithm framework combining the multi-modal large model and the physical constraint verification mechanism in some embodiments.

[0171] The embodied intelligent navigation algorithm framework includes a multi-modal large model, an LLM large model, an atomic instruction verification submodule, a task logic verification submodule, and a robot execution module. Through the close linkage of these modules, the system ensures that accurate and executable navigation and operation instructions are generated in the case that the user inputs high-level short instructions to the robot.

[0172] Among them, the multi-modal large model mainly takes the visual input of the environment where the robot is located and the language instruction as input, generates detailed environment description information by fusing visual and language features, and outputs. The LLM large model takes the environment description information and the task instruction (natural language command) as input, and generates the step-by-step instructions (the first atomic instruction combination) executable by the robot according to the environment description and the task instruction and outputs. In addition, the LLM large model can query the external knowledge base as needed. The physical constraint module mainly includes: the atomic instruction verification submodule and the task logic verification submodule. The atomic instruction verification submodule takes the step-by-step instructions output by the LLM large model as input, verifies whether these instructions exist in the atomic instruction library of the robot, and performs necessary instruction decomposition, and then outputs the execution instructions that exist in the post-atomic instruction library after verification. The task logic verification submodule takes the verified execution instructions output by the atomic instruction verification submodule and the preset task logic rules as input, verifies and adjusts these instructions by mathematical optimization method to ensure that the instructions comply with the task logic and have no conflicts, and finally outputs the execution instructions (behavior instructions) that pass the verification and / or optimization. The robot execution module takes the execution instructions output by the task logic verification submodule as input, and outputs the actual robot actions to control the behavior instructions executed by the robot to complete the specified task.

[0173] In some embodiments, the system can implement the linkage between the various functional modules according to the following processes 1 to 5 using the embodied intelligent navigation algorithm framework described above.

[0174] 1. Initialization: The system generates environment description information and preliminary instructions using a multi-modal large model when the robot receives high-level short instructions input by the user. 2. Environment perception: The system extracts visual and language features through a multi-modal large model to generate environment description information. 3. Preliminary instruction generation: The system generates preliminary execution instructions through an LLM large model combined with environment description information and task instructions. The LLM large model can query an external knowledge base as needed. 4. Physical constraint verification and optimization: The system ensures that the instructions are in the atomic instruction library through an atomic instruction verification submodule and performs instruction decomposition. Furthermore, the system also has a task logic verification submodule that ensures the logical correctness of the instructions through mathematical formulas and optimization algorithms. 5. Execution and update: The system performs operations according to the optimized instructions through a robot execution module to complete the task.

[0175] Exemplarily, please refer to Figure 7 , Figure 7 The overall architecture schematic diagram involved in the behavior instruction generation method provided by the embodiments of the present application in some embodiments. As shown in Figure 7 , the system can complete the perception of the environment where the robot is located through a multi-modal large model, and summarize the environment description information in text form. Then, the system inputs the environment description information and natural language commands into an LLM large model for robot task planning, combining the environment description information and natural language commands. The LLM large model disassembles the commands in combination with the current state information indicated by the environment description information, and introduces an external knowledge base during the reasoning process according to the situation. Finally, the LLM large model outputs a first atomic instruction combination of several atomic instructions to complete the given task. After that, before issuing the multiple atomic instructions in the first atomic instruction combination to the execution unit of the robot, the system first verifies the multiple atomic instructions from two angles of atomic instruction verification and task logic verification through a physical constraint module. Only the behavior instructions that pass the verification will be issued to the action execution unit of the robot to control the robot to perform the execution of the instructions.

[0176] Furthermore, if the multiple atomic instructions in the first atomic instruction combination do not pass the physical constraint verification (do not satisfy the physical constraints represented from two angles of atomic instruction verification and / or task logic verification), the system will also feed back the conflicts corresponding to the instructions to the LLM large model for adjustment. After that, the system can continue to perform robot task planning using the adjusted LLM large model.

[0177] Next, the complete embodiment of the behavior instruction generation method provided by the embodiments of the present application is proposed.

[0178] Please refer to Figure 8 , Figure 8 The application process schematic diagram involved in the behavior instruction generation method provided by the embodiments of the present application.

[0179] As shown in Figure 8 When a user inputs a high-level short natural language command to a robot, the system processes the environment perception information (visual input and language input) of the environment where the robot is located through a multi-modal large model to fuse visual and language features to generate detailed environment description information. Then, the system uses a LLM large model to combine the environment description information and the natural language command for robot task planning, thereby generating the robot's preliminary execution instructions (first atomic instruction combination). Then, the system inputs the instructions to the physical constraint module for physical constraint checking from two angles of atomic instruction verification and task logic verification. Finally, in the case that the behavior instructions pass the atomic instruction verification and also pass the task logic verification, the system executes the behavior instructions through the robot execution module. In addition, in the case that the instructions do not pass the atomic instruction verification and / or the task logic verification, the system feeds back the corresponding conflicts of the instructions to the LLM large model for adjustment. Then, the system can use the adjusted LLM large model to re-infer to continue robot task planning.

[0180] Please refer to Figure 9 The embodiment of the present application also provides a behavior instruction generation device, which can implement the above behavior instruction generation method. The device comprises an environment description module 901, a task planning module 902, an instruction verification module 903, and an instruction generation module 904. Wherein,

[0181] The environment description module 901 is configured to process the environment perception information of the robot to generate the environment description information of the environment where the robot is located.

[0182] The task planning module 902 is configured to perform robot task planning based on the environment description information and the natural language command to obtain the first atomic instruction combination of the robot; wherein the first atomic instruction combination comprises a plurality of atomic instructions.

[0183] The instruction verification module 903 is configured to match the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the plurality of atomic instructions to obtain a processing result.

[0184] The instruction generation module 904 is configured to, in the case that the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, take the at least one atomic instruction as the behavior instruction of the robot.

[0185] In some embodiments, the instruction verification module 903 is further configured to match the plurality of atomic instructions in the first atomic instruction combination with atomic instructions in an atomic instruction library of the robot; and in a case where at least one atomic instruction in the plurality of atomic instructions matches an atomic instruction in the atomic instruction library, confirm that the at least one atomic instruction belongs to the atomic instruction library and perform task logic verification on the at least one atomic instruction.

[0186] In some embodiments, the instruction verification module 903 is further configured to, in a case where a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, decompose the first target atomic instruction into a plurality of target sub-instructions; match the plurality of target sub-instructions with atomic instructions in the atomic instruction library; and in a case where the plurality of target sub-instructions match atomic instructions in the atomic instruction library, confirm that the first target atomic instruction belongs to the atomic instruction library.

[0187] In some embodiments, the instruction verification module 903 is further configured to generate an instruction sequence based on the plurality of atomic instructions; perform verification on a possession logic state of an instruction in the instruction sequence based on a preset possession logic constraint condition to obtain a verification result; and in a case where the verification result indicates that the instruction in the instruction sequence satisfies the possession logic constraint condition, perform verification on a position logic state of the instruction in the instruction sequence based on a preset position logic constraint condition.

[0188] In some embodiments, the instruction verification module 903 is further configured to, in a case where the processing result indicates that the plurality of atomic instructions belong to the atomic instruction library, and a second target atomic instruction in the plurality of atomic instructions passes task logic verification and a third target atomic instruction in the plurality of atomic instructions does not pass task logic verification, optimize the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein a number of the plurality of atomic instructions is equal to a sum of a number of the second target atomic instructions and a number of the third target atomic instructions; and use the second target atomic instructions and the optimized sub-instruction as behavior instructions of the robot.

[0189] In some embodiments, the task planning module 902 is further configured to input the environment description information and the natural language command into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command through the large language model, and obtain a first atomic instruction combination for the robot; if the processing result indicates that multiple atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, and / or the multiple atomic instructions fail to pass the task logic verification, adjust the large language model; input the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command through the adjusted large language model, and obtain a second atomic instruction combination for the robot;

[0190] The instruction verification module 903 is further configured to match the plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the plurality of atomic instructions;

[0191] The instruction generation module 904 is also used to use the fourth target atomic instruction as the behavior instruction of the robot when the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification.

[0192] In some embodiments, the environmental perception information includes visual input information and voice input information. The environmental description module 901 is further configured to extract features from the visual input information using a preset multimodal macro model to obtain environmental visual features corresponding to the visual input information, extract features from the voice input information using the multimodal macro model to obtain environmental language features corresponding to the voice input information, and fuse the environmental visual features and the environmental language features using the multimodal macro model to generate environmental description information of the robot's environment.

[0193] The specific implementation of the behavior instruction generation device provided in the embodiment of the present application is basically the same as the specific implementation of the behavior instruction generation method described above, and will not be repeated here.

[0194] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for generating behavioral instructions. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0195] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0196] The processor 1001 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided by the embodiments of the present application.

[0197] The memory 1002 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1002 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, relevant program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the behavior instruction generation method of the embodiments of the present application.

[0198] The input / output interface 1003 is configured to implement information input and output.

[0199] The communication interface 1004 is configured to implement communication interaction between the device and other devices. The communication can be implemented by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0200] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device.

[0201] The processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are connected to each other by the bus 1005 to realize communication connection between them in the device.

[0202] The embodiments of the present application further provide a robot, and the robot is configured with an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and the processor implements the behavior instruction generation method when executing the computer program.

[0203] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the behavior instruction generation method.

[0204] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0205] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements steps substantially the same as those of the above-mentioned embodiment of the method for generating behavior instructions, and thus will not be described herein again.

[0206] The embodiments described in the present application are for more clearly illustrating the technical solutions of the present application, and do not constitute a limitation on the technical solutions provided by the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the present application are also applicable to similar technical problems.

[0207] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps or different steps.

[0208] The above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0209] Those skilled in the art can understand that all or some of the steps in the above-disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0210] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order unless otherwise stated below.

[0211] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0212] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0213] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0214] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0215] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0216] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method of generating a behavior instruction, characterized by, The method comprises: processing environment perception information of the robot to generate environment description information of an environment in which the robot is located; based on the environment description information and the natural language command, robot task planning is performed to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination comprises a plurality of atomic instructions; matching the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions to obtain a processing result; wherein, in the case that a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, the first target atomic instruction is decomposed into a plurality of target sub-instructions, the plurality of target sub-instructions are matched with the atomic instructions in the atomic instruction library, and in the case that the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, it is confirmed that the first target atomic instruction belongs to the atomic instruction library; in the case that the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is taken as a behavior instruction of the robot; the at least one atomic instruction includes the first target atomic instruction.

2. The method of claim 1, wherein, The matching of the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot comprises: matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot; The task logic verification on the plurality of atomic instructions comprises: in the case that at least one atomic instruction in the plurality of atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and the at least one atomic instruction is subjected to task logic verification.

3. The method of claim 1, wherein, The task logic verification on the plurality of atomic instructions comprises: generating an instruction sequence based on the plurality of atomic instructions; verifying the holding logic state of the instructions in the instruction sequence based on a preset holding logic constraint condition to obtain a verification result; in the case that the verification result indicates that the instructions in the instruction sequence satisfy the holding logic constraint condition, verifying the position logic state of the instructions in the instruction sequence based on a preset position logic constraint condition.

4. The method of claim 1, wherein, The method further comprises: in the case that the processing result indicates that the plurality of atomic instructions belong to the atomic instruction library, a second target atomic instruction in the plurality of atomic instructions passes the task logic verification, and a third target atomic instruction in the plurality of atomic instructions does not pass the task logic verification, optimizing the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein the number of the plurality of atomic instructions is equal to the sum of the number of the second target atomic instruction and the number of the third target atomic instruction; taking the second target atomic instruction and the optimized sub-instruction as the behavior instruction of the robot.

5. The method of claim 1, wherein, The robot task planning based on the environment description information and the natural language command obtains a first atomic instruction combination of the robot, and the robot task planning based on the environment description information and the natural language command comprises: inputting the environment description information and the natural language command into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command by the large language model, and obtain a first atomic instruction combination of the robot; The method further comprises: adjusting the large language model in a case where the processing result indicates that a plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, and / or the plurality of atomic instructions do not pass the task logic verification; inputting the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command by the adjusted large language model, and obtain a second atomic instruction combination of the robot; matching a plurality of atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions; in a case where a fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification, taking the fourth target atomic instruction as a behavior instruction of the robot.

6. The method according to any one of claims 1 to 5, characterized in that, The environment perception information comprises visual input information and voice input information. The environment perception information of the robot is processed to generate environment description information of an environment in which the robot is located, and the environment description information of the environment in which the robot is located comprises: extracting features of the visual input information by a preset multi-modal large model to obtain environment visual features corresponding to the visual input information, and extracting features of the voice input information by the multi-modal large model to obtain environment language features corresponding to the voice input information; fusing the environment visual features and the environment language features by the multi-modal large model to generate the environment description information of the environment in which the robot is located.

7. An action instruction generating apparatus characterized by comprising: The device comprises: an environment description module configured to process environment perception information of a robot to generate environment description information of an environment in which the robot is located; a task planning module configured to perform robot task planning based on the environment description information and a natural language command to obtain a first atomic instruction combination of the robot, wherein the first atomic instruction combination comprises a plurality of atomic instructions. The instruction verification module is configured to match the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the plurality of atomic instructions to obtain a processing result; in a case where a first target atomic instruction in the plurality of atomic instructions does not match any atomic instruction in the atomic instruction library, the first target atomic instruction is decomposed into a plurality of target sub-instructions, the plurality of target sub-instructions are matched with atomic instructions in the atomic instruction library, and in a case where the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, it is confirmed that the first target atomic instruction belongs to the atomic instruction library; The instruction generation module is configured to, in a case where the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, take the at least one atomic instruction as the behavior instruction of the robot; the at least one atomic instruction includes the first target atomic instruction.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the behavior instruction generation method of any one of claims 1 to 6 when executing the computer program.

9. A robot, characterized in that The robot includes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the behavior instruction generation method of any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the behavior instruction generation method of any one of claims 1 to 6.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the behavior instruction generation method of any one of claims 1 to 6.

Images