Behavior instruction generation method and device, equipment, robot, medium and product

Abstract

The invention provides a behavior instruction generation method and device, equipment, a robot, a medium and a product, and belongs to the technical field of robots, and the method comprises the steps: processing the environment perception information of the robot to generate the environment description information of the environment where the robot is located; robot task planning is carried out based on the environment description information and the natural language command, and a first atomic instruction combination of the robot is obtained; wherein the first atomic instruction combination comprises a plurality of atomic instructions; matching the plurality of atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions to obtain a processing result; and under the condition that the processing result indicates that at least one atomic instruction belongs to the atomic instruction library and the at least one atomic instruction passes task logic verification, taking the at least one atomic instruction as a behavior instruction of the robot. According to the invention, the flexibility and reliability of understanding and executing the behavior instruction by the intelligent system can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a method, device, equipment, robot, medium and product for generating behavioral instructions. Background Art

[0002] With the continuous development and progress of artificial intelligence and robotics, the application prospects of artificial intelligence systems with physical bodies (embodied intelligent systems) in daily life, industrial automation, medical assistance and other fields are becoming increasingly broad. In related technologies, embodied intelligent systems mainly rely on detailed behavioral instructions to gradually perform corresponding operations, but the understanding and execution of short and high-level behavioral instructions in complex and dynamic environments (for example, the behavioral instruction "go to the kitchen to get milk" used by people in daily life) will appear to be inflexible and inefficient, and there may even be a situation where they cannot respond to the behavioral instructions output by people. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, device, equipment, robot, medium and product for generating behavioral instructions, aiming to improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavioral instructions, so as to efficiently perform corresponding operational tasks in complex and dynamic environments.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a behavior instruction, the method comprising:

[0005] Processing the robot's environmental perception information to generate environmental description information of the robot's environment;

[0006] Perform robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes multiple atomic instructions;

[0007] Matching multiple atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions to obtain a processing result;

[0008] When the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is used as the behavior instruction of the robot.

[0009] In some embodiments, matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot includes:

[0010] Matching multiple atomic instructions in the first atomic instruction combination with atomic instructions in an atomic instruction library of the robot;

[0011] The task logic verification of the multiple atomic instructions includes:

[0012] When at least one atomic instruction among the multiple atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and task logic verification is performed on the at least one atomic instruction.

[0013] In some embodiments, after matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot, the method further comprises:

[0014] When a first target atomic instruction among the multiple atomic instructions does not match any atomic instruction in the atomic instruction library, decomposing the first target atomic instruction into multiple target sub-instructions;

[0015] Matching the multiple target sub-instructions with the atomic instructions in the atomic instruction library;

[0016] In a case where the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, it is confirmed that the first target atomic instruction belongs to the atomic instruction library.

[0017] In some embodiments, performing task logic verification on the plurality of atomic instructions includes:

[0018] generating an instruction sequence based on the plurality of atomic instructions;

[0019] Verifying the object logic state of the instructions in the instruction sequence based on the preset object logic constraint condition to obtain a verification result;

[0020] When the verification result indicates that the instructions in the instruction sequence satisfy the object logic constraint condition, the position logic state of the instructions in the instruction sequence is verified based on the preset position logic constraint condition.

[0021] In some embodiments, the method further comprises:

[0022] When the processing result indicates that the multiple atomic instructions belong to the atomic instruction library, and the second target atomic instruction among the multiple atomic instructions passes the task logic verification, and the third target atomic instruction among the multiple atomic instructions fails to pass the task logic verification, optimizing the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein the number of the multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions;

[0023] The second target atomic instruction and the optimized sub-instruction are used as the behavior instructions of the robot.

[0024] In some embodiments, performing robot task planning based on the environment description information and the natural language command to obtain the first atomic instruction combination of the robot includes:

[0025] The environment description information and the natural language command are input into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command through the large language model to obtain a first atomic instruction combination of the robot.

[0026] In some embodiments, the method further comprises:

[0027] When the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library and / or the plurality of atomic instructions fail to pass task logic verification, adjusting the large language model;

[0028] Inputting the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command through the adjusted large language model to obtain a second atomic instruction combination for the robot;

[0029] Matching a plurality of atomic instructions in the second atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions;

[0030] When the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification, the fourth target atomic instruction is used as the behavior instruction of the robot.

[0031] In some embodiments, the environmental perception information includes: visual input information and voice input information;

[0032] The processing of the robot's environmental perception information to generate environmental description information of the robot's environment includes:

[0033] Extracting features from the visual input information using a preset multimodal large model to obtain environmental visual features corresponding to the visual input information, and extracting features from the voice input information using the multimodal large model to obtain environmental language features corresponding to the voice input information;

[0034] The environmental visual features and the environmental language features are fused through the multimodal large model to generate environmental description information of the environment where the robot is located.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a device for generating a behavior instruction, the device comprising:

[0036] An environment description module, used to process the robot's environmental perception information to generate environmental description information of the robot's environment;

[0037] A task planning module, used to perform robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes a plurality of atomic instructions;

[0038] An instruction verification module, used for matching multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions to obtain a processing result;

[0039] An instruction generation module is used to use the at least one atomic instruction as a behavior instruction of the robot when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification.

[0040] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a robot, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0042] To achieve the above objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the sixth aspect of the embodiments of the present application proposes a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method provided in the first aspect above is implemented.

[0044] The behavioral instruction generation method, device, electronic device, robot, computer-readable storage medium and computer program product proposed in the present application process the robot's environmental perception information to generate environmental description information of the robot's environment; perform robot task planning based on the environmental description information and natural language commands to obtain a first atomic instruction combination for the robot; wherein the first atomic instruction combination includes multiple atomic instructions; match the multiple atomic instructions in the first atomic instruction combination with the robot's atomic instruction library, and perform task logic verification on the multiple atomic instructions to obtain a processing result; when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, use the at least one atomic instruction as the behavioral instruction of the robot.

[0045] That is to say, after the embodiment of the present application performs robot task planning based on the environment description information and the natural language commands to the robot and obtains the first atomic instruction combination of the robot, it further performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination to determine whether the multiple atomic instructions belong to sub-instructions in the atomic instruction library of the robot, and determine whether the atomic instructions conform to the task logic. Only when the physical constraint verification is passed to ensure that the atomic instruction is both in the robot's atomic instruction library and conforms to the logic of the current task, the atomic instruction is used as the robot's behavior instruction.

[0046] Thus, when the embodiments of the present application are applied to an embodied intelligent system, the embodied intelligent system can generate atomic instructions by combining environmental description information and natural language commands, thereby being able to respond to high-level, brief natural language commands input by the user, and thus being able to efficiently plan and execute tasks in a dynamic and complex environment. In addition, the embodied intelligent system performs physical constraint checks on the generated atomic instructions to ensure that these instructions are both in the robot's atomic instruction library and conform to the logic of the current task, thereby avoiding robot execution conflicts and erroneous operations, thereby improving the executability of these instructions and improving the success rate of executing instructions to achieve task goals. That is, the embodiments of the present application can improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavioral instructions, thereby being able to efficiently execute corresponding operational tasks in a complex and dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1A schematic diagram of the steps of the method for generating a behavior instruction provided in an embodiment of the present application in some embodiments;

[0048] Figure 2 for Figure 1 Schematic diagram of detailed process flow of step S101;

[0049] Figure 3 A schematic diagram of the steps of the method for generating a behavior instruction provided in an embodiment of the present application in other embodiments;

[0050] Figure 4 A schematic diagram of the steps of the method for generating a behavior instruction provided in an embodiment of the present application in some other embodiments;

[0051] Figure 5 for Figure 1 Schematic diagram of detailed process flow of step S103;

[0052] Figure 6 A schematic diagram of the steps of the method for generating a behavior instruction provided in an embodiment of the present application in some other embodiments;

[0053] Figure 7 A schematic diagram of the overall architecture involved in some embodiments of the method for generating behavioral instructions provided in the embodiments of the present application;

[0054] Figure 8 A schematic diagram of an application flow involved in a method for generating a behavior instruction provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the structure of a device for generating behavioral instructions provided in an embodiment of the present application;

[0056] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, the professional and technical terms involved in the embodiments of the present application are explained.

[0061] Embodied intelligent systems.

[0062] An embodied intelligent system refers to an artificial intelligence system with a physical body. An embodied intelligent system is able to perceive, act, and interact in a real physical environment. Embodied intelligent systems combine technologies from multiple fields such as computer vision, natural language processing, robotics, and artificial intelligence, enabling robots and other intelligent agents to understand and adapt to their physical environment and perform complex tasks. The characteristics of embodied intelligent systems include: embodied intelligent systems are able to perceive information about the surrounding environment through sensors (such as cameras, microphones, etc.), embodied intelligent systems are able to move and manipulate objects in the environment through physical components (such as robotic arms, wheels, etc.) to perform physical actions, and embodied intelligent systems are also able to understand task instructions and perform corresponding tasks in the physical environment.

[0063] Atomic instruction library.

[0064] The atomic instruction library refers to the basic instruction set that the robot can execute. The basic instructions in the atomic instruction library are the smallest unit of robot operation. The basic qualities in the atomic instruction library can be such as "move to a certain position", "grab an object", "turn on a switch", etc. The atomic instruction library can be set up in the form of "predefined instruction set", that is, a set of basic operation instructions are predefined according to the function of the robot to construct the atomic instruction library of the robot.

[0065] Next, the overall concept of the method for generating behavioral instructions provided in the embodiment of the present application is described.

[0066] With the continuous development and progress of artificial intelligence and robotics, the application prospects of artificial intelligence systems with physical bodies (embodied intelligent systems) in daily life, industrial automation, medical assistance and other fields are becoming increasingly broad. In related technologies, embodied intelligent systems mainly rely on detailed behavioral instructions to gradually perform corresponding operations, but the understanding and execution of short and high-level behavioral instructions in complex and dynamic environments (for example, the behavioral instruction "go to the kitchen to get milk" used by people in daily life) will appear to be inflexible and inefficient, and there may even be a situation where they cannot respond to the behavioral instructions output by people.

[0067] To this end, the embodiments of the present application provide a method, device, equipment, robot, medium and product for generating behavioral instructions, aiming to improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavioral instructions, so as to efficiently perform corresponding operational tasks in complex and dynamic environments.

[0068] The embodiment of the present application processes the robot's environmental perception information to generate environmental description information of the robot's environment; performs robot task planning based on the environmental description information and natural language commands to obtain a first atomic instruction combination for the robot; wherein the first atomic instruction combination includes multiple atomic instructions; matches the multiple atomic instructions in the first atomic instruction combination with the robot's atomic instruction library, and performs task logic verification on the multiple atomic instructions to obtain a processing result; when the processing result indicates that there is at least one atomic instruction that belongs to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is used as a behavior instruction for the robot.

[0069] That is to say, after the embodiment of the present application performs robot task planning based on the environment description information and the natural language commands to the robot and obtains the first atomic instruction combination of the robot, it further performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination to determine whether the multiple atomic instructions belong to sub-instructions in the atomic instruction library of the robot, and determine whether the atomic instructions conform to the task logic. Only when the physical constraint verification is passed to ensure that the atomic instruction is both in the robot's atomic instruction library and conforms to the logic of the current task, the atomic instruction is used as the robot's behavior instruction.

[0070] Thus, when the embodiments of the present application are applied to an embodied intelligent system, the embodied intelligent system can generate atomic instructions by combining environmental description information and natural language commands, thereby being able to respond to high-level, brief natural language commands input by the user, and thus being able to efficiently plan and execute tasks in a dynamic and complex environment. In addition, the embodied intelligent system performs physical constraint checks on the generated atomic instructions to ensure that these instructions are both in the robot's atomic instruction library and conform to the logic of the current task, thereby avoiding robot execution conflicts and erroneous operations, thereby improving the executability of these instructions and improving the success rate of executing instructions to achieve task goals. That is, the embodiments of the present application can improve the flexibility and reliability of the embodied intelligent system in understanding and executing behavioral instructions, thereby being able to efficiently execute corresponding operational tasks in a complex and dynamic environment.

[0071] Next, the method, device, equipment, robot, medium and product for generating behavioral instructions provided in the embodiments of the present application are specifically described through the following embodiments, and the method for generating behavioral instructions provided in the embodiments of the present application is first described in detail.

[0072] It should be noted that the method for generating behavioral instructions provided in the embodiment of the present application relates to the field of robot technology. The method for generating behavioral instructions provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a terminal configured by a robot, or can be an electronic device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. associated with the robot. The terminal and the robot are associated with each other, which means that the terminal can communicate and interact with the robot based on the network. The server side can be a backend server terminal device of the robot, which can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or can be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. Software can be an application, a computer program, and a storage medium carrying the computer program, etc., that implements the method for generating behavioral instructions. It should be understood that, based on different design requirements of actual applications, in different feasible embodiments, the terminal, server, and software that use the method for generating behavioral instructions provided in the embodiments of the present application may of course also be in other forms not listed here, and the method for generating behavioral instructions provided in the embodiments of the present application does not specifically limit this.

[0073] In addition, the present application can also be used in many general or special computer system environments or configurations. For example: embodied intelligent systems, robots, personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] For ease of understanding and explanation, the following text will take the method for generating behavioral instructions provided by the embodiment of the present application applied by the embodied intelligent system as an example to explain in detail the various specific embodiments of the present application. The embodied intelligent system can apply the method for generating behavioral instructions provided by the embodiment of the present application to control its own physical body, or the embodied intelligent system can also be integrated on the robot or associated with the robot to manage the process of planning and executing tasks by the robot. The embodied intelligent system is referred to as the system in the following text. The implementation of the method for generating behavioral instructions provided by the embodiment of the present application applied by any of the above-mentioned forms of the subject matter can refer to the process of the method for generating behavioral instructions applied by the electronic device described later.

[0075] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0076] Please refer to Figure 1 , Figure 1 The following is a flow chart of the steps of the method for generating a behavior instruction in some embodiments of the present application. Figure 1 The subsequent flowcharts of other steps show the execution order of some method steps, but based on different design requirements of actual applications, the method for generating behavior instructions provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1 The order of the steps of the method shown does not constitute a limitation on the execution logic order of the method for generating the behavior instruction provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the steps of the method shown should be included in the protection scope of the method for generating behavioral instructions provided in the embodiments of the present application.

[0077] like Figure 1 As shown, in some embodiments, the system applies the method for generating behavioral instructions provided by the embodiments of the present application, which may include steps S101 to S104.

[0078] Step S101: Processing the environment perception information of the robot to generate environment description information of the environment in which the robot is located.

[0079] When the user issues a high-level brief instruction to the robot to instruct the robot to perform a task, the system processes the robot's environmental perception information, and generates environmental description information of the robot's environment based on the environmental perception information.

[0080] In some embodiments, the system can receive commands input by the user to the robot through natural language by controlling the robot's microphone, and then control the robot to perceive the information of the surrounding environment through sensors (such as cameras, microphones, etc.) and upload it to the system. In this way, the system can obtain the robot's environmental perception information and further process it to generate corresponding environmental description information.

[0081] In other embodiments, the system may also control the robot to receive instructions manually input by the user to instruct the robot to perform tasks through a visual human-computer interaction interface. Alternatively, the system may also receive instructions input by the user on a visual human-computer interaction interface provided by a terminal device associated with the robot. After receiving the instructions input by the user, the system may further control the robot to perceive information about the surrounding environment, thereby performing an operation of processing the robot's environmental perception information to generate corresponding environmental description information.

[0082] Step S102: Perform robot task planning based on the environment description information and natural language commands to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes multiple atomic instructions.

[0083] It should be noted that the first atomic instruction combination is a combination of a series of basic instructions (atomic instructions) that the robot needs to execute in order to respond to high-level short instructions (natural language commands) input by the user and perform the tasks instructed by the user.

[0084] After generating environmental description information of the robot's environment, the system combines the environmental description information with high-level, brief instructions (natural language commands) input by the user to the robot, and plans the tasks that the user instructs the robot to perform, thereby obtaining multiple atomic instructions that the robot needs to execute in response to the natural language commands input by the user, and uses the multiple atomic instructions as the first atomic instruction combination.

[0085] Step S103: Match the multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the multiple atomic instructions to obtain a processing result.

[0086] After obtaining the first atomic instruction combination of the robot, the system does not directly use the multiple atomic instructions in the first atomic instruction combination as the final behavior instructions of the robot, but first matches the multiple atomic instructions with the atomic instruction library of the robot that has been set in advance, so as to confirm whether the multiple atomic instructions are basic instructions existing in the atomic instruction library. In addition, the system also performs task logic verification on the multiple atomic instructions to confirm whether each of the multiple atomic instructions conforms to the current task logic of the robot. After matching the multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions, the system obtains the corresponding processing results.

[0087] In some embodiments, when the system matches multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on multiple atomic instructions, all the atomic instructions can be matched with the atomic instruction library one by one, and then task logic verification can be performed on all the atomic instructions one by one.

[0088] In some embodiments, the system can also simultaneously perform task logic verification on a target instruction among multiple atomic instructions when matching it with the atomic instruction library, and thus take one target instruction at a time to simultaneously perform matching with the atomic instruction library and task logic verification until all atomic instructions have been matched with the atomic instruction library and task logic verification.

[0089] Step S104: when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, use the at least one atomic instruction as the behavior instruction of the robot.

[0090] It should be noted that the atomic instruction belongs to the atomic instruction library means that the system matches the atomic instruction with the atomic instruction library, thereby confirming that the atomic instruction is a basic instruction existing in the atomic instruction library.

[0091] After the system matches multiple atomic instructions in the first atomic instruction combination with the robot's atomic instruction library, and performs task logic verification on the multiple atomic instructions to obtain corresponding processing results, if the processing result indicates that there is at least one atomic instruction among the multiple atomic instructions that is a basic instruction existing in the atomic instruction library, and the at least one atomic instruction has also passed the system's task logic verification, then in this case, the system will use the at least one atomic instruction as one of the behavioral instructions to be executed by the robot in response to natural language commands to perform tasks.

[0092] In the embodiment of the present application, when the user issues a high-level short instruction to the robot to instruct the robot to perform a task, the system processes the robot's environmental perception information, thereby generating environmental description information of the robot's environment. Afterwards, the system combines the environmental description information and the high-level short instruction (natural language command) input by the user to the robot to perform robot task planning, thereby obtaining multiple atomic instructions that the robot needs to execute in response to the natural language command input by the user, and uses the multiple atomic instructions as the first atomic instruction combination. After that, the system matches the multiple atomic instructions in the first atomic instruction combination with the robot's atomic instruction library, and performs task logic verification on the multiple atomic instructions to obtain corresponding processing results. Finally, if the processing result indicates that there is at least one atomic instruction among the multiple atomic instructions that is a basic instruction in the atomic instruction library, and the at least one atomic instruction has also passed the system's task logic verification, then in this case, the system uses the at least one atomic instruction as one of the behavioral instructions to be executed by the robot in response to the natural language command to perform the task.

[0093] In this way, in the embodiment of the present application, the system generates atomic instructions by combining environmental description information and natural language commands, so that the robot can respond to high-level, brief natural language commands input by the user, and thus can efficiently plan and execute tasks in a dynamic and complex environment. In addition, the system performs physical constraint verification on the generated atomic instructions to ensure that these instructions are both in the robot's atomic instruction library and conform to the logic of the current task, which can avoid robot execution conflicts and erroneous operations, thereby improving the robot's executability and logical correctness for these instructions, and thus improving the robot's success rate in executing instructions to achieve task goals. That is, the embodiment of the present application can improve the flexibility and reliability of the system's understanding and execution of behavioral instructions, so that the robot can efficiently perform corresponding operation tasks in a complex and dynamic environment.

[0094] In some embodiments, the system controls the robot to perceive the surrounding environment through sensors to obtain environmental perception information, which may include: visual input information (such as image data and / or video data, etc.) of the surrounding environment perceived by the robot through a camera, and voice input information (such as environmental noise data and / or user voice data, etc.) of the surrounding environment perceived by the robot through a microphone. In view of this, the system can introduce a multimodal large model to process the robot's environmental perception information, thereby generating accurate and comprehensive environmental description information of the robot's environment. Among them, the multimodal large model is an artificial intelligence model that can process multiple input modes (such as vision and language) at the same time.

[0095] Please refer to Figure 2 , Figure 2 for Figure 1Schematic diagram of the detailed process flow of step S101.

[0096] like Figure 2 As shown, in some embodiments, the above step S101: processing the robot's environmental perception information to generate environmental description information of the robot's environment may include steps S201 and S202 as shown below.

[0097] Step S201: extract features of the visual input information through a preset multimodal large model to obtain environmental visual features corresponding to the visual input information, and extract features of the voice input information through the multimodal large model to obtain environmental language features corresponding to the voice input information.

[0098] It should be noted that the preset multimodal large model can be an open source MINI-GEMINI multimodal large model. It should be understood that based on the different design requirements of actual applications, in other different feasible embodiments, the system can of course also use other multimodal large models different from the MINI-GEMINI multimodal large model to process the robot's environmental perception information, and the method for generating behavioral instructions provided in the embodiment of the present application is not limited to the specific type of the multimodal large model.

[0099] When the robot's environmental perception information includes visual input information and voice input information, the system inputs the visual input information into the MINI-GEMINI multimodal large model to extract features from the visual input information through the MINI-GEMINI multimodal large model, thereby obtaining environmental visual features corresponding to the visual input information. In addition, the system also inputs the voice input information into the MINI-GEMINI multimodal large model to extract features from the voice input information through the MINI-GEMINI multimodal large model, thereby obtaining environmental language features corresponding to the voice input information.

[0100] Step S202: fusing the environmental visual features and the environmental language features through the multimodal large model to generate environmental description information of the environment in which the robot is located.

[0101] After the system obtains the environmental visual features corresponding to the visual input information and the environmental language features corresponding to the voice input information through the MINI-GEMINI multimodal large model, it further performs feature fusion processing on the environmental visual features and environmental language features through the MINI-GEMINI multimodal large model, thereby generating accurate and comprehensive environmental description information of the robot's environment. Among them, the environmental description information obtained by the system through the MINI-GEMINI multimodal large model fusing the environmental visual features and environmental language features can be the environmental state description information in text form.

[0102] In this embodiment, the system introduces the open source MINI-GEMINI multimodal large model to integrate the visual and language features in the environmental perception information, thereby generating accurate and comprehensive environmental description information for the robot's environment. In this way, the robot can respond to the high-level short instructions input by the user with the perception accuracy of the surrounding environment and the task understanding ability for the high-level short instructions. Furthermore, the system uses the environmental description information generated by the MINI-GEMINI multimodal large model together with the high-level short instructions input by the user to perform robot task planning to obtain the first atomic instruction combination of the robot, and combines the physical constraint verification of multiple atomic instructions in the first atomic instruction combination, so as to generate accurate and executable navigation and operation instructions for the robot, thereby improving the system's ability to control the robot to perform tasks in complex and dynamic environments.

[0103] In addition, in some embodiments, the system can also fine-tune and optimize the MINI-GEMINI multimodal large model to adapt the MINI-GEMINI multimodal large model to the specific task requirements of the robot, thereby further improving the accuracy of environmental perception and the subsequent generation of atomic instructions for the robot.

[0104] In some embodiments, the system can also use a reinforcement learning algorithm to process the robot's environmental perception information, thereby generating environmental description information of the robot's environment. Thus, with the help of reinforcement learning, the characteristics of the strategy can be gradually optimized in a dynamic and complex environment, and the perception accuracy of the robot's surrounding environment can be improved. For example, the system continuously interacts with the environment through a reinforcement learning model to learn the best strategy, thereby generating detailed environmental description information based on the robot's environmental perception information. Among them, the system can pre-train the reinforcement learning model using reinforcement learning algorithms such as Deep Q-Learning (DQN) or Proximal Policy Optimization (PPO), and in the process of training the reinforcement learning model, the system can optimize its action strategy through a reward mechanism.

[0105] In addition, during the process of training the reinforcement learning model, the system can use a high-precision environmental simulator to provide a training environment, so that the reinforcement learning model can perform a large amount of training in a virtual environment and accumulate experience.

[0106] In some embodiments, the system may perform robot task planning by combining environment description information and high-level brief instructions of the user through a large language model (LLM), thereby generating a first atomic instruction combination for the robot.

[0107] Based on this, the above step S102: performing robot task planning based on the environment description information and the natural language command to obtain the first atomic instruction combination of the robot may include the following steps:

[0108] The environment description information and the natural language command are input into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command through the large language model to obtain a first atomic instruction combination of the robot.

[0109] It should be noted that the preset large language model is the large language model LLM mentioned above. The large language model LLM is also called the large language model LLM or the LLM large model. For unified expression, the LLM large model is used in the following description of the embodiments to explain the embodiments in detail.

[0110] After generating environmental description information of the robot's environment, the system inputs the environmental description information and the natural language commands input by the user to the robot into the LLM large model, so as to plan the robot's tasks through the LLM large model in combination with the environmental description information and the natural language commands, and generate step-by-step instructions executable by the robot. The step-by-step instructions are the instructions (multiple atomic instructions) that the robot needs to execute in response to the natural language commands input by the user. The system uses the step-by-step instructions as the first atomic instruction combination of the robot.

[0111] In some embodiments, the system can pre-install an external knowledge base for the LLM large model, that is, the system establishes an association between the LLM large model and the knowledge base provided by the big data platform, so that the LLM large model can query the knowledge base for the required analysis and judgment. In this way, when the system performs robot task planning through the LLM large model combined with environmental description information and natural language commands, the LLM large model can query the external knowledge base as needed to generate step-by-step instructions that the robot can execute.

[0112] In some embodiments, when the system uses the LLM large model in combination with environment description information and natural language commands to perform robot task planning to generate a first atomic instruction combination, if the system subsequently performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination and finds that the multiple atomic instructions have failed the physical constraint verification, the system can also adjust the LLM large model to re-plan the robot and generate new instructions.

[0113] Please refer to Figure 3 , Figure 3 A flowchart of steps of the method for generating behavioral instructions provided in the embodiment of the present application in other embodiments.

[0114] like Figure 3 As shown, in some embodiments, when the system uses the LLM large model in combination with environmental description information and natural language commands to perform robot task planning to generate a first atomic instruction combination, the method for generating behavioral instructions provided in the embodiment of the present application may also include steps S301 to S304 as shown below.

[0115] Step S301: When the processing result indicates that multiple atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library and / or the multiple atomic instructions fail to pass task logic verification, adjust the large language model.

[0116] After the system uses the LLM large model in combination with the environment description information and the natural language command to generate the first atomic instruction combination, the system matches the multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on the multiple atomic instructions to obtain the corresponding processing results. If the processing result indicates that the multiple atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, or the processing result indicates that the multiple atomic instructions in the first atomic instruction combination have not passed the task logic verification, or the processing result indicates that the multiple atomic instructions in the first atomic instruction combination neither belong to the atomic instruction library nor pass the task logic verification, the system confirms that the multiple atomic instructions currently generated by the LLM large model have not passed the physical constraint verification, and thus, the system adjusts the LLM large model.

[0117] It should be noted that the system can adjust the LLM large model as follows: the system analyzes multiple atomic instructions that have not passed the physical constraint check, thereby determining the conflicts that exist in the robot's execution of the multiple atomic instructions, and then the system feeds back the conflicts to the LLM large model so that the LLM large model can avoid these conflicts when making new robot task planning.

[0118] Step S302: Input the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command through the adjusted large language model to obtain a second atomic instruction combination for the robot.

[0119] After adjusting the LLM large model, the system continues to input the environment description information and natural language commands into the adjusted LLM large model, and then uses the adjusted LLM large model to recombine the environment description information and natural language commands for robot task planning, thereby generating the second atomic instruction combination of the robot.

[0120] It should be noted that since the system fed back conflicts in atomic instructions that had not passed the physical constraint check when adjusting the LLM large model, the adjusted LLM large model will avoid the conflicts fed back by the system when re-planning the robot task in combination with the environmental description information and natural language commands, thereby generating more accurate step-by-step instructions. The system further uses the step-by-step instructions as the second atomic instruction combination of the robot.

[0121] Step S303: Match the multiple atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and perform task logic verification on the multiple atomic instructions.

[0122] After obtaining the second atomic instruction combination of the robot, the system first matches the multiple atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot that has been set in advance, just like the processing for the first atomic instruction combination, so as to confirm whether the multiple atomic instructions are basic instructions existing in the atomic instruction library. In addition, the system also performs task logic verification processing on the multiple atomic instructions in the second atomic instruction combination, so as to confirm whether each of the multiple atomic instructions conforms to the current task logic of the robot. After matching the multiple atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions, the system obtains the corresponding processing results.

[0123] Step S304: When the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification, the fourth target atomic instruction is used as the behavior instruction of the robot.

[0124] After the system matches multiple atomic instructions in the second atomic instruction combination with the robot's atomic instruction library, and performs task logic verification on the multiple atomic instructions to obtain the corresponding processing results, if the processing result indicates that there is at least one atomic instruction (the fourth target atomic instruction) in the second atomic instruction combination that is a basic instruction existing in the atomic instruction library, and the at least one atomic instruction (the fourth target atomic instruction) has also passed the system's task logic verification, then in this case, the system will use the at least one atomic instruction (the fourth target atomic instruction) as one of the behavioral instructions to be executed by the robot in response to a natural language command to perform a task.

[0125] In this embodiment, the system inputs the environment description information and the natural language commands input by the human user into the LLM large model for robot task planning, so as to use the LLM large model to disassemble the command in combination with the current state information, and in the reasoning process, the external knowledge base can be queried according to the situation, so as to output a combination of several atomic instructions. In this way, the LLM large model generates preliminary instructions based on the environment description and the user instructions, and then the instructions are physically checked to ensure that they are both in the robot atomic instruction library and conform to the logic of the current task, thereby improving the robot's executability of these instructions, and improving the robot's success rate in achieving the task objectives based on the execution of these instructions.

[0126] In some embodiments, when the system performs physical constraint verification on multiple atomic instructions, after confirming that a certain target sub-instruction belongs to the robot's atomic instruction library, it can further perform task logic verification on the target sub-instruction.

[0127] Based on this, the above-mentioned step S103 of "matching the multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot" may include the following steps:

[0128] Matching multiple atomic instructions in the first atomic instruction combination with atomic instructions in the atomic instruction library of the robot.

[0129] When the system performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination, it first matches the multiple atomic instructions one by one with the atomic instructions (basic instructions) in the robot's atomic instruction library.

[0130] Furthermore, the above-mentioned step S103 of “performing task logic verification on multiple atomic instructions” includes the following steps:

[0131] When at least one atomic instruction among the multiple atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and task logic verification is performed on the at least one atomic instruction.

[0132] When at least one atomic instruction among multiple atomic instructions matches an atomic instruction in the atomic instruction library (for example, they are consistent or the robot actions they represent are consistent), the system confirms that the at least one atomic instruction is an instruction belonging to the atomic instruction library, that is, the at least one atomic instruction is a basic instruction existing in the atomic instruction library. In this way, the system further performs task logic verification on the at least one atomic instruction to confirm whether the at least one atomic instruction also conforms to the current task logic of the robot.

[0133] In some embodiments, after the system matches multiple atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library, if there is one or more first target atomic instructions among the multiple atomic instructions that do not match any atomic instruction in the atomic instruction library, the system decomposes the first target atomic instruction, and then continues to match the multiple target sub-instructions obtained after the decomposition with the atomic instructions in the atomic instruction library, and when the target sub-instruction matches the atomic instruction in the atomic instruction library, confirms that the first target atomic instruction is also an atomic instruction in the atomic instruction library.

[0134] Please refer to Figure 4 , Figure 4 A flowchart of steps in some further embodiments of the method for generating behavioral instructions provided in the embodiments of the present application.

[0135] like Figure 4 As shown, in some embodiments, after the above-mentioned "matching the multiple atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot", the method for generating behavioral instructions provided in the embodiment of the present application also includes steps S401 to S403 as shown below.

[0136] Step S401: when a first target atomic instruction among the multiple atomic instructions does not match any atomic instruction in the atomic instruction library, decompose the first target atomic instruction into multiple target sub-instructions.

[0137] When the system performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination, after matching the multiple atomic instructions with the atomic instructions in the robot's atomic instruction library, if the first target atomic instruction among the multiple atomic instructions does not match any atomic instruction in the atomic instruction library, the system decomposes the first target atomic instruction to obtain multiple target sub-instructions of the first target atomic instruction.

[0138] Step S402: Match the multiple target sub-instructions with the atomic instructions in the atomic instruction library.

[0139] After decomposing the first target atomic instruction to obtain multiple target sub-instructions, the system further matches the multiple target sub-instructions with the atomic instructions in the atomic instruction library one by one, so as to confirm whether the multiple target sub-instructions are basic instructions existing in the atomic instruction library.

[0140] Step S403: when the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, confirming that the first target atomic instruction belongs to the atomic instruction library.

[0141] After the system matches the multiple target sub-instructions with the atomic instructions in the atomic instruction library, thereby confirming that the multiple target sub-instructions are basic instructions existing in the atomic instruction library, the system confirms that the first target atomic instruction in the first atomic instruction combination also belongs to the robot's atomic instruction library. In this way, the system can further perform task logic verification on the first target atomic instruction.

[0142] It should be noted that after the system confirms that the first target atomic instruction has passed the task logic verification and thus uses the first target atomic instruction as the robot's behavior instruction, the robot can sequentially execute multiple target sub-instructions obtained by decomposing the first target atomic instruction when executing the first target original value instruction.

[0143] The system's physical constraint verification for multiple atomic instructions in the first atomic instruction combination includes atomic instruction verification and task logic verification. Among them, atomic instruction verification refers to matching atomic instructions with the atomic instruction library to ensure that the atomic instructions generated by the system are in the robot's atomic instruction library, as well as verifying and decomposing complex instructions (first target atomic instructions) into atomic instructions. In addition, task logic verification uses mathematical optimization methods (such as linear programming or integer programming optimization methods) to ensure the logical correctness of the instructions generated by the system and avoid robot conflicts and erroneous operations. The system can significantly improve the reliability of instruction execution by performing task logic verification on multiple sub-instructions.

[0144] Please refer to Figure 5 , Figure 5 for Figure 1Schematic diagram of the detailed process flow of step S103.

[0145] like Figure 5 As shown, in some embodiments, the "performing task logic verification on the multiple atomic instructions" in the above-mentioned step S103 may include steps S501 to S503 as shown below.

[0146] Step S501: Generate an instruction sequence based on the multiple atomic instructions.

[0147] When the system performs task logic verification on multiple atomic instructions, it first generates an instruction sequence based on the multiple atomic instructions.

[0148] Step S502: verifying the object logic state of the instructions in the instruction sequence based on the preset object logic constraint condition to obtain a verification result.

[0149] It should be noted that the preset object holding logic condition may be: if the robot holds an object when executing the i-th instruction, then the robot must also hold the object when executing the i+1-th instruction.

[0150] After the system generates an instruction sequence based on multiple atomic instructions, it first verifies the object holding logic state of the instructions in the instruction sequence one by one based on the object holding logic condition to obtain a verification result. The verification result is used to characterize whether the object holding logic state of the instructions in the instruction sequence satisfies the object holding logic constraint, that is, the verification result specifically indicates whether the object holding logic state of the instructions in the instruction sequence satisfies the robot holding the object when executing the i-th instruction in the instruction sequence, and the robot also holds the object when executing the i+1-th instruction.

[0151] Step S503: When the verification result indicates that the instructions in the instruction sequence satisfy the object logic constraint condition, the position logic state of the instructions in the instruction sequence is verified based on the preset position logic constraint condition.

[0152] It should be noted that the preset position logic constraint condition may be: the robot must be located at a specific position (such as "in front of the refrigerator") before performing certain specific operations (such as "opening the refrigerator").

[0153] After the system obtains the verification result of the object holding logic state of the instructions in the instruction sequence, if the verification result indicates that the instructions in the instruction sequence meet the object holding logic constraint condition, that is, the verification result indicates that the object holding logic state of the instructions in the instruction sequence meets the situation that the robot holds the object when executing the i-th instruction in the instruction sequence, and the robot also holds the object when executing the i+1-th instruction. In this case, the system further verifies the position logic state of the instructions in the instruction sequence one by one based on the position logic constraint condition.

[0154] Exemplarily, the system defines an instruction sequence {a1, a2, ..., an} based on multiple atomic instructions, where each atomic instruction ai has a corresponding logical state si, and the logical state si may include and in, Indicates whether the robot holds an object when executing instruction ai, and Indicates the position of the robot when executing instruction ai.

[0155] After that, the system first verifies the object holding logic state for the atomic instruction ai in the instruction sequence {a1, a2, …, an}. That is, if the robot holds the object when the instruction ai is executed, the robot cannot execute the "pick up the object" instruction again before executing the "put down the object" instruction.

[0156] formula: Describes that in a series of execution instructions, when the robot holds an object at a certain time step (or instruction), it must continue to hold the object at the next time step (or instruction) until the "put down the object" instruction is executed. This is a state-keeping constraint that ensures that the robot follows the correct logical sequence when completing the task. Among them, if the robot holds the object (i.e. is true), then when executing the i+1th instruction, the robot must also hold the object (i.e. is also true).

[0157] In addition, the system also verifies the position logic state of the atomic instruction ai in the instruction sequence {a1, a2, …, an}. That is, The robot must be in a specific location (such as "in front of the refrigerator") before it can perform certain specific actions (such as "open the refrigerator").

[0158] In this embodiment, the system performs physical constraint verification on the first atomic instruction combination of the robot generated by combining the environment description information and the natural language command. The physical constraint verification includes atomic instruction verification and task logic verification. These verifications can ensure that multiple atomic instructions in the first atomic instruction combination are both in the robot's atomic instruction library and conform to the robot's current task logic, thereby effectively avoiding conflicts and erroneous operations when the robot executes instructions.

[0159] In addition, since the system can integrate visual and language information by introducing the MINI-GEMINI multimodal large model, thereby generating more accurate and comprehensive environmental description information, the system combines the multimodal large model with physical constraint verification, first generates accurate and comprehensive environmental description information based on the multimodal large model, then combines the environmental description information with natural language commands to generate the first atomic instruction combination, and then performs physical constraint verification on multiple atomic instructions in the first atomic instruction combination. In this way, the adaptability, flexibility and reliability of robot task planning and execution in dynamic and complex scenarios can be significantly improved.

[0160] In addition to performing physical constraint checking on multiple atomic instructions in the first atomic instruction combination, the system can also optimize the atomic instructions in combination with the physical constraint checking.

[0161] Please refer to Figure 6 , Figure 6 A flowchart of steps in some further embodiments of the method for generating behavioral instructions provided in the embodiments of the present application.

[0162] like Figure 6 As shown, in some embodiments, the method for generating behavioral instructions provided in the embodiments of the present application may also include step S601 and step S602 as shown below.

[0163] Step S601: When the processing result indicates that the multiple atomic instructions belong to the atomic instruction library, and the second target atomic instruction among the multiple atomic instructions passes the task logic verification, and the third target atomic instruction among the multiple atomic instructions fails to pass the task logic verification, optimize the third target atomic instruction based on preset task logic rules to obtain optimized sub-instructions; wherein the number of the multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions.

[0164] After the system matches multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performs task logic verification on the multiple atomic instructions to obtain corresponding processing results, if the processing result indicates that the multiple atomic instructions are all basic instructions (belonging to the atomic instruction library) existing in the atomic instruction library, but the second target atomic instruction in the multiple atomic instructions passes the task logic verification, and the remaining third target atomic instruction in the multiple atomic instructions does not pass the task logic verification, in this case, the system optimizes the third target atomic instruction based on the preset task logic rules to obtain the optimized sub-instruction, and then performs logical task verification on the optimized sub-instruction. Among them, the number of multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions.

[0165] It should be noted that the system can optimize the third target atomic instruction by verifying and adjusting the third target atomic instruction through the above-mentioned data optimization method. For example, in the case that the third target atomic instruction does not satisfy the above-mentioned object logic constraint, the system adjusts the object logic state of the third target atomic instruction based on the object logic constraint condition, thereby obtaining an optimized sub-instruction, and the object logic state of the optimized sub-instruction satisfies the object logic constraint. Alternatively, in the case that the third target atomic instruction does not satisfy the above-mentioned position logic constraint, the system adjusts the position logic state of the third target atomic instruction based on the position logic constraint condition, thereby obtaining an optimized sub-instruction, and the position logic state of the optimized sub-instruction satisfies the position logic constraint.

[0166] Step S602: Use the second target atomic instruction and the optimized sub-instruction as the behavior instruction of the robot.

[0167] After the system optimizes the third target atomic instruction to obtain the optimized sub-instruction, and further confirms that the optimized sub-instruction passes the task logic verification, the system uses the second target atomic instruction that has previously passed the task logic verification and the optimized sub-instruction as one of the behavioral instructions to be executed by the robot in response to the user's natural language command.

[0168] In this embodiment, the system uses mathematical optimization methods to optimize the third target atomic instruction that has not passed the task logic verification, thereby ensuring that it is both in the robot atomic instruction library and conforms to the logic of the current task (ensuring the logical correctness of the instruction). This can avoid conflicts and erroneous operations when the robot executes instructions, thereby improving the executableness of the instructions and the success rate of the task.

[0169] In some embodiments, the system can also combine the rules and constraints provided by the expert system to ensure the safety and logical correctness of the system behavior. For example, the system pre-designs a set of expert system rules to define the task logic and operation constraints when the robot executes instructions (such as prohibiting the robot from putting down an object without holding the object). Then, after the system combines the environment description information and the natural language command to generate the first atomic instruction combination, the system rules can be used to perform physical constraint checks on multiple atomic instructions in the first atomic instruction combination, thereby ensuring the logical correctness of the robot operations corresponding to these instructions with the help of the clear rules and constraints provided by the expert system rules.

[0170] Next, the method for generating behavioral instructions provided in the embodiments of the present application is proposed, which involves an embodied intelligent navigation algorithm framework that combines a multimodal large model and a physical constraint verification mechanism in some embodiments.

[0171] The embodied intelligent navigation algorithm framework includes a multimodal large model, an LLM large model, an atomic instruction verification submodule, a task logic verification submodule, and a robot execution module. Through the close linkage of these modules, the system ensures that when the user inputs high-level and brief instructions to the robot, accurate and executable navigation and operation instructions are generated.

[0172] Among them, the multimodal large model mainly takes the visual input and language instructions of the robot's environment as input, and generates and outputs detailed environment description information by integrating visual and language features. The LLM large model takes the environment description information and task instructions (natural language commands) as input, and generates and outputs the robot-executable step-by-step instructions (first atomic instruction combination) according to the environment description and task instructions. In addition, the LLM large model can query the plug-in knowledge base as needed. The physical constraint module mainly includes: an atomic instruction verification submodule and a task logic verification submodule. The atomic instruction verification submodule takes the step-by-step instructions output by the LLM large model as input, verifies whether these instructions are in the robot's atomic instruction library, and performs necessary instruction decomposition, and then outputs the execution instructions that are verified to exist in the post-atomic instruction library. The task logic verification submodule takes the verified execution instructions output by the atomic instruction verification submodule and the preset task logic rules as input, verifies and adjusts these instructions through mathematical optimization methods to ensure that the instructions conform to the task logic and are conflict-free, and finally outputs the verified and / or optimized execution instructions (behavior instructions). The robot execution module takes the execution instructions output by the task logic verification submodule as input, and outputs the actual robot action to control the robot to execute the behavioral instructions to complete the specified task.

[0173] In some embodiments, the system uses the above-mentioned embodied intelligent navigation algorithm framework to achieve linkage between various functional modules according to the following process 1 to 5.

[0174] 1. Initialization: When the robot receives high-level brief instructions input by the user, the system uses the multimodal large model to generate environmental description information and preliminary instructions. 2. Environmental perception: The system extracts visual and language features through the multimodal large model to generate environmental description information. 3. Preliminary instruction generation: The system generates preliminary execution instructions through the LLM large model combined with environmental description information and task instructions. The LLM large model can query the plug-in knowledge base as needed. 4. Physical constraint verification and optimization: The system ensures that the instructions are in the atomic instruction library through the atomic instruction verification submodule and decomposes the instructions. In addition, the system also uses the task logic verification submodule to ensure the logical correctness of the instructions through mathematical formulas and optimization algorithms. 5. Execution and update: The system performs operations according to the optimized instructions through the robot execution module to complete the task.

[0175] For example, please refer to Figure 7 , Figure 7 The overall architecture diagram of the method for generating behavioral instructions provided in the embodiments of the present application in some embodiments. Figure 7 As shown, the system can complete the perception of the robot's environment through a multimodal large model, and summarize it into textual description information of the environment, and then combine it with human natural language commands, and input the environment description information and natural language commands into the LLM large model for robot task planning. The LLM large model disassembles the command in combination with the current state information indicated by the environment description information, and introduces the plug-in knowledge base according to the situation during the reasoning process. Finally, the LLM large model outputs the first atomic instruction combination of several atomic instructions to complete the current given task. Afterwards, before the system sends the multiple atomic instructions in the first atomic instruction combination to the robot's execution unit, it first verifies the multiple atomic instructions from the two perspectives of atomic instruction verification and task logic verification through the physical constraint module. Only the behavioral instructions that pass the verification will be sent to the robot's action execution unit to control the robot to execute the command action.

[0176] Furthermore, if multiple atomic instructions in the first atomic instruction combination fail to pass the physical constraint check (do not satisfy the physical constraints represented by the two angles of atomic instruction verification and / or task logic verification), the system will also feed back the conflict corresponding to the instruction to the LLM large model for adjustment. After that, the system can use the adjusted LLM large model to continue robot task planning.

[0177] Next, a complete embodiment of the method for generating behavioral instructions provided in the embodiments of the present application is proposed.

[0178] Please refer to Figure 8 , Figure 8 A schematic diagram of an application flow involved in a method for generating behavioral instructions provided in an embodiment of the present application.

[0179] like Figure 8 As shown in the figure, when the user inputs a high-level and brief natural language command to the robot, the system processes the environmental perception information (visual input and language input) of the robot's environment through the multimodal large model to generate detailed environmental description information by integrating visual and language features. After that, the system uses the LLM large model to combine the environmental description information and natural language commands for robot task planning, thereby generating preliminary execution instructions for the robot (the first atomic instruction combination). Then, the system inputs the instructions into the physical constraint module to perform physical constraint verification from two perspectives: atomic instruction verification and task logic verification. Finally, when the behavior instruction passes the atomic instruction verification and the task logic verification, the system executes the behavior instruction through the robot execution module. In addition, when the instruction does not pass the atomic instruction verification and / or the task logic verification, the system feeds back the conflict corresponding to the instruction to the LLM large model to adjust it. After that, the system can use the adjusted LLM large model to re-reason to continue the robot task planning.

[0180] See also Fig. 9 The embodiment of the present application also provides a device for generating a behavior instruction, which can implement the above-mentioned method for generating a behavior instruction. The device includes: an environment description module 901, a task planning module 902, an instruction verification module 903 and an instruction generation module 904. Among them,

[0181] The environment description module 901 is used to process the environment perception information of the robot to generate environment description information of the environment where the robot is located;

[0182] A task planning module 902 is used to perform robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes multiple atomic instructions;

[0183] An instruction verification module 903 is used to match multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and to perform task logic verification on the multiple atomic instructions to obtain a processing result;

[0184] The instruction generation module 904 is used to use the at least one atomic instruction as the behavior instruction of the robot when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification.

[0185] In some embodiments, the instruction verification module 903 is also used to match multiple atomic instructions in the first atomic instruction combination with atomic instructions in the atomic instruction library of the robot; and, when at least one atomic instruction among the multiple atomic instructions matches the atomic instruction in the atomic instruction library, confirm that the at least one atomic instruction belongs to the atomic instruction library, and perform task logic verification on the at least one atomic instruction.

[0186] In some embodiments, the instruction verification module 903 is also used to decompose the first target atomic instruction among the multiple atomic instructions into multiple target sub-instructions when the first target atomic instruction does not match any atomic instruction in the atomic instruction library; match the multiple target sub-instructions with the atomic instructions in the atomic instruction library; and, when the multiple target sub-instructions match the atomic instructions in the atomic instruction library, confirm that the first target atomic instruction belongs to the atomic instruction library.

[0187] In some embodiments, the instruction verification module 903 is also used to generate an instruction sequence based on the multiple atomic instructions; verify the object logic state of the instructions in the instruction sequence based on a preset object logic constraint to obtain a verification result; and, when the verification result indicates that the instructions in the instruction sequence satisfy the object logic constraint, verify the position logic state of the instructions in the instruction sequence based on a preset position logic constraint.

[0188] In some embodiments, the instruction verification module 903 is also used to optimize the third target atomic instruction based on preset task logic rules to obtain optimized sub-instructions when the processing result indicates that the multiple atomic instructions belong to the atomic instruction library, and the second target atomic instruction among the multiple atomic instructions passes the task logic verification, and the third target atomic instruction among the multiple atomic instructions fails to pass the task logic verification; wherein the number of the multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions; and, use the second target atomic instruction and the optimized sub-instructions as behavior instructions of the robot.

[0189] In some embodiments, the task planning module 902 is further used to input the environment description information and the natural language command into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command through the large language model, and obtain a first atomic instruction combination of the robot; when the processing result indicates that multiple atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library, and / or the multiple atomic instructions fail to pass the task logic verification, adjust the large language model; input the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command through the adjusted large language model, and obtain a second atomic instruction combination of the robot;

[0190] The instruction verification module 903 is further used to match the multiple atomic instructions in the second atomic instruction combination with the atomic instruction library of the robot, and to perform task logic verification on the multiple atomic instructions;

[0191] The instruction generation module 904 is also used to use the fourth target atomic instruction as the behavior instruction of the robot when the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification.

[0192] In some embodiments, the environmental perception information includes: visual input information and voice input information. The environmental description module 901 is also used to extract features from the visual input information through a preset multimodal large model to obtain environmental visual features corresponding to the visual input information, and to extract features from the voice input information through the multimodal large model to obtain environmental language features corresponding to the voice input information; and to fuse the environmental visual features and the environmental language features through the multimodal large model to generate environmental description information of the environment where the robot is located.

[0193] The specific implementation of the device for generating behavior instructions provided in the embodiment of the present application is basically the same as the specific implementation of the method for generating behavior instructions described above, and will not be repeated here.

[0194] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for generating behavior instructions when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0195] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0196] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0197] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the method for generating the behavior instruction in the embodiments of this application;

[0198] Input / output interface 1003, used to implement information input and output;

[0199] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0200] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0201] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0202] An embodiment of the present application also provides a robot, which is equipped with an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method of generating behavioral instructions when executing the computer program.

[0203] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for generating the above-mentioned behavior instructions is implemented.

[0204] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0205] The embodiment of the present application also provides a computer program product, including a computer program. The steps implemented when the computer program is executed by a processor are basically the same as the specific embodiment of the above-mentioned behavior instruction generation method, and will not be repeated here.

[0206] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0207] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0210] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0211] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0212] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0213] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0216] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a behavior instruction, characterized in that: The method comprises: Processing the robot's environmental perception information to generate environmental description information of the robot's environment; Perform robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes multiple atomic instructions; Matching multiple atomic instructions in the first atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions to obtain a processing result; When the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification, the at least one atomic instruction is used as the behavior instruction of the robot.

2. The method according to claim 1, characterized in that The matching of the plurality of atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot includes: Matching multiple atomic instructions in the first atomic instruction combination with atomic instructions in an atomic instruction library of the robot; The task logic verification of the multiple atomic instructions includes: When at least one atomic instruction among the multiple atomic instructions matches an atomic instruction in the atomic instruction library, it is confirmed that the at least one atomic instruction belongs to the atomic instruction library, and task logic verification is performed on the at least one atomic instruction.

3. The method according to claim 2, characterized in that After matching the plurality of atomic instructions in the first atomic instruction combination with the atomic instructions in the atomic instruction library of the robot, the method further includes: When a first target atomic instruction among the multiple atomic instructions does not match any atomic instruction in the atomic instruction library, decomposing the first target atomic instruction into multiple target sub-instructions; Matching the multiple target sub-instructions with the atomic instructions in the atomic instruction library; In a case where the plurality of target sub-instructions match the atomic instructions in the atomic instruction library, it is confirmed that the first target atomic instruction belongs to the atomic instruction library.

4. The method according to claim 1, characterized in that: The performing task logic verification on the multiple atomic instructions includes: generating an instruction sequence based on the plurality of atomic instructions; Verifying the object logic state of the instructions in the instruction sequence based on the preset object logic constraint condition to obtain a verification result; When the verification result indicates that the instructions in the instruction sequence satisfy the object logic constraint condition, the position logic state of the instructions in the instruction sequence is verified based on the preset position logic constraint condition.

5. The method according to claim 1, characterized in that The method further comprises: When the processing result indicates that the multiple atomic instructions belong to the atomic instruction library, and the second target atomic instruction among the multiple atomic instructions passes the task logic verification, and the third target atomic instruction among the multiple atomic instructions fails to pass the task logic verification, optimizing the third target atomic instruction based on a preset task logic rule to obtain an optimized sub-instruction; wherein the number of the multiple atomic instructions is equal to the sum of the number of the second target atomic instructions and the number of the third target atomic instructions; The second target atomic instruction and the optimized sub-instruction are used as the behavior instructions of the robot.

6. The method according to claim 1, characterized in that The robot task planning is performed based on the environment description information and the natural language command to obtain the first atomic instruction combination of the robot, including: Inputting the environment description information and the natural language command into a preset large language model, so as to perform robot task planning based on the environment description information and the natural language command through the large language model, and obtain a first atomic instruction combination of the robot; The method further comprises: When the processing result indicates that the plurality of atomic instructions in the first atomic instruction combination do not belong to the atomic instruction library and / or the plurality of atomic instructions fail to pass task logic verification, adjusting the large language model; Inputting the environment description information and the natural language command into the adjusted large language model, so as to perform robot task planning based on the environment description information and the natural language command through the adjusted large language model to obtain a second atomic instruction combination for the robot; Matching a plurality of atomic instructions in the second atomic instruction combination with an atomic instruction library of the robot, and performing task logic verification on the plurality of atomic instructions; When the fourth target atomic instruction in the second atomic instruction combination belongs to the atomic instruction library and the fourth target atomic instruction passes the task logic verification, the fourth target atomic instruction is used as the behavior instruction of the robot.

7. The method according to any one of claims 1 to 6, characterized in that The environmental perception information includes: visual input information and voice input information; The processing of the robot's environmental perception information to generate environmental description information of the robot's environment includes: Extracting features from the visual input information using a preset multimodal large model to obtain environmental visual features corresponding to the visual input information, and extracting features from the voice input information using the multimodal large model to obtain environmental language features corresponding to the voice input information; The environmental visual features and the environmental language features are fused through the multimodal large model to generate environmental description information of the environment where the robot is located.

8. A device for generating a behavior instruction, characterized in that: The device comprises: An environment description module, used to process the robot's environmental perception information to generate environmental description information of the robot's environment; A task planning module, used to perform robot task planning based on the environment description information and the natural language command to obtain a first atomic instruction combination of the robot; wherein the first atomic instruction combination includes a plurality of atomic instructions; An instruction verification module, used for matching multiple atomic instructions in the first atomic instruction combination with the atomic instruction library of the robot, and performing task logic verification on the multiple atomic instructions to obtain a processing result; An instruction generation module is used to use the at least one atomic instruction as a behavior instruction of the robot when the processing result indicates that there is at least one atomic instruction belonging to the atomic instruction library and the at least one atomic instruction passes the task logic verification.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for generating a behavior instruction as described in any one of claims 1 to 7 when executing the computer program.

10. A robot, characterized in that: The robot includes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method for generating behavior instructions as described in any one of claims 1 to 7 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a behavior instruction according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating behavioral instructions according to any one of claims 1 to 7 are implemented.

Images