Robot control method and device, server, warehousing system and storage medium

Abstract

The application relates to the technical field of intelligent storage systems, and provides a robot control method and device, a server and a storage system. The method comprises the following steps: acquiring multi-modal instruction information input by a user; acquiring current environment information from system data of a storage system; inputting the multi-modal instruction information and the current environment information into a multi-modal large model; analyzing the multi-modal instruction information based on the current environment information by using the multi-modal large model, so as to obtain a target entity and a user instruction indicated by the multi-modal instruction information; searching for related system data related to the target entity from the system data, and inputting the related system data into the multi-modal large model; performing dependency verification on the user instruction according to the current environment information, the related system data and a pre-posed dependency relationship by using the multi-modal large model, and obtaining a to-be-executed instruction after the verification; and calling an application programming interface corresponding to the to-be-executed instruction, so as to control a robot to execute the instruction. The application can accurately analyze a composite instruction.

Description

Technical Field

[0001] This application relates to the field of intelligent warehousing system technology, specifically to a robot control method, a robot control device, a server, a warehousing system, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of the logistics industry, intelligent warehousing systems have been widely adopted, improving warehousing efficiency, reducing labor costs, and minimizing human error. Traditional intelligent warehousing systems mostly use a GUI (User-Guided Interface) as the human-computer interaction method, where users issue instructions according to predetermined logic through a human-computer interaction terminal. However, traditional GUI interaction is inefficient, prone to errors, and unable to parse unstructured instructions.

[0003] Developing interactive technology towards natural language interaction is one possible direction, but existing related technologies only support fixed phrase recognition and cannot accurately parse compound instructions. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a robot control method, a robot control device, a server, a warehousing system, and a computer-readable storage medium to solve the problem that the prior art cannot accurately parse compound instructions.

[0005] According to a first aspect of the embodiments of this application, a robot control method is provided, the method comprising: acquiring multimodal instruction information input by a user through a human-machine interaction terminal; acquiring current environmental information from system data of a warehousing system; inputting the multimodal instruction information and the current environmental information into a multimodal large model, wherein the multimodal large model is a model trained based on basic information of the warehousing system, the basic information including entity information and instruction pre-dependencies of the warehousing system; parsing the multimodal instruction information based on the current environmental information using the multimodal large model to obtain the target entity indicated by the multimodal instruction information and the user instruction; retrieving associated system data related to the target entity from the system data and inputting the associated system data into the multimodal large model; performing dependency verification on the user instruction using the multimodal large model based on the current environmental information, the associated system data, and the instruction pre-dependencies, and obtaining an instruction to be executed after successful verification; and calling an application programming interface corresponding to the instruction to be executed to control a robot to execute the instruction to be executed.

[0006] In some embodiments, parsing the multimodal instruction information based on the current environment information using the multimodal large model to obtain the target entity and user instruction indicated by the multimodal instruction information includes: parsing the multimodal instruction information based on the current environment information using the multimodal large model to obtain the entity description of the target entity and the user instruction indicated by the multimodal instruction information; determining whether the entity description of the target entity contains an entity identifier; if the entity description of the target entity does not contain the entity identifier, querying the entity identifier of the target entity from the system data using the multimodal large model; and updating the entity description of the target entity using the entity identifier of the target entity.

[0007] In some embodiments, the basic information further includes atomic instructions. The step of querying the entity identifier of the target entity from the system data using the multimodal large model includes: calling a query instruction through the multimodal large model to query the entity identifier of the target entity from the system data. The query instruction is an instruction within the atomic instructions, and the parameters of the query instruction are entity information contained in the entity description of the target entity. The atomic instruction refers to a minimal, indivisible instruction.

[0008] In some embodiments, obtaining current environmental information from the system data of the warehousing system includes: obtaining the current environmental information configured to be read in full by default from the system data of the warehousing system, wherein the current environmental information includes shelf information, bin information, workstation information, robot information and current task information.

[0009] In some embodiments, the step of performing dependency verification on the user instruction based on the current environment information, the associated system data, and the instruction pre-dependency relationship, and obtaining the instruction to be executed after the verification passes, includes: performing dependency verification on the user instruction based on the current environment information, the associated system data, and the instruction pre-dependency relationship; if the dependency verification passes, obtaining the instruction to be executed containing the user instruction; if the dependency verification fails, adding supplementary instructions to the user instruction, until the dependency verification passes, obtaining the instruction to be executed containing the supplementary instructions and the user instruction.

[0010] In some embodiments, performing dependency verification on the user instruction based on the current environment information, the associated system data, and the instruction pre-dependency relationship includes: determining whether the current state satisfies the instruction pre-dependency relationship corresponding to the user instruction based on the current environment information and the associated system data; if the current state satisfies the instruction pre-dependency relationship corresponding to the user instruction, determining that the dependency verification of the user instruction has passed; if the current state does not satisfy the instruction pre-dependency relationship corresponding to the user instruction, determining that the dependency verification of the user instruction has failed.

[0011] In some embodiments, if the dependency check fails, a supplementary instruction is added to the user instruction until the dependency check passes, resulting in the execution instruction containing the supplementary instruction and the user instruction. This includes: determining a supplementary instruction that must be executed before the user instruction is executed based on the current environment information, the associated system data, and the instruction's pre-dependency relationship; adding the supplementary instruction to the user instruction, wherein the execution order of the supplementary instruction is before the user instruction; performing the dependency check on the user instruction after adding the supplementary instruction; and if the dependency check passes, determining the supplementary instruction and the user instruction as the execution instruction.

[0012] In some embodiments, the instruction to be executed includes at least one supplementary instruction executed sequentially and the user instruction. The step of calling the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction includes: generating an instruction chain based on the at least one supplementary instruction, the user instruction, and the instruction execution order, for sequentially calling multiple application programming interfaces corresponding to the at least one supplementary instruction and the user instruction; executing the instruction chain to call the application programming interface corresponding to the instruction to be executed, thereby controlling the robot to execute the instruction to be executed.

[0013] In some embodiments, before invoking the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction, the method further includes: outputting the instruction to be executed to the user; and in response to the user's confirmation operation of the instruction to be executed, performing the step of invoking the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction.

[0014] In some embodiments, the instruction to be executed includes at least one supplementary instruction executed sequentially and the user instruction. After invoking the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction, the method further includes: after the robot executes each instruction, obtaining the execution result of the current instruction, the execution result including execution success and execution failure; if the execution result is execution failure, generating a rollback instruction corresponding to the preceding instruction, the rollback instruction being used to restore the state before the execution of the preceding instruction; and invoking the application programming interface corresponding to the rollback instruction to control the robot to execute the rollback instruction.

[0015] According to a second aspect of the embodiments of this application, a robot control device is provided, the device comprising: a first acquisition module, configured to acquire multimodal instruction information input by a user through a human-computer interaction terminal; a second acquisition module, configured to acquire current environment information from system data of a warehousing system; an input module, configured to input the multimodal instruction information and the current environment information into a multimodal large model, the multimodal large model being a model trained based on basic information of the warehousing system, the basic information including entity information, atomic instructions, and instruction pre-dependencies of the warehousing system; and a parsing module, configured to use the multimodal large model based on the current environment information... The environmental information is used to parse the multimodal instruction information to obtain the target entity and user instruction indicated by the multimodal instruction information; the retrieval module is used to retrieve the associated system data related to the target entity from the system data and input the associated system data into the multimodal large model; the verification module is used to perform dependency verification on the user instruction through the multimodal large model based on the current environmental information, the associated system data and the instruction pre-dependency relationship, and obtain the instruction to be executed after the verification is passed; the invocation module is used to invoke the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction to be executed.

[0016] According to a third aspect of the embodiments of this application, a server is provided, comprising: a processor and a memory, wherein the memory stores executable instructions, and the processor is capable of executing the executable instructions to implement the robot control method as described in any of the above embodiments.

[0017] According to a fourth aspect of the embodiments of this application, a warehousing system is provided, including a human-machine interaction terminal, a server, and a robot; the human-machine interaction terminal is used to receive multimodal instruction information input by a user and send the multimodal instruction information to the server; the server is used to execute the robot control method as described in any of the above embodiments; the robot is used to execute the instruction to be executed in response to the instruction to be executed issued by the server through an application programming interface.

[0018] According to a fifth aspect of the present application, a computer-readable storage medium is provided, wherein executable instructions are stored therein, which, when executed on a server, cause the server to perform the robot control method as described in any of the preceding embodiments.

[0019] This application embodiment obtains the target entity and user instructions by fusing multimodal instruction information through a multimodal large model. It retrieves associated system data related to the target entity from the system data and performs dependency verification on the user instructions based on the acquired current environment information, the retrieved associated system data, and the instruction pre-dependencies learned during pre-training. If the verification passes, the resulting instruction to be executed satisfies the instruction pre-dependencies. Through this method, this application embodiment can understand and decompose unstructured user instructions, achieving rapid and accurate parsing of multimodal composite instructions, improving operational efficiency in the warehousing system, and avoiding errors caused by multiple layer-by-layer operations by the user, thus reducing the error rate.

[0020] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0022] In the attached diagram:

[0023] Figure 1 A schematic diagram illustrating an application scenario of an embodiment of this application is shown;

[0024] Figure 2 A schematic diagram of the interaction flow of the robot control method provided in an embodiment of this application is shown;

[0025] Figure 3 A flowchart illustrating the robot control method provided in an embodiment of this application is shown;

[0026] Figure 4 A schematic diagram of the structure of the robot control device provided in an embodiment of this application is shown;

[0027] Figure 5 A schematic diagram of the server structure provided in an embodiment of this application is shown. Detailed Implementation

[0028] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0030] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists, A and B exist simultaneously, and B exists. In addition, the character " / " in this document generally indicates that the related objects before and after it have an "or" relationship.

[0033] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0034] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0035] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0036] Intelligent warehousing systems are automated solutions that integrate multiple advanced technologies and are widely used in the modern warehousing and logistics field. Their main functions cover key warehousing operations such as automatic storage and retrieval of goods, inventory management, order picking, goods handling and route planning, aiming to improve warehousing efficiency, reduce labor costs and reduce human error.

[0037] Traditional intelligent warehousing systems mostly use graphical user interfaces (GUIs) as the main way for human-computer interaction. Users can use a PC with a display screen (hereinafter referred to as the human-computer interaction terminal) to issue instructions to the robot by clicking menu options or buttons layer by layer according to the established interaction logic, so as to complete tasks such as moving goods to designated storage locations and picking goods according to order requirements.

[0038] The instructions given by the user to the robot can be atomic instructions or compound instructions.

[0039] Atomic instructions are the most basic operational units in intelligent warehousing systems. They are minimal, indivisible instructions that directly control robots or other warehousing equipment to complete a single action. For example, the inbound instruction "Move bin 1 from the temporary storage area to storage location 001 on the second floor of shelf 1 in area A" is an atomic inbound instruction, explicitly specifying the goods (bin 1), the starting position (temporary storage area), and the target position (storage location 001 on the second floor of shelf 1 in area A). Similarly, the outbound instruction "Retrieve bin 2 from storage location 003 on the fifth floor of shelf 2 in area B and deliver it to workstation 1" is an atomic outbound instruction, explicitly specifying the goods (bin 2), its storage location (storage location 003 on the fifth floor of shelf 2 in area B), and the target position (workstation 1). Other examples include the inventory management instruction "Move the bin on storage location 001 on the second floor of shelf 1 to storage location 003 on the fifth floor of shelf 2," and the charging instruction "Robot 1, go to charge," etc.

[0040] Composite instructions consist of multiple atomic instructions, used to implement more complex and complete warehousing business processes. For example, the outbound-charging composite instruction "Robot 1 moves the toy box from shelf 1, second shelf, location 001 to workstation 1 and then goes to charge it." This composite instruction includes the operations of two atomic instructions, and these two atomic instructions have an execution order. First, there is the outbound operation: Robot 1 moves the toy box from shelf 1, second shelf, location 001 to workstation 1; then, there is the charging operation: Robot 1 goes to the charging location to charge.

[0041] Traditional GUI interactions are inefficient, especially for complex commands, requiring users to click multiple times to select and issue them. Warehouse PCs are often small, and intelligent warehousing systems are feature-rich; operating on such small screens results in a poor user experience and is prone to errors. Users sometimes issue unstructured commands in natural language format. These commands lack strict adherence to fixed syntax, format, and semantic rules, and are often unstandardized and ambiguous. Traditional GUI interactions lack semantic understanding and cannot parse unstructured commands.

[0042] The software systems within an intelligent warehousing system include an Intelligent Equipment Scheduling System (ESS) and various business systems. These business systems include an Order Management System (OMS), a Warehouse Management System (WMS), a Warehouse Control System (WCS), and a Warehouse Execution System (WES). Each of these systems has a separate entry point, requiring users to operate each system individually and frequently switch between them.

[0043] In summary, while traditional GUI-based interaction modes can achieve a certain degree of warehouse automation, they suffer from problems such as cumbersome operation processes, low efficiency, error-proneness, and inability to parse structured instructions. They are ill-suited to the growing demand for efficient and convenient interaction and the flexibility requirements of business processes in warehouse scenarios.

[0044] Developing the interaction technology of intelligent warehousing systems from traditional GUIs to natural language interaction is a possible direction. In related technologies, some solutions in intelligent warehousing systems based on natural language interaction simplify voice commands, supporting fixed phrase recognition such as "stop" and "charge," but still unable to handle context-dependent complex commands. Some solutions utilize large models to achieve preliminary conversion from natural language to commands, but can only complete single-command conversions, failing to consider the multi-action dependency breakdown and necessary operation completion in warehousing scenarios. For example, for the input "charge," the large model conversion only yields the "charge" command. However, if the robot is currently carrying a toy box and performing a handling task, the robot actually needs to cancel the current task and return the toy box to the warehouse (i.e., return the box) before performing the charging operation. The correct converted command should be "cancel task → return box → charge."

[0045] Therefore, how to accurately parse compound instructions is an urgent problem to be solved.

[0046] This application provides a robot control method. Figure 1 A schematic diagram illustrating an application scenario of an embodiment of this application is shown. For example... Figure 1 As shown, the warehousing system includes a human-machine interface terminal 10, a server 20, and a robot 30. The server 20 is communicatively connected to both the human-machine interface terminal 10 and the robot 30.

[0047] The human-machine interface terminal 10 is used to receive multimodal command information input by the user and send the multimodal command information to the server 20. The human-machine interface terminal 10 can receive multimodal command information input by the user through a GUI. The human-machine interface terminal can be a smartphone, tablet, industrial control terminal, or other device equipped with various types of input devices, such as a touchscreen for receiving finger touch input, a microphone for receiving voice commands, a camera for capturing gesture commands, capturing facial expression commands, and taking photos of the scene. The human-machine interface terminal can be fixedly installed in some locations in a warehouse or can be a mobile handheld terminal.

[0048] Server 20 is equipped with software systems such as ESS, OMS, WMS, WCS, and WES, and also deploys a multimodal large model. The multimodal large model parses multimodal instructions to obtain instructions to be executed, and calls the application programming interfaces (APIs) provided by each business system to control the robot to execute the instructions to be executed.

[0049] The robot responds to the API call from the server and executes the pending instructions, such as bin handling or charging operations. The robot can be any type of material handling robot, and this application does not limit its application to this type.

[0050] Figure 2 This application provides an embodiment of a robot control method, which illustrates the interactive flow diagram. Figure 2 As shown, the robot control method includes the following steps:

[0051] S101, the human-computer interaction terminal receives multimodal command information input by the user.

[0052] Multimodal instruction information refers to instructions that contain data in multiple modalities, including text, images, voice, and video. Unlike plain text instructions, multimodal instructions integrate data from multiple modalities, enabling the expression of more complex tasks. Human-computer interaction (HCI) terminals have input devices such as touchscreens, microphones, and cameras, which can receive multimodal instruction information from users. For example, a user can input a voice command, "Move all the boxes on shelf 2 in area A to the shelf shown in the diagram," through the HCI terminal's microphone, or take a photo through the HCI terminal's camera, the photo containing shelf number L1 or a graphic code indicating shelf number L1, thus inputting a multimodal instruction into the HCI terminal.

[0053] S102, the human-computer interaction terminal sends multimodal instruction information to the server.

[0054] The human-computer interaction terminal can send multimodal instruction information to the server through a wired or wireless connection, and the server can obtain the multimodal instruction information.

[0055] S103, the server obtains the current environment information from the system data of the warehousing system.

[0056] The warehousing system stores data including inventory information, goods receiving and shipping information, entity information, user information (i.e., operator information), historical operating data and statistical reports, current task information, and early warning and alarm information. Entity information includes shelf information, bin information, workstation information, and robot information. Shelf information and workstation information can also be aggregated in the warehousing system's map information.

[0057] Upon receiving the multimodal instruction, the server retrieves the current environment information configured for default full reading from the warehouse system's system data. This includes information such as shelf information (shelf location, storage location status, etc.), bin information (bin distribution), workstation information (workstation location, function, load, etc.), robot information (number of robots, status, location, battery level, etc.), and current task information (completed tasks, ongoing tasks, pending tasks, etc.). "Default full reading" means that when reading this type of information, no distinction is made; all information within that type is read directly.

[0058] S104, the server inputs multimodal instruction information and current environment information into the multimodal large model.

[0059] The server is deployed with pre-trained multimodal large models. A multimodal large model is an artificial intelligence model capable of understanding and processing various types of data, including text, images, audio, and video. Compared to traditional single-modal models, multimodal large models can capture information more comprehensively, possess stronger integrated understanding capabilities, a wider range of applicability, more efficient resource utilization, and more natural interaction methods, and are better able to understand natural language commands. Multimodal large models can be GPT-4, Claude 3, Gemini, and other similar large models.

[0060] The multimodal large model deployed on the server is trained based on the basic information of the warehousing system, which includes the entity information of the warehousing system and the pre-determined dependencies of instructions.

[0061] The entity information of the warehousing system includes shelf information, bin information, workstation information, robot information, etc. By learning the entity information of the warehousing system, the multimodal large model understands which entities are supported by the scenario of this warehousing system, so that it can subsequently identify entities belonging to this warehousing system in the multimodal instruction information, thereby parsing out the instructions to be executed.

[0062] Instruction pre-dependency refers to the requirement that the execution of certain instructions depends on the execution results or states of other instructions, or that the execution of certain instructions depends on the fulfillment of specific conditions. For example, for a robot to recharge, the specific condition that there are no extra material bins on the robot is required. By learning instruction pre-dependencies, a multimodal large model can infer the necessary supplementary pre-instructions based on the multimodal instruction information input by the user, thereby obtaining complete and executable instructions to be executed, and achieving the correct completion of complex tasks.

[0063] Instruction dependencies can be described using a Directed Acyclic Graph (DAG). When training a multimodal large model, atomic instructions and the relationships between them are input into the model using a DAG description, enabling the model to understand the dependencies between different instructions. For example, A->B means that instruction B depends on instruction A.

[0064] S105, the server parses the multimodal instruction information based on the current environment information using the multimodal large model to obtain the target entity and user instruction indicated by the multimodal instruction information.

[0065] The multimodal large model parses multimodal instruction information based on received environmental information such as shelf information, bin information, workstation information, robot information, and current task information to obtain the target entity and user instruction (i.e., user intent) indicated by the multimodal instruction information. Step S105 can be further implemented in the following way:

[0066] The multimodal large model parses multimodal command information based on current environment information to obtain the entity description of the target entity indicated by the multimodal command information and the user command. The entity description of the target entity refers to the original description of the target entity in the user-input multimodal command information. This original description may directly contain an entity identifier (i.e., entity ID) or may not. Each entity corresponds to a unique entity identifier, which can be used to identify a specific entity. Then, it is determined whether the entity description of the target entity contains an entity identifier. If the entity description of the target entity obtained after parsing the multimodal command information by the multimodal large model does not contain an entity identifier, the multimodal large model queries the system data for the entity identifier of the target entity and then updates the entity description of the target entity using the entity identifier, so that the entity description contains the entity identifier. If the entity description of the target entity obtained after parsing the multimodal command information by the multimodal large model contains an entity identifier, then there is no need to update the entity description of the target entity. Since the parameters of the user command include the target entity, the user command can only be executed if the entity description of the target command contains an entity identifier. By using the above methods, we can ensure that the entity description of the target entity contains the entity identifier of the target entity, so that user commands can be executed.

[0067] The basic information used in the pre-training of the multimodal large model also includes atomic instructions. When querying the entity identifier of a target entity from the system data, the multimodal large model can call the query instruction in the atomic instructions to retrieve the entity identifier of the target entity from the system data. The parameter of the query instruction is the entity information contained in the entity description of the target entity.

[0068] For example, a user inputs the voice command "Move all the boxes on shelf 2 in zone A to the shelf shown in the diagram," and takes a photo containing shelf number L1. Based on its understanding of the user's multimodal command information and pre-trained content, the multimodal large model can extract the entity description of the target entity and the user command from the multimodal command information. The entity description of the target entity includes: Box (X): All boxes on shelf 2 in zone A; Optional destination storage group (Y): All storage locations on shelf L1. The user command includes "move(X,Y)". Next, it determines whether the entity description of the target entity contains an entity identifier for further extraction and verification. For the target entity Box (X), its entity description "All boxes on shelf 2 in zone A" does not contain an entity identifier. Therefore, entity extraction is performed on this entity description, and the multimodal large model converts this description into a query command, such as the query command "get_totes(zone=A,rack=2)", where the parameters zone and rack are the entity information "A(zone)" and "(shelf)2" contained in the entity description. The aforementioned user commands and query commands were learned during the pre-training process of the multimodal large model. If the multimodal large model cannot find a specific command during parsing, an exception handling process is initiated, prompting the user to refine the command or cancel the task. After calling the query command, all box identifiers {B1, B2, B3, ...} for shelf 2 in area A are obtained. The target entity of the optional destination storage location group (Y) is extracted in the same way to obtain the specific storage location numbers {P1, P2, P3, ...}.

[0069] S106, the server retrieves the associated system data related to the target entity from the system data and inputs the associated system data into the multimodal large model.

[0070] After obtaining the target entity through step S105, the server can retrieve related system data from the system data based on the entity description of the target entity. This related system data includes information such as the target entity's current state and current task. For example, if the target entity is robot number 1, the retrieved related system data includes robot number 1's location, battery level, currently executing task, and its status over the past 10 minutes. After retrieving the related system data, this data is input into the multimodal large model. The related system data of the target entity can provide a more complete context for the multimodal large model.

[0071] S107, the server uses a multimodal large model to perform dependency verification on user instructions based on current environment information, related system data, and instruction pre-dependencies. Once the verification is successful, the instruction to be executed is obtained.

[0072] When performing dependency validation on user instructions, the validation can be performed based on the current environment information, associated system data, and the instruction's prerequisite dependencies. If the dependency validation passes, an execution instruction containing the user instruction is obtained. If the dependency validation fails, supplementary instructions are added to the user instruction until the dependency validation passes, resulting in an execution instruction containing both the supplementary instructions and the user instruction.

[0073] When performing dependency validation on user commands, it is determined whether the current state satisfies the pre-dependency relationship with the corresponding user command. If the current state satisfies the pre-dependency relationship, the dependency validation of the user command is considered successful. If the current state does not satisfy the pre-dependency relationship, the dependency validation of the user command is considered failed.

[0074] Based on the current environment information obtained in step S103 and the associated system data related to the target entity retrieved in step S106, the multimodal large model determines whether the current state satisfies the instruction pre-dependency relationship corresponding to the user instruction based on the instruction pre-dependency relationship learned during pre-training.

[0075] For example, for the user instruction "move(X,Y)" obtained from the above parsing, the multimodal large model incorporates the entity identifier bins (X){B1,B2,B3,…} of the target entity and the optional destination storage locations (Y){P1,P2,P3,…} into the instruction "move" to perform dependency checks, determining whether the instruction "move" can be executed for the inputs X and Y in the current state. During pre-training, the multimodal large model learns that executing the "move" instruction requires "the existence of a robot with available battery power that is not low on battery, and a sufficient number of destination storage locations that are not occupied."

[0076] If the robot currently has a battery level that is not low, and all of the target storage locations (Y){P1,P2,P3,…} are unoccupied and the number of storage locations is greater than or equal to the number of boxes (X), then the dependency verification passes, and the resulting instruction to be executed contains the user instruction, which is “move(X,Y)”.

[0077] If the multimodal large model finds that the only idle robot in the entire field is robot 1, and robot 1 is in a low-battery state, the move command cannot be executed, then the dependency check fails. In the case of a failed dependency check, the multimodal large model, based on the current environment information, associated system data, and command pre-dependencies, determines the supplementary command that must be executed before the user command; it adds the supplementary command to the user command, and the execution order of the supplementary command is before the user command; the dependency check is performed again on the user command after the addition of the supplementary command; if the dependency check passes, the supplementary command and the user command are determined as commands to be executed.

[0078] Continuing with the user instruction "move(X,Y)" obtained from the above analysis as an example, to execute the move instruction, the charging task of robot 1 must be completed first. Therefore, the charging instruction is a supplementary instruction. A supplementary instruction "charge(R)" is added to the user instruction "move(X,Y)". "charge(R)" must be executed before "move(X,Y)", where R represents robot 1. A dependency check is performed again on "charge(R)→move(X,Y)". It is confirmed that there are robots with power levels not low, and that all selectable destination storage groups (Y){P1,P2,P3,…} are unoccupied and the number of storage locations is greater than or equal to the number of boxes (X). If the dependency check passes, the resulting instruction to be executed includes the supplementary instruction and the user instruction. The instruction to be executed is "charge(R)→move(X,Y)".

[0079] For example, the user instruction "robot go to charge" depends on the premise that "the robot has no material box on it". If the robot is currently moving a material box, the multimodal model needs to add a supplementary instruction "put down the material box" or "place the material box in an empty storage location" to the user instruction "robot go to charge" based on the instruction's prerequisite dependency. The final instruction to be executed is "put down the material box" / "place the material box in an empty storage location" → "robot go to charge".

[0080] Furthermore, dependency validation for user commands includes both static and dynamic validation. Static validation involves determining whether the current state satisfies the pre-dependencies of the corresponding user command. Static validation can also include syntax verification (verifying whether command parameters conform to specifications), permission verification (verifying whether the user role has the necessary permissions), and other reasonable checks. Dynamic validation can involve performing a Directed Acyclic Graph (DAG) simulation in a sandbox environment to determine if the DAG can be successfully navigated.

[0081] S108, the server calls the API corresponding to the instruction to be executed.

[0082] The server controls the robot to execute instructions by calling the API corresponding to the instructions to be executed. Related technologies require manual user intervention for cross-business system instruction coordination. For example, after parsing a user instruction to obtain the "return box" instruction, the user needs to manually operate on the human-machine interface terminal to trigger the WMS interface. This application's embodiment achieves cross-business system instruction coordination by generating instruction chains. The server generates an instruction chain based on at least one supplementary instruction and the user instruction, as well as the instruction execution order, to sequentially call multiple APIs corresponding to at least one supplementary instruction and the user instruction. The instruction chain is then executed, thereby calling the API corresponding to the instruction to be executed. The instruction chain can be expressed in formats such as JSON, XML, YAML, and MCP. Through this method, automatic API calls from various business systems are achieved, unifying the operation entry point for multiple business systems. Users do not need to be aware of multiple business systems, nor do they need to frequently switch between business systems, reducing tedious and erroneous operations caused by manual intervention, and realizing cross-business system instruction coordination.

[0083] S109, the robot executes the instruction to be executed.

[0084] After receiving the instruction to be executed, the robot executes the instruction.

[0085] This application embodiment obtains the target entity and user instructions by fusing multimodal instruction information through a multimodal large model. It retrieves related system data concerning the target entity from system data and performs dependency verification on the user instructions based on the acquired current environment information, the retrieved related system data, and the instruction pre-dependencies learned during pre-training. If the verification passes, the resulting instruction to be executed satisfies the instruction pre-dependencies. Through this method, this application embodiment can understand and decompose unstructured user instructions, achieving rapid and accurate parsing of multimodal composite instructions, improving operational efficiency in the warehousing system, and avoiding errors caused by multiple layer-by-layer operations by the user, thus reducing the error rate.

[0086] In some embodiments, before the server calls the API corresponding to the instruction to be executed (S108), the server also outputs the instruction to be executed to the user. If the user confirms the instruction to be executed, the server responds to the user's confirmation operation and executes step S108. This method allows the user to confirm the instruction to be executed, and the robot is only controlled to execute the instruction after the user confirms it, thus preventing instructions that should not be executed from being executed directly, such as canceling or deleting a task.

[0087] The execution result of robot instructions includes success and failure. For example, a user might want to pick up a bin and deliver it to a certain location, but the robot fails to deliver it after picking it up, meaning the task of delivering the bin to the designated location fails. The server needs to handle this exception. In some embodiments, when the instruction to be executed includes at least one supplementary instruction and a user instruction executed sequentially, after the server calls the API corresponding to the instruction to be executed to control the robot to execute the instruction, the server obtains the execution result of the current instruction after the robot executes each instruction. If the execution result is failure, the server generates a rollback instruction corresponding to the preceding instruction. The rollback instruction is used to restore the state before the execution of the preceding instruction. Finally, the server calls the API corresponding to the rollback instruction to control the robot to execute the rollback instruction. The preceding instruction can be one or more instructions. For example, the instructions to be executed include three atomic instructions A→B→C. If the robot fails to execute the current instruction B, the preceding instruction before the current instruction B is instruction A. At this time, the preceding instruction is one instruction, and the rollback instruction is used to restore the state before the execution of the preceding instruction A. If the robot fails to execute the current instruction C, the preceding instructions before the current instruction C include instructions A and B. At this time, the preceding instruction is two instructions, and the rollback instruction will correspondingly contain two instructions. The first rollback instruction is used to restore the state before the execution of the preceding instruction B, and the second rollback instruction is used to restore the state before the execution of the preceding instruction A. Through the two rollback instructions, the state before the execution of the instructions to be executed is gradually restored.

[0088] If any instruction in the pending instructions fails to execute, then the instructions executed before that instruction are also meaningless. In this way, when the robot fails to execute an instruction in the pending instructions, it automatically reverts to the initial state before the execution of that instruction, thus achieving exception handling when instruction execution fails. For example, in the case above, if the robot cannot deliver the bin to the designated location after retrieving it, a rollback command can be used to control the robot to return the bin to its original location.

[0089] Each time the robot executes an instruction, it sends the result to the server. The server can then update the status of the software system corresponding to the result and provide feedback to the user, for example, by outputting the result through a human-computer interaction interface.

[0090] The robot control method of this application embodiment is described in detail below with a more specific flowchart. Figure 3 A flowchart illustrating a robot control method provided in an embodiment of this application is shown. This method is applied to the aforementioned server. Figure 3 As shown, the robot control method includes the following steps:

[0091] S201 acquires multimodal instruction information input by the user through a human-computer interaction terminal.

[0092] S202, Obtain current environmental information from the system data of the warehousing system.

[0093] S203 inputs multimodal command information and current environment information into the multimodal large model.

[0094] The multimodal large model is a model trained based on the basic information of the warehousing system, which includes the entity information of the warehousing system and the pre-determined dependencies of instructions.

[0095] S204: The multimodal large model parses the multimodal instruction information based on the current environment information to obtain the entity description of the target entity indicated by the multimodal instruction information and the user instruction.

[0096] S205, determine whether the entity description of the target entity contains an entity identifier. If yes, proceed to step S208; otherwise, proceed to step S206.

[0097] S206, query the entity identifier of the target entity from the system data through a multimodal large model.

[0098] S207, Update the entity description of the target entity using the entity identifier of the target entity.

[0099] S208, retrieve the associated system data related to the target entity from the system data, and input the associated system data into the multimodal large model.

[0100] S209 uses a multimodal large model to perform dependency verification on user commands based on current environmental information, associated system data, and command pre-dependencies.

[0101] S210, determine whether the dependency validation passes. If yes, proceed to step S213; otherwise, proceed to step S211.

[0102] S211, based on the current environment information, associated system data, and instruction prerequisite dependencies, determine the supplementary instructions that must be executed before executing the user instruction.

[0103] S212, add supplementary instructions to the user command, and return to execution step S209.

[0104] The execution order of supplementary instructions precedes that of user instructions. When executing step S209 after adding supplementary instructions to user instructions, a dependency check is performed on the user instructions following the addition of supplementary instructions.

[0105] The above process is repeated several times until the dependency validation passes. If the dependency validation fails after a preset time or a preset number of iterations, the entire process is terminated and an error message is returned to the user.

[0106] S213, the instruction to be executed is obtained.

[0107] If the dependency check passes on the first attempt, the resulting instruction to be executed is the initial user instruction. If the dependency check passes only after instruction supplementation, the instruction to be executed includes both supplementary instructions and user instructions.

[0108] S214, Based on the instructions to be executed and the order of instruction execution, generate an instruction chain for sequentially calling multiple APIs corresponding to each instruction in the instructions to be executed.

[0109] If the instruction to be executed is an initial user instruction, the instruction chain is used to call the API corresponding to that initial user instruction. If the instruction to be executed contains at least one supplementary instruction and a user instruction, the server generates an instruction chain based on the at least one supplementary instruction and the user instruction, and the order in which the instructions are executed, to sequentially call multiple APIs corresponding to the at least one supplementary instruction and the user instruction, respectively.

[0110] S215 executes the instruction chain, thereby calling the API corresponding to the instruction to be executed, in order to control the robot to execute the instruction.

[0111] Figure 4 A schematic diagram of the robot control device provided in an embodiment of this application is shown. Figure 4 As shown, the robot control device 400 includes:

[0112] The first acquisition module 401 is used to acquire multimodal instruction information input by the user through a human-computer interaction terminal;

[0113] The second acquisition module 402 is used to acquire current environmental information from the system data of the warehousing system;

[0114] The input module 403 is used to input multimodal instruction information and current environment information into the multimodal large model. The multimodal large model is a model trained based on the basic information of the warehousing system. The basic information includes the entity information of the warehousing system, atomic instructions, and instruction pre-dependencies.

[0115] The parsing module 404 is used to parse the multimodal instruction information based on the current environment information through the multimodal large model to obtain the target entity and user instruction indicated by the multimodal instruction information;

[0116] The retrieval module 405 is used to retrieve associated system data related to the target entity from the system data and input the associated system data into the multimodal large model.

[0117] The verification module 406 is used to perform dependency verification on user instructions based on the current environment information, related system data and instruction pre-dependencies using a multimodal large model. Once the verification is successful, the instruction to be executed is obtained.

[0118] Module 407 is used to call the API corresponding to the instruction to be executed, so as to control the robot to execute the instruction.

[0119] The robot control device 400 of this application embodiment also includes other modules for performing the steps of the above method embodiments, which will not be described in detail here.

[0120] Figure 5 A schematic diagram of the server structure provided in an embodiment of this application is shown. Figure 5 As shown, the server 500 may include a processor 502 and a memory 504.

[0121] The processor 502 is used to execute the computer program 506, which can specifically perform the relevant steps described above in the robot control method embodiment.

[0122] Specifically, computer program 506 may include computer-executable instructions.

[0123] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The server includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0124] Memory 504 is used to store computer program 506. Memory 504 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0125] This application provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is run on a server, it causes the server to perform the operation of the robot control method as described in the above embodiment.

[0126] This application provides a computer program product, including a computer program that, when executed by a processor, implements the robot control method described in the above embodiment.

[0127] This application provides a computer program that can be called by a processor to cause a server to perform the robot control method described in the above embodiment.

[0128] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0129] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0130] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various aspects of the invention, features of the embodiments of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of this application. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim.

[0131] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0132] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A robot control method, characterized in that, The method includes: The system acquires multimodal command information input by the user through a human-computer interaction terminal. Obtain current environmental information from the system data of the warehousing system; The multimodal instruction information and the current environment information are input into the multimodal large model, which is a model trained based on the basic information of the warehousing system. The basic information includes the entity information of the warehousing system and the instruction pre-dependencies. The multimodal big model parses the multimodal instruction information based on the current environment information to obtain the target entity and user instruction indicated by the multimodal instruction information; Retrieve associated system data related to the target entity from the system data, and input the associated system data into the multimodal large model; The multimodal large model performs dependency verification on the user instruction based on the current environment information, the associated system data, and the instruction pre-dependency relationship. Once the verification is successful, the instruction to be executed is obtained. The application programming interface corresponding to the instruction to be executed is invoked to control the robot to execute the instruction.

2. The method according to claim 1, characterized in that, The step of parsing the multimodal instruction information based on the current environment information using the multimodal large model to obtain the target entity and user instruction indicated by the multimodal instruction information includes: The multimodal large model parses the multimodal instruction information based on the current environment information to obtain the entity description of the target entity indicated by the multimodal instruction information and the user instruction; Determine whether the entity description of the target entity contains an entity identifier; If the entity description of the target entity does not contain the entity identifier, the entity identifier of the target entity is queried from the system data through the multimodal large model; Update the entity description of the target entity using the entity identifier of the target entity.

3. The method according to claim 2, characterized in that, The basic information also includes atomic instructions, and the step of querying the entity identifier of the target entity from the system data through the multimodal large model includes: The multimodal large model invokes a query instruction to query the entity identifier of the target entity from the system data. The query instruction is an instruction in the atomic instruction, and the parameter of the query instruction is the entity information contained in the entity description of the target entity. The atomic instruction refers to the minimal, indivisible instruction.

4. The method according to claim 1, characterized in that, The process of obtaining current environmental information from the system data of the warehousing system includes: The current environment information, configured to be read in full by default, is obtained from the system data of the warehousing system. The current environment information includes shelf information, bin information, workstation information, robot information, and current task information.

5. The method according to claim 1, characterized in that, The process involves performing dependency verification on the user instruction based on the current environment information, the associated system data, and the instruction's pre-dependencies. Upon successful verification, an instruction to be executed is obtained, including: Based on the current environment information, the associated system data, and the instruction prerequisite dependencies, the user instructions are subject to dependency verification. If the dependency verification passes, the instruction to be executed, containing the user's instructions, is obtained; If the dependency verification fails, a supplementary instruction is added to the user instruction until the dependency verification passes, resulting in the execution instruction containing the supplementary instruction and the user instruction.

6. The method according to claim 5, characterized in that, The step of performing dependency verification on the user command based on the current environment information, the associated system data, and the command prerequisite dependencies includes: Based on the current environment information and the associated system data, determine whether the current state satisfies the instruction prerequisite dependency relationship corresponding to the user instruction; If the current state satisfies the instruction prerequisite dependency relationship corresponding to the user instruction, it is determined that the dependency verification of the user instruction has passed; If the current state does not satisfy the instruction prerequisite dependency relationship corresponding to the user instruction, it is determined that the dependency verification of the user instruction has failed.

7. The method according to claim 5, characterized in that, If the dependency verification fails, a supplementary instruction is added to the user instruction until the dependency verification passes, resulting in the execution instruction containing the supplementary instruction and the user instruction, including: Based on the current environment information, the associated system data, and the instruction prerequisite dependencies, determine the supplementary instructions that must be executed before executing the user instruction; The supplementary instruction is added to the user instruction, and the execution order of the supplementary instruction is before the user instruction; Perform the dependency verification on the user instructions after adding the supplementary instructions; If the dependency verification passes, the supplementary instruction and the user instruction are identified as the instruction to be executed.

8. The method according to claim 5, characterized in that, The instructions to be executed include at least one supplementary instruction to be executed sequentially and the user instruction. The step of calling the application programming interface corresponding to the instructions to be executed to control the robot to execute the instructions includes: Based on the at least one supplementary instruction and the user instruction, and the order of instruction execution, an instruction chain is generated for sequentially calling multiple application programming interfaces corresponding to the at least one supplementary instruction and the user instruction, respectively. The instruction chain is executed, thereby invoking the application programming interface corresponding to the instruction to be executed, so as to control the robot to execute the instruction to be executed.

9. The method according to claim 1, characterized in that, Before invoking the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction, the method further includes: Output the instruction to be executed to the user; In response to the user's confirmation of the instruction to be executed, the step of calling the application programming interface corresponding to the instruction to be executed to control the robot to execute the instruction to be executed is performed.

10. The method according to claim 1, characterized in that, The instructions to be executed include at least one supplementary instruction executed sequentially and the user instruction. After invoking the application programming interface corresponding to the instructions to be executed to control the robot to execute the instructions, the method further includes: After the robot executes each instruction, it obtains the execution result of the current instruction, which includes execution success and execution failure. If the execution result is an execution failure, a rollback instruction corresponding to the preceding instruction before the current instruction is generated, and the rollback instruction is used to restore the state before the execution of the preceding instruction. The application programming interface corresponding to the rollback instruction is invoked to control the robot to execute the rollback instruction.

11. A robot control device, characterized in that, The device includes: The first acquisition module is used to acquire multimodal instruction information input by the user through a human-computer interaction terminal; The second acquisition module is used to acquire current environmental information from the system data of the warehousing system; The input module is used to input the multimodal instruction information and the current environment information into the multimodal large model. The multimodal large model is a model trained based on the basic information of the warehousing system. The basic information includes the entity information, atomic instructions, and instruction pre-dependencies of the warehousing system. The parsing module is used to parse the multimodal instruction information based on the current environment information using the multimodal large model to obtain the target entity and user instruction indicated by the multimodal instruction information; The retrieval module is used to retrieve associated system data related to the target entity from the system data and input the associated system data into the multimodal large model; The verification module is used to perform dependency verification on the user instruction based on the current environment information, the associated system data and the instruction pre-dependency relationship using the multimodal large model. After the verification is passed, the instruction to be executed is obtained. The calling module is used to call the application programming interface corresponding to the instruction to be executed, so as to control the robot to execute the instruction to be executed.

12. A server, characterized in that, include: A processor and a memory, wherein the memory stores executable instructions, and the processor is capable of executing the executable instructions to implement the robot control method as described in any one of claims 1-9.

13. A warehousing system, comprising a human-computer interaction terminal, a server, and a robot; The human-computer interaction terminal is used to receive multimodal instruction information input by the user and send the multimodal instruction information to the server; The server is used to execute the robot control method as described in any one of claims 1-10; The robot is used to execute the pending instruction issued by the server through the application programming interface.

14. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, which, when executed on the server, cause the server to perform the robot control method as described in any one of claims 1-10.

Images