Robot control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a robot control method and device, electronic equipment and a storage medium, and belongs to the technical field of logistics. The method comprises the steps of obtaining a logistics task text; according to the logistics task text, a pre-constructed topological map is called to perform node matching query, target pose data corresponding to the logistics task text is obtained, the topological map comprises a plurality of nodes, and different nodes are connected according to a path reachable relationship; each node comprises pose data of the robot at a corresponding logistics site position and a site description text of the logistics site when the topological map is constructed; and determining the current operation state and the current pose data of the robot, and performing motion control on the robot according to the current operation state, the current pose data and the target pose data. According to the embodiment of the invention, the robot control flexibility can be improved.

Description

Technical Field

[0001] This application relates to the field of logistics technology, and in particular to a robot control method and device, electronic equipment and storage medium. Background Technology

[0002] With the rapid development of e-commerce and smart logistics, the demand for automation and intelligence in warehousing and logistics centers and transit hubs is becoming increasingly urgent.

[0003] Currently, autonomous mobile robots (AMRs) have become key equipment for improving logistics efficiency. In related technologies, robot navigation often relies on pre-laid magnetic strips, QR codes, or static maps based on laser SLAM. This approach has drawbacks such as fixed paths and an inability to understand high-level task instructions, thus affecting the flexibility of robot control. Summary of the Invention

[0004] The main objective of this application is to provide a robot control method, apparatus, electronic device, and storage medium that can improve the flexibility of robot control.

[0005] To achieve the above objectives, a first aspect of this application proposes a robot control method, the method comprising: Obtain the logistics task text; The pre-built topology map is invoked based on the logistics task text to perform node matching query, thereby obtaining the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to path reachability. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map was constructed, as well as the site description text of the logistics site. The robot's current operating state and current pose data are determined, and motion control of the robot is performed based on the current operating state, the current pose data, and the target pose data.

[0006] To achieve the above objectives, a second aspect of this application provides a robot control device, the device comprising: The task acquisition unit is used to acquire logistics task text. The node query unit is used to call a pre-built topology map to perform node matching query based on the logistics task text to obtain the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to path reachability. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map was constructed, as well as the site description text of the logistics site. The robot control unit is used to determine the robot's current operating state and current pose data, and to perform motion control on the robot based on the current operating state, the current pose data, and the target pose data.

[0007] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any one of the embodiments of the first aspect.

[0008] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the embodiments of the first aspect.

[0009] The robot control method, device, electronic device, and storage medium proposed in this application acquire logistics task text rich in semantic information and perform node matching queries using a pre-constructed topological map that integrates pose and scene description text. This achieves accurate semantic parsing and target localization of complex logistics instructions, effectively overcoming the problem of poor flexibility caused by the inability of traditional navigation systems to understand high-level abstract instructions and fixed paths. In other words, the embodiments of this application can improve the flexibility of robot control. Attached Figure Description

[0010] Figure 1 This is a flowchart of an embodiment of the robot control method provided in this application; Figure 2 This is a schematic diagram of the system architecture of the robot control method provided in the embodiments of this application; Figure 3 yes Figure 1 A flowchart of an embodiment of step S120; Figure 4 This is a flowchart illustrating the upper-layer VLN robot navigation module provided in an embodiment of this application; Figure 5 This is a flowchart illustrating the lower-level robot control module provided in an embodiment of this application; Figure 6 This is a flowchart of an embodiment of the topology map construction method provided in this application; Figure 7 This is a flowchart illustrating the environment perception and modeling module provided in an embodiment of this application; Figure 8 This is a schematic diagram of a robot control device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0012] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., used in the specification, claims, and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0013] With the rapid development of e-commerce and smart logistics, the demand for automation and intelligence in warehousing and logistics centers and transit hubs is becoming increasingly urgent.

[0014] Currently, autonomous mobile robots (AMRs) have become key equipment for improving logistics efficiency. Among related technologies, robot navigation largely relies on pre-laid magnetic strips, QR codes, or static maps based on laser SLAM. This approach has the following drawbacks: 1. Fixed paths result in poor flexibility; 2. It cannot understand high-level, abstract task instructions (such as "pick up goods in area A and deliver them to the temporary storage point in area B"); 3. In dynamic, dense environments with multiple robots working collaboratively, congestion and deadlocks are prone to occur, leading to low scheduling efficiency.

[0015] Based on this, embodiments of this application provide a robot control method and apparatus, electronic device and storage medium, which can improve the flexibility of robot control.

[0016] The robot control method provided in this application relates to the field of logistics technology. The logistics transfer method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the robot control method, but is not limited to the above forms.

[0017] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0018] Please see Figure 1 , Figure 1 This is an optional flowchart of the robot control method provided in the embodiments of this application. In some embodiments of this application, Figure 1 The specific methods may include, but are not limited to, steps S110 to S130.

[0019] Step S110: Obtain the logistics task text; Step S120: Based on the logistics task text, call the pre-built topology map to perform node matching query to obtain the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to the path reachability relationship. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map is built, as well as the site description text of the logistics site. Step S130: Determine the robot's current running state and current pose data, and perform motion control on the robot based on the current running state, current pose data, and target pose data.

[0020] In step S110 of some embodiments, such as Figure 2 As shown, order information can be obtained by interfaceing with a Warehouse Management System (WMS) or Enterprise Resource Planning (ERP) system, or by directly obtaining task instructions sent by users. Order information (or task instructions) is usually in the form of structured data. Through a task parsing and instruction generation engine (such as a task management and scheduling center), order information (or task instructions) can be decomposed and assigned as tasks, and a precise natural language text instruction (i.e., logistics task text) can be generated based on each task. The logistics task text contains clear operational actions, target objects, and location information. For example, based on order information, the following logistics task text can be obtained: Robot_05, please retrieve 2 items SKU-123456 from shelf A1-05 in storage area, and then transport them to packing station P02.

[0021] In step S120 of some embodiments, a pre-built and continuously updated topology map can be used to parse and match the acquired logistics task text. The topology map can refer to an abstract representation of the logistics site, and may include multiple "nodes" and "edges" connecting these nodes. Nodes can represent key locations in the logistics site that the robot arrives at or passes through during the map-building phase. Edges can represent path reachability relationships between nodes (i.e., whether nodes are passable). Furthermore, each node contains the robot's pose data at that point during map building and site description text. The pose data can refer to quantified data of the robot's position and orientation, such as coordinates and orientation angles in the global coordinate system. The site description text can be a semantic description generated after understanding the site visual data collected by the robot at the node.

[0022] Thus, by matching the logistics task text with the site description text of all nodes in the topology map, the corresponding target node can be obtained. Finally, the target pose data, which serves as the navigation endpoint, can be extracted from the target node. It is understandable that the matching query operation can be performed by the upper-layer VLN robot navigation module, which can be deployed in the cloud for the robot's global localization.

[0023] Reference Figure 3 In some embodiments, step S120 may include, but is not limited to, steps S310 to S340.

[0024] Step S310: Obtain preset task prompt words, and call the preset visual language model to perform text analysis on the logistics task text based on the task prompt words to obtain the task target object; Step S320: Based on the task target object, call the topology map to perform node matching query, and determine the nodes corresponding to the site description text containing the task target object as candidate nodes; Step S330: For each candidate node, determine the preceding and following adjacent nodes of the candidate node in the topology map based on the edges connected to the candidate node; Step S340: Select the target node from the candidate nodes based on the site description text contained in the adjacent nodes, and determine the pose data contained in the target node as the target pose data.

[0025] In step S310 of some embodiments, such as Figure 4 As shown, the upper-layer VLN robot navigation module can include a navigation cue word manager and a visual language model. The navigation cue word manager can send cue words to the visual language model to guide it through step-by-step reasoning. These cue words include task cue words, which can refer to text modules or instruction sets used to guide and standardize the visual language model's task analysis during reasoning. For example, a task cue word might be: first, parse the core actions and target locations in the text; then, ignore irrelevant descriptive words; finally, output a standardized target location name. In this way, the visual language model can perform text analysis on the logistics task text based on the task cue words to obtain the task target object (i.e., the key object for performing the task). For example, the task target object might include "packing station P02" and "storage area A1-05 shelf".

[0026] In step S320 of some embodiments, after obtaining the task target object, the task target object can be used as a query keyword and semantically matched with the site description text carried by all nodes in the topology map. Thus, nodes in the topology map can be initially filtered based on semantic similarity, achieving coarse localization of the target object. The filtered nodes (such as nodes with semantic similarity greater than a preset threshold) are determined as candidate nodes.

[0027] In step S330 of some embodiments, the spatial structural relationships (such as "edges") of the topology map can be used to further filter the identified candidate nodes to determine the final target node. Specifically, for each candidate node, the network structure in the topology map is queried to find the two nodes directly connected to the candidate node via "edges" (i.e., adjacent nodes). Adjacent nodes include front adjacent nodes and back adjacent nodes. Front adjacent nodes and back adjacent nodes can refer to two nodes in the topology map that are directly connected to the candidate node in different directions. Adjacent nodes constitute the local spatial context of the corresponding candidate node, providing a reasoning basis for finally determining the target node.

[0028] In step S340 of some embodiments, spatial context information can be used to accurately identify the target node that truly corresponds to the task target object from multiple candidate nodes, thereby achieving fine localization. Specifically, for each candidate node, the site description text corresponding to the nodes before and after the candidate node can be obtained, and the site description text can be combined with the site description text of the candidate node itself to form a complete local environmental semantic scene. In this way, the combined scene can be comprehensively analyzed by a visual language model or semantic matching algorithm to verify whether the real environment in which the candidate node is located is consistent with the expected spatial logic of the task target object. For example, when the task target object is "shelf A1-05 in storage area", coarse localization may match multiple candidate nodes whose site description text contains the words "storage area". At this time, the fine localization process will check the nodes before and after each candidate node. If the site description text of the node before the first candidate node includes "located in the main passage of the storage area, with the sign A1 visible on the right", and the site description text of the node after the first candidate node includes "near the end of the main passage of the storage area". The site description text of the preceding adjacent node of the second candidate node includes "left aisle of shelf A1-04", and the site description text of the following adjacent node includes "right aisle of shelf A1-06". Through comparative analysis, it can be determined that the descriptions of the preceding and following adjacent nodes of the second candidate node form a continuous and coherent spatial sequence of "shelf A1-04 → shelf A1-05 → shelf A1-06". Therefore, the second candidate node can be identified as the target node. Finally, the pose data contained in this target node is used as the navigation endpoint (i.e., the target pose data).

[0029] It is understood that, in some embodiments, historical action analysis can also be used to determine the target node. For example, the historical target node determined when a task to "storage area A1-05 shelf" was executed within a certain period of time can be identified. In this way, the target node for executing the current task can be determined by combining the historical target node with the site description text of the adjacent nodes before and after each candidate node, thereby improving the accuracy of target node determination.

[0030] This application employs a two-stage node localization strategy of "preliminary semantic matching - fine-grained contextual filtering." First, it uses the task target object for coarse-grained candidate node filtering. Then, it comprehensively utilizes the local spatial context formed by the site description text of the candidate nodes and their adjacent nodes for verification. This allows the application to overcome the localization ambiguity problem that may arise from relying solely on single-point semantic matching. This localization mechanism significantly improves the accuracy and robustness of target localization in large-scale, highly similar logistics scenarios. It ensures that even when faced with multiple potential locations with similar descriptions, reliable decisions can be made by analyzing surrounding environmental information. This provides a precise navigation endpoint for lower-level motion control, improving the intelligence level of robot control and the success rate of task execution.

[0031] In step S130 of some embodiments, such as Figure 5 As shown, the current operating state can refer to relevant information obtained based on the robot's own sensors, such as movement speed, movement distance, and deflection angle. The robot can refer to autonomous mobile robots such as humanoid robots. The current pose data can refer to the robot's real-time position and orientation in the global coordinate system (i.e., the current node in the topology map). Thus, the target pose data can be used as the final navigation target, the current pose data as the starting point, and combined with the robot's dynamic constraints reflected in the current operating state to perform motion control on the robot. For example, the robot's motion control can be performed through a lower-level robot motion control module. This lower-level module can obtain global path guidance by querying the topology map, and perform local path planning based on this. Simultaneously, it integrates the robot's real-time operating state to achieve dynamic obstacle avoidance, ultimately generating safe and smooth control commands to drive the robot's movement. Local path planning refers to the process of dynamically determining a trajectory from the current node to nearby nodes that effectively avoids sudden obstacles and meets motion constraints, based on information about the robot's surrounding environment collected by its own sensors and sensors placed in the logistics area, as well as the node connection relationships provided by the topology map.

[0032] This application embodiment obtains logistics task text rich in semantic information and uses a pre-constructed topological map that integrates pose and scene description text for node matching query, thereby achieving accurate semantic parsing and target positioning of complex logistics instructions. This effectively overcomes the problem of poor flexibility caused by the inability of traditional navigation systems to understand high-level abstract instructions and fixed paths.

[0033] In some embodiments, the method for determining the robot's current pose data may include, but is not limited to, the following steps: Obtain preset environmental description prompts and the robot's current visual data; The visual language model is invoked to perform visual analysis on the robot's current visual data based on environmental description prompts, and the first visual description text is obtained. The topology map is invoked based on the first visual description text to perform node matching query. The node corresponding to the site description text that matches the visual description text is determined as the current node, and the pose data contained in the current node is determined as the current pose data.

[0034] In the embodiments of this application, such as Figure 4 As shown, the navigation cue word manager can also send environment description cue words, which can be used to guide and constrain the visual language model's content focus during inference. For example, environment description cue words could be: "Please describe the main objects, spatial structures, visible text labels, and relative positional relationships between objects in the current image." Furthermore, current visual data can be acquired through visual sensors mounted on the robot (such as RGB cameras, RGB-D cameras, etc.). Current visual data can refer to single-frame or multi-frame temporal images containing information about the scene in front of the robot.

[0035] In this way, the visual language model can be invoked to perform visual analysis on the current visual data based on the guidance of environmental description cues, identifying key elements within the data. For example, the visual language model might identify elements such as "shelf A-10 is directly in front," "sorting station P05 is one meter to the left," and "there are yellow guide lines on the ground." Based on these identified elements, the visual language model can also generate a structured first visual description text. This first visual description text can be an objective and detailed textual summary of the robot's current visual scene. Based on this first visual description text, raw, unstructured visual information can be transformed into semantic information that can be used for efficient querying and matching.

[0036] The generated first-view description text is used as a query condition to perform node matching queries on the global topology map. Specifically, the semantic similarity between the first-view description text and the site description text contained in each node in the topology map can be calculated. The node with the highest similarity is determined as the current node, and the pose data contained in the current node is determined as the current pose data. In this way, global localization of the robot's current pose is achieved.

[0037] This application's embodiments enable rapid and accurate global localization of a robot's pose in complex logistics environments without relying on GPS or pre-laid physical markers. Furthermore, by transforming the complex image matching problem into a more robust semantic text matching problem, the failure of traditional visual localization methods due to changes in lighting, perspective differences, or minor scene modifications can be reduced.

[0038] In some embodiments, the lower-level robot motion control module (hereinafter referred to as the motion control model) can be pre-trained based on reinforcement learning methods. Specifically, the training method for the motion control model may include, but is not limited to, the following steps: Acquire the robot's sample starting pose data, sample ending pose data, and sample running status in the simulation environment; The motion control model is invoked to control the robot's motion based on the sample's starting point pose data, the sample's ending point pose data, and the sample's running state. The control reward score is determined based on the motion control results and the preset reward function, and the motion control model is trained based on the control reward score.

[0039] In this embodiment, the simulation environment can refer to a virtual environment that models the physical characteristics and geometric layout of a real logistics site. The sample starting pose data can refer to the robot's initial position and posture information, randomly or strategically set for each training session in the simulation environment. Similarly, the sample ending pose data can refer to the position and posture information of the robot's target point. The sample starting pose data and sample ending pose data can be determined based on corresponding nodes in the topology map to ensure consistency between the training scenario and the real navigation task. The sample running state can refer to data characterizing the robot's dynamic state, which can be simulated by virtual sensors. Thus, the acquired sample starting pose data, sample ending pose data, and sample running state can be used as input data for the motion control model. The motion control model can be a deep neural network, which processes the input data and generates corresponding control commands. The control commands are sent to the robot model in the simulation environment for execution, driving the robot model to produce corresponding pose changes, thereby completing a motion control interaction.

[0040] After the motion control model outputs control commands and these commands are executed, the simulation environment provides feedback on the motion control results. These results can include the robot's new pose, whether a collision occurred, and whether the target was reached. The motion control results can then be quantitatively evaluated based on a reward function to obtain a control reward score. The reward function can include positive and negative rewards. For example, a positive reward is given when the robot gets closer to the target (i.e., the position indicated by the sample endpoint pose data), and a high positive reward is given when the target is successfully reached. A negative reward is applied when the robot collides, lingers for an extended period, or deviates from the global path planning. In this way, the network weight parameters of the motion control model can be updated based on the control reward score, enabling the model to make decisions that yield higher cumulative rewards in the future, thus iteratively developing the optimal motion control strategy.

[0041] This application's embodiments effectively overcome the shortcomings of traditional pre-programmed control methods, such as poor adaptability and rigid obstacle avoidance logic in dynamic and dense environments, by introducing a reinforcement learning training method based on a simulation environment into the lower-level robot motion control module. This endows the robot with powerful online decision-making and real-time obstacle avoidance capabilities, enabling it to smoothly, safely, and efficiently execute target points issued by the upper-level VLN robot navigation module. This significantly improves the robustness, adaptability, and overall operating efficiency in complex and realistic logistics scenarios.

[0042] In some embodiments, such as Figure 6 As shown, the method for constructing the topology map may include, but is not limited to, steps S610 to S650.

[0043] Step S610: In the map building stage, the robot is controlled to run multiple times in the logistics site according to the preset running data until the node creation stopping condition is met. Step S620: After each operation of the controlled robot, obtain the site vision data and robot pose of the robot at the corresponding logistics site location. Step S630: Call the preset visual language model to perform visual analysis on the site visual data to obtain site description text; Step S640: Perform attention encoding on the site description text and robot pose, and create nodes based on the encoding results; Step S650: Connect all the created nodes based on path reachability to obtain a topology map.

[0044] In step S610 of some embodiments, the map building phase can refer to a mode specifically set up for building an environmental model of the logistics site, distinct from the actual task execution mode. Preset operating data can refer to a sequence of instructions controlling the robot's movement. Preset operating data can be generated in the following ways: predefined global coverage paths, control instructions based on random exploration algorithms, operation records manually controlled by the operator, etc. Thus, the robot can be repeatedly driven to move within the logistics site according to the preset operating data until the node creation stopping condition is met. The node creation stopping condition can refer to the conditions used to determine when the map building process terminates. For example, it can include: stopping creation when the robot's cumulative mileage reaches a preset threshold, stopping creation when the rate of new node creation is lower than a preset threshold (indicating that the environment has been basically covered), or stopping creation when all key semantic areas (such as all shelving areas, workstations, etc.) are detected to have been represented by at least one node; no specific limitation is made here.

[0045] In step S620 of some embodiments, after each movement and pause of the robot based on preset operating data (this pause point is a potential node position), the site visual data and robot pose can be acquired through sensors mounted on the robot body. The site visual data can refer to images of the visual scene reflecting the robot's current location, obtained through RGB cameras or RGB-D cameras. The robot pose can refer to spatial state information including position and orientation, which can be acquired through odometry, inertial measurement units, laser SLAM (simultaneous localization and mapping), etc. This ensures that each data acquisition point simultaneously possesses visual appearance information and accurate spatial coordinates.

[0046] In step S630 of some embodiments, a visual language model can be invoked to perform visual analysis on site visual data based on cue words (such as environmental description cue words). This includes identifying elements in the site visual data such as objects (such as shelves, pallets, workstations, etc.), text (such as area labels, shelf numbers, etc.), and spatial structures (such as aisles, corners, etc.), as well as the relationships between these elements. Based on the recognition results, the visual language model can also generate a site description text in natural language format, such as "located at the intersection of a cross passage, with shelf label B2-08 directly in front."

[0047] In step S640 of some embodiments, such as Figure 7 As shown, attention encoding can be performed on the site description text and robot pose using the encoding module in the environment perception and modeling module. This allows for the fusion of the site description text and robot pose using an attention mechanism, generating a unified comprehensive feature vector rich in environmental semantics and spatial information. Then, nodes can be created based on this comprehensive feature vector. Each node is treated as a data structure, storing the comprehensive feature vector, site description text, robot pose, and site visual data together, becoming a basic element in the topology map.

[0048] Understandably, the above operations can be performed on all the locations the robot moves to, thereby creating multiple nodes.

[0049] In step S650 of some embodiments, after stopping node creation, the association between the created nodes can be established based on the robot's actual mobility. Path reachability refers to whether there is a physical path between two nodes that the robot can directly traverse. Path reachability can be determined by analyzing the robot's actual motion trajectory data during the map building phase. For example, if the robot can move directly from node A to node B, then a path reachability relationship exists between node A and node B. Thus, an edge can be established between every two nodes with a path reachability relationship to connect them, thereby connecting all the created discrete nodes into a connected network graph. The network graph constructed from nodes (representing corresponding positions, and the semantics and pose of the corresponding positions) and edges (representing connectivity relationships) is determined as a topology map. Therefore, a topology map can abstractly describe the spatial layout and connectivity of a logistics site.

[0050] This application embodiment constructs a hybrid topology map that is compatible with both geometric and high-level semantic information. This method reduces the shortcomings of traditional SLAM maps, which only contain geometric features and lack semantic understanding capabilities. It also reduces the problem of pure semantic maps lacking accurate spatial coordinate support for precise navigation. This provides a direct and efficient data foundation for the upper-level VLN robot navigation module to realize natural language-based instruction parsing and target query.

[0051] Understandably, the topology map can be continuously updated during the task execution phase. This update strategy is explained below.

[0052] In some embodiments, after step S130, the robot control method provided in this application may further include, but is not limited to, the following steps: During the motion control of the robot, the first running visual data collected by the robot at intermediate nodes on the path connecting the current node and the target node is acquired. The visual language model is invoked to perform visual analysis on the first running visual data based on environmental description prompts, and the second visual description text is obtained. Update the site description text of the intermediate nodes based on the second visual description text.

[0053] In this embodiment, when the lower-level robot motion control module controls the robot to move from the current node to the target node, the robot travels along the connected paths formed by edges in the topology map. Intermediate nodes can refer to other nodes on the travel path besides the current node and the target node. The first operational visual data can refer to scene images or video frames collected in real time by sensors mounted on the robot body when the robot passes through intermediate nodes. It is understood that, unlike the map building phase, the first operational visual data is collected synchronously during the robot's execution of logistics tasks, without interrupting the task to perform dedicated mapping operations, thus achieving invisible data collection.

[0054] After acquiring the first running visual data, a visual language model can be invoked to perform visual analysis on the first running visual data based on environmental description prompts, resulting in a second visual description text. This second visual description text can then be compared and analyzed with the original site description text contained in the corresponding intermediate node. For example, the semantic similarity or difference between the two description texts can be calculated. When the difference exceeds a preset threshold, it indicates that the visual scene of the intermediate node has changed significantly, and therefore the site description text needs to be replaced with the second visual description text. Alternatively, if the original site description text of the intermediate node is "End of the passage, blank wall on the right," and the new second visual description text is "End of the passage, new shelf installed on the right," then the description of the site description text can be updated using the second visual description text. This achieves online, incremental maintenance of the semantic content of the topology map, ensuring that the text description bound to each node in the map remains synchronized with the real environment without re-executing the time-consuming global mapping process. It is understood that in some embodiments, the above-mentioned site description text update operation can also be performed on the target node, which will not be elaborated further.

[0055] In some embodiments, after step S130, the robot control method provided in this application may further include, but is not limited to, the following steps: During the motion control of the robot, the obstacle avoidance position of the robot during obstacle avoidance is determined, and the second running visual data and obstacle avoidance pose of the robot at the obstacle avoidance position are acquired. The visual language model is invoked to perform visual analysis on the second running visual data based on environmental description prompts, and the third visual description text is obtained. Create new nodes based on the third-view description text and obstacle avoidance pose, and connect the new nodes to the nodes traversed before the obstacle avoidance operation.

[0056] In this embodiment, when the robot is performing a task, if the lower-level robot motion control module determines that there is a temporary obstacle (such as temporarily stacked goods) in front of the robot based on the sensors on the robot and / or the sensors in the logistics area, the lower-level robot motion control module can control the robot to perform obstacle avoidance. Obstacle avoidance can refer to the process by which the robot deviates from the originally calculated global path (i.e., the navigation path determined based on the current node, the target node, and the topology map) to bypass the obstacle. When the robot completes obstacle avoidance and enters a stable state, the stable position of the robot can be determined as the obstacle avoidance position. In addition, the second running visual data and obstacle avoidance pose of the obstacle avoidance position can also be obtained. The second running visual data can be data obtained based on the visual sensors on the robot, reflecting the visual scene image corresponding to the obstacle avoidance position. The obstacle avoidance pose can be obtained by fusing positioning data such as odometry and laser SLAM, and the obstacle avoidance pose records the spatial position coordinates and attitude angle of the robot in the global coordinate system after completing obstacle avoidance.

[0057] The visual language model is invoked to perform visual analysis on the second-run visual data based on environmental description cues, resulting in a third visual description text. This third visual description text is a semantic definition of the new location discovered by the obstacle avoidance behavior. For example, the third visual description text could be "Located at the entrance of the backup passage in the main passage, with no obstacles ahead." Thus, the third visual description text and obstacle avoidance pose can be used to construct new nodes in the topology map. These new nodes contain semantic information about the obstacle avoidance location and their precise spatial pose. Then, this new node can be connected to the last path node recorded by the robot before starting this obstacle avoidance run (i.e., the node on the original path that triggered the obstacle avoidance behavior) by establishing an "edge." This newly created edge represents a physically reachable path from the original path node to the newly discovered obstacle avoidance location. This operation allows the robot to plan and utilize this path or location explored through obstacle avoidance as an effective navigation option when performing future tasks.

[0058] This application embodiment achieves dynamic updating of the topology map through the above-described method, reducing the inherent limitations of traditional pre-built static maps in recording and utilizing temporary or permanent new paths explored by the robot during actual operation. This allows the topology map to continuously enrich and improve its coverage and path diversity as the robot operates over a long period. This capability significantly enhances the adaptability and robustness of the entire control system in the face of dynamic changes and complex unstructured environments. It not only improves the efficiency of subsequent task execution and provides more optimization options for path planning, but also fundamentally enhances the overall mobility and collaborative efficiency of multi-robot systems in dense, dynamic environments.

[0059] The robot control method provided in this application introduces a visual language model, enabling the overall system to understand and execute unconventional, complex, and abstract natural language instructions without the need for pre-programming every possible instruction, thus enhancing the system's ability to handle complex and ever-changing tasks. By leveraging the cross-modal reasoning capabilities of the visual language model to dynamically construct and maintain a hybrid topology map that integrates semantic information and spatial relationships, a query foundation rich in environmental context is provided for global navigation, allowing the robot not only to find its way but also to "understand" its environment. A navigation cue word manager is designed to guide the visual language model in reasoning, giving the decision-making process human-like logic and interpretability. Simultaneously, a hierarchical control architecture is adopted, with the upper-level VLN navigation module outputting high-level instructions, and the lower-level robot motion control module, trained through reinforcement learning, responsible for converting these high-level instructions into low-level control quantities and performing real-time, safe local path planning and dynamic obstacle avoidance, achieving a combination of high-level intelligent decision-making and low-level agile execution. In this way, the robot's ability to parse abstract instructions, its semantic perception and adaptation in dynamic environments, and the collaborative efficiency of multi-robot systems are improved. At the same time, the dependence on fixed navigation infrastructure is reduced, enabling the system to exhibit higher intelligence, robustness and maintainability in complex logistics scenarios.

[0060] Reference Figure 8 This application also provides a robot control device, which includes: Task acquisition unit 810 is used to acquire logistics task text; The node query unit 820 is used to call a pre-built topology map to perform node matching query based on the logistics task text, and obtain the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to the path reachability relationship. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map is built, as well as the site description text of the logistics site. The robot control unit 830 is used to determine the robot's current operating state and current pose data, and to perform motion control on the robot based on the current operating state, current pose data, and target pose data.

[0061] It should be noted that the robot control device provided in this application embodiment is used to implement the robot control method provided in the above embodiment, and the specific implementation process corresponds to the robot control method in the above embodiment. It can be referred to the aforementioned robot control method, and will not be repeated here.

[0062] This application also provides an electronic device (i.e., a computer device), which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement any of the robot control methods described in the above embodiments. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0063] Reference Figure 9 , Figure 9 This illustration shows the hardware structure of an electronic device according to another embodiment, the electronic device comprising: The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and called and executed by the processor 910 using the robot control method of the embodiments of this application. The input / output interface 930 is used to implement information input and output; The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940); The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0064] This application also provides a computer-readable storage medium storing a computer program for causing a computer to execute the robot control method described in the above embodiments.

[0065] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] This invention also provides a computer program product that stores program instructions, which, when executed by a computer, cause the computer to implement the robot control method described in any of the above embodiments.

[0067] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0068] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0071] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0074] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robot control method, characterized in that, The method includes: Obtain the logistics task text; The pre-built topology map is invoked based on the logistics task text to perform node matching query, thereby obtaining the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to path reachability. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map was constructed, as well as the site description text of the logistics site. The robot's current operating state and current pose data are determined, and motion control of the robot is performed based on the current operating state, the current pose data, and the target pose data.

2. The method according to claim 1, characterized in that, The step of calling a pre-built topology map based on the logistics task text to perform node matching and querying to obtain the target pose data indicated by the logistics task text includes: Obtain preset task prompt words, and call a preset visual language model to perform text analysis on the logistics task text based on the task prompt words to obtain the task target object; Based on the task target object, the topology map is invoked to perform node matching query, and the nodes corresponding to the site description text containing the task target object are determined as candidate nodes; For each candidate node, the preceding and following adjacent nodes of the candidate node are determined in the topology map based on the edges connected to the candidate node; The target node is selected from the candidate nodes based on the site description text contained in the preceding and following adjacent nodes, and the pose data contained in the target node is determined as the target pose data.

3. The method according to claim 2, characterized in that, Methods for determining the robot's current pose data include: Obtain preset environmental description prompts and the robot's current visual data; The visual language model is invoked to perform visual analysis on the robot's current visual data based on the environmental description prompts, and a first visual description text is obtained. The topology map is invoked based on the first visual description text to perform node matching query. The node corresponding to the site description text that matches the visual description text is determined as the current node, and the pose data contained in the current node is determined as the current pose data.

4. The method according to claim 3, characterized in that, After performing motion control on the robot based on the current operating state, the current pose data, and the target pose data, the method further includes: During the motion control of the robot, the robot acquires the first operational visual data collected by the intermediate node on the path connecting the current node and the target node; The visual language model is invoked to perform visual analysis on the first running visual data based on the environment description prompts, to obtain the second visual description text; The site description text of the intermediate node is updated based on the second visual description text.

5. The method according to claim 3, characterized in that, After performing motion control on the robot based on the current operating state, the current pose data, and the target pose data, the method further includes: During the motion control of the robot, the obstacle avoidance position of the robot during obstacle avoidance is determined, and the second running visual data and obstacle avoidance pose of the robot at the obstacle avoidance position are acquired. The visual language model is invoked to perform visual analysis on the second running visual data based on the environmental description prompts, to obtain the third visual description text; Create new nodes based on the third-view description text and the obstacle avoidance pose, and connect the new nodes to the nodes passed before the obstacle avoidance operation.

6. The method according to claim 1, characterized in that, The method for constructing the topology map includes: During the map building phase, the robot is controlled to run multiple times in the logistics site according to the preset operating data until the node creation stopping condition is met. After each operation, acquire the site vision data and robot pose of the robot at the corresponding logistics site location; A preset visual language model is invoked to perform visual analysis on the site visual data to obtain site description text; Attention encoding is performed on the site description text and the robot pose, and nodes are created based on the encoding results; Connect all created nodes based on path reachability to obtain a topology map.

7. The method according to claim 1, characterized in that, The pre-trained motion control model invoked performs motion control on the robot based on the current operating state, the current pose data, and the target pose data. The training method of the motion control model includes: Acquire the robot's sample starting pose data, sample ending pose data, and sample running status in the simulation environment; The motion control model is invoked to perform motion control on the robot based on the sample's starting point pose data, the sample's ending point pose data, and the sample's running state. The control reward score is determined based on the motion control result and a preset reward function, and the motion control model is trained based on the control reward score.

8. A robot control device, characterized in that, The device includes: The task acquisition unit is used to acquire logistics task text. The node query unit is used to call a pre-built topology map to perform node matching query based on the logistics task text to obtain the target pose data corresponding to the logistics task text. The topology map includes multiple nodes, and different nodes are connected according to path reachability. Each node contains the pose data of the robot at the corresponding logistics site location when the topology map was constructed, as well as the site description text of the logistics site. The robot control unit is used to determine the robot's current operating state and current pose data, and to perform motion control on the robot based on the current operating state, the current pose data, and the target pose data.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images